Multiple Admissibility: Judging Grammaticality Using Unlabeled Data in Language Learning

Multiple Admissibility in Language Learning: Judging Grammaticality Using Unlabeled Data

Anisia Katinskaia, Sardana Ivanova, Roman Yangarber University of Helsinki, Department of Computer Science, Finland [email protected]

Abstract because negative feedback for an acceptable answer misleads and discourages the learner. We present our work on the problem of detection Multiple Admissibility (MA) in language We examine MA in the context of our language learning. Multiple Admissibility occurs when learning system, Revita (Katinskaia et al., 2018). more than one grammatical form of a word fits Revita is available online2 for second language syntactically and semantically in a given con- (L2) learning beyond the beginner level. It is in text. In second-language education—in partic- use in official university-level curricula at several ular, in intelligent tutoring systems/computer- major universities. It covers several languages, aided language learning (ITS/CALL), systems many of which are highly inflectional, with rich generate exercises automatically. MA implies that multiple alternative answers are possi- morphology. Revita creates a variety of exercises ble. We treat the problem as a grammatical- based on input text materials, which are selected ity judgement task. We train a neural network by the users. It generates exercises and assesses with an objective to label sentences as gram- the users’ answers automatically. matical or ungrammatical, using a “simulated For example, consider the sentence in Finnish: learner corpus”: a dataset with correct text “Ilmoitus vaaleista tulee kotiin postissa .” and with artificial errors, generated automat- (“Notice about elections comes to the house in the ically. While MA occurs commonly in many mail.”) languages, this paper focuses on learning Rus- sian. We present a detailed classification of In practice mode, Revita presents the text to the the types of constructions in Russian, in which learner in small pieces—“snippets” (about 1 para- MA is possible, and evaluate the model using graph each)—with all generated exercises. This a test set built from answers provided by users is important, because grammatical forms in exer- of the Revita language learning system. cises usually depend on a wider context. 1 Introduction For the given example, Revita can generate cloze exercises hiding several tokens (surface The problem of Multiple Admissibility (MA) oc- forms) and providing their lemmas as hints: curs in the context of language learning. In “Ilmoitus vaali tulla koti posti .” “cloze” exercises (fill-in-the-blank), the learner re- (“Notice election come house mail .”). ceives a text with some word removed, and a base For the verb ”tulla” (”come”) and nouns ”vaali” form1 of the removed word as a hint. The task is to (“election”), ”koti” (“home”) and ”posti” (”mail”) produce the correct grammatical form of the miss- the learner should insert the correct grammatical ing word, given the context. The answer given by forms. Revita expects them to be the same as the the user is checked automatically by the language forms in the original text, and will return negative learning system. Therefore, the system should feedback otherwise. However, in this example, be able to accept more than one answer, if there postitse (“via email”) is also acceptable, although are grammatically and semantically valid alterna- it is not in the original text. tives in the given context. Otherwise, the language The MA problem is also relevant in the context learning system returns negative (actually incor- of exercises that require a free-form answer, such rect) feedback to the learner. This is a problem, as an answer to a question or an essay. The learner 1The base or “dictionary” form will be referred to as lemma in this paper. 2https://revita.cs.helsinki.fi/

Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 12–22, Florence, Italy, 2 August 2019. c 2019 Association for Computational Linguistics can produce “unexpected” but nevertheless valid larger base of Revita users. grammatical forms. 2.1 Types of Multiple Admissibility In the current work, we restrict our focus to MA of multiple surface forms of the same lemma, In analyzing the answers given by our users, we given the context; we do not consider synonyms, discovered several types of the most frequent con- which can also fit the same context. The latter texts where MA appears. topic is part of the future work. Present/Past tense: The most clear case of MA The structure of the paper is as follows: In sec- in Russian (as in many other languages) is the case tion2 we formulate the problem and provide a de- of interchangeable forms of present and past tense tailed classification of the types of MA found in of verbs. Russian has three tenses (present, past, the learner data. In section3 we describe our ap- future), of which past and future tenses can have proach, in particular, the procedure for generating perfective or imperfective aspect. artificial grammatical errors and creating test sets. In the next example, if both verbs are chosen 3 Section4 presents previous work on artificial er- for as exercises and the learner sees two lem- ror generation. Section5 describes our model and mas (”задержаться” and ”вернуться”), she has experimental setup. In section6 we discuss the re- a possibility to produce verb forms in any tense de- sults and error analysis, and conclude in section7. pending on the context beyond the given sentence. It may be narrative past tense or present tense, or 2 Multiple Admissibility: Problem the text may be a message where the communica- Overview tive goal of the speaker is to inform the reader about future events, so future tense is expected. We use data from several hundred registered stu- 4 dents learning Russian (and other languages), “Мы задержались(PST) у друзей и practicing with exercises based on texts, or an- вернулись(PST) домой поздно.” swering questions in test sessions. Currently, Re- “We were delayed with friends and returned vita does not provide a mode for written essays, home late.” because it does not check free-form answers. The following option may be acceptable: Except for cloze exercises, Revita also generates exercises where the users are asked to se- “Мы задержались(PST) у друзей и lect among various surface forms based on a given вернемся(FUT) домой поздно.” lemma—a correct surface form and a set of auto- “We were delayed with friends and matically generated distractors; or to type the word will return home late.” they hear. This allows us to collect a wide range The future cannot precede the past in this sen- of errors, though not all kinds of possible errors; tence, so the next variant answer is grammatically e.g., currently we do not track punctuation errors, incorrect: word order errors, insertion errors, errors resulting from choosing an incorrect lemma, etc. *“Мы задержимся(FUT) у друзей и We annotated 2884 answers from the Revita вернулись(PST) домой поздно.” database, which were automatically marked as “We will be delayed with friends and “not matching the original text.” This work returned home late.”5 was done by two annotators (90% agreement) Some cases are more difficult because the both with native-level competency in Russian, and choice of tense can depend on knowledge beyond background in linguistics and teaching Russian. the text: Among all annotated answers, 7.5% were actually correct, but differed from the original text. “Ученым удалось установить, что Also, we checked answers given by three students у минойцев существовало (PST) with C1 CEFR proficiency level in Russian (es- несколько видов письма.” tablished independently by their teachers); 15.8% 3If one of the verbs is chosen as an exercise, the user may of these answers were grammatically and seman- get a hint from the surface form of the other verb, that they tically valid in the context. Thus, for advanced should be coordinated. 4We use standard abbreviations from the Leipzig users, the problem of MA is twice as relevant as on Glossing Rules. average; we plan to investigate these results with a 5*Star is used to mark an incorrect sentence.

13 “Scientists were able to establish that the Mi- “Она была загадочной (INS) и noans had several types of writing systems.” непонятной (INS) для меня”. In a non-fictional narrative, only the past tense “She was mysterious and incomprehensible can be used, since the Minoans do not exist in the to me”. present. These examples show that deciding which Genitive/Accusative case: usage of genitive tenses are acceptable in the context is difficult. vs. accusative is a complex topic, beyond the Singular/Plural: Singular and plural nouns can scope of this paper. We mention a few examples be grammatically valid in the same context, if briefly, where MA can appear—denotation of a there is no dependency on words beyond the sen- part of the whole and negation. Usually the gen- tence boundaries. For instance: itive is used to denote a part of the whole. In the “Из последних разработокскафандр following example, usage of the accusative is in- для работы (SG.) в открытом космосе.” correct: “Из последних разработокскафандр “Пожалуйста, отрежь хлеба” (GEN) для работ (Pl.) в открытом космосе.” *“Пожалуйста, отрежь хлеб” (ACC) “From the latest developments—the space- suit for work in open space.” “Please, cut some bread.” Short/Full adjective:6 in many constructions, In some contexts both meanings are possible— short and full forms of adjectives can be used as a of a part and of the whole—resulting in MA: part of a compound predicate (Vinogradov, 1972). “Нам оставили хлеба и вина” (GEN) The difference between these is that the short form typically expresses temporal meaning and that it “We were left with some bread and wine.” is a phenomenon of literary language, whereas its full alternative form sounds more colloquial. “Нам оставили хлеб и вино” (ACC) “Вы ужасно болтливы (short) и “We were left with bread and wine.” непоседливы “ (short) (from I. Bunin) If both words appear in exercises, Revita should “You are being terribly talkative (PRED) and accept genitive or accusative (if the context speci- restless (PRED).” fies the expected meaning nowhere else). “Вы ужасно болтливые и The next case is negation constructions. The непоседливые”. genitive is usually used where negation is stressed, “You are terribly talkative (DESCR) and whereas the accusative weakens the negation, restless (DESCR)”. (Rosenthal et al., 1994). However, it is worth noticing that the difference can be difficult to un- We treat these examples as MA, because even derstand even for native Russian speakers. for native Russian speakers in many cases the choice can be unclear. So it would be too strict “До вас никто еще этого браслета to treat one of the variants as incorrect. (GEN) не надевал.” (from Kuprin.) Nominative/Instrumental case: nouns as part “No one has worn this bracelet before you.” of compound named predicates can be in the nominative or instrumental case. The difference in Compare with a similar example in the accusative: meaning is similar to that of short/full adjectives “Он не отвергнул тогда с презрением (Rosenthal et al., 1994): nominative indicates a эти сто (ACC) рублей.” (Dostoevsky) constant feature of a subject, whereas instrumen- “He did not reject those one hundred rubles tal indicates a temporary feature of a subject. We with contempt.” consider the following examples as MA, because of a subtle difference in meaning: It is not always possible for the learner to know “Она была загадочная (NOM) и which case expected in the sentence, because it непонятная (NOM) для меня”. (from I. implies that she should know which type of nega- Bunin) tion was mentioned. always possible, and both options fit semantically. 6So-called full/short forms of adjectives correspond to de- scriptive/predicative adjectives in other languages: compare Perfective/Imperfective aspect: errors in as- “the hungry(DESCR) dog” vs. “the dog is hungry(PRED)”. pect are very common (Rozovskaya and Roth,

14 2019). Without going into detail, we show ex- Other cases: some examples of MA contexts amples from (Rakhilina et al., 2014)—a heritage are exceptionally interesting and rare. 7 learner corpus. Both sentences can be interpreted “На этой почве хорошо росла трава, что as correct, with a subtle difference in meaning: обеспечивало (Neu) пастбищами овец.” “Он пишет ей, чтобы она не перестала “На этой почве хорошо росла трава, что (PFV) любить его.” обеспечивала (Fem) пастбищами овец.” “He writes to her that she should not stop lov- “The grass grew well on this soil, what ing him.” provided sheep with pasture.” “Он пишет ей, чтобы она не переставала These sentences expresses similar meaning, al- (IPFV) любить его.” though the verb “обеспечивать” (“to provide”) “He writes to her that she continue to love appears in the neuter gender in the first and fem- him.” inine in the second. This happens because in the first sentence the subordinative pronoun “что” Gerund/Other verb forms: MA occurs in (“which”) refers to the entire preceding clause, some contexts where gerunds are used. In the fol- whereas in the second it refers only to the word lowing example, both sentences can express the “трава” (“grass”), which is feminine. meaning of two actions happening at the same We observe many other types of constructions time. The only argument against using a past tense with MA. This is not an exhaustive list, but covers verb form is that it sounds somewhat unnatural only some of the types actually found among the without a conjunction ”and” between verbs (”was answers given by the learners of Russian in Revita. saying and thinking”): The list should give us some intuitions about the “... говорил я себе, думая (GERUND) problem of MA, and how difficult it is to identify об Охотном ряде” (from Bunin) automatically. “... I was saying to myself, thinking about 3 Overview of the Approach Okhotny Ryad” “... говорил я себе, думал (PST) об How can we identify instances of Multiple Admis- Охотном ряде” sibility? One approach to this problem is to train a model with a language modeling objective— “... I was saying to myself, was thinking referred to as “LM-trained” in the literature. In about Okhotny Ryad” such a scenario, the task of the model is to pre- Prepositions with multiple cases: some prepo- dict the next word at every point in the sentence, sitions can govern two different cases of the fol- e.g., for the sentence ”The keys to the cabinet lowing noun, with no change in meaning: [is/are] here”8 the task of the model is to predict that P (are|C) > P (is|C), where C is the Она спряталась под одеялом “ .” (INS) context. Linzen (2016) experimented with this “Она спряталась под одеяло.” (ACC) kind of language modeling in three setups: with- “She hid under a blanket.” out any grammatically relevant supervision, with supervision on grammaticality (predicting which Second Genitive (Partitive): and other forms of two sentences9 is grammatical/ungrammatical), common in spoken language can be valid alterna- and number prediction—predicting between two tives, often unfamiliar to L2 learners. classes, ”singular” or ”plural”. The last two setups “Я привозил ей коробки шоколаду are strongly supervised. The poorest results were (2GEN), новые книги...” obtained using a LM-trained model, despite using “Я привозил ей коробки шоколада a large-scale language model (Jozefowicz et al., (GEN), новые книги...” 2016). “I was bringing her boxes of chocolate, new Later, Gulordava (2018) reevaluated these re- books...” sults for the task of predicting long-distance agreement: for several languages, including Russian, 7Heritage learners are persons with a cultural con- nection to or proficiency in the language through fam- 8The example is from (Linzen et al., 2016). ily, community, or country of origin. (Definition from 9”The keys to the cabinet are here” vs. ”The keys to the http://www.cal.org/heritage/research/) cabinet is here”

15 an LM-trained RNN approached the accuracy of relate to collecting data from language learners the supervised models described in (Linzen et al., and very expensive annotation procedures. 2016). Marvin (2018) performed a targeted evalu- Revita at present creates exercises only for ation of several LMs—N-gram, LM-trained RNN, words which do not exhibit lemma ambiguity (ho- and Multitask RNN10—on the task of detecting er- mography).12 Lemma ambiguity occurs when rors in several grammatical constructions for En- a surface form has more than one lemma. An glish.11 The results of even the strongest LM var- example of this type of ambiguity is the token ied considerably depending on the syntactic com- “стекло”, which has two morphological analyses: plexity of the construction: 100% accuracy in the стечь (“to flow down”) Verb+Past+Sing+Neut, case of simple subject-verb agreement, and 0.52% стекло (“glass”) Noun+Sing+Nom/Acc.13 accuracy for subject-verb agreement across an ob- In this setting, we do not need to generate errors ject relative clause (without “that”). for surface forms with lemma ambiguity. In light of these results from prior research, we Training instances are generated by sliding a decided to approach the problem as a supervised window of radius r over the list of input tokens, grammaticality judgement task—a two-class clas- with the target token in the middle. The target is sification task, (Linzen et al., 2016)). every n-th token (n is the stride). If the target to- Since MA answers are correct answers, sen- ken is unambiguous, is above a frequency thresh- tences with alternative grammatical forms of the old,14 and has a morphological analysis, it is re- same lemma would be grammatically correct. One placed by a random grammatical form from the of the problems with this approach is the lack of paradigm to which the token belongs.15 annotated training data. The Revita database had We use r = 10, which results in a window only 7156 answers labeled “incorrect” for Rus- wider than an average sentence in Russian, and we sian, at the time when these experiments began. are interested in including wide context in training Therefore we generated a training dataset by sim- instances. All generated windows are labeled as ulating grammatical errors. We describe the sim- negative/ungrammatical. The training dataset con- ulation procedure in the following subsection, and sists of a balanced number of grammatical and un- briefly review prior approaches to generating ar- grammatical instances. Part of the generated data tificial errors for grammatical error detection and was removed from training dataset and used as a correction tasks (GED/GEC). Every instance in validation set for training the model. the simulated dataset is labeled as correct or in- Of the automatically generated errors that we correct. The network reads the entire sequence in checked, some appear very natural, while others a bidirectional fashion and receives a supervised may be less likely to be made by real students. signal at the end. We describe the model and the As Linzen (2016) notes, some of the gener- experiments in the following sections. ated instances will not in fact be ungrammatical. We analysed 500 randomly chosen generated win- 3.1 Generating Artificial Errors dows; 3% of them happened to be grammatical (in First, we describe the process of generating train- Table1 we refer to them as Multi-admissible). We ing data: the source of data, preprocessing steps provide the interpretation for all labels in Table1 and a brief analysis of what types of errors we ob- in the following subsection. tain in the simulated data. In the following sub- 3.2 Test Data Analysis section, we proceed to describe the test sets which were build from real users’ data containing “natu- To create test sets, we took 2884 answers from Re- ral” errors. vita’s database, which were automatically marked Generating training datasets with artificial er- as “incorrect,” and manually annotated them using rors is a common approach, because obtaining the labels below: large error-annotated learner corpora is extremely 12Because it currently does not attempt to perform disam- difficult and costly, (Granger, 2003): difficulties biguation, and only one lemma can be shown as the hint. 13This is an example not only of lemma ambiguity, but also 10Language modeling objective, combined with a super- of word sense and morphological ambiguity. vised task of sequence tagging with CCG supertags. 14We count frequencies from the entire corpus used for 11The authors expected that a LM would assign a higher building the training set, to exclude words appearing once. probability to a grammatical sentence than to an ungrammat- 15We generate paradigms of inflected words using the py- ical one. morphy2 morphological analyzer (Korobov, 2015).

16 Label (1) Training set (2) Real student data (3) Advanced students Grammatical error 83.0% 21.4% 39.4% Non-word error — 20.8% 12.2% Multiple-choice — 12.0% 17.0% Multi-admissible 3.0% 7.5% 15.8% Pragmatic error 2% 1.6% 2.9% Broken 12% 36.7% 12.7% Total instances annotated 500 2884 170

Table 1: Data we annotated for veriﬁcation and testing: (1) subset of the set of errors automatically generated for training (randomly sampled and manually annotated), (2) learners’ answers (randomly sampled), marked by the System as incorrect, (3) subset of learner’ incorrect answers—for advanced learners only (CEFR level C1/C2). “Broken”: discarded instances (technical problems, too many unknown words, numbers, punctuation marks, etc.)

• Grammatical error: answer was a valid gram- which were given only by advanced learners (the matical form of the word (exists in paradigm), but third column). incorrect in the given context. This group includes We separate the real, manually annotated data only errors made in cloze exercises. into four test sets (see Table2). • Non-word error: spelling error—the word was A. The first test set contains only sentences ex- rejected by the morphological analyzer. hibiting MA. • Multiple-choice: error in a choice of word from B. The second test set is randomly chosen cor- a list of options. rect sentences from a separate corpus (for a total of • Multi-admissible: as mentioned above, we con- 500 instances) which was not used for generating sider these to be correct answers. training data. • Pragmatic error: a separate type of error where C. The third test set is made to test the ability the given answer can fit grammatically, but is se- of our model to distinguish between grammatical mantically/pragmatically unnatural in the context. and ungrammatical sentences (as it was trained to We provide one example of the last kind of er- do)—thus it contains: ror; it requires further investigation: C1. sentences with grammatical errors made by “У меня машина сломалась, и мне Revita users; пришлось звонить в автосервис (ACC)”. C2. correct sentences from Revita’s database. “My car broke down and I had to call (to) D. The fourth test set contains sentences only the auto repair”. with pragmatic errors. *“ У меня машина сломалась, и мне In the next section, we shortly review prior work пришлось звонить в автосервисe related to artificial error generation (AEG) for the (LOC)”. grammaticality judgement task. * “My car broke down and I had to call (while 4 Related Work being) in a car-service station”. Preposition “в” (“in”) governs two cases— Felice(2016) divides methods of AEG into deter- Nominative and Locative—but the second sen- ministic vs. probabilistic. The deterministic ap- tence does not make sense pragmatically. We have proach consists of methods that generate errors in begun a more detailed annotation of all learner an- systematic ways, which do not make use of learner swers (i.e., the types of grammatical errors). This error distributions. Izumi et al.(2003) introduced topic is beyond the scope of this paper. a system for correction of article errors made by • Broken: discarded instances (technical prob- English learners, native in Japanese. The system lems, words not in our training vocabulary, too was trained on artificial data where a, an, the or many numbers, punctuation marks, answers given the zero article were replaced with a different op- in languages different from expected, etc). tion chosen randomly. Table1 represents the number of all mentioned Sjobergh¨ and Knutsson(2005) created an artifi- data types in the real learners’ answers (the second cial corpus consisting of two of the most frequent column) and in the subset of these real answers types of errors among non-native Swedish speak-

17 ers: split compounds and word order errors. corpus,16 which is arranged by genre into sev- Brockett et al.(2006) describe a statistical ma- eral segments. We used all news segments, and chine translation (SMT) system for correcting a set part of the literary text segment, for a total of of 14 countable and uncountable nouns which are 809M words. We exclude social media, film subti- often confused by learners of English. They used tles, and poems, because their language has more rules to change quantifiers (e.g. much–many), to deviations from the literary standard. All docu- generate plural forms, and to insert unnecessary ments were lowercased, tokenized, and morpho- determiners. Lee and Seneff(2008) created an ar- logically analyzed using Crosslator (Klyshinsky tificial corpus of verb form errors. They changed et al., 2011).17 We replace all punctuation marks verbs in the original text to different forms, such with a special token, to preserve information about as to-infinitive, 3rd person singular present, past, sentence/clause boundaries. The size of the train- or -ing participle. Ehsan and Faili(2013) used ing vocabulary was around 1.2M words (after re- SMT for AEG to correct grammatical errors and moving words with frequency less than 2). For context-sensitive spelling mistakes in English and validation, we randomly chose 5% of all generated Farsi. Training corpora were obtained by inject- data. ing artificial errors into well-formed treebank sen- Model architecture: our baseline neural net- tences using predefined error templates. work (NN) is implemented in TensorFlow. Its ar- Probabilistic approach: Rozovskaya and Roth chitecture is a one-layer bidirectional LSTM with describe several methods for AEG which include dropout (0.2), which has 512 hidden units. The creation of article (Rozovskaya and Roth, 2010b) hidden state of the BiLSTM is then fed to an and preposition errors (Rozovskaya and Roth, Multi-layer Perceptron (MLP). The MLP uses one 2010a, 2011) based on statistics from an English hidden layer with 1024 neurons, and Leaky ReLU as a Second Language (ESL) corpora. They in- activation function. The size of the output layer is ject errors into Wikipedia sentences using differ- 1, since we have only two classes to predict. The ent strategies (e.g., distribution before and after output of the MLP is then fed to a sigmoid activa- correction, L1-specific error distributions). tion function to obtain a prediction for the entire Rozovskaya et al.(2012) proposed an inflation input sequence. To encode words, we use the Fast- method, which preserves the ability of the model Text 300-dimensional pre-trained embeddings.18 to take into account learner error patterns. While The network and the word embeddings were also increasing the model’s recall, this method trained in an end-to-end fashion. Optimization reduced the confidence that the system has in was done using Adam, dropout, and early stopping the source word. Improvement in F-scores was based on the loss on the validation set. We trained achieved by this method when correcting deter- the network over only half of an epoch, since it miners and prepositions. Further, this method was was showing signs of overfitting—because we use used by other researchers (Felice and Yuan, 2014; a sliding window, the number of training instances Putra and Szabo´, 2013; Rozovskaya et al., 2013, was over 90M. The averaged accuracy on the val- 2014, 2017). idation set was 95 %. Table2 reports the accuracy Dickinson(2010) introduce an approach to gen- on the test sets, averaged across 5 runs. erate artificial syntactic errors and morphologi- 6 Results cal errors for Russian. Imamura et al.(2012) adapt the method of Rozovskaya and Roth(2010b) Table2 shows the results of our experiments in for particle correction in Japanese. Cahill et al. terms of accuracy. 85.9% accuracy was achieved (2013) examine automatically-compiled sentences across all types of MA. However, we should stress from Wikipedia revisions for correcting errors in that in the test set marked Multi-admissible (MA), prepositions. Kasewa et al.(2018) use an off-the- the majority of the instances belong to the MA shelf attentive sequence-to-sequence NN (Bah- types of Present/Past tense and Singular/Plural. danau et al., 2014) to learn to introduce errors. Since the test set has a small number of instances of MA contexts with gerund/other verb 5 Model and Experiment 16https://tatianashavrina.github.io 17This analyzer was chosen because it is a part of Revita’s Data: For generating the training/validation text processing pipeline. datasets, we use the open-source “Taiga” Russian 18https://fasttext.cc/docs/en/crawl-vectors.html

18 Test set # Acc ing/validation data, and are not present in the Re- A. Multi-admissible 178 85.9 vita database. On random correct sentences, the B. Random correct 500 92.3 model achieved substantially better results than for C. Correct & incorrect 1290 81.0 MA instances (92.3%). It is interesting that the C1. Grammatically correct 650 73.3 model has more difficulty with the syntactic struc- C2. Grammatically incorrect 640 88.6 ture of contexts with known MA than with some D. Pragmatic errors 46 54.3 random correct contexts. Another test set (C.) is made up of correct sen- Table 2: Percent accuracy of our NN model. Random tences from the Revita database, and sentences correct: test set built from sentences which were not in- cluded in the training and validation sets and did not ap- with grammatical errors made by the learners. We pear in Revita’s database, randomly selected sentences evaluate the model on this test set to gain insight from normal texts. Grammatically incorrect: test set into how well it can differentiate between various with real grammatical errors from students’ data. Prag- incorrect vs. correct sentences. The discussion of matic errors: test set with real pragmatic errors from results for different grammatical error types is be- students’ data. yond the scope of this paper, and is left for the further work. MA types # Acc The pragmatic error test set (D.) was used to 1. Perf/Imperf + Gerund/Other 14 92.8 find out how difficult it is to predict labels for 2. Case 24 91.7 sentences which are correct grammatically but in- 3. Present/Past 53 88.7 correct semantically/pragmatically. Clearly these 4. Singular/Plural 78 82.0 instances pose the greatest challenge to the cur- 5. Short/Full adj 9 77.7 rent model; (it was not explicitly trained to detect Table 3: Percent accuracy of our NN model for dif- them). ferent MA contexts. Case combines all types of MA Of the nearly 3000 manually annotated in- contexts listed in the Subsection 2.1 which differ by stances, the number of instances found to be prag- case (Nominative/Instrumental, Genitive/Accusative matic errors and MA was not large. and others). 6.1 Error Analysis forms and MA contexts which differ by perfec- We analysed some of the errors the model made on tive/imperfective aspect, we grouped them to- the MA test dataset. For all types of MA, we found gether for testing the model. On these two types some similar patterns: the model assigns very low of MA the model achieved the highest accu- scores to short sequences (which are padded), con- racy, 92.8% (see Table3). For the same rea- texts with too many punctuation marks or names, sons we grouped together MA contexts which and context with non-Russian words which are un- differ by case (Nominative/Instrumental, Geni- known to the model. For example, for construc- tive/Accusative, Second Genitive and other). The tions with Present/Past tense, the network made overall accuracy for these contexts is 91.7%. wrong predictions if the subject was a name or a number, which in most cases corresponds to the We plan to test all combined MA types sepa- token “UNK” in our model’s vocabulary. The rately as soon as we have more annotated data. same happens if the subject is outside the window. The accuracy for Present/Past tense is 88.7%. Sometimes the model confuses certain nouns or The accuracy for Number agreement (including pronouns next to the verb with its subject, for ex- subject-verb agreement on number) is 81.0%. ample: The lowest accuracy was achieved for Short/Full adjectives—77.7%. Some discussion of errors is “Поезда (SUBJ) метро задерживаются in the following subsection. (PRED). We use additional test sets to assess other as- “Trains of the metro are delayed.” pects of the trained NN model. The “Random correct” test set (B.) contains 500 randomly sampled In this case, the model might suppose that sentences without errors, to compare with the MA the (genitive) singular noun “метро” (“metro”) is test set. These sentences were sampled from a the subject of the plural verb “задерживаются” corpus which was not used for generating train- (“are delayed”), which is incorrect—the actual

19 subject is the plural noun “поезда” (“trains”)— lacking labeled training data was approached by but the genitive is closer to the predicate. As a re- generating a dataset with artificial errors. We also sult, the model will identify the sentence as gram- observed that the MA problem is more relevant matically incorrect, believing that the subject and for advanced language learners. Another observa- predicate conflict in number. tion is that for a trained model it is more difficult We also should note that some instances marked to make prediction about MA contexts than about by the model as incorrect are actually incorrect, random correct sentences. but marked as MA by annotators, which means We plan to extend and improve our training data that MA instances need to be double-checked and by marking numbers with special tokens, or by that the model is able to identify ungrammatical mapping them into words. We also plan to mark contexts. names with a name tag by using some of the exist- It is difficult to compare our results directly ing NER models, and mark rare words with their with prior work, because we have not yet found part of speech. We also believe that providing in the previous work a problem similar to Multiple the model with syntactic information (parsing) can Admissibility for Russian. A similar problem— help, so that we can train a model in a multitask grammatical acceptability judgment—is presented fashion: predict tags of words, as well as their cor- in (Warstadt et al., 2018), for English only. The rectness. Also, it is worth trying to use new large- best results they achieved in terms of percent ac- scale language models, which proved to be more curacy is 77.2%. The average human accuracy is effective on a variety of tasks. 85%. Additional annotation of student data collected For the task of grammatical error detection, the in Revita’s database is needed; in the current work, results obtained for Russian are much lower than the annotation was done by two experts, and all for English. For example, the highest precision in disagreements were resolved. We plan to ex- (Rozovskaya and Roth, 2019) for errors in number tend our experiments to other languages available agreement is 56.7. in Revita. Each has its own language-specific Concerning the grammaticality judgement task, types of MA. Generating all paradigms of a word Marvin (2018) reported accuracies for subject- could be problematic for some highly inflected verb agreement from 50% to 99% depending on languages (e.g., Finnish, etc.). the syntactic complexity of the sentence (e.g., re- The goal of this paper is to introduce the prob- lations across relative clause). This is similar to lem of Multiple Admissibility and to attract more Present/Past tense construction in our setup. attention to experimenting with morphologically Linzen et al.(2016) also concludes that the rich languages and languages other than English grammaticality judgment objective is more diffi- in this context. cult than, for example, the number prediction objective. The LSTM model can have up to 23% References error on this task, as sentence complexity grows. This work studied only number agreement and Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly subject-verb dependencies in English. learning to align and translate. arXiv preprint arXiv:1409.0473. 7 Conclusions and Future Work Chris Brockett, William B Dolan, and Michael Gamon. We address the problem of Multiple admissibility 2006. Correcting ESL errors using phrasal SMT techniques. In Proceedings of the 21st International in automatically generated exercises, by approach- Conference on Computational Linguistics and the ing it as a grammaticality judgment task. We of- 44th annual meeting of the Association for Compu- fer a detailed study of examples, where our lan- tational Linguistics, pages 249–256. Association for guage learning system mistakenly assesses admis- Computational Linguistics. sible answers as incorrect. We classify these con- Aoife Cahill, Nitin Madnani, Joel Tetreault, and Diane texts into 10 types, where only some of these types Napolitano. 2013. Robust systems for preposition have been in the focus of prior research, especially error correction using Wikipedia revisions. In Pro- ceedings of the 2013 Conference of the North Amer- for Russian. We train a NN model with the gram- ican Chapter of the Association for Computational maticality objective, independent of the type of Linguistics: Human Language Technologies, pages test set we use for evaluation. The problem of 507–517.

20 Markus Dickinson. 2010. Generating learner-like mor- Mikhail Korobov. 2015. Morphological analyzer and phological errors in russian. In Proceedings of the generator for Russian and Ukrainian languages. In 23rd International Conference on Computational International Conference on Analysis of Images, So- Linguistics (Coling 2010), pages 259–267. cial Networks and Texts, pages 320–332. Springer.

Nava Ehsan and Heshaam Faili. 2013. Grammatical John Lee and Stephanie Seneff. 2008. Correcting mis- and context-sensitive error correction using a sta- use of verb forms. Proceedings of ACL-08: HLT, tistical machine translation framework. Software: pages 174–182. Practice and Experience, 43(2):187–206. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Mariano Felice. 2016. Artiﬁcial error generation 2016. Assessing the ability of lstms to learn syntax- for translation-based grammatical error correction. sensitive dependencies. Transactions of the Associ- Technical report, University of Cambridge, Com- ation for Computational Linguistics, 4:521–535. puter Laboratory. Rebecca Marvin and Tal Linzen. 2018. Targeted syn- Mariano Felice and Zheng Yuan. 2014. Generating ar- tactic evaluation of language models. arXiv preprint tiﬁcial errors for grammatical error correction. In arXiv:1808.09031 Proceedings of the Student Research Workshop at . the 14th Conference of the European Chapter of the Desmond Darma Putra and Lili Szabo.´ 2013. UdS at Association for Computational Linguistics , pages CoNLL 2013 Shared Task. In Proceedings of the 116–126. Seventeenth Conference on Computational Natural Sylviane Granger. 2003. Error-tagged learner corpora Language Learning: Shared Task, pages 88–95. and CALL: A promising synergy. CALICO journal, 20(3):465–480. E.V. Rakhilina, A.S. Vyrenkova, and M.S. Polinskaya. 2014. Grammar of errors and grammar of construc- Kristina Gulordava, Piotr Bojanowski, Edouard Grave, tions: ”Heritage” (”inherited”) Russian language. Tal Linzen, and Marco Baroni. 2018. Color- Questions of linguistics, 3(2014):3–19. less green recurrent networks dream hierarchically. arXiv preprint arXiv:1803.11138. Ditmar Elyashevich Rosenthal, E.V. Dzhandzhakova, and N.P. Kabanova. 1994. Reference on Spelling, Kenji Imamura, Kuniko Saito, Kugatsu Sadamitsu, and Pronunciation, and Literary Editing. Moscow Inter- Hitoshi Nishikawa. 2012. Grammar error correc- national School of Translators. tion using pseudo-error sentences and domain adaptation. In Proceedings of the 50th Annual Meeting Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, of the Association for Computational Linguistics: and Dan Roth. 2013. The University of Illinois Short Papers-Volume 2, pages 388–392. Association system in the CoNLL-2013 shared task. In Pro- for Computational Linguistics. ceedings of the Seventeenth Conference on Compu- tational Natural Language Learning: Shared Task, Emi Izumi, Kiyotaka Uchimoto, Toyomi Saiga, Thep- pages 13–19. chai Supnithi, and Hitoshi Isahara. 2003. Automatic error detection in the Japanese learners’ English spo- Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, ken data. In The Companion Volume to the Proceed- Dan Roth, and Nizar Habash. 2014. The Illinois- ings of 41st Annual Meeting of the Association for Columbia system in the CoNLL-2014 shared task. Computational Linguistics. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Task, pages 34–42. Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint Alla Rozovskaya and Dan Roth. 2010a. Generating arXiv:1602.02410. confusion sets for context-sensitive error correction. Sudhanshu Kasewa, Pontus Stenetorp, and Sebastian In Proceedings of the 2010 Conference on Empiri- Riedel. 2018. Wronging a right: Generating better cal Methods in Natural Language Processing, pages errors to improve grammatical error detection. arXiv 961–970. Association for Computational Linguis- preprint arXiv:1810.00668. tics. Anisia Katinskaia, Javad Nouri, and Roman Yangarber. Alla Rozovskaya and Dan Roth. 2010b. Training 2018. Revita: a language-learning platform at the paradigms for correcting errors in grammar and us- intersection of ITS and CALL. In Proceedings of age. In Human language technologies: The 2010 LREC: 11th International Conference on Language annual conference of the north american chapter of Resources and Evaluation, Miyazaki, Japan. the association for computational linguistics, pages 154–162. Association for Computational Linguis- E.S. Klyshinsky, N.A. Kochetkova, M.I. Litvinov, tics. and V.Yu. Maximov. 2011. Method of POS- disambiguation using information about words co- Alla Rozovskaya and Dan Roth. 2011. Algorithm occurrence (for Russian). Proceedings of GSCL, selection and model adaptation for ESL correction pages 191–195. tasks. In Proceedings of the 49th Annual Meeting of

21 the Association for Computational Linguistics: Hu- man Language Technologies-Volume 1, pages 924– 933. Association for Computational Linguistics. Alla Rozovskaya and Dan Roth. 2019. Grammar error correction in morphologically rich languages: The case of Russian. Transactions of the Association for Computational Linguistics, 7:1–17. Alla Rozovskaya, Dan Roth, and Mark Sammons. 2017. Adapting to learner errors with minimal supervision. Computational Linguistics, 43(4):723– 760. Alla Rozovskaya, Mark Sammons, and Dan Roth. 2012. The UI system in the HOO 2012 shared task on error correction. In Proceedings of the Seventh Workshop on Building Educational Applications Us- ing NLP, pages 272–280. Association for Computa- tional Linguistics. Jonas Sjobergh¨ and Ola Knutsson. 2005. Faking errors to avoid making errors: Very weakly supervised learning for error detection in writing. In Proc. RANLP, volume 2005. Viktor Vladimirovich Vinogradov. 1972. Russian language: The grammatical doctrine of the word, volume 21. High School. Alex Warstadt, Amanpreet Singh, and Samuel R Bow- man. 2018. Neural network acceptability judg- ments. arXiv preprint arXiv:1805.12471.