Identifying negative language transfer in learner errors using POS information Leticia Farias Wanderley Carrie Demmans Epp EdTeKLA Research Group EdTeKLA Research Group Department of Computing Science Department of Computing Science University of Alberta University of Alberta [email protected] [email protected] Abstract guage usage and, consequently, increase their gram- matical awareness (Karim and Nassaji, 2020). A common mistake made by language learners is the misguided usage of first language rules One metalinguistic phenomenon that often when communicating in another language. In causes confusion and errors in learner writing is this paper, n-gram and recurrent neural net- the occurrence of language transfer. The language work language models are used to represent transfer phenomenon is characterized by learners language structures and detect when Chinese reusing rules from their first languages (L1s) when native speakers incorrectly transfer rules from communicating in a second one1 (L2) (Lado, 1957). their first language (i.e., Chinese) into their When the L1 and L2 grammars diverge, learners English writing. These models make it pos- sible to inform corrective error feedback with who apply L1 rules make mistakes. This particu- error causes, such as negative language trans- lar version of language transfer is called negative fer. We report the results of our negative language transfer, and it is a significant source of language detection experiments with n-gram errors in second language learner speaking and and recurrent neural network models that were writing. trained using part-of-speech tags. The best per- forming model achieves an F1-score of 0.51 This paper explores the application of language when tasked with recognizing negative lan- models to represent language structures and detect guage transfer in English learner data. negative language transfer in learner essays written by Chinese native speakers. The proposed meth- 1 Introduction ods aim to identify incorrect learner utterances that Advances in grammatical error correction (GEC) have structural patterns that are more similar to the research allow writing editors to provide real-time learners’ L1 (Chinese) than to their L2 (English). corrective feedback to language learners. GEC sys- In this paper, we demonstrate that a recurrent neural tems are trained on parallel erroneous and corrected network trained to identify a part-of-speech (POS) learner data to detect and suggest corrections for tag sequence’s source language outperformed POS user errors. State-of-the-art GEC systems apply n-gram models in negative language transfer de- neural machine translation to learn how to correct tection. The RNN model achieved an F1-score of grammatical errors in English learner data (Bryant 0.51 on the negative language transfer detection et al., 2019). These systems also use metadata, task analysing POS tag sequences extracted from such as the learners’ native languages and their pro- learner errors. The output of this language model ficiency levels in the target language, to support the can be used to inform metalinguistic feedback, ty- GEC task (Nadejde and Tetreault, 2019). ing the incorrect utterances to differences between The direct corrective feedback derived from the L1 and English rules. By enabling negative GEC systems’ predictions helps learners improve language transfer detection in learner writing, lan- their writing and increase their language under- guage learners will have access to interpretable in- standing. However, direct corrective feedback is formation about their errors’ potential causes along not the only type of information that can aid lan- with direct corrective feedback for those errors. guage learning (Bacquet, 2019). Learners also ben- efit from metalinguistic feedback. This feedback 1By second language, we mean any additional language type can support learners’ reflections on their lan- beyond the learner’s mother tongue. 64 Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 64–74 April 20, 2021 ©2021 Association for Computational Linguistics 2 Related work trastive analysis hypothesis (Lado, 1957), as re- ported in papers by Wong and Dras(2009) and When writing in another language, writers are Berzak et al.(2015). Berzak et al.(2015) found a prone to leave behind certain traces of their na- correlation between structural error distribution in tive languages. This fact is the foundation for the English as a second language learners’ writing and native language identification task. This task makes typological differences between the learners’ L1s use of computational models to determine the L1 and English. Their work used the L1’s structural of a text’s author (Tetreault et al., 2013; Malmasi properties to predict challenging grammatical areas et al., 2017). Native language identification mod- in English as a second language writing. els aim to uncover usage patterns that distinguish native speakers of a certain language when writ- One of the challenges for language learners is ing in English. Rabinovich et al.(2018) explored that they are often unaware that they reuse struc- the usage of cognates in non-native English writ- tures from their L1s (Wanderley and Demmans ing to cluster the authors’ text according to their Epp, 2020). One way to alleviate this difficulty native languages’ families. Cognates are words is to provide error feedback that contains informa- from distinct languages that are similar in meaning tion about possible error sources (Bacquet, 2019). and form. As demonstrated by Rabinovich et al. Linguists and education researchers have long de- (2018), non-native English writers show a prefer- bated what is the best way to provide feedback ence for using English words that have cognates in a second language setting. Some argue that no in their L1. Furthermore, Flanagan et al.(2015) feedback should be given, while others posit that have shown that it is possible to employ the authors’ learners should have access to a different range of error patterns to discriminate between L1s, as En- error feedback types, such as direct, indirect, com- glish learners’ writing errors are highly correlated prehensive, and focused (Bitchener et al., 2005). to their native languages. More recent research has found that providing met- Learner writing errors are useful in yet another alinguistic feedback can increase language learn- computational task. The aforementioned grammat- ers’ writing accuracy (Bacquet, 2019; Karim and ical error correction task aims to detect and cor- Nassaji, 2020). Unlike direct corrective feedback, rect errors in learner data (Ng et al., 2013, 2014; which simply highlights and corrects learner errors, Bryant et al., 2019). State-of-the-art GEC systems metalinguistic feedback can help learners under- model the task as a neural machine translation pro- stand their errors by calling attention to the errors cedure in which the erroneous learner data is the and describing possible causes (Lyster and Ranta, source language, and its corrected version is the 1997). To understand whether students value met- target language (Bryant et al., 2019). Throughout alinguistic feedback about negative language trans- the years, researchers have explored the impact of fer, Watts(2019) examined Japanese students’ fa- using L1-specific learner data to train GEC sys- miliarity with the phenomenon. These learners tems: see Rozovskaya and Roth(2011), Chollam- highlighted how being aware of language transfer patt et al.(2016), and Nadejde and Tetreault(2019) effects helped improve their English writing. for examples. These researchers have found that While natural language processing (NLP) tasks whether GEC models are exclusively trained with and language learners benefit from information L1-specific data or whether this data is only used about negative language transfer, we are not aware to fine-tune the model, GEC benefits from infor- of previous research that automatically detects this mation about learners’ L1s. In addition to that, phenomenon. Recently, Monaikul and Di Euge- Nadejde and Tetreault(2019) explored fine-tuning nio(2020) applied traditional and neural natural GEC systems to learners’ English proficiency lev- language processing methods to detect preposition els and L1s. They found that information about omission errors with the aim of informing nega- both learner aspects improves GEC performance. tive language transfer feedback. Preposition errors English non-native speakers transfer patterns and are one of the most common types of negative lan- rules from their L1s into English writing. This phe- guage transfer errors in second language learning. nomenon has been explored by native language In their paper, the authors proposed applying the identification and grammatical error correction sys- preposition error detection output to generate met- tems. Moreover, the phenomenon has been demon- alinguistic contrastive feedback for language learn- strated in experimental investigations of the con- ers and teachers (Monaikul and Di Eugenio, 2020). 65 Dataset Number of sentences Error type Count Global Voices 138 582 Structural negative language transfer 1457 WMT19 11 960 Structural not negative language transfer 914 Combined 150 542 Total 2371 Table 1: Training datasets sizes Table 2: Structural negative language transfer error counts from the test dataset We envision a similar application of our results. However, our focus is on detecting additional neg- 3.2 Test data ative language