Analyzing Linguistic Complexity and Accuracy in Academic Language Development of German Across Elementary and Secondary School
Total Page:16
File Type:pdf, Size:1020Kb
Analyzing Linguistic Complexity and Accuracy in Academic Language Development of German across Elementary and Secondary School Zarah Weiss and Detmar Meurers University of Tubingen¨ Department for General and Computational Linguistics fzweiss,[email protected] Abstract such as the morphologically richer German. In We track the development of writing complex- this article, we target this gap with three contri- ity and accuracy in German students’ early butions. We build classification models for early academic language development from first to academic language development in German from eighth grade. Combining an empirically broad first to eighth grade, based on a uniquely broad approach to linguistic complexity with the set of linguistically informed measures of com- high-quality error annotation included in the plexity and accuracy. Our results indicate that two Karlsruhe Children’s Text corpus (Lavalley phases of academic language development can be et al., 2015) used, we construct models of Ger- man academic language development that suc- distinguished: Initial academic language and writ- cessfully identify the student’s grade level. We ing acquisition focusing on the writing process it- show that classifiers for the early years rely self, best characterized in terms of accuracy devel- more on accuracy development, whereas de- opment, with little development in terms of com- velopment in secondary school is better char- plexity. A second stage is characterized by the acterized by increasingly complex language in increasing linguistic complexity, in particular in all domains: linguistic system, language use, the domains of lexis and syntactic complexity at and human sentence processing characteris- the phrasal level. We demonstrate the robustness tics. We demonstrate the generalizability and robustness of models using such a broad com- and generalizability of the models informed by the plexity feature set across writing topics. broad range of linguistic characteristics – a major concern not only for obtaining practically relevant 1 Introduction approaches for real-life use, but also for charac- We model the development of linguistic com- terizing machine learning going beyond focused plexity and accuracy in German early academic task to approaches capable of capturing general language and writing acquisition from first to language characteristics. eighth grade. Complexity and Accuracy are well- The article is structured as follows: We first give established notions from Second Language Ac- a brief overview of research on writing develop- quisition (SLA) research. Together with Fluency, ment in terms of complexity and accuracy. We they form the CAF triad that has successfully then present the Karlsruhe Children’s Text corpus be used to characterize second language develop- used as empirical basis of our work. In Section4, ment (Housen et al., 2012). Accuracy here is de- we spell out our approach to assessing writing fined as a native-like production error rate (Wolfe- in terms of complexity and accuracy, before sec- Quintero et al., 1998) and Complexity as the elab- tions5,6, and7 report on three studies designed orateness and variation of the language which may to address the research issues introduced above. be assessed across various linguistic domains (El- lis and Barkhuizen, 2005). 2 Related Work While there has been substantial research on the link between linguistic complexity analysis and The main strand of research analyzing the com- second language proficiency and writing devel- plexity and accuracy constructs targets the assess- opment for English (cf., e.g., Bulte´ and Housen, ment of second language development. Linguis- 2014; Kyle, 2016), much less is known about aca- tic complexity measures have been successfully demic language development for other languages, used to model the language acquisition of English 380 Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 380–393 Florence, Italy, August 2, 2019. c 2019 Association for Computational Linguistics as a Second Language (ESL) learners (Bulte´ and results for accuracy. This is in line with findings Housen, 2014; Crossley and McNamara, 2014). by Yoon and Polio(2016), who investigate the ef- Work on first language writing development for fect of genre differences on CAF constructs. Yoon English has also been conducted, but it is less (2017) focuses on the effect of topic on the syn- common (Crossley et al., 2011) The same holds tactic, lexical, and morphological complexity of for the development of accuracy (Larsen-Freeman, ESL learners’ writings and shows a significant in- 2006; Yoon and Polio, 2016). Most studies focus fluence on the complexity of writings of the same on adult ESL learners’ development during peri- learners, similar to findings in Yang et al.(2015). ods of instruction. Vercellotti(2015) finds an in- Such task effects have mostly been discussed from crease in syntactic and lexical complexity, overall a theoretical perspective, considering their impli- accuracy, and fluency in adult ESL speech over the cations for the development of CAF constructs and course of several months. Crossley and McNa- the two main task frameworks (Robinson, 2001; mara(2014) find that advanced adult ESL learn- Skehan, 1996). From a more practical perspec- ers phrasal and clausal complexity significantly in- tive, task, genre, and topic effects have been rec- creases over the course of one semester of writ- ognized as an important issue for machine learn- ing instruction in particular with regard to nominal ing for readability assessment or Automatic Es- modification and number of clauses. These find- say Scoring (AES). For the real-world applica- ings are corroborated by Bulte´ and Housen(2014). bility of such approaches it is crucial for them For uninstructed settings, however, this does not to account for differences due to genre or topic. hold. Knoch et al.(2014, 2015) study university In their readability assessment system READ-IT students’ ESL development over 12 months and for Italian, Dell’Orletta et al.(2014) use this is- three years without instruction in an immersion sue to motivate favoring a ranking-based over a context and found that only fluency but not gram- classification-based approach. A recent AES ap- matical and lexical complexity developed. proach discussing the issue is the placement sys- tem for ESL by Yannakoudakis et al.(2018). Research on languages other than English is starting to appear (Hancke et al., 2012; Velle- 3 Data man and van der Geest, 2014; Pilan´ and Volod- ina, 2016; Reynolds, 2016). As for English, re- Our studies are based on the Karlsruhe Children’s search on German writing development has pre- Text (KCT) corpus by Lavalley et al.(2015). 1 It dominantly focused on German as a Second Lan- is a cross-sectional collection of 1,701 German guage (GSL) in instructed settings (Byrnes, 2009; texts produced by students in German elementary Byrnes et al., 2010; Vyatkina, 2012). Their find- and secondary school students from first to eighth ings suggest that as for ESL learners’ writing, grade. The secondary school students in the cor- clausal complexity progressively increases. For pus attended one of two German school tracks, ei- lexical complexity results have been more mixed ther a basic school track (Hauptschule) or an inter- depending on the proficiency of GSL learners’ mediate school track (Realschule). The texts were proficiency level. The development of writing written on a topic chosen by the students from a accuracy has also been assessed in some corpus set of age-appropriate options: Elementary school studies using automated or manual error annota- students were asked to continue one of two sto- tion (Lavalley et al., 2015;G opferich¨ and Neu- ries, one about children playing in a park, and the mann, 2016). In Weiss et al.(2019) we analyze other about a wolf who learns how to read. Sec- the impact of linguistic complexity and accuracy ondary school students wrote about a hypothetical on teacher grading behavior. day spent with their idol or their life in 20 years. All student texts in the corpus are made available One challenge for the assessment of language in the original, including all student errors, and a performance in terms of complexity that is start- normalized version, where errors and misspellings ing to receive attention is the influence of the task. were corrected. The data is enriched with error an- Alexopoulou et al.(2017) demonstrate task ef- notations covering word splitting, incorrect word fects, specifically task complexity and task type, choices and repetitions, grammar, and legibility. on the complexity of English as a Second Lan- For our studies analyzing writing development guage writers in the EF-Cambridge Open Lan- guage Database (EFCAMDAT) and show mixed 1https://catalog.ldc.upenn.edu/LDC2015T22 381 in terms of development across the grade levels, and grammatical errors. Annotations at the sen- we made use of the normalized texts and the error tence level mark content deletions, insertions, and annotation. Some grade levels in the corpus in- incorrect word choices. In addition, we developed clude only few texts, such as the 42 cases of first an approach to automatically derive additional er- grade writings compared to the other grade levels ror types by comparing the original student writ- with 189 to 283 writings. We thus grouped ad- ings with their normalized sentence-aligned target jacent grade levels, i.e., grades 1 and 2 together, hypotheses. This