DOCSLIB.ORG

Modelling Language Learning of Duolingo Users

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Modelling Language Learning of Duolingo Users Modelling language learning of Duolingo users Floor Dikker STUDENT NUMBER: 2017217 THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE IN DATA SCIENCE & SOCIETY DEPARTMENT OF COGNITIVE SCIENCE & ARTIFICIAL INTELLIGENCE SCHOOL OF HUMANITIES AND DIGITAL SCIENCES TILBURG UNIVERSITY Thesis committee: Dr. Hendrickson Dr. Alishahi Tilburg University School of Humanities and Digital Sciences Department of Cognitive Science & Artificial Intelligence Tilburg, The Netherlands June, 2019 Data Science and Society 2019 2 Data Science and Society 2019 Preface Dear reader, I present to you my thesis about modelling language learning of users of Duolingo. First of all, I would like to thank Dr. Hendrickson for his guidance. I have found our meetings and your suggestions very helpful. Moreover, your enthusiasm is very contagious which made writing this thesis a more fun experience. I would also like to thank my parents and brother for offering support during my entire studies and always offering a listening ear. Finally, I would like to thank my boyfriend for supporting me all the way. I hope you enjoy reading my thesis, Floor Dikker 3 Data Science and Society 2019 Table of Contents 1. Introduction ....................................................................................................... 6 2. Related Work..................................................................................................... 9 2.1 The SLAM challenge .................................................................................... 9 2.2 Relevant features ........................................................................................ 10 2.2.1 Linguistic features................................................................................ 10 2.2.2 Learning process features ...................................................................... 12 2.3 Unbalanced data ......................................................................................... 15 2.4 Added value of the thesis ............................................................................. 15 3. Experimental Setup .......................................................................................... 16 3.1 The data set ............................................................................................... 16 3.1.1 Description of the data set used in the experiment .................................... 16 3.1.2 Data structure ...................................................................................... 16 3.1.3 Software ............................................................................................. 17 3.2 Data manipulation ...................................................................................... 17 3.2.1 Preprocessing steps .............................................................................. 17 3.2.2 Feature engineering .............................................................................. 18 3.2.3 Outliers and missing values ................................................................... 19 3.3 Explanatory Data Analysis ........................................................................... 21 3.3.1 The target feature ................................................................................. 21 3.3.2 Linguistic features................................................................................ 21 3.3.3 Learning process features ...................................................................... 23 3.4 Methods .................................................................................................... 25 3.4.1 Logistic Regression .............................................................................. 25 3.4.2 Random Forest Classifier ...................................................................... 25 3.5 Experimental procedure............................................................................... 25 3.5.1 The Baseline ....................................................................................... 25 3.5.2 Algorithms .......................................................................................... 26 3.5.3 Hyperparameters.................................................................................. 26 3.5.4 Evaluation metrics ............................................................................... 26 3.5.5 Down sampling ................................................................................... 27 4. Results ............................................................................................................ 27 4.1 The baseline model ..................................................................................... 27 4.1.1 The LR baseline................................................................................... 27 4.1.2 The ZeroR Classifier ............................................................................ 28 4.2 The other models ........................................................................................ 29 4.3 Feature importance ..................................................................................... 31 4 Data Science and Society 2019 4.4 Error analysis ............................................................................................. 32 4.4.1 Number of characters ........................................................................... 32 4.4.2 Morphological features ......................................................................... 33 4.4.3 Word .................................................................................................. 34 4.4.4 Part of Speech ..................................................................................... 34 4.4.5 Edge Label.......................................................................................... 35 5. Discussion ....................................................................................................... 35 6. Conclusion ...................................................................................................... 39 References .......................................................................................................... 40 Appendix A: Features in the original data set........................................................... 44 Appendix B: Parameter settings of the models ......................................................... 45 Appendix C: Plots of the logistic regression models ................................................. 46 Appendix D: Plots of the random forest models ....................................................... 47 Appendix E: Visual representation of the features and the error ................................. 48 Appendix F: Distribution comparisons train and test set ............................................ 49 5 Data Science and Society 2019 Modelling language learning of Duolingo users Floor Dikker This thesis looks to conclude which features are most effective when modelling second language acquisition. In order to do so in a transparent manner, a division is made between learning process features and linguistic features. Both categories of features are modelled with a random forest classifier and a logistic regression. The linguistic features, modelled with a random forest classifier, are concluded to be most effective to predict word learning. Moreover, an error analysis is conducted to further investigate the errors made by the best performing model. Based on the error analysis, the conclusion is made that the model is more prone to predict certain classes or ranges within the features incorrectly, which is assumed to be caused by a different distribution of the features in the train versus the test set. 1. Introduction When learning a new language, a learner must devote attention to numerous interacting elements of a language. For instance, not only the vocabulary must be learned, but also the structure of the language must be assimilated. As these various elements cannot be dealt with in isolation, Second Language Acquisition (SLA) is found to be challenging for many people. Traditionally, one of the most common ways to learn a language has been by means of classroom education. However, in recent years, there has been a large increase in the number of learners that have chosen to acquire a second language via educational software. This new trend is often referred to as ‘mobile learning’ or ‘m-learning’, which implies that learners cannot only learn where they want but also when they want (Kim & Yeonhee, 2012). The rapid growth of mobile learning poses new challenges for researchers. People learn in a different manner when using such platforms compared to the more traditional ways and, as such, the field is said not to align with established SLA theories and other pedagogical research (Heil, Wu, Lee, & Schmidt, 2017). Goodwin & Highfield (2012) state that researchers in these fields have failed to keep pace with the exponential growth of the amount of educational software available for language learning. Nonetheless, the issue has been addressed to some extent by the teams participating in the Second Language Acquisition Modelling (SLAM) challenge, using a data set from Duolingo. Duolingo is an award-winning language learning platform on which more than 200 million students have enrolled. According to the company, their success comes from offering personalized education, a fun learning experience and accessibility from every location1. A benefit of the many learners
Load more

Recommended publications
  • A Queueing-Theoretic Foundation for Optimal Spaced Repetition
    A Queueing-Theoretic Foundation for Optimal Spaced Repetition Siddharth Reddy [email protected] Department of Computer Science, Cornell University, Ithaca, NY 14850 Igor Labutov [email protected] Department of Electrical and Computer Engineering, Cornell University, Ithaca, NY 14850 Siddhartha Banerjee [email protected] School of Operations Research and Information Engineering, Cornell University, Ithaca, NY 14850 Thorsten Joachims [email protected] Department of Computer Science, Cornell University, Ithaca, NY 14850 1. Extended Abstract way back to 1885 and the pioneering work of Ebbinghaus (Ebbinghaus, 1913), identify two critical variables that de- In the study of human learning, there is broad evidence that termine the probability of recalling an item: reinforcement, our ability to retain a piece of information improves with i.e., repeated exposure to the item, and delay, i.e., time repeated exposure, and that it decays with delay since the since the item was last reviewed. Accordingly, scientists last exposure. This plays a crucial role in the design of ed- have long been proponents of the spacing effect for learn- ucational software, leading to a trade-off between teaching ing: the phenomenon in which periodic, spaced review of new material and reviewing what has already been taught. content improves long-term retention. A common way to balance this trade-off is spaced repe- tition, which uses periodic review of content to improve A significant development in recent years has been a grow- long-term retention. Though spaced repetition is widely ing body of work that attempts to ‘engineer’ the process used in practice, e.g., in electronic flashcard software, there of human learning, creating tools that enhance the learning is little formal understanding of the design of these sys- process by building on the scientific understanding of hu- tems.
    [Show full text]
  • Spaced Learning Strategy in Teaching Mathematics Ace T
    International Journal of Scientific & Engineering Research, Volume 8, Issue 4, April-2017 851 ISSN 2229-5518 Spaced Learning Strategy in Teaching Mathematics Ace T. Ceremonia, Remalyn Q. Casem Abstract—Students’ low mastery of the lesson in Mathematics is one of the alarming problems confronted by Mathematics teachers. It is in this light that this study was conceptualized to determine the effectiveness of spaced learning strategy on the performance and mastery of DMMMSU Laboratory High School students (Grade 7) in Mathematics. This study used the true experimental design, specifically the pretest-posttest control group design. The main instrument used to gather data was the pretest-posttest which was subjected to validity and reliability tests. It was found out that the experimental and control groups were comparable in the pretest and posttest. Comparison on their gain scores revealed significant difference with the performance of the experimental higher than the control group. It was also found out that the effect size of using the spaced learning strategy was large. This indicates that the intervention is effective in increasing the performance and mastery of high school students in Mathematics. It is recommended the use of the Spaced Learning Strategy to improve the performance of the high school students in Mathematics. Index Terms— basic education, effect size, mathematics performance, spaced learning strategy. —————————— —————————— 1 INTRODUCTION 1.1 Situation Analysis results, such as the 2003 TIMSS (Trends in International Mathematics and Science Study), the Philippines ranked 34th out uality education is reflected on the exemplary outputs and of 38 countries in HS II Math and ranked 43rd out of 46 countries Q outstanding performances of students in academics and in HS II Science; for grade 4, the Philippines ranked 23rd out of 25 related co-curricular activities (Glossary Education Reform, participating countries in both Math and Science.
    [Show full text]
  • Serial Position Effects and Forgetting Curves: Implications in Word
    Studies in English Language Teaching ISSN 2372-9740 (Print) ISSN 2329-311X (Online) Vol. 2, No. 3, 2014 www.scholink.org/ojs/index.php/selt Original Paper Serial Position Effects and Forgetting Curves: Implications in Word Memorization Guijun Zhang1* 1 Department of Foreign Languages, China Pharmaceutical University, Nanjing, China * Guijun Zhang, E-mail:[email protected] Abstract Word memorization is important in English learning and teaching. The theory and implications of serial position effects and forgetting curves are discussed in this paper. It is held that they help students understand the psychological mechanisms underlying word memorization. The serial position effects make them to consider the application the chunking theory in word memorization; the forgetting curve reminds them to repeat the words in long-term memory in proper time. Meanwhile the spacing effect and elaborative rehearsal effect are also discussed as they are related to the forgetting curve. Keywords serial position effects, forgetting curves, word memorization 1. Introduction English words are extraordinarily significant for English foreign language (EFL) learners because they are the essential basis of all language skills. As Wilkins said, “...while without grammar very little can be conveyed, without vocabulary nothing can be conveyed” (Wilkins, 1972). Effective word memorization plays a significant role in the process of vocabulary learning. Researchers have and are still pursuing and summarizing the effective memory methods. Schmitt, for example, classified vocabulary memory strategies into more than twenty kinds (Schmitt, 1997, p. 34). However, it is hard to improve the efficiency of the vocabulary memory in that different students remember the huge amount of words with some certain method or methods that may not suit them.
    [Show full text]
  • Where Companies Go Wrong with Learning and Development by Steve Glaveski
    REPRINT H055J1 PUBLISHED ON HBR.ORG OCTOBER 02, 2019 ARTICLE BUSINESS EDUCATION Where Companies Go Wrong with Learning and Development by Steve Glaveski This article is made available to you with compliments of EGN for your personal use. Further posting, copying, or distribution is not permitted. BUSINESS EDUCATION Where Companies Go Wrong with Learning and Development by Steve Glaveski OCTOBER 02, 2019 GREGOR SCHUSTER/GETTY IMAGES Organizations spent $359 billion globally on training in 2016, but was it worth it? Not when you consider the following: • 75% of 1,500 managers surveyed from across 50 organizations were dissatisfied with their company’s Learning & Development (L&D) function; COPYRIGHT © 2019 HARVARD BUSINESS SCHOOL PUBLISHING CORPORATION. ALL RIGHTS RESERVED. 2 • 70% of employees report that they don’t have mastery of the skills needed to do their jobs; • Only 12% of employees apply new skills learned in L&D programs to their jobs; and • Only 25% of respondents to a recent McKinsey survey believe that training measurably improved performance. Not only is the majority of training in today’s companies ineffective, but the purpose, timing, and content of training is flawed. Learning for the Wrong Reasons Bryan Caplan, professor of economics at George Mason University, and author of The Case Against Education, says in his book that education often isn’t so much about learning useful job skills, but about people showing off, or “signaling.” Today’s employees often signal through continuous professional education (CPE) credits so that they can make a case for a promotion. L&D staff also signal their worth by meeting flawed KPIs, such as the total CPE credits employees earn, rather than focusing on the business impact created.
    [Show full text]
  • Teaching the Science of Learning Yana Weinstein1* , Christopher R
    Weinstein et al. Cognitive Research: Principles and Implications (2018) 3:2 Cognitive Research: Principles DOI 10.1186/s41235-017-0087-y and Implications TUTORIALREVIEW Open Access Teaching the science of learning Yana Weinstein1* , Christopher R. Madan2,3 and Megan A. Sumeracki4 Abstract The science of learning has made a considerable contribution to our understanding of effective teaching and learning strategies. However, few instructors outside of the field are privy to this research. In this tutorial review, we focus on six specific cognitive strategies that have received robust support from decades of research: spaced practice, interleaving, retrieval practice, elaboration, concrete examples, and dual coding. We describe the basic research behind each strategy and relevant applied research, present examples of existing and suggested implementation, and make recommendations for further research that would broaden the reach of these strategies. Keywords: Education, Learning, Memory, Teaching Significance educators who might be interested in integrating the Education does not currently adhere to the medical strategies into their teaching practice, (b) science of learn- model of evidence-based practice (Roediger, 2013). ing researchers who are looking for open questions to help However, over the past few decades, our field has made determine future research priorities, and (c) researchers in significant advances in applying cognitive processes to other subfields who are interested in the ways that princi- education. From this work, specific recommendations ples from cognitive psychology have been applied to can be made for students to maximize their learning effi- education. ciency (Dunlosky, Rawson, Marsh, Nathan, & Willing- While the typical teacher may not be exposed to this ham, 2013; Roediger, Finn, & Weinstein, 2012).
    [Show full text]
  • Nicolas Schweighofer's Curriculum Vitae
    CURRICULUM VITAE NICOLAS SCHWEIGHOFER, PhD PERSONAL INFORMATION University Address: Division of Biokinesiology and Physical Therapy University of Southern California Los Angeles, CA, 90089-9006 Phone: (1) 323 -442 -1838 Email: [email protected] Lab website: http://pt.usc.edu/labs/cnrl EDUCATION AND PROFESSIONAL APPOINTMENTS UNIVERSITY EDUCATION 1991-1995 PhD, University of Southern California, Los Angeles: Neuroscience, (Computational Neuroscience Emphasis) (Advisor: Dr. M. Arbib) 1988-1990 Diplôme d’Ingénieur, Ecole Nationale Supérieure de Mécanique, Nantes, France (since ‘Ecole Centrale de Nantes’): Mechanics and Control System Engineering 1985-1987 Undergraduate training: Mathématiques Spéciales P’, Lycée Descartes, Tours, France: Advanced Mathematics and Physics POST-GRADUATE TRAINING 1996-1997 Postdoctoral Fellow, Human Information Science, ATR, Kyoto, Japan: Computational Neuroscience (Mentor: Dr. M. Kawato) APPOINTMENTS 2011-Present Associate Professor (with Tenure), Division of Biokinesiology & Physical Therapy, University of Southern California 2011-Present Joint Appointment (courtesy), Department of Biomedical Engineering, Viterbi School of Engineering, University of Southern California 2004-Present Joint Appointment (courtesy), Department of Computer Science, Viterbi School of Engineering, University of Southern California 2004-Present Joint Appointment (courtesy), Neuroscience Graduate Program, University of Southern California 2004-2010 Assistant Professor (Tenure track), Division of Biokinesiology & Physical Therapy, University
    [Show full text]
  • Spacing Effects and Their Implications for Theory and Practice
    Eéucational Psychology Review, Vol. 1, No. 4, 1989 Spacing Effects and Their Implications for Theory and Practice Frank N. Dempster! ee, on differentnoes evidence, gathered in a variety of settings and across realestpees of materials and procedures, that spaced repetitions — or success ether they are in theform of additional study opportunities Research yu tests —are a highly effective means of promoting learning. inplications spacing effects is reviewed andits theoretical and educational iaderstood| are examined. it is concluded that spacing effects can best be ito in terms of the accessibility hypothesis, and that spaced repe- titionshave considerable potential for improving classroomlearning. KEYEY WORDS:W memory;practice; learning; cognition. THE SPACING EFFECT . Ask almost any student of learning and memory and he/she will tell ; Uthat the spacing effect refers to the finding that for a given amount of “udy time, spaced presentationsyield significantly better learning than do esentations that are massed moreclosely together in time. The spacing cf- “tis, in fact, one of the most thoroughly studied phenomena in psycholo- ey. However, what is not so widely appreciated is just how remarkable the Pacing effect is. __ Consider, for example, its impressive length of service as a subject of “entific inquiry. The earliest documented studies of the spacing effect were corded by Ebbinghausin his seminal work on memory, originally published 11885. With himself as the subject, Ebbinghau s noted that for a single I2-syllable series, 68 immediately successive repetitions had the effect of Las Department of Educational Psychology, College of Education, University of Nevada, 310 Dempster making possible an errorless recital after seven additionalrepetitionsonthe following day.
    [Show full text]
  • Spacing Learning Events Over Time: What the Research Says
    Spacing Learning Events Over Time: What the Research Says Will Thalheimer, PhD A Work-Learning Research, Inc. Publication. © Copyright 2006 by Will Thalheimer. You may share this report as long as you share it intact and whole and don’t charge a price for it or include it in any priced event or product. Published February 2006 Spacing Learning Over Time Will Thalheimer, PhD How to cite this report using APA style: Thalheimer, W. (2006, February). Spacing Learning Events Over Time: What the Research Says. Retrieved November 31, 2006, from http://www.work-learning.com/catalog/ Obviously, you should substitute the date on which the document was downloaded for the fictitious November 31 date. Published 2006 The Author: Will Thalheimer is a learning-and-performance consultant and research psychologist specializing in learning, cognition, memory, and performance. Dr. Thalheimer has worked in the workplace learning field, beginning in 1985, as an instructional designer, simulation architect, project manager, product leader, trainer, consultant, and researcher. He has a PhD from Columbia University and an MBA from Drexel University. He founded Work- Learning Research in 1998 to help client organizations create learning environments to maximize performance and help learning professionals—instructional designers, e-learning developers, trainers, performance consultants, talent managers, and chief learning officers— utilize research-based knowledge to build effective learning-and-performance solutions. Dr. Thalheimer can be contacted at [email protected] or at the Work-Learning Research phone number below for inquiries about learning audits, workshops, speaking engagements, high-level instructional design, and consulting on e-learning, classroom training, learning measurement, learning strategy, and business strategy for learning.
    [Show full text]
  • Distributed Practice: the Research Base
    Distributed Practice: The Research Base Studies of the effects of the timing of practice on learning and retention have a long history. The positive effects of “spaced” rather than “massed” practice were recognized as early as 1885 when the German psychologist Hermann Ebbinghaus published his seminal work on memory. Over the past century, Ebbinghaus’s findings have been repeatedly confirmed and extended. Strong positive effects of spaced practice have been found in a wide variety of contexts. Carlous Caple summarized this body of research as follows: The spacing effect is an extremely robust and powerful phenomenon, and it has been repeatedly shown with many kinds of material. Spacing effects have been demonstrated in free recall, in cued recall of paired associations, in the recall of sentences, and in the recall of text material. It is important to note that these spacing results do generalize to textbook materials, meaning that subjects such as science can be manipulated by spacing effects. Also the effect of spaced study can be very long-lasting. (Caple, 1996, p. 22) The role of distributed, or spaced, practice in the learning of mathematics has also been studied. Marilyn Suydam’s 1985 ERIC digest (ED 260891) summarized research about the role of review in mathematics instruction. She wrote, “Long-term retention is best served if assignments on a particular skill are spread out in time rather than concentrated within a short interval.” Suydam also noted that short periods of intensive review are better than long periods, and that games provide effective review. The relevance of this body of research to the construction of elementary school mathematics curriculum was shown by the translations of Russian textbooks carried out by the University of Chicago School Mathematics Project in the 1980s.
    [Show full text]
  • Why Interleaving Enhances Inductive Learning: the Roles of Discrimination and Retrieval
    Mem Cogn DOI 10.3758/s13421-012-0272-7 Why interleaving enhances inductive learning: The roles of discrimination and retrieval Monica S. Birnbaum & Nate Kornell & Elizabeth Ligon Bjork & Robert A. Bjork # Psychonomic Society, Inc. 2012 Abstract Kornell and Bjork (Psychological Science 19:585– Later in life, we might learn to distinguish between dif- 592, 2008) found that interleaving exemplars of different ferent species of butterflies or birds, as in the present categories enhanced inductive learning of the concepts based research. Such inductive learning is critical in making on those exemplars. They hypothesized that the benefit of sense of events, objects, and actions—and, more generally, mixing exemplars from different categories is that doing so in structuring and understanding our world. In the present highlights differences between the categories. Kang and Pash- research, we examined how exemplars of to-be-learned cate- ler (Applied Cognitive Psychology 26:97–103, 2012) obtained gories should be sequenced and spaced in order to optimize results consistent with this discriminative-contrast hypothesis: inductive learning. Interleaving enhanced inductive learning, but temporal spac- Kornell and Bjork (2008) investigated the effect of study ing, which does not highlight category differences, did not. We schedules on inductive learning—specifically, learning further tested the discriminative-contrast hypothesis by exam- artists’ painting styles from exemplars of their paintings. ining the effects of interleaving and spacing, as well as their Images of six paintings by each of 12 artists were presented combined effects. In three experiments, using photographs of for study, with the artist’s name displayed below each paint- butterflies and birds as the stimuli, temporal spacing was ing.
    [Show full text]
  • The IALLT Journal a Publication of the International Association for Language Learning Technology
    The IALLT Journal A publication of the International Association for Language Learning Technology A CROWDSOURCING APPROACH TO CHINESE VOCABULARY LEARNING Jianxiong Wu Southern Connecticut State University ABSTRACT The fast development of new technology, particularly web-based and mobile technology, has been transforming ways how languages are taught and learned. The current study examines the use of Memrise, a crowdsourcing spaced repetition program, by college students of Chinese, and its effects on Chinese character learning. The findings of the study demonstrate the effectiveness of the program. The study has also identified important factors and conditions that should exist in order for the use of the program to be successful. Based on the findings of the study, suggestions are made for language teachers in regards to using Memrise or similar programs in vocabulary teaching. Even though the current study focuses on Chinese character learning, the findings of the study can be applied to any other languages. INTRODUCTION The biggest challenge in studying Chinese is to recognize and write Chinese characters. The learning of Chinese characters is even more difficult for students whose native language is English or any other Roman alphabetic language. Teachers of Chinese always seek better strategies and tools for teaching Chinese characters. The development and application of new technology have opened up new opportunities for making this challenging task easier and more successful. 43 1 Vol. 44 (2) 2015 A Crowdsourcing Approach to Chinese Vocabulary Learning The majority of the existing research demonstrates that the use of technology enhances foreign and second language vocabulary learning and acquisition (e.g.
    [Show full text]
  • Continual Reinforcement Learning with Memory at Multiple Timescales
    Imperial College Department of Computing Continual Reinforcement Learning with Memory at Multiple Timescales Christos Kaplanis Submitted in part fulfilment of the requirements for the degree of Doctor of Philosophy in Computing of the University of London and the Diploma of Imperial College, April 2020 Declaration of Originality and Copyright This is to certify that this thesis was composed solely by myself. Except where it is stated otherwise by reference or acknowledgment, the work presented is entirely my own. The copyright of this thesis rests with the author. Unless otherwise indicated, its contents are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Licence (CC BY NC-SA). Under this licence, you may copy and redistribute the material in any medium or format. You may also create and distribute modified versions of the work. This is on the condition that; you credit the author, do not use it for commercial purposes and share any derivative works under the same licence. When reusing or sharing this work, ensure you make the licence terms clear to others by naming the licence and linking to the licence text. Where a work has been adapted, you should indicate that the work has been changed and describe those changes. Please seek permission from the copyright holder for uses of this work that are not included in this licence or permitted under UK Copyright Law. i ii Abstract In the past decade, with increased availability of computational resources and several improve- ments in training techniques, artificial neural networks (ANNs) have been rediscovered as a powerful class of machine learning methods, featuring in several groundbreaking applications of artificial intelligence.
    [Show full text]