DEGREE PROJECT IN COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2020 Designing and using gamification elements to improve students’ user experience in a video-based mobile language learning app THOR GALLE KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE 1 Abstract With the rise of the smartphone industry, the domain of mobile-assisted language learning (MALL) has increasingly grown. A large number of language learning applications have been developed aiming to support individuals’ second language acquisition on various levels, e.g., by teaching vocabulary and grammar to improve reading and listening comprehension. The viability of these applications has been examined in literature and shows overall positive but mixed results. On one side, their success is partly attributed to gamified design elements. These are reported to improve the user experience (UX) and boost learners' motivation. On the other side, the primary reliance on decontextualized vocabulary and grammar exercises is criticized. In response, one such application, SVT Språkplay developed by the Swedish non-profit ​ ​ Språkkraft, incorporated television programs as a longer form of context. This introduced novel video-based learning functions. The first aim of this thesis was to start filling a gap in research by evaluating the usability and user experience of these functions. This was performed through user tests and interviews with seven second language students who used the app to learn Swedish over a period of at least two weeks. The second aim of the thesis was to improve the usability and user experience of the problematic learning functions through a user-centred design process with the ultimate goal to improve learner support and vocabulary acquisition outcomes. The study participants consisted of doctoral researchers and students recruited from a basic Swedish course at KTH. They represent a demographic that benefits from learning Swedish to improve their job opportunities. The initial evaluation results were analysed through the lens of the MUUX-E theoretical framework [10], a framework for evaluating the “user experience and educational features ​ ​ of m-learning environments”. The evaluation showed that the core vocabulary learning aids directly integrated into the video watching experience were perceived as useful. Conversely, the gamified learning functions outside of the video watching experience were found to be scarcely used as intended. The subsequent user-centered design process improved upon the design of problematic learning functions by adhering to the principles of the MUUX-E framework. Concretely, more varied contextualized vocabulary exercises were designed, more options for user customization were included and feedback and progress metrics such as “streaks” were highlighted. An evaluation of the design with the same participants as the initial evaluation suggests that these changes would improve the usability and user experience of the application. Further research should evaluate an implemented end-product based on the proposed designs in a real-life setting. In that case, its pedagogical merit should also be evaluated. In summary, this thesis found that mobile video-based MALL apps such as Språkplay can provide usable and enjoyable language learning functions. 1 1.1 Keywords mobile-assisted language learning, video-based language learning, usability, UX, usability, user experience and educational features of m-learning environments (MUUX-E), gamification 2 Abstract | 3 2 Sammanfattning Med tillväxten av mobiltelefonbranschen har domänen för mobilassisterad språkinlärning (MALL) ökat alltmer. Ett stort antal språkinlärningsapplikationer har utvecklats för att stödja individers förvärv av andraspråk på olika nivåer, t.ex. genom att lära ut ordförråd och grammatik samt för att förbättra läs- och hörförståelsen. Dessa applikationer har undersökts i litteraturen och visar positiva men blandade resultat. Å ena sidan tillskrivit deras framgång delvis spelelementen. Dessa rapporteras förbättra användarupplevelsen (UX) och öka elevernas motivation. Å andra sidan kritiseras det primära förlitandet på dekontekstualiserade ordförråd och grammatikövningar. Som ett svar skapades en sådan applikation, SVT Språkplay, utvecklad av den svenska ​ ​ ideella föreningen Språkkraft, vilken använder TV-program som en längre form av språkligt sammanhang. Detta introducerade nya videobaserade inlärningsfunktioner. Det första syftet med denna uppsats var att börja fylla ett hål i forskningen genom att utvärdera användbarheten och användarupplevelsen av dessa funktioner. Det gjordes genom att genomföra användartester och intervjuer med sju andraspråkstudenter som använde appen för att lära sig svenska under en period av två veckor. Det andra syftet med arbetet var att förbättra användbarheten och användarupplevelsen för dessa inlärningsfunktioner genom en användarcentrerad designprocess med det slutliga målet att förbättra studentens stöd. Studiedeltagarna bestod av doktorander och studenter rekryterade från en nybörjarkurs i svenska på KTH. De representerar en demografisk nytta av att lära sig svenska för att öka deras tillgång till den svenska arbetsmarknaden. De första utvärderingsresultaten analyserades genom tillämpning av MUUX-E-ramverket, ett ramverk för att utvärdera “user experience and educational features of m-learning ​ ​ ​ ​ ​ ​ environments” [10]. Det visade att de grundläggande ordförrådets inlärningshjälpmedel ​ ​ ​ som direkt integrerades i video upplevdes som användbara. Omvänt användes knappt alls de spelifierade inlärningsfunktionerna utanför videon. Den efterföljande användarcentrerade designprocessen förbättrades vid design av problematiska inlärningsfunktioner genom att följa principerna i MUUX-E-ramverket. Konkret utformades mer varierade kontextualiserade vokabulärövningar, fler alternativ för användaranpassning inkluderades och feedback- och framstegsmetriker som “streaks” lyftes fram. En utvärdering av designen med samma deltagare som den första utvärderingen tyder på att dessa förändringar skulle förbättra användbarheten och användarupplevelsen. Ytterligare forskning bör utvärdera en implementerad slutprodukt baserad på de föreslagna designförbättringarna i en verklig miljö. I så fall bör dess pedagogiska meriter också utvärderas. Sammanfattningsvis fann vi att videobaserade MALL-appar som Språkplay kan ge användbara och roliga språkinlärningsfunktioner. 3 2.1 Nyckelord mobile-assisted language learning, video-based language learning, usability, UX, usability, user experience and educational features of m-learning environments (MUUX-E), gamification 4 3 Acknowledgments First of all I would like to thank Niss Jonas Carlsson, CEO of Språkkraft and industrial adviser to this thesis, for allowing me to examine and reimagine the app developed by his non-profit. The regular discussions we had and insights from Språkkrafts’ earlier R&D efforts were more than once an inspiration for my work. My views did not always align with the direction of the app development, so I appreciate Niss Jonas’ willingness to welcome a different perspective on the design of the app. I genuinely believe Språkkraft is contributing to an important social challenge in an innovative way, and I am grateful that my thesis could contribute to that. Next, I would like to thank my supervisor Olga Viberg for the continuous support in the planning and writing of this thesis. Her knowledge of the field and regular and precise feedback were an invaluable source of help. In this context, I should also mention the group supervision sessions where my peers Edward Lindén, Xinyan Luo and Martin Wedberg taught me useful lessons, and where I could share my progress and concerns once in a while. Finally, I want to express my gratitude to the seven study participants who voluntarily spent their time on interviews and user tests to help improve Språkplay. Without them, this study would have been difficult. Stockholm, May 2020 Thor Galle 5 4 Table of contents 1 Abstract 1 1.1 Keywords 2 2 Sammanfattning 3 2.1 Nyckelord 4 3 Acknowledgments 5 4 Table of contents 6 5 List of acronyms and abbreviations 12 6 Introduction 1 6.1 Context of this work 2 6.1.1 Societal background 2 6.1.2 Learning language with television content 3 6.1.3 SVT Språkplay: the solution? 3 6.1.4 Reception of Språkplay 4 6.2 Problem 5 6.3 Purpose 7 6.4 Goals 7 6.5 Research Methodology 7 6.6 Delimitations 8 6.7 Summary 9 6.8 Structure of the thesis 9 7 Description of the examined application 10 7.1 Video watching functions 10 7.1.1 Control functions (pause, jump forward, jump backward) 10 7.1.2 The translation pop-up 11 7.1.3 The color system 12 Initial color determination 12 Changing word color 13 7.2 Active learning functions 13 7.2.1 Word test 13 7.2.2 Word list 14 6 7.3 Self-monitoring functions 15 7.3.1 Profile page 15 7.3.2 Activity statistics dashboard 15 8 Background 16 8.1 Video and language learning 16 8.1.1 Media and language learning 16 8.1.2 Learning with Closed Captions 16 8.1.3 Control and “glossed” captions 17 8.1.4 Implications for a video-based MALL application 17 8.2 Mobile assisted language learning (MALL) 17 8.2.1 Defining MALL 17 8.2.2 MALL with immigrants 18 8.2.3 Video-based learning in MALL 18 8.3 The role of gamification 19 8.3.1 Defining gamification 19 8.3.2 Game-based learning and gamification within MALL 19 8.3.3 The value of gamification for vocabulary learning 20 8.3.4 Implications for this project 20 8.4 Usability,