EME6208 Interactive Media - IMA2: Usability testing report

IMA 2 - Usability testing report

Duolingo

Alia Hadid Carla Butel Muhittin Cavusoglu Rabea Alfahad

October 4, 2016

1 2TEST REPORT

Introduction Duolingo is a free, science-based, language-learning application available on both IOS and Android mobile devices as well as through a website. The lessons incorporate speaking, listening, translation and multiple choice exercises presented in a game-like environment. As with Rosetta Stone, instead of being taught concepts or grammar, the user is immersed in the language; however, unlike Rosetta Stone, the translations are crowdsourced.

In this usability report, only the mobile version was tested. Eight participants were selected; four participants used Duolingo on an IOS device and four participants used Duolingo on an Android device. Surprisingly, overall user satisfaction seemed to be impacted by the platform used with Android users reporting greater satisfaction than IOS users. In order to reduce the stress of performing an unknown task in an unfamiliar environment, the study was conducted by testers with whom the participants already knew while in a comfortable location.

Methodology

Sessions The participants were known to the testers prior to the study; therefore, there was an easy, relaxed atmosphere during the sessions, and the participants were able to perform the tasks without the stress of being observed by a stranger. The locations were either in the participant’s home or the tester’s home. The purpose of the study was explained to the participants, and a brief background survey was conducted (see Attachment A). While the participants performed the tasks, the testers recorded participant comments, the time it took to complete task, and the number of clicks needed to complete the task (see Attachment B). Participants were encouraged to think aloud as they performed the tasks.

After completing the assigned tasks, the participants were given three minutes to study Duolingo. The session concluded with a carefully designed survey composed of a combination of closed-ended, open-ended and scale-based questions. These questions were designed to test the learnability, usability and possibility of producing errors by the mobile application and the participants during the session.

Participants The participants for the study were chosen to represent the broad spectrum of users who might use Duolingo: professionals, students, teachers, and retired individuals. The ages ranged from 20-69 years old. Eight participants were interviewed from September 24 through September 29; seven were female and one was male.

[Type text] [Type text] [Type text] EME6208 Interactive Media - IMA2: Usability testing report

Evaluation Tasks/Scenarios The participants were asked to perform the following tasks:  Select a language to learn  Create a user profile  Change the language to learn  Turn off/on practice reminder  Turn off/on sound effects  Deactivate the speaking option  Choose or change the daily goal

Results

Task Completion Success Rate A total of 8 participants tried Duolingo. All testers took notes while the participants attempted to perform the tasks; the participants were not prompted with the correct method. Four of the participants used the IOS system, whereas the other four used Android. All of the participants successfully completed Task 1 (finding one language they would like to learn and selecting it), Task 2 (creating a profile), Task 3 (changing the language to another one they’d like to learn), Task 5 (turning on/off sound effects), Task 6 (deactivating speaking), and Task 7 (picking a daily goal or changing the daily goal). All of the participants except one, participant 7, completed Task 4 (turning on/off practice reminder). Some of them took longer than others; this has to do with how digitally literate they are and the fact that they were using different systems, which may prove to be easier or more difficult when completing certain tasks.

Table 1: Task Completion Rates Participant Task 1 Task Task 3 Task Task 5 Task 6 Task 7 2 4 IOS 1 √ √ √ √ √ √ √ 2 √ √ √ √ √ √ √ 3 √ √ √ √ √ √ √ 4 √ √ √ √ √ √ √ Success 4 4 4 4 4 4 4 Completion Rates 100% 100% 100% 100% 100% 100% 100% ANDR 5 √ √ √ √ √ √ √ OID 6 √ √ √ √ √ √ √ 7 √ √ √ - √ √ √ 8 √ √ √ √ √ √ √

3 4TEST REPORT

Success 4 4 4 3 4 4 4 Completion Rates 100% 100% 100% 75%% 100% 100% 100% TOTAL Success in Total 8 8 8 7 8 8 8

Learnability: Time to complete a task successfully All testers noted how long it took the participants to complete each task. IOS and Android have different interfaces which accounts for time differences between the two. For example, it took IOS users 104.5 seconds to complete Task 4 (turning on/off practice reminder) while it took Android users only 85.75 seconds. In general, all users found Task 4 the most challenging; participant 7 quit after trying to complete it. The second most difficult was Task 2(setting up a profile). It took IOS users 66.25 seconds, whereas it took Android users an average of 72 seconds. The easiest task was Task 6 (deactivating speaking). It took IOS users on average 12.5 seconds to complete it whereas it took Android users 16.25 seconds.

Table 2: Time on task IOS ANDROID IOS+ANDROID P1 P2 P P4 TO Avg. P P6 P7 P8 TO Avg. TOT Avg. 3 T 5 T Task 1 52 20 11 12 95 23.75 9 15 14 16 54 13.5 149 18.63 Task 2 63 45 50 107 265 66.25 75 145 50 41 311 77.75 576 72 Task 3 90 28 8 12 138 34.5 10 17 65 62 154 38.5 292 36.5 Task 4 123 168 9 118 418 104.5 53 35 100 80 268 67 686 85.75 Task 5 30 24 5 7 66 16.5 15 14 10 21 60 15 126 15.75 Task 6 15 15 4 16 50 12.5 12 33 5 15 65 16.25 115 14.38 Task 7 30 31 6 40 107 26.75 35 23 41 110 209 52.25 316 39.5

Errors To understand how easily or with what difficulty the participants completed the tasks, testers recorded the number of clicks per task. Task 4 (turn off/on practice reminders) yielded the greatest number of errors with IOS users making more errors. Task 2 (creating user profile) was the next in error rate; IOS users made slightly more errors than Android users. Task 3 (change language) was third in error rate; this time Android users had the greatest number of errors—almost double the error rate. Similarly, Task 7 (change daily goal) caused Android users almost double the number of errors compared to IOS users. Task 6 (deactivate speaking) had the greatest disparity in error rate between Android and IOS users, 6 times more. The number of errors for the rest of the tasks was not significantly different between both group with the average number of errors ranging between 0.63 and 2.25.

[Type text] [Type text] [Type text] EME6208 Interactive Media - IMA2: Usability testing report

5 6TEST REPORT

Table 3: Number of Errors per Task

IOS ANDROID IOS+ANDROID P1 P2 P3 P4 TOT Avg. P5 P6 P7 P8 TOT Avg. TOT Avg. Task 1 1 0 0 0 1 0.25 1 1 1 1 4 1.00 5 0.63 Task 2 1 2 10 13 26 6.50 5 14 2 4 25 6.25 51 6.38 Task 3 10 3 1 2 16 4.00 1 1 14 16 32 8.00 48 6.00 Task 4 23 22 1 23 69 17.25 9 3 7 9 28 7.00 97 12.13 Task 5 2 2 1 3 8 2.00 2 1 3 4 10 2.50 18 2.25 Task 6 0 0 1 1 2 0.50 1 6 2 4 13 3.25 15 1.88 Task 7 0 1 0 6 7 1.75 8 3 2 3 16 4.00 23 2.88

Summary of Data Table 4 below is a summary of the data. Low completion rates, high learnability rates, and high number of errors are in bold. A single participant was unable to complete one task, the only low completion rate. IOS users had the highest learnability ratings and the highest number of errors for Task 4 (turning on/off practice reminder). Android users had the highest learnability ratings and the highest number of clicks for Task 2 (creating a profile). Task 1 (select a language) took IOS users longer to complete even though they made fewer clicks. Similarly, IOS users made fewer clicks with Task 5 (turn off/on sound effects) but took longer than Android users. However, with Task 7 (change daily goal) Android users clicked 1.25 times more than IOS users but spent twice as long to complete the task.

Table 4: Summary

IOS ANDROID IOS+ANDROID Task Learnability Errors Task Learnability Errors Task Learnability Errors Completion Completio Completio n n Task 1 4 23.75 0.25 4 13.5 1 8 18.63 0.63 Task 2 4 66.25 6.5 4 77.75 6.25 8 72.00 6.38 Task 3 4 34.5 4 4 38.5 8 8 36.50 6.00 Task 4 4 104.5 17.25 3 67 7 7 85.75 12.13 Task 5 4 16.5 2 4 15 2.5 8 15.75 2.25 Task 6 4 12.5 0.5 4 16.25 3.25 8 14.38 1.88 Task 7 4 26.75 1.75 4 52.25 4 8 39.50 2.88

Satisfaction

One hundred percent of IOS users agreed that Task 5 (turning on/off sound effects) and Task 6 (deactivating speaking) were easy or super easy as compared to 25% and 75% for Android users. However, IOS users reported that Task 2 (creating a profile) and Task 4 (turning on/off practice reminders) were not easy at all while Android users agreed by 25% that the tasks were easy or super easy. Both IOS and Android users agreed that

[Type text] [Type text] [Type text] EME6208 Interactive Media - IMA2: Usability testing report

Task 1 (selecting a language), Task 3 (changing the language), and Task 7 (changing daily goal) were easy tasks.

Fifty percent of IOS users expressed their satisfaction with how easy the app is, how efficient and comfortable they were using the app, the way the interface looked and the way the information was organized as opposed to 75% of Android users. It is interesting to note that 100% of IOS users felt that it was not easy to find the information and that this app would help them become productive, whereas 50% of Android users felt that the information was easy to find and that this app can help them become productive. In fact, the overall satisfaction for IOS users is 25% as opposed to 75% for Android users. There are noticeable differences between IOS and Android versions.

7 EME6208 Interactive Media - IMA2: Usability testing report

Table 5: Task Questionnaire

IOS ANDROID IOS+ANDROID Not H S.H M.R S. Not S. P. E H S.H S.E. E. E./H. . . . P.A. E. E. E./H. H. H. M.R. A. S.E. . Not E./H. . . M.R. P.A. Task 75 1 1 2 1 2.3 75% 2 1 1 1.8 % 3 3 1 1 2.0 75% Task 25 2 1 3 3.8 0% 1 3 3.5 % 1 1 6 3.6 13% Task 50 3 2 1 1 2.8 50% 1 1 1 1 2.5 % 1 3 2 2 2.6 50% Task 25 4 4 5.0 0% 1 2 1 3.0 % 1 2 5 4.0 13% Task 25 5 2 2 1.5 100% 1 3 3.3 % 3 2 3 2.4 63% Task 75 6 2 2 1.5 100% 1 2 1 2.3 % 3 4 1 1.9 88% Task 75 7 1 2 1 2.0 75% 3 1 2.8 % 1 5 1 1 2.4 75% Note: Super Easy=S.E.; Easy=E.; Not easy or hard=Not E./H.; Hard=H.; Super hard=S.H.; Mean Rating=M.R.; Percent Agree=P.A. *Percent Agree (%) = Super Easy & Easy Responses combined

8 EME6208 Interactive Media - IMA2: Usability testing report

Table 6: Satisfaction Questionnaire IOS ANDROID IOS+ANDROID 1 2 3 4 7 M.R. P.A. 1 2 3 4 5 M.R. P.A. 1 2 5 6 7 M.R. P.A. 1. 1 1 1 4.5 50% 1 5.0 75% 2 4 1 4.8 63% O v e r al l, I a m s at is fi e d w it h h o w e a s y it is to u s e th is

9 EME6208 Interactive Media - IMA2: Usability testing report

la n g u a g e le a r ni n g a p p. 2. It 1 1 3.8 25% 1 5.0 75% 1 1 1 4 4.4 50% w a s si m pl e to u s e th is la n g u a g e

10 EME6208 Interactive Media - IMA2: Usability testing report

le a r ni n g a p p. 3. I 1 1 4.8 50% 1 1 1 4.8 25% 1 2 1 4.8 38% c o ul d ef fe ct iv el y c o m pl et e th e ta s k s u si n g th

11 EME6208 Interactive Media - IMA2: Usability testing report

is la n g u a g e le a r ni n g a p p. 4. I 3 4.5 25% 1 1 2 4.0 0% 1 2 1 4.3 13% w a s a bl e to c o m pl et e th e ta s k s

12 EME6208 Interactive Media - IMA2: Usability testing report

q ui c kl y u si n g th is la n g u a g e le a r ni n g a p p. 5. I 2 5.0 50% 1 5.0 75% 1 5 5.0 63% w a s a bl e to ef fi

13 EME6208 Interactive Media - IMA2: Usability testing report

ci e nt ly c o m pl et e th e ta s k s u si n g th is la n g u a g e le a r ni n g a p

14 EME6208 Interactive Media - IMA2: Usability testing report

p.

6. I 1 4.8 50% 1 6.3 75% 1 1 2 3 5.5 63% fe lt c o m fo rt a bl e u si n g th is la n g u a g e le a r ni n g a p p.

15 EME6208 Interactive Media - IMA2: Usability testing report

7. It 1 4.8 50% 1 5.3 75% 1 1 5 5.0 63% was eas y to lear n to use this lang uag e lear ning app. 8. I 2 2 3.0 0% 1 1 5.3 50% 2 1 2 4.1 25% b el ie v e I c o ul d b e c o m e p r o d u ct

16 EME6208 Interactive Media - IMA2: Usability testing report

iv e q ui c kl y u si n g th is la n g u a g e le a r ni n g a p p. 9. 4 1.0 0% 1 1 2 4.3 0% 4 2 2.6 0% T hi s la n g u

17 EME6208 Interactive Media - IMA2: Usability testing report

a g e le a r ni n g a p p g a v e e rr o r m e s s a g e s th at cl e a rl y to ld

18 EME6208 Interactive Media - IMA2: Usability testing report

m e h o w to fi x p r o bl e m s. 10. 1 1 1 4.0 50% 1 1 1 4.3 25% 1 2 1 2 1 4.1 38% W h e n e v e r I m a d e a m is ta k e u si

19 EME6208 Interactive Media - IMA2: Usability testing report

n g th e la n g u a g e le a r ni n g a p p, I c o ul d r e c o v e r e a si ly a

20 EME6208 Interactive Media - IMA2: Usability testing report

n d q ui c kl y. 11. 2 1 1 2.8 25% 1 1 1 4.0 25% 3 1 1 2 3.4 25% T h e in fo r m at io n (s u c h a s o n - li n e h el p, o n - s

21 EME6208 Interactive Media - IMA2: Usability testing report

cr e e n m e s s a g e s a n d ot h e r d o c u m e nt at io n ) p r o vi d e d

22 EME6208 Interactive Media - IMA2: Usability testing report

w it h th is la n g u a g e le a r ni n g a p p w a s cl e a r. 12. 2 2 3.0 0% 1 1 4.5 50% 1 2 1 1 3.8 25% It w a s e a s y

23 EME6208 Interactive Media - IMA2: Usability testing report

to fi n d th e in fo r m at io n I n e e d e d. 13. 2 1 1 3.8 25% 3 3.0 25% 5 1 1 3.4 25% T h e in fo r m at io n p r o vi d e

24 EME6208 Interactive Media - IMA2: Usability testing report

d fo r th is la n g u a g e le a r ni n g a p p w a s e a s y to u n d e rs ta n d.

25 EME6208 Interactive Media - IMA2: Usability testing report

14. 1 1 2 2.8 0% 1 1 4.8 50% 1 2 1 2 3.8 25% T h e in fo r m at io n w a s ef fe ct iv e in h el pi n g m e c o m pl et e th e ta s

26 EME6208 Interactive Media - IMA2: Usability testing report

k s. 15. 1 1 1 4.3 50% 1 6.0 75% 1 1 3 2 5.1 63% T h e o r g a ni z at io n of in fo r m at io n o n th e la n g u a g e le a

27 EME6208 Interactive Media - IMA2: Usability testing report

r ni n g a p p s cr e e n s w a s cl e a r. 16. 1 1 1 4.5 50% 1 6.3 75% 1 1 4 5.4 63% T h e in te rf a c e of th is la n g u

28 EME6208 Interactive Media - IMA2: Usability testing report

a g e le a r ni n g a p p ( b ut to n s, m e n u s, g r a p hi c s) w a s pl e a s

29 EME6208 Interactive Media - IMA2: Usability testing report

a nt . 17. I 1 3 3.5 0% 1 1 1 4.8 25% 1 1 1 4.1 13% li k e d u si n g th e in te rf a c e of th is la n g u a g e le a r ni n g a

30 EME6208 Interactive Media - IMA2: Usability testing report

p p ( b ut to n s, m e n u s, g r a p hi c s) . 18. 1 3 3.5 0% 3 4.5 25% 1 1 4.0 13% T hi s la n g u a g e le a r ni n

31 EME6208 Interactive Media - IMA2: Usability testing report

g a p p h a s al l th e fu n ct io n s a n d c a p a bi lit ie s I e x p e ct it to h

32 EME6208 Interactive Media - IMA2: Usability testing report

a v e. 19. 2 3.8 25% 1 6.0 75% 2 1 2 2 4.9 50% O v e r al l, I a m s at is fi e d w it h th is la n g u a g e le a r ni n g

33 EME6208 Interactive Media - IMA2: Usability testing report

a p p. Note: 1=Strongly Disagree; 4=Do not agree or disagree; 7=Strongly Agree; M.R.=Mean Rating , P.A.=Percent Agree *Percent Agree (%) = Strongly Agree & Agree Responses combined

User experience

Although the participants gave both positive and negative comments about the app, there were more negative comments. The immediate feedback was seen as encouraging (Participant 1) and the app was considered a good supplement to anyone learning another language (Participant 2). However, the graphics were only okay (Participant 1) and the directions were difficult (Participant 5) or confusing (Participants 1, 3 and 4). The app was deemed more appropriate for adults than children (Participant 5) due to the lack of user-friendly directions. One lesson contained a translation mistake (Participant 1) and the inability to skip lessons or questions limits users (Participant 6). The limited number of words in each lesson/level was viewed as a negative point (Participant 6).

34 EME6208 Interactive Media - IMA2: Usability testing report

Recommendations Based on feedback captured by the testers while participants performed tasks and end- of-session questionnaires, the following are recommendations we suggest to improve user satisfaction with the app:

Table 7: Recommendations for Change Change Justification Severity Learnability 1. Add real-time 1. This notification based guide will 5 guiding notifications help the users by prompting suggestions about next steps. 2. Add a FAQ (Frequently Asked 2. FAQ section will make it easier 5 Questions) section for the new users to resolve generic issues. Usability 1. Integrate an auto- 1. The auto wizard will make the 8 wizard option to make usability and processing easier for installation easier for new or beginning level users. new or beginning level users Errors 1. Add an 1. An automated bug reporting 9 automated bug module helps notify the back end detection and reporting development team about possible tool that can be malfunctions, errors and bugs. Using integrated for these notification reports, the enhancing the quality developers can improve the quality of of the app. the app.

Conclusion With the exception of one participant on one task, all of the assigned tasks were completed. It would appear that Duolingo is a successful mobile language-learning app were not user satisfaction also being considered. According to the Duolingo site, users are motivated to keep learning, and the “gamification poured into every lesson” makes learning fun.

During the testing, however, the participants were often confused or frustrated by the lack of clear directions, hardly a fun scenario which would motivate continued use. Offering an optional tutorial before the lesson begins or having easy access to a tutorial screen would alleviate the main complaint of the participants. Attachments Attachment A – Introduction and task list template Attachment B – Observation sheet template

35 EME6208 Interactive Media - IMA2: Usability testing report

Attachment C – End-of-session questionnaires template

36