Word Games the Effects of a Gaming Element on Non-Vocabulary Retention in Adults

Word games The effects of a gaming element on non-vocabulary retention in adults.

Michelle Pigmans ANR: 317075

Bachelor’s Thesis Communication and Information Sciences Specialization: Human Aspects of Information Technology

Faculty of Humanities Tilburg University, Tilburg (The Netherlands)

Supervisor: Dr. A. Alishahi Second reader: Dr. G. A. Chrupala

June, 2014 WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Table of contents

Page

Abstract 4 1. Introduction 5 1.1 Problem statement 5 1.2 Research questions 6 2. Theoretical background 6 2.1 Vocabulary learning in a new language 6 2.2 Games and learning 7 2.3 Game elements 7 2.4 Research hypotheses 10 3. Game design 10 3.1 Typing game 11 3.2 Pretests 14 4. Main experiment 16 4.1 Participants 16 4.2 Design 17 4.3 Materials 17 4.4 Procedure 18 5. Results 18 5.1 Difference total performance between conditions on vocabulary retention 18 5.2 Effect of word length on vocabulary retention 21 5.3 Experiences of participants on both conditions 22 6. Discussion and conclusion 24 6.1 Discussion 25 6.2 Limitations 26 6.3 Conclusion 26 References 27 Appendix 29 - Appendix A: Word association results (translated into English) 29 - Appendix B: Results picture recognition test (translated into English) 30 - Appendix C: Pre-test 2 transcribed responses (translated into English) 34

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

- Appendix D: The sequences in the main experiment: game and non-game 35 - Appendix E: The survey (for both conditions) 38 - Appendix F: Means & standard deviation of correct four, five and six -letter words by condition 40

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Abstract Learning the vocabulary of a new language is difficult: it takes time and motivation. Adults often do not have the time in their regular work week and their motivation might be lacking. This study focused on the effect of the learning environment on the retention of words and the participants’ enjoyment level by using a game and a non-game condition. It used a between-subjects design of which one group was assigned to the game condition and the other group of participants to the non-game condition of the experiment (which uses a more traditional version of vocabulary learning). The differences between the two conditions do not seem to significantly increase or decrease the performance of participants. Adults also do not seem to learn new words better in a more engaging program: their score is more or less the same. There was also no significance in retention between shorter and longer words. However, an interaction effect on language preference and condition was found. This experiment showed that there is a positive correlation between enjoyment and vocabulary retention. Male and Dutch participants seemed to enjoy the non-game more, while female and English-preferring participants enjoyed the game more. Keywords: vocabulary learning, game-based learning, typing game, second language learning, learn by playing.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

1. Introduction

1.1 Problem statement Learning a new language is difficult, especially as an adult. There is less time to learn because of other commitments, like work, and it is difficult to stay engaged. Therefore, vocabulary retention is taking a longer time than one would wish. By playing a game it is possible to keep the engagement high and thus learn a new language in a non-boring way (Frank, 2007). There are already many “serious games” designed for children, which have proven to be beneficial (Susi, 2007). Besides the goal of learning a certain school subject, these children could also improve their attention behavior, self-monitoring, problem recognition and developed better short-term and long-term memory and social skills by playing a game. These tests were done by using a between-subjects design: in one group there were children who played games regularly, while the other group rarely played games. These nine to twelve years old children got similar tasks that tested the previously mentioned skills. Even though computer games work great as a learning tool, many learning tools for adults keep using the traditional flashcard method, which is not as engaging as a game. Is it possible to design a game that improves word retention by adults, just like the studies with children have shown? This study will focus on the effects of a game and a non-game on the retention of new words. For this, words from a non-existing language will be used. Previous research used English (Turgut and Irgin, 2009), which caused the users to already have a slight advantage because the participants already had at least a basic knowledge of the language. Earlier studies also used children instead of adults and a regular game instead of a serious game. The goal of a serious game is to teach the user something specific like a language, while the goal of a regular game is mainly to entertain the user. In the following section, the research question and subquestions will be presented. Then, chapter two will present the theoretical background which includes vocabulary learning in a new language, games and learning and the elements that create a good game. This chapter will be concluded by the research hypotheses. Chapter three will present the design of the conditions and chapter four will present the main experiment. The results will be shown in chapter five and chapter six will conclude this research by discussing these results, describing its limitations and present a conclusion.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

1.2 Research questions

The main research question is:

“What are the effects of a gaming element when learning the vocabulary of a new language by adults?”

The subquestions will be as follows: 1. Do adults have better vocabulary retention when they are playing a game compared to when they are just reviewing words? 2. Do adults learn better when they are more engaged?

2. Theoretical background

In this chapter the theoretical background that shaped this thesis will be discussed. First, section 2.1 will describe how people learn words in a new language. Secondly, section 2.2 will show the link between games and learning. Thirdly, section 2.3 will delve deeper into the specific elements that will improve the retention of words in a learning game. Lastly, the research hypotheses will be presented.

2.1 Vocabulary learning in a new language Vocabulary is an important part of learning a new language: they are the building blocks of a language. Hatch and Brown (1995) formulated 5 steps for vocabulary learning (via Renskers, 2013): 1. Having sources for encountering new words 2. Getting a clear image, either visual, auditory or both, of the forms of the new words 3. Learning the meaning of the words 4. Making a strong memory connection between the forms and the meanings of the words 5. Using the words Rieder (2003) argued that learning a vocabulary can be implicit, but not completely. A word’s form seems to be implicitly learned, but the semantic properties are explicitly learned. Learners also try to infer meaning by any cues they get, which can be both implicit and explicit. Adults seem to have more constraints when learning a new language compared to children (Newport, 1990). Normal language learning begins early in life when the child is

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS. exposed to other people conversing in that language. When adults are exposed to a new language their performance is worse than when children are exposed to the same new language. One reason for this is that there is a sensitive period effect where people can learn languages easily (early in life). After this period, these abilities weaken the older people get. Another hypothesis for this phenomenon is that the language learning abilities decline over age because the nonlinguistic cognitive abilities naturally increase (because of life experiences). This increase makes it harder for adults to reach the ability to learn a new language.

2.2 Games and learning Games can be helpful for learning a variety of skills and school subjects like a new language or leaning how to pilot a plane (Susi, 2007). Not only school subjects like math and English, but lots of other skills can be taught by a game (for example, games increase attention behavior, self-monitoring, problem recognition and improve short-term and long- term memory and social skills). Turgut and Irgin (2009) researched English vocabulary learning by using a game like Counter Strike or FIFA08. The results showed that playing games improved the English vocabulary of children between the ages of 10-14 years old. They stated that the games motivated them to learn words. Aghlara and Tamjid (2011) compared the effect of a game versus a traditional method of learning on children vocabulary retention. For this, they used 6- 7 year old children and a second language. They found that the children who played digital games scored better than the children that learned by using a more traditional method. Research by Renskers (2013) found that auditory and visual elements play a significant role in vocabulary retention. By using both elements in a game, the retention of words was much higher than when there were less audiovisual cues. This means that the characteristics of a game are helpful for teaching users subjects that allow the use of text, audio and visual elements.

2.3 Game elements In order for a learning game to be successful, certain aspects need to be defined. Malone (1980, via Renskers, 2013) defined three aspects of games that lead to more engagement: challenge, fantasy and curiosity. Short term goals were found to be more motivating than long-term goals and fixed goals. When there is more engagement in a game, the game will usually be more fun, leading to more retention.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Frank (2007) confirms the three aspects Malone gave, and expanded on this idea by using the research of Garris, Ahlers and Driskell (2002). When creating a serious game according to Garris et al., it is necessary to focus on three design goals: creating an engaging game, making sure that the content is relevant to the training objective and designing for the whole context of use, including learning methods and domain specific constraints. These three design goals will be explained in detail below. To create and enhance engagement (the first design goal) Frank (2007) mentions five motivational dimensions: fantasy, rules/goals, sensory stimuli, challenge and mystery.

 Fantasy: this means that the action in the game has no impact on the real world. Garris claims that the fantasy element enables players to “interact in situations that are not part of normal activities, yet they are insulated from real consequences”. Fantasy also helps to cover up the fact that the user is learning, but should be used carefully. If the fantasy element is used incorrectly, the element might interfere with the dimension of relevancy.  Rules/goals: these are the rules that set the limits for the players. It is a fixed guideline that they have to follow. These rules can be explicit like a manual, but can also contain implicit rules such as “etiquette”. In a game a clear and specific goal is necessary so that the performance can be measured. A goal is something the player can push for; this can be seen as a motivational aspect of the game.  Sensory stimuli: audio-visually appealing games can grab the players’ attention and lead to stronger motivation. Sound effects, music and visual effects enhance the sensation of the player but might affect the objective of the game.  Challenge: a game should have challenges that are not too hard nor too simple. Usually a game starts simple and gradually becomes harder, so that everyone can play it. This means that when the game ends, all players will be at the same level.  Mystery: there is a difference between mystery and curiosity. Mystery is a feature of the game while curiosity is something the player senses. Mystery can cause curiosity in a player, which causes more motivation.

Learners also need a training objective, something that they can and want to work towards. Frank used earlier research to divide this second dimension into three groups: scenario, relevancy and validity.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

 Scenario: the scenario should be relevant to the subject that is being taught. For example, history lessons should be in a historic setting. However, a subject can have multiple scenarios to choose from and scenarios are not necessarily clear from the start.  Relevancy: the game needs to be relevant in order for players to pick up the objective and use it in the real world. If for example, a flying simulator takes place in a futuristic setting with aliens in space, the player might not use the knowledge they gained in the real world piloting. It should also be consistent: if you immediately come back to life after death in the game, the information that is being taught in the game will be perceived as less relevant for use in the “real world”.  Validity: games are a simplification of the world. In a serious game, these simplifications should preferably relate to the real world in some way.

To keep learners from slacking off the context is also necessary. This means that for example, a teacher figure might help to motivate players to stay sharp. Another example is the time available to complete the objective. A shorter time to respond in a game might push people to learn faster. Analysis of the learners (for example, their prior experiences with the subject, or whether they played games at all) can show the developer of the game what the difficulty should be for the game and how the interface should be designed. Liu, Horton, Kang, Kimmons, and Lee (2013) support that games are beneficial to learning, and claim that you should not make a game too hard. In their research they found that middle school students learn better in playful environments. Therefore it is necessary to put the element of fun in the game. The flow theory by Csikszentmihalyi (1992, via Renskers, 2013) adds to this by stating that players need a state of flow: being involved in an activity in such a way that nothing else seems to matter. By having factors like clear goals, immediate feedback and concentration on the task added in the game it will be viewed as more enjoyable. Even though the experiment of Liu et al. was conducted on students their findings can also be applied on adults, because Renskers found that the state of the players influenced their vocabulary retention. However, Renskers (2013) did not find that time influenced the retention of vocabulary like Frank (2007) suggested. This, as stated in the discussion of the paper, probably occurred due to the participants paying more attention to the game itself rather than trying to memorize the words. Thus, a more elaborate game design to let participants actively use the words might be necessary in order to gather results about the influence of time.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

In summation, to create a successful game the design should focus on engaging the player, making sure that the content is relevant to the training objective and designing for the whole context of use.

2.4 Research hypotheses Hypothesis 1 (H1): Adults learn new words better when playing a game. Earlier research proved that children can learn a wide variety of school subjects and life skills by playing games (Turgut and Irgin, 2009; Aghlara and Tamjid, 2011). However, adult learners have more constraints when learning new words than children have (Newport, 1990). Auditory and visual elements play a significant positive role in retention (Renskers, 2013). A learning game can provide these cues and therefore it can be assumed that adults will learn new words better when playing a game. Hypothesis 2 (H2): Adults learn new words better when the learning program is more engaging. As Frank (2007) stated, it is important to create an engaging game in order to get learners to actively participate. The main motivational properties a learning game needs to have are: fantasy, rules/goals, sensory stimuli, challenge, mystery and control. Liu et al. (2013) showed that games are beneficial to learning, but that the difficulty shouldn’t be too hard. These properties should assure that adults learn new words better when the learning program is more engaging.

3. Game design

In this chapter, the game design will be discussed. First, the main experiment (the typing game) will be introduced and explained. It will include the workings of the two conditions, the choice of word stimuli, the art and sound used and the software that was used. Then, the two pretests will be discussed. In order to preserve the validity of this experiment, the words and pictures were tested on associations. It was necessary to have pictures with high association (for example, a picture of a cat will cause viewers to immediately associate the picture with a cat), so that the presented objects would not need extra explanation by using language. The non-words were expected to have low associations, so that participants would not be able to associate the non-words with existing associations that could help or confuse them. The second pretest was constructed in order to find out whether there were any problems regarding unfair advantages in any condition and whether the program itself worked well.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

3.1 Typing game In the experiment there were two conditions: game and non-game. With these conditions it is possible to measure the effect of enjoyment and the gaming aspect on the retention of words. The program allows the participants to learn new words and consists of two phases. In the first phase, the participants are in both conditions presented with words that they need to learn. The second phase will test the participants in both conditions by letting them type the solutions (the words) into the program. Both conditions will be explained in more detail in the second part of this section. The order in which the objects are presented is random in both phases and in both conditions. An added feature is that players are able to use the backspace button if they made a spelling error in the second phase. The answer the players give is only checked when all panels have been filled and when they press the space button. Figure 1 shows a screenshot of the first phase. A more detailed set of screenshots can be found in Appendix D. The full game can be found on: http://tinyurl.com/gameBTMP and the non-game can be found on: http://tinyurl.com/non-gameBTMP.

Figure 1. Screenshot of the program.

In the first phase of the game the words and their corresponding pictures are shown and the moment they disappear, the participant has to reproduce the word by typing them. After this, an animation will show the character on the left of the screen “catching” the object. They will repeat this twice for all words. If they make a spelling error, the full word will be shown again, after which they have to type it again. During the second phase of the game the participants will be tested. They are presented with the image and then have to type the word. As a hint, the word length is shown. When they are right, the character on the left of the screen catches the object. When wrong, the object flies away. After this, the score will be

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS. revealed. The non-game has the same setup and is the same in many aspects to the game condition. However, in the entire non-game there are no catching or flying animations and there is no score at the end. They also are not allowed to type in the first phase (Appendix D3). Instead, they see the words for 8 seconds (the time the participants need in the game), after which the program moves on. After completing either the game or non-game, the participants are linked to a small survey, which can be reviewed in Appendix E. The choices for the design of the game were inspired by Frank (2007), Renskers (2013) and Liu et al. (2013). In order to create engagement, a fantasy world has been created. This world is made out of paper and it seems that the moving parts are being controlled by a person that remains unseen by the player. The rules and goals of the game are simple: try to remember as many words as possible to catch the objects! Only the right combinations of letters are allowed, otherwise the object will escape. This is also the challenge of the game. The sensory stimuli used are text (the words), pictures (the environment and the objects that need to be learned) and audio (the words are pronounced, there are sound effects and there is background music playing). There is not a lot of mystery to the game, mainly because the game does not have a story. The game is not about teaching history, so a historic scenario did not seem valid. Instead, the objects are all made out of paper, which you can mentally connect to writing. The way you play is relevant to the goal of the game: it is necessary to know the words by heart and typing lets you actively review the words. The game still connects to the real world by taking real concepts (for example, a cat) and connects them to non-existing words. In terms of context, this game does not use a teacher to motivate learners. Instead, there are big panels that explain the game to you, and there are sounds to tell you whether you answered right or wrong. The participants that are going to play this game can be any age, which is why the game needs to stay simple in use. The participants will need to have a basic understanding of using a computer, which means that they will know how to use a mouse and a keyboard. However, it is not possible to use a time constraint to motivate people to answer more quickly, because not all participants will be as fast as each other (there can be a tech- savvy 20 year old, but also a 60 year old that is new to computers playing the game). There are only two phases in the game, which are reasonable in length, and the difficulty will be the same for all players. This creates more flow, which will hopefully enhance the engagement in the game. By enhancing the engagement, the enjoyment participants feel will increase.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

The non-game will have less motivational aspects. For example, there is less fantasy (you do not catch the object anymore), less challenge in the first phase (you only review the words) and less immediate feedback (there is no sound when you are right or wrong).

Word stimuli The words used in this experiment are all non-words created by an online random word generator (Wolfram Demonstrations Project, n.d.). The full list of words can be viewed in Appendix A. In the experiment, a total of 15 words were used that had no strong associations. These words were divided into 4 groups, sorted by the length of the words. In total, there were three 3-letter words, and four 4-letter, 5-letter and 6-letter words. Table 1 shows the words together with their meaning. The choice of the final words was influenced by the pre-test, which will be explained in section 3.2.

Table 1. Words used in the game. English New word Cat Rona Table Blune Heart Terp Crown Las Eye Muhby Book Leoris T-shirt Dohi Monkey Fonuf Sun Noosto Car Shu Lightbulb Wisnax Banana Vunre Phone Eph Foot Buvixi Tree Voov

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Art The art of the game was made in Photoshop CS5. The pictures were selected as random as possible, so that there were a variety of objects to be connected to the words. Before the experiment took place, a pretest was held to verify that these pictures represented the right concepts (explained in section 3.2). The animations were hardcoded in Javascript.

Audio The music used in the game is copyright free and the spoken conditions of the words are recorded especially for this experiment.

Software The experiment is programmed in Unity by using the language Javascript and is hosted online by Google Drive. The participants had to follow a link, after which the program decided on which condition they were going to do. After the experiment, the participants got forwarded to a survey, which was made in Google Documents and also hosted online by Google Drive.

3.2 Pretests Before the experiment could start, two pre-tests were conducted to preserve the validity of this experiment. Firstly, the images and non-words had to be checked for (strong) associations. The second pretest was conducted to find out if there were any unfair advantages in either condition and to find out if everything was working accordingly.

Pretest 1: word and picture associations. The aim of this pretest was to find out the associations participants have with the pictures and non-words. It is necessary for the images to have strong associations so that everyone will know that the image of a cat actually represents a cat, and thus can be used for people speaking any language. It is important for participants to not have strong associations with the non-words, so that it is not “cheating”. Participants and materials There were 4 participants in total. Age minimum was 21 and maximum 53 (M= 30.00, SD: 15.36). There were two males and two females. The pretest used 16 images and 24 non- words. The survey was made in Google documents. An overview of the used images and words can be found in Appendix A and Appendix B.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Procedure The participants got to see the images and the words and had to write down their associations. They also had to rate the strength of the word association on a Likert scale so that it could be determined whether people wrote down just because they could or because they had actual associations. The goal was to have the images as clear as possible so that everyone could understand it, and the words to have as little association as possible. Any words that had strong associations will not show up in the experiment. Results and discussion By the results of the first pre-test, it was concluded that only two images had slight problems with their design. The monkey had a banana in his hand, which is why participants also included the banana in their answer. The flower seemed a bit vague because it did not have the round center on which the petals are connected. The non-words turned out to have more associations than anticipated. Six words in total had two or more participants with the same associations and thus had to be deleted. This caused there to be too little 3-letter words, and thus one image was left out of the experiment (the flower). It turned out that the strength factor is not really measurable, as some participants put “nothing” as their association, which is a strong association. These answers were not used in the final decision of which words to keep. Instead, it was checked whether participants had the same associations.

Pretest 2: Program check The second pretest was conducted to find out if there were any unfair advantages in either condition, and to find out if everything was working accordingly. Participants and materials Four participants did the experiment, of which two played the game and two played the non-game. There were two males and two females, which were evenly divided over the two conditions. The minimum age of the participants was 21 years old, and the maximum age was 24 years old (M=23.00, SD=1.41). All of the participants were Dutch students who have experience with playing web-based games. The participants got the final draft of the program to play and got a small interview afterwards. Procedure The participants had to play the condition that they were assigned to and afterwards a small interview was conducted. The transcribed and translated interviews can be found in Appendix C.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Results and discussion Both groups scored similar results: all participants got 5 correct answers. This means that the difficulty is the same for both conditions. They all agreed that the look and use of the programs were good and that the instructions were clear. However, there were some points of critique. First of all, the animations were too slow. Secondly, they would prefer a longer learning phase and thirdly they would like to have some sort of progress bar. Lastly, the participants of the game noted that the learning phase in the game made them focus on the typing instead of learning the words. These responses caused a couple of changes in the final product: the animations are two times as fast, all the words are presented twice in the learning phase and there is a progress count in a cloud on the top right of the screen. The biggest change occurred in the learning phase of the game: instead of being able to see the words while typing them, the words disappear after which you have to type them. When you get the word wrong, the full word is presented again and after a few seconds it disappears.

4. Main experiment

4.1 Participants In total there were 61 participants with a minimum age of 17 and a maximum age of 53. On average, the age of the participants was 26.52 (SD: 9.712) There were 35 male participants (M = 25.8; SD: 8.8) and there were 26 female participants (M = 27.5; SD: 10.9). The distribution of the gender, education level and language preference can be found in Table 1. Out of all participants, 65.6% chose Dutch as their preferred language and 34.4% chose English as their preferred language. 54.7% of the participants were male, and 34.4% of the participants were female. Most of the participants’ educational background is bachelor (68.9%), 19.7% of the participants chose college, 8.2% chose high school and 1.6% chose master’s as their educational background. Only one respondent chose to not answer this question. In this experiment, everyone was allowed to participate. However, in order to be able to complete the experiment, a basic level of computer literacy was needed. The participants also needed to be proficient in either Dutch or English. These conditions made it possible to broaden the target group and reach out to people in other countries. All of the participants

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS. were randomly selected and were assigned to one of the two conditions at random (by using a randomizer in the code of the program).

Table 1 Distribution of the participants by Gender, education and language (standard deviation between brackets) Gender Education level Language Male Female Low level High level English Dutch (n=35) (n=26) (n=5) (n=55) (n=21) (n=40) Age 25.8 27.5 29.4 26.3 23.1 28.3 (8.8) (10.9) (16.1) (9.2) (2.85) (26.5)

4.2 Design The participants were randomly assigned to one of the two conditions: the game or the non-game. After the study, the participants got the same set of survey questions. A thorough explanation on the design of the experiment itself can be found in section 3.1. The independent variables of the experiment are the two conditions: the game and the non-game. The dependent variables are the total score, the level of enjoyment, boredom, self-rated level of learning and whether they would want to play a program like this more often when learning new words from a new language. The level of enjoyment and boredom are both measured because lack of enjoyment could mean that the gameplay is not engaging enough, while boredom could mean that the game is not fast-paced enough. For example, an open world game can be enjoyable just to drive through, but if you have to wait two seconds at every intersection the pace of the game becomes boring. The two separate conditions makes this experiment suitable for a between subjects design.

4.3 Materials The experiment consisted of a small survey and a game, all hosted by Google Drive. The art for this game was made in Photoshop CS5, and programmed in Unity by using the language Javascript. Google Drive was be used to get the survey results and measure the participants’ performance in-game. A more in-depth explanation can be found in section 3.1. The survey was constructed as follows: first, there were questions to determine the demographic (age, gender, language preference, education level). The participants could answer with “would rather not say” in regards of education level. Then, the participants got questions regarding the experiment. The first question contained what score they got. The score they got seemed arbitrary, but it is actually encodes a count on what kind of words they got correctly. The choice for a seemingly arbitrary number was made so that it would be difficult for participants to cheat in the survey. In the game condition the participants also got

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS. to see their score on a 0 to 15 scale, but this score was not used to process the results. For every 3-letter word they got correct, 1 point was given. For every correct 4-letter word they got 10 points, every 5-letter words 100 and every 6-letter word got 1000 points. The maximum score that was possible to achieve was 4443. The following questions were in what degree they enjoyed the experiment, if they were bored and how much they think they learned. The participants could answer by the use of a Likert scale (score of 1-7). The last question was whether they would like to do this activity more often when learning words from a new language. It is a closed question, with only yes or no as possible answers. All of these questions were obligatory to answer. The survey can be reviewed in Appendix E.

4.4 Procedure First, the participants choose their preferred language (Appendix D1), get a small explanation about the experiment and then they will do the experiment. Afterwards, participants get a survey to determine the demographics (such as age, sex and level of schooling) and a couple of questions regarding their experiences (Appendix E).

5. Results

This section will show the results of this experiment. First, the differences in total performance between the conditions on vocabulary retention will be discussed. Then, the scores will be analyzed on whether word length had an influence on retention. Lastly, the experiences of the participants will be discussed.

5.1 Difference total performance between conditions on vocabulary retention.

The first research question concerned whether participants would perform better when playing a game compared to when reviewing the words. In the experiment there was one respondent that did not answer what his or her score was in a correct way, and thus was excluded from the analyses. The Saphiro-Wilk Test for normality showed that neither condition deviated from a normal distribution (non-game: (D(30) = .950, p = .359), and game: (D(30) = .930, p = .480)). This means that a t-test for independent samples can be applied. The Levene’s Test of homogeneity of variance showed that the variances in overall vocabulary retention scores for both conditions were statistically equivalent (F(1,58) = 1.379, p = .245).

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Figure 2. Mean Vocabulary Retention Score per Condition

Figure 2 shows that the participants in the non-game condition performed better than the participants in the game condition. Participants who played the game (M= 5.23, SD= 4.05) remembered fewer words than the participants that played the more traditional program (M= 6.13, SD: 3.46). Figure 2 also shows an overlap of the mean vocabulary retention scores. The independent-samples t-test showed that there was no significant difference between the two conditions (t(58) = .925, p = .359). This means that no effect of the condition on word retention could be detected with this sample size. In addition to these results there are differences between the vocabulary retention and the educational background, gender, language preference and age of the participants. The Saphiro-Wilk test showed that the participants were normally distributed over the six education types (D(61) = .715, p < .05). Results of the Levene’s test confirmed that the variances in vocabulary retention for each education type were not found to be statistically significant (F(2,55) = .730, p = .486). A two-factor analysis on variance showed no significant effect of education level on vocabulary retention (F(4) = .525, p =.718). Participants in the non-game condition scored higher than participants in the game condition. There was no significant interaction found between education level and the condition (F(2) = 3.124, p =.052). Table 2 shows that the vocabulary retention mean scores of “college” are slightly higher than “bachelor”, and that “high school” scored higher in the game condition than in the traditional condition. However, this could not be proven significant. This means that the

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS. educational background did not influence the vocabulary retention. The gender types also did not seem to influence the vocabulary retention. The Saphiro- Wilk Test showed that the participants were not normally distributed over the two gender types (D(61)=.629, p < .05), and the Levene’s Test showed that the variances in vocabulary retention scores for the two gender types were statistically equivalent (F(1,58)=.375 p =.543). The two-factor analysis of variance showed no significant effect of gender on vocabulary retention (F(1) = .205, p = .653), nor a significant effect of the condition on vocabulary retention (F(1) = 1.090, p = .301). Male participants seemed to score higher in the game condition and female participants seemed to score higher in the non-game condition. There was no significant interaction effect between gender and condition (F(1) = 1.472, p = .230). This means that it was not found whether gender influences the vocabulary retention. The preferred language of the participants did seem to matter on vocabulary retention. A Saphiro-Wilk Test showed that the participants were not normally distributed over the two language types (D(61)=.600, p < .05). The Levene’s Test showed that the variances in vocabulary retention scores for the two language types were statistically equivalent (F(1,58)= .194 p = .661). A two-factor analysis of variance showed no significant effect of language preference on vocabulary retention (F(1) = .489, p = .487). However, a significant interaction effect between language preference and condition has been found (F(1) = 4.968, p < .05). This means that the language preference influenced the vocabulary retention. Dutch participants seemed to remember fewer words than English participants in the game condition, while Dutch participants remembered more words in the non-game condition than English participants. Lastly, a significant negative correlation between age and vocabulary retention has not been found (r(60)= -.142, p = .278). This means that in this experiment it was not possible to determine whether younger participants remembered more or less than older participants.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Table 2. Vocabulary Retention Scores: type of Education, Gender and Language (Means and Std.) Educational background Non-game Game Master 9.00 (0.00) - Bachelor 5.38 (3.32) 5.55 (3.87) College 5.80 (1.48) 5.86 (4.88) High school 11.00 (3.60) 2.50 (0.71) Nursery school - - None - - Gender Male 5.83 (3.33) 6.00 (4.59) Female 6.58 (3.75) 4.36 (3.27) Language English 5.23 (3.22) 7.57 (5.19) Dutch 6.82 (3.58) 4.52 (3.46)

5.2 Effect of word length on vocabulary retention The data gathered from the experiment allowed a more detailed look into the effect of word length on vocabulary retention. There were four types of word lengths: three, four, five and six-letter words. The Saphiro-Wilk Test for normality showed that both conditions deviated from a normal distribution, non-game: D(30)= .852 p < .05 and game: D(30)= .873 p < .05. This means that a Welsh t-test can be applied. The Levene’s Test of homogeneity of variance showed that the variances in three-letter vocabulary retention scores for both conditions were statistically equivalent (F(58)= .183 p = .670).

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Figure 2. Mean Three-letter Word Retention Score per Condition

Figure 2 shows the participants from the non-game condition performed better than the participants in the game condition on three-letter vocabulary retention (non-game: M= 1.67, SD: .922 and game: M= 1.27, SD: 1.015). The figure also shows an overlap of the mean three-letter vocabulary retention scores. The Welch t-test showed that there were no significant differences found between the two conditions on three-letter vocabulary retention (t(1,57) = 2.55, p = .116). Four, five and six-letter words showed the same results as the retention on three-letter words and can be found in Appendix F). The Saphiro-Wilk Test showed that all of the groups deviated from a normal distribution, and the Levene’s Test showed that the variances in four, five and six-letter vocabulary retention scores for both conditions were statistically equivalent. The Welsh T-test confirmed that there were no significant differences found between the two conditions on four, five or six-letter vocabulary retention.

5.3 Experiences of participants on both conditions The last research questions concerned whether participants’ experiences influence vocabulary retention: does the level of enjoyment or boredom influence the vocabulary retention score? And do they think they learned as much as the score showed? The Saphiro- Wilk Test for normality showed that both conditions did not deviate from a normal distribution, non-game:, D(30)= .921, p = .029 and game: D(30)=.926, p = .039. This means

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS. that a t-test for independent samples can be applied. The Levene’s Test of homogeneity of variance showed that the variances in enjoyment scores for both conditions were statistically equivalent (F(59)= .175, p = .677). Table 3 shows that participants in the non-game condition performed better than the participants in the game condition. Participants who played the game (M= 4.26, SD=1.93) enjoyed themselves slightly less than the participants that played the more traditional program (M= 4.33, SD= 1.94). The t-test for independent samples showed that there was no significant difference between the two conditions (t(59) = .152, p = .880). This means that the game element did not affect the enjoyment. A correlation test between enjoyment and vocabulary retention score showed that there is a positive correlation between the two variables (r(60)= .463, p < .05). The amount of boredom that participants might have experienced did not seem to influence vocabulary retention. The Saphiro-Wilk test showed that both conditions were normally distributed (non-game D(30)= .882, p < .05 and game:D(31)= .939, p = .078). A Levene’s test showed that the variances in both conditions were statistically equivalent F(59)= 3.456, p = .648. Participants in the non-game condition were slightly more bored (M=4.30, SD= 2.81) than the participants in the game condition (M=4.06, SD=1.80). The independent samples T- test confirmed that there was no significant difference in experienced boredom in either condition found (t(59) = .460, p = .648). The correlation between boredom and vocabulary retention is not significant (r(60)= -.157, p = .232). The participants also had to rate how much they had learned in the experiment on a scale of 1 to 7. The Saphiro-Wilk test showed that both conditions were not normally distributed (non- game: D(30)= .910, p < .05 and game: D(31)=.865, p < .05), and the Levene’s test showed that the variances in both conditions were equivalent (F(59)= 1.074, p = .310). Participants in the non-game condition rated their learning level slightly higher (M= 3.37, SD=1.65) than the participants in the game condition (M=3.29, SD=2.04).The Welch T- test showed that there is no significance found between the condition and the self-rated learning level (t(1,57)=.026, p = .873). A correlation test showed that there is a significant positive correlation between the self-rated learning level and vocabulary retention score (r(60)= .544, p < .05).

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Table 3. Mean scores and SD of Enjoyment, Boredom and Self-rated learning level. Non-game Game (N=30) (N=31) Enjoyment 4.33 (1.94) 4.26 (1.93) Boredom 4.30 (2.81) 4.06 (1.80) Rated learning 3.37 (1.65) 3.29 (2.04)

On the question whether the participants would like to do the activity more often in order to learn a new language, 53.3% said “no” in the non-game condition. In the game condition, 51.6% of the participants would rather not do this activity in order to learn a new language. A correlation test showed that there is a positive significant correlation between wanting to play it more often and vocabulary retention score for the game condition (Non- game: (r(30)= -219, p = .246 and game: (r(30)= -.410, p = .024). Table 4 shows how the scores are divided between gender and language. Male participants would rather play the non-game (61% versus 41%), while females would rather play the game (57% versus 25%). Also, Dutch participants seem to prefer the non-game slightly (59% versus 48%), while English-preferring participants preferred the game condition (50% versus 31%). However, these differences were not found to be significant.

Table 4 Whether participants would play this more often for learning a new language by condition, gender and language (in percentages) Game Non-game Gender Language Gender Language Male Female English Dutch Male Female English Dutch (n=17) (n=14) (n=8) (n=23) (n=18) (n=12) (n=13) (n=17) Yes 41 57 50 48 61 25 31 59 No 59 43 50 52 39 75 69 41

6. Discussion and conclusion

This study was devised in order to find out whether a game aspect and word length had any influence on vocabulary retention and the experiences of participants. In this paragraph, the results of this experiment will be discussed. After this, limitations that arose during the study will be revised and lastly, a conclusion will be given.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

6.1 Discussion The first hypothesis (H1) assumed that “Adults learn new words better when playing a game.” In this experiment, no significant differences were found between vocabulary retention and condition. Younger participants did not remember more or less words than older participants. However, there was an interaction effect on language preference and condition. The results of this study did not correspond with previous research by Turgut and Irgin (2009) and Aghlara and Tamjid (2011). The difference could be that their study was conducted on children and that they did not focus on learning words from a new language. Additionally, the results of the effect of word length on word retention were shown in section 5.2. Neath and Naime (2009) did not find a lot of evidence that shorter words are remembered better than long words in the short-term memory. Their hypothesis was that longer words would be harder to remember because it takes learners longer to pronounce them. The findings of Neath and Naime are compatible with the findings of this study. This experiment showed no influence of word length on vocabulary retention, although it seems that shorter words were more suitable for non-game and longer words were more suitable for the game variant. The second hypothesis (H2) assumed that “Adults learn new words better when the learning program is more engaging.”. This experiment showed that there is a positive correlation between enjoyment and vocabulary retention. There was also a positive correlation between self-rated learning and vocabulary retention, which means that the participants had the right idea about their own skills. By looking at the answers on the question whether the participants would like to play the program more often to learn new words, male participants seemed to enjoy the non-game better, and females seemed to enjoy the game better. Dutch participants seemed to enjoy the non-game only slightly more and English speaking seemed to prefer the game condition. This means that there is no one straight answer to this hypothesis; further investigation will be necessary. The research by Frank (2007), Liu et al (2013) and Renskers (2013) showed extensively how to make a game more engaging. These guidelines were proven to work by them, but this could not be proven in this experiment. A difference could be that this learning game focused on learning words from a fictional language and that not all guidelines were followed. For example, there was no time limit because the expected participants could have differed greatly in response times.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

6.2 Limitations There were a couple of limitations that could have influenced this experiment. Firstly, the experiment was done via the internet. This means that there it was not possible to check for any disturbances during the experiment. Secondly, the music might have influenced the retention. However, the music occurred in both conditions. Thirdly, the conditions only had minor differences which might have made the difference in experiences which the respondents had a bit too similar. Further studies can focus on what participants actually answered. In this experiment, it was not possible to check whether participants only missed one or two letters of the word, or whether they only entered gibberish. Also, the participants could have been less motivated due to the words not being presented as words from a different language that they wanted to learn. Another study could delve deeper into the differences in experiences by adding a third condition with plain flashcards and no fancy graphics or audio.

6.3 Conclusion This study focused on the retention of words and their image in two different conditions: a game and a non-game. By using a between-subjects design, one group of participants got the game condition of the experiment while the other group got a more traditional condition of the experiment. It turned out that the differences do not seem to significantly increase or decrease the performance of participants. Adults also do not seem to learn new words better in a more engaging program: their score is more or less the same. There was also no significance in retention between shorter and longer words. An interaction effect on language preference and condition was found. This experiment showed that there is a positive correlation between enjoyment and vocabulary retention. Male and Dutch participants seemed to enjoy the non-game more, while female and English- preferring participants enjoyed the game more. In following studies, a detailed description of the answers of participants on the task is necessary.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

References

Aghlara, L. & Tamjid, N.H. (2011). The effect of digital games on Iranian children's vocabulary retention in foreign language acquisition. Procedia - Social and Behavioral Sciences. 29, 552–560. Retrieved on June 17, 2014 from http://www.sciencedirect.com/science/article/pii/S1877042811027364

Frank, A. (2007). Balancing Three Different Foci in the Design of Serious Games: Engagement, Training Objective and Context. Conference proceedings of DiGRA. 567-574.

Garris, R. Ahlers, R. and Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33 (4). 441-467.

Liu, M., Horton, L., Kang, J., Kimmons, R. And Lee, J. (2013). Using a Ludic Simulation to Make Learning of Middle School Space Science Fun. The International Journal of Gaming and Computer-Mediated Simulations, 5(1). 2-41.

Neath, I. & Nairne, J.S. (1995). Word-length effects in immediate memory: Overwriting trace decay theory. Psychonomic Bulletin & Review, 2 (4). 429-441.

Newport, E.L. (1990). Maturational Constraints on Language Learning. Cognitive Science, 14. 11-28.

Renskers, J. (2013). Learn by Playing: The Effects of a Digital Memory game and its Pictorial and Auditorial Elements on Adults’ Vocabulary Retention. Masterthesis Tilburg University, Tilburg.

Rieder, A. (2003). Implicit and Explicit Learning in Incidental Vocabulary Acquisition. Vienna English Working Papers, 12(2). 24-39.

Susi, T., Johannesson, M. & Backlund, P. (2007). Serious Games – An Overview. Technical Report HS-IKI-TR-07-001. University of Skövde, Sweden. 1-28.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Turgut, Y. & Irgin, P. (2009). Young learners‘ language learning via computer games. Procedia Social and Behavioral Sciences, 1, 760–764

Wolfram Demonstrations Project. (n.d.). Random Word Generation for Fictional Languages. Retrieved from http://demonstrations.wolfram.com/RandomWordGenerationFor FictionalLanguages

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Appendix

Appendix A: Word association results (translated into English)

The fictional words with their associations and strength by 4 Dutch adults note; (removed) means that these words were removed in the actual experiment. Fictional words Associations 3-letter words Nuz (removed) - No - Nose - Nutella - Chocolate bar Las - Welding - Lassie, the dog. Also “girl” in Scottish - Reading - Joint compound Shu - Something edible - Quiet - Go away - Shoe Peh (removed) - Breakdown - Poeh (expression showing fatigue) - Meh (expression showing a lack of interest) - Bad luck Pri (removed) - ? - In front of - Prefix which is not the same as “pre” - Togepi (Pokemon character) sound Eph - ? - ? - Name of a location in a game - Abbreviation in mathematics 4-letter words Terp - Derp! Turpentine - Tulip that is hanging - Refuge - Terrace Pinn (removed) - Getting money by using pin - Pin code - Surname of a person - Pin Tori - ? - Tree - Conversation in Surinamese - Surinamese word Voov - Abbreviation - ? - Volvo - Volvo

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Dohi - ? - ? - Delhi. - African name 5-letter words Fonuf - ? - Enough - From - Trick Blune - Flower - Sunflower - Blue - Mp3 player shaped like a flower Onama (removed) - Obama - Obama - Japanese word - Resembling obama Eraow (removed) - ? - Mistake - Cat sound - Cat food brand Vunre - ? - Noting - Dirty - Tunnel Muhby - Maybe - “Mayby” - Ice cream - Nothing 6- letter words Leoris - A rising lion - Lion - Name ear - Reading Noosto - Nose - ? - Non stop - “whut?” Wisnax - ? - Nothing - Wax - Know-all Buvixi - ? - Pet name - Asterix comic - Chic restaurant Nedril (removed) - ? - Nose drill - Pain killer - Dutch spawn - ?

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Roinad (removed) - ? - ? - Steroid - Clothing detergent brand

Appendix B: Results picture recognition test (translated into English)

Table result picture recognition test Picture Answers - Cat op stick - Cat - Cat on pole - Cat food brand

- Lamp on stick - Lamp bulb - Light - Light bulb

- Foot ice cream stick - Foot on stick - Foot - Foot

- Crown on stick - Crown on ice cream stick - Crown, king - Crown

- Heart, love - Small heart - Heart - Heart on stick

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

- Tree, pine tree - Pine tree on stick - Pine tree - Tree

- Car - Car on stick - Car - Car

- Flower - Flower on stick - Flower - Flower

Note: removed from experiment - Telephone on stick - Telephone - Telephone - Telephone

- Monkey - Monkey with banana - Monkey with banana on stick - Monkey

old versus new design - T-shirt - Shirt on stick - Shirt - Shirt

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

- Table - Table on stick - Square table - Table

- Eye - Eye on stick - Left eye - Eye

- Sun - Sun on stick - Sunshine - Sun

- Banana on stick - Banana - Banana - Banana

- Book on stick - Book - Book - Book (thick)

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Appendix C: Pre-test 2 transcribed responses (translated into English)

Game Age: 24 Gender: Male Score: 3110 What were the good qualities of the program? The instructions were clear and the interface was easy to use. What were the bad qualities of the program? The Animations are too slow! It gets boring! Do you have any tips to improve the program? Speed up the animations!

Age: 21 Gender: Female Score: 221 What were the good qualities of the program? The graphics look very cute! What were the bad qualities of the program? It was pretty hard to remember all the words in the second phase. I was busier with the typing itself than with actually trying to remember the words. Do you have any tips to improve the program? Maybe repeat the words in the first phase?

Non-game Age: 23 Gender: Male Score: 1031 What were the good qualities of the program? The program looked nice, the music was funny. What were the bad qualities of the program? Reviewing the words was boring and slow, but I guess this type of learning is not for me. Do you have any tips to improve the program? I don’t know!

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Age: 24 Gender: Female Score: 3020 What were the good qualities of the program? It looked very creative and there was not a lot of text (but enough to know what to do). What were the bad qualities of the program? I didn’t know what my progress was, which made it seem endless at times. Do you have any tips to improve the program? A progress bar!

Appendix D: The sequences in the main experiment: game and non-game

The experiment: choosing the language

The introduction screen of both the game and non-game

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

The first phase of the game versus the non-game

Game Non-game

Both conditions show the word

In the non-game, the word shows up and disappears again. In the game, the word shows up, then disappears and afterwards you need to type the word. When you are right, you catch the object.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

The second phase for both conditions

A correct answer in the second phase (game and non-game).

Game Non-game

A wrong answer in the second phase (game and non-game).

Game Non-game

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

The score screen and link to the survey in both conditions.

Game Non-game

Appendix E: The survey (for both conditions). Nieuwe Woorden Leren/Learning New Words

Phase 3: Thank you for participating in this experiment! Please answer the following questions. It should take no longer than 2-3 minutes.

Fase 3: Dank u voor het meedoen in dit experiment! Beantwoord alstublieft de volgende vragen. Dit zal niet langer duren dan 2-3 minuten.

1. Preferred language - English - Dutch 2. Gender - Male - Female

3. Age -any number between 0-99 4. What is your level of education? - none - nursery school - high school - college

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

- bachelor - master - would rather not say 5. What was the number that you got? - 0 – 4443 6. How much did you enjoy this experiment? - 1 (not at all) – 7 (a lot) 7. Were you bored during the experiment? - 1 (not at all) – 7 (a lot) 8. How much did you learn during the experiment? - 1 (nothing) – 7 (a lot) 9. Would you like to do this activity more often in order to learn words from a new language? - Yes - No

Thank you for participating! Your response has been recorded.

Bedankt voor het invullen! Uw antwoorden zijn genoteerd.

Appendix F: Means & standard deviation of correct four, five and six -letter words by condition

Means & standard deviation of correct four -letter words by condition.

Means & standard deviation of correct five-letter words by condition.

WORD GAMES: THE EFFECTS OF A GAMING ELEMENT ON NON-WORD RETENTION ON ADULTS.

Means & standard deviation of correct six-letter words by condition.