How Does the Complexity of Game Mechanics Affect the Understanding of the Gameplay in Serious Games

SUBMITTED IN PARTIAL FULLFILLMENT FOR THE DEGREE OF MASTER OF SCIENCE

NAVDEEP SINGH 10004603

MASTER INFORMATION STUDIES GAME STUDIES

FACULTY OF SCIENCE UNIVERSITY OF AMSTERDAM

August 28, 2017

1st Supervisor 2nd Supervisor Dr. Frank Nack Mr. Daniel Buzzo ISLA, UvA ISLA, UvA How Does the Complexity of Game Mechanics Affect the Understanding of the Gameplay in Serious Games

Navdeep Singh University of Amsterdam 10004603 Supervisor: Dr. F. Nack [email protected]

ABSTRACT However, the rise of serious games was met with failure as 80 In this paper it is hypothesized that the understanding of percent of the serious games predicted not to work by 2014 gameplay is affected by the complexity of game mechanics due to poor design [3]. Research also showed that serious in serious games. Complex game mechanics are considered games are effective in terms of learning and retention, but here as complex systems consisting of smaller mechanics that not more motivational than conventional instruction meth- influence each other. In this article it is investigated if the ods [1]. complexity of the game mechanics in Minecraft1 influence the understanding of the gameplay within the context of Serious games try to tackle a problem by either designing a a serious game that addresses the teaching of digital root game that intrinsically merges the gameplay with the prob- in math. Two versions of a Minecraft custom map will be lem domain (mainly tasks) or design gameplay that incor- created and tested. This custom map will be known as Num- porates the addressed problem in some loose metaphorical bers Prison, and there will be a complex and a simple ver- manner. In other words serious games try to merge two sion. problem spaces. The first addresses the comprehension of the domain problem, e.g. the educational purpose, such as The study shows that the players of the complex version of particular math strategies. Here the ”player” tries to gain Numbers Prison enjoyed the game more than the players knowledge and/or learn skills through playing. The second of the simple version. The gameplay was experienced more problem space addresses the gameplay (i.e. the mechanics). difficult by the players of the complex version. Finally, the The ”player” needs to understand the game mechanics to players of the complex versions forgot more often the ac- make progress in the serious game. In video games, for ex- tivities they performed in the game than the players of the ample, the core mechanics are mostly hidden, but players simple version. will learn to understand them while playing. Expert players will deduce what the core mechanics must be by watching It can be concluded that complexity of the game mechanics the behavior of the game many times; they will learn how affect the understanding of the gameplay in serious games in to use a game’s core mechanics to their advantage [4]. a negative sense, however more research must be conducted to investigate where this effect comes from. In serious games the ”player”faces these two problems spaces simultaneously. This can be problematic because the player might focus mainly on one problem area and therefore lose 1. INTRODUCTION access to a large part of the game experience. This might With the growth of video games, researchers started to see be one reason why serious games are not as successful as as- the usefulness of video games for the purpose of learning sumed [3]. Another problem can be the complexity of tasks and instruction. Games in this context are referred to as to be performed that degrades the appeal of serious games. serious games [1]. Serious games are grounded in active The player must learn the game mechanics, understand the learning and encourages learning activities by building on learning material of the serious game, and understand what engagement and challenges to achieve the intended learning the relation is between the learning material and the game objectives [2]. mechanics in order to figure out the gameplay.

1https://minecraft.net/en-us/ Most studies regarding serious games were to be found fo- cusing whether players understand the learning material [5], engagement and motivation [1]. However, no studies (as of this date) were found that investigate whether players were able to understand the gameplay of the serious game, which provides an essential contribution to its appeal. The appeal comes from play which is a fundamental part of the moral self-being, of the healthy and mature human life [6].

In this paper it is investigated whether the understanding of the gameplay is affected by the complexity of the game • have a conflict mechanics in serious games. Gameplay is a set of activi- ties that can be performed by the player and other entities • have rules that belong to the virtual world during the experience, as a • can be won or lost response to the actions of the player [7]. • are interactive The thesis is structured in the following way to explore the topic mentioned in the previous paragraph. First, the paper • have challenges starts with discussing related work. This is where it is ex- plored and established what (serious) games are. Next the • can create their own internal value paper dives into different components of games, and focuses • engage players on a specific component, i.e. game mechanics. This is fol- lowed by looking at complexity, and a definition for complex • are closed and formal systems. game mechanics is given based on complex systems. Based on what serious games are and how (complex) game mechan- ics work, two problem spaces are identified and explained. Another definition concerning video games state that video This is where the research question will be formulated based games are designed for players to actively engage with their on previous research. Finally, the methodology will be dis- systems and for these systems, in turn, to react to players’ cussed, followed by the results from the experiment, discus- agentive behaviors [14]. sion, and conclusion. So, games can be described as having goals, rules, challenges, 2. RELATED WORK interaction and stimulation. These definitions essentially 2.1 (Serious) Games show that games can be seen as a problem-solving activity in form of some sort of conflict or puzzle (goal / challenge) Throughout time many researchers and philosophers have that is structured by rules. given their definition for what a game is. A game is consid- ered to be a free and fun activity standing quite consciously Serious games are games that have other purposes next to outside ordinary life as being not serious, but at the same entertainment. Serious games have the properties of the def- time absorbing the player intensely and utterly [8]. initions as described before. These type of games are being used for training [15], education [16] and health [17]. Seri- One of the earliest philosopher to discuss games was Ludwig ous games are games that engage the user, and contribute Wittgenstein [1953]. He argued that elements such as play, to the achievement of a purpose other than pure entertain- rules, and competition fail to define what games are. The ment. This purpose may be formulated by the user self or term game is applied to a range of disparate human activities by the designer of the game. Therefore, commercial off-the- [9]. shelf (COTS) games, used for non-entertainment purposes, can be considered as serious games [18]. According to Roger Caillois [1957] a game is an activity with the following characteristics: fun, separate (as it is circum- Minecraft2 is such an example. Minecraft is an open-world, scribed in time and place), uncertain (where the outcome sandbox video game [19]. Key purposes of this game is the of the activity is unforeseen), non-productive (participation survival of the player and the exploration of the procedu- does not accomplish anything useful), rules, and fictitious rally generated world. The player can harvest numerous of (there is an awareness of a different reality) [10]. materials in the world, which can be used to craft other ma- terials and items. Next to the survival mode, the game has An interesting definition comes from Salen and Zimmerman a creative mode. In this mode, the survival element of the [2003]. They define a game as a system in which players en- game is disabled. Also, the player has infinite resources to gage in an artificial conflict, defined by rules, that results in all the materials and items at its disposal. This allows the a quantifiable outcome [11]. Noticeable here is the mention player to create and manipulate the world as he/she sees fit. conflict, which indicates the competitive nature of games. This has led to many worlds created by different players. This paved way to the application of Minecraft in the world Crawford [2003], in his definition, talks about conflict in of serious games. more detail. If no goals are associated with a plaything, it is a toy. If it has goals, a plaything is a challenge. If a challenge Minecraft also introduced an educational edition, where stu- has no ”active agent against whom you compete with,” it is dents can be taught various subjects using the mechan- a puzzle; if there is one, it is a conflict. If the player can only ics of Minecraft [20]. This was original a mod created by outperform the opponent, but not attack them, the conflict TeacherGaming 3 called Minecraftedu [21]. This version is a competition. However, if attacks are allowed, then the of Minecraft includes features that benefit teachers using conflict qualifies as a game [12]. Minecraft in the classroom. Teachers can switch their view- point for that of an avatar to spectate mode, which allows A game is a problem-solving activity, approached with a teacher fast mobility and a bird’s eye view to easily survey playful attitude [13]. According to Schell [2008] games: the students [20]. The teacher can freeze all students in the Minecraft world to draw their full attention [20]. Exploring • are entered willfully 2https://minecraft.net/en-us/ • have goals 3https://teachergaming.com/ a pre-built world, the teacher can disable the ability of stu- This will be discussed in more detail later on. dents to mine and place blocks [20]. This way the teacher can limit the actions of the students, to keep them focused Game mechanics are the rules, processes, and data at the on the task at hand. Another feature in the educational heart of a game. They define how play progresses, what version is the ability to give assignments [20]. This is an happens when, and what conditions determine victory or in-game item which contains instructions. The students can defeat[4]. fill it in as well, which can be graded by the teacher. The teacher is also able to provide students with certain items Core mechanics is a term often used to indicate mechanics which is needed to complete assignments, using simple com- that are the most influential, affecting many aspects of a mands [20]. This way the students do not need to craft the game and interacting with mechanics of lesser importance, items themselves (if it is not a part of the game), and the such as those that control only a single game element [4]. teacher does not need to travel to individual students in the Primary mechanics can be directly applied to solving chal- game. lenges that result in the desired end state. These mechanics are available from the start, which are explained in the early One example of a non-COTS serious games in education a stages of the game, and are constant throughout the game serious game called ”The Math Garden” [22]. In this game experience. In Grand Theft Auto IV5, primary mechanics pupils try to maintain their garden. This is achieved by play- are shooting, melee fighting, and driving. These mechan- ing small games in the context of mathematics. Correlations ics are always available to the player, mapped to the most show a high relation between the use of Math Garden and obvious and tradition-conforming controller inputs and re- the dutch CITO4 examinations scores. Other research also main consistent throughout the game experience. Secondary support the effectiveness of serious games in math. Research mechanics ease the player’s interaction with the game to- shows that even relatively short mathematical game inter- wards reaching the end state. Secondary mechanics are ei- ventions can be effective. Well designed games can engage ther available occasionally or require their combination with learners and facilitate a positive learning attitude. Games a primary mechanic in order to be functional. The cover me- can provide non-intrusive ways to assess learners mathemat- chanic in Grand Theft Auto IV is an example: it cannot be ical knowledge and skills. And it is possible to achieve trans- used exclusively to solve the main challenges of the game, fer [23]. but once mastered, it can prove of help to achieve the end state of the game [26]. Research shows mixed results when it comes to the effective- ness of such serious games. A meta analysis shows that a Mechanics of serious games are similar to those found in en- few studies found that games improved learning, while other tertainment video games. Since the main objective of each is studies shows no positive effects on knowledge and skill ac- different, there are some differences to be aware of. Whereas quisition when compared to traditional methods of learning entertainment video games main purpose is to entertain and [5]. the mechanics used in them allows for various modes of play. Serious games for education have a different path. Educa- Another study shows the possibility of increasing primary tional games focus on learning outcomes that are dependent school students multiplicative reasoning ability through an upon an appropriate pedagogy and the underlying game me- intervention with multiplicative mini-games. The mini-games chanics and how the content is integrated into the game so were found to be most effective when played at home and the learning is intrinsic to play [27]. Serious Game Mechanic afterwards debriefed at school. When utilized in this way, (SGM) can be defined as the design decision that transitions mini-games were found to promote students multiplicative the learning goal into a mechanical element of game-play operation skills (procedural knowledge) as well as their in- [28]. SGMs act as the game elements/aspects linking ped- sight in multiplicative number relations (conceptual knowl- agogical practices (represented through learning mechanics) edge) [24]. to game mechanics related to the player’s actions.

2.2 Game Mechanics 2.3 Complexity According to the Mechanics Dynamics and Aesthetics (MDA) The science of complexity mainly focuses on vast, complex model [25] a video game consists of these three main com- systems [4]. There is no precise definition of a ”complex sys- ponents. Mechanics describes the particular components of tem” [29]. Most researchers do agree that it is a system com- the game, at the level of data representation and algorithms. posed of many interacting parts. The collective behaviour This can be seen as the rule-set of the game. Dynamics de- of those parts together is more than the sum of their indi- scribes the run-time behaviour of the mechanics acting on vidual behaviors. The collective behaviours are sometimes player inputs and each others outputs over time. Aesthetics also called ”emergent” behaviours [29]. describes the desirable emotional responses evoked in the player, when he/she interacts with the game system [25]. In terms of complex game mechanics, not much research Some of the components tie in with the previous mentioned was found. However, complex systems can be observed in definitions. The mechanics of the game are considered as games as well. Complex game mechanics can be considered a rule-set which help structure the play. The dynamics is as smaller game mechanics that influence each other. This caused by the interaction of the player with the systems and can be observed in tower defence games which consist of these systems, in turn, react to the player. In this paper, a number of relatively simple parts [30]. Enemies follow a the focus will be on the mechanics component of the game. fixed path toward the player’s fortress. Each enemy has a

4http://www.cito.nl/ 5https://www.rockstargames.com/IV/ particular speed, and a number of hit points. The player places towers to defend his fortress. Each tower fires pro- jectiles at enemies within a certain range and at a certain rate. Some of the defending towers of the player deal damage while others produce other effects (e.g. such as slowing en- emies down). Sometimes towers will boost the performance of neighboring towers [29]. In this example smaller simpler mechanics influence each other. The relation between these mechanics then create a more complex system, which can be considered as complex game mechanics. Therefore, simple mechanics are regarded to be mechanics that are not inter- acting with other mechanics. Whereas complex mechanics are mechanics that do interact with other mechanics, creat- ing a complex mechanics system.

3. METHOD Figure 1: Problem spaces in complex serious games Previous studies show that players will deduce what the core mechanics must be by watching the behavior of the game many times [4]. This could be seen as a problem space by itself, which is the understanding of the game mechanics. iterative process of adding numbers until left with a single Complex mechanics can be seen as complex systems, that digit. The digits of any number are added together, and would make the problem space of game mechanics bigger. then the digits of the resulting number are added together. However, no studies were found exploring the effect of com- This cycle continues until the resulting number is a single plex mechanics in serious games. digit. That single digit is the digital root of the original number. It is assumed that this approach is relative easy, Serious games have an educational goal [31]. This can be but overall unknown to people. seen as a problem space where a game ( or its elements) is being added to, e.g. learning math. Research shows mixed The serious game is constructed using Minecraft version 1.12 results regarding the effectiveness of serious games [1]. with the help of a mod. A mod (stand short for modifica- tion) is an alteration made to the game. The mod used to So, a serious game can be considered having two problem help construct Numbers Prison is called WorldEdit 7. The spaces, one being the being the game part and the other be- narrative of the game is that the player is locked inside an ing learning material part. As the player must figure its way underground prison. The underground prison consists of five through both problem spaces, this could result in problems rooms. The player starts with a book and several pieces of in either the understanding the gameplay of a serious games paper in the inventory. The book was written by someone or its learning material, or both. who already had escaped the prison. This book contains instructions on how to play the game, how to escape, and In summary, research already shows how (serious) games hints for each room (see figure 2). Each room has a puzzle work, and how to define game mechanics. However, not that the player needs to solve, a locked chest, and an anvil. much research is found on the complexity of game mechan- ics, and the effect of those type of mechanics in serious games. Therefore, the research question in this paper is: how does the complexity of game mechanics affect the un- derstanding of the gameplay in serious games.

To answer the research question a quantitative experiment will be conducted. In this experiment mechanics will be ma- nipulated in terms of complexity. Complex game mechanics will be considered as a number of smaller mechanics creating a complex mechanics system. This might disturb the un- derstanding of the gameplay in serious games, as the player will also be confronted with the mathematical problem space (see figure 1). Simple game mechanics will be considered as number of smaller mechanics that do not affect each other, while the player faces the mathematical problem space. A Figure 2: Screenshot of instructions serious game called ”Numbers Prison” for learning the ”Dig- ital Root” will be created using Minecraft6. These puzzles are based on the digital root method. Each room contains a set of hidden numbers (see figure 3). The 3.1 Numbers Prison player searches for these numbers then applies the digital Numbers Prison is a serious game that teaches the players root method. The hidden numbers are added, and if the about the so-called ”Digital Root”. It is a mathematical resulting number is a double-digit number then process of 6https://minecraft.net/en-us/ 7https://minecraft.curseforge.com/projects/worldedit adding continues until left with a single digit number. This number is the pass code needed to open the chest. The player is instructed to change the name of the paper to the right pass code using the anvil. The chest will contain an item that helps the player to proceed to the next room. This is the layout of the simple version of the game.

Figure 5: Screenshot of trading

4.1 Setup Numbers Prison will be shared online with existing Minecraft communities. For each version of the game a post has been made onto either PlanetMinecraft 8 or on Minecraft Curse- 9 Figure 3: Screenshot of crafting Forge website with the description of the study, a descrip- tion of the game, and a link to the online questionnaire. Both version will not be posted onto the same website. This The complex version is similar to the simple version of the will be done to make sure that the same players do not play game, but with added complex game mechanics. Through- both versions, as this is a between-subjects study. This way out the underground prison there are other prisoners trapped. carry-over effects are avoided. The simple version will be The player needs to find these prisoners so that the player uploaded first. After sufficient amount of participants have can trade with them. To trade the player needs to craft the played the simple version, this version will be taken offline, required item. To craft the player needs to find the right and the complex version will be uploaded. materials. After finding the right materials the player needs to find a crafting table to craft the item. The item that A power analysis was conducted to determine the sample the player receives from the trade helps the player to move size. With the confidence level of 95 percent, and the confi- onto the next room. So, the complexity is added with the dence interval of 10. The power analysis shows a sample size crafting (see figure 4) and trading systems (see figure 5). of 97 participants. Around 100 participants are expected to participate. That will be 50 participants per condition.

Members of the community can download the Numbers Prison save file and play it using their version of Minecraft. This is the PC version of the game, and it is required to play with keyboard and mouse. The players will play the game alone (single player mode). The game is set in adventure mode, and the difficulty is set on peaceful. This means that the player cannot manipulate any of the structures (e.g. cannot destroy or place blocks), and the players cannot die in the game. The expected duration of the simple version is 10 minutes, whereas the expected duration of the complex ver- sion is 20 minutes. It is not required to finish the game to fill in the questionnaire. These participants can be compared to the ones that did complete the game in the analysis. Par- Figure 4: Screenshot of crafting ticipants who did not finish the game can be considered as either not understanding the gameplay or not understand- 4. EXPERIMENT ing the Digital Root concept. After the player finishes the game, an in-game message will prompt asking them to fill The goal of the experiment is to see whether the complexity in the questionnaire with a link to the online questionnaire. of game mechanics affects the understanding of gameplay in The questionnaire is does not take more than 5 minutes. serious games. And if it does, how does this effect look like. The experiment will consist of two experimental conditions. In the simple condition the participants are going to play 4.2 Assessment the simple version of the game. In the complex condition the The questionnaire used for this study derives from the Core participants are playing the complex version. Participants Elements of the Gaming Experience Questionnaire (CEGEQ) of both conditions will fill in the same questionnaire. 8https://www.planetminecraft.com/ 9https://minecraft.curseforge.com/ [32] to measure the understanding of the gameplay. The pants. This is less than the participants needed according questionnaire consists of different scales, which are: Enjoy- to the power analysis. In the discussion section this will be ment, Frustration, Video-game, Puppetry, Game-Play, En- discussed in more detail. vironment, Control, Ownership and Facilitators. The aim of the study is to see if players understand the game mechanics First, it is interesting to see whether the amount of Minecraft in serious games, only several scales were needed. played per week and the level of expertise differentiated be- tween the participants of both conditions. Therefore, the The following scales were included for the assessment: En- dependent variables are: playtime per week in hours and joyment, Frustration, Puppetry (control), and Gameplay. expertise level, and the independent variable is the experi- Enjoyment and frustration can tell how the player perceived mental conditions. This might influence the understanding the gameplay. Low enjoyment and high frustration can in- of the gameplay as more experienced Minecraft players will dicate to a low understanding of the gameplay. Puppetry have less trouble with the mechanics problem space than (control) is split into two separate scales. One scale assesses less experienced Minecraft players. To analyze this an in- if the players understood the controls of the game. This is dependent t-test was utilized. The analysis shows no statis- to see whether the game controls influenced the players in tical difference on average playtime per week between both either version. The second Puppetry (control) scale judges conditions. Also, no statistical difference was found on the to what degree the players understood the gameplay. This expertise level between both groups. In other words, partic- is to see if players indicate that they knew what they were ipants of both groups played an equal amount of Minecraft suppose to do. Gameplay scale focus more directly on their per week, and indicated the same expertise level (see table opinion of the gameplay of Numbers Prison. This is to see 1). to what degree the players found the gameplay difficult and challenging. This led to 16 questions from the CEGEQ. The Table 1: Score playtime hours per week and questions use a 7-point likert scale. expertise level (higher score equals higher expertise) The questionnaire includes an open question asking the par- ticipants to state all of the activities the player performed in the game. This is done to see if the participants actually un- Condition Mean Standard deviation derstood the gameplay by reproducing the knowledge they Playtime hours per week gained about the gameplay in Numbers Prison. The an- Simple 14.32 11.06 swers will be compared with an answer model. The answer Complex 15.21 13.51 model contains all the activities that can be performed in Expertise level the game that are crucial for the progression of the game. Simple 3.73 0.83 The model distinguishes two kind of answers. Answers that Complex 3.84 0.83 focus on mathematical related mechanics, and answers that focus of the non-mathematical mechanics. Whenever the participants failed to state an activity that was crucial for The next step is to see whether the participants of each the progression of the game that was marked as forgotten. condition differentiated on the scales of the CEGEQ. As mentioned before, there are in total of five scales (enjoy- The questionnaire also includes several general questions ment, frustration, puppetry (control) I and II, and game- about the player and their experience with Minecraft. It play). These are also the dependent (continuous) variables. is asked how much Minecraft they play per week, and what The (categorical) independent variable is the experimental their expertise level is. The expertise level the participants conditions. Based on the types of variables independent t- can choose from are: novice, beginner, intermediate, ad- tests were utilized to analyze the data. vanced, and expert. This questionnaire has been programmed 10 online into Qualtrics . Table 2: Playtime per week and expertise level

5. RESULTS CEGEQ Scales t-value p-value The data gained from the online questionnaire is analyzed Enjoyment 2.35 0.02 using IBM’s SPSS 11. The results section is divided into Frustration 0.95 0.35 several sections. First, the participants will be discussed. Puppetry Control I -1.42 0.16 Second, the different scales are compared between the two Puppetry Control II -1.46 0.16 conditions. Third, the analysis will focus more on the game- Gameplay 2.15 0.04 play and activities the participant performed in the game.

A total of 42 participants filled in the online questionnaire. The analysis shows a significant difference on the Enjoyment The average age off the participants was 18.40 with a stan- scale (see table 2). The Levene’s test for equality was not dard deviation of 5.01. The participants indicated to play an significant (F = 1.63, p = .21), therefore equal variances average of 14.73 hours per week Minecraft with a standard were assumed. The participants of the complex condition deviation of 12.11. The simple condition covers 22 partic- scored higher on the enjoyment scale than the participants ipants, and the complex condition consisted of 20 partici- of the simple condition (see table 3).

10 There was a significant difference in score found on the 11https://www.ibm.com/analytics/us/en/technology/spss/ Gameplay scale (see table 2). There is a significant result on the Levene’s test for equality, F = 5.23, p = .03. Therefore, 69 percent of the participants indicated that their expertise equal variances is not assumed. To compensate for this a level of the game is advanced or higher. slight adjustment is made to the degree of freedom. The participants of the complex condition scored higher on the The second Puppetry (control) scale focuses on the partic- Gameplay scale than the participants of the simple condition ipants perception of their understanding of the gameplay. (see table 3). The analysis does show that participants of both condition indicated to understand the gameplay well, and there was no significant difference between those conditions. However, Table 3: Mean scores and standard deviation (SD) combining this result with the previous result regarding the on the CEGEQ scales (lower number means higher Gameplay scale some light can be shed on the research ques- score) tion. The analysis shows that the participants of the com- plex version have the perception that their understanding of CEGEQ Scales Condition Mean SD the gameplay is high. However, further analysis show that Enjoyment Simple 2.92 1.09 the participants found the gameplay more challenging and Complex 2.18 0.94 difficult. This might indicate that their actual understand- Frustration Simple 5.55 1.01 ing of the gameplay is lacking compared to the participants Complex 5.18 1.44 of the simple version. To further research this the open Puppetry Control I Simple 1.70 0.47 question was included. Complex 2.00 0.88 Puppetry Control II Simple 1.87 0.68 The analysis shows that the participants of the complex Complex 2.38 1.43 condition forgot to state activities significantly more than Gameplay Simple 3.23 1.04 the participants from the simple condition. Even though Complex 2.63 0.73 the participants stated to have a good understanding of the game. They also stated that the gameplay was more chal- lenging and difficult, and forgot to mention gameplay ac- tivities more than the participants from the simple version. There were no significant results found on the puppetry However, the analysis of the open question shows a correla- (control) scales (see table 2). For the first scale it means tion. Therefore, nothing can be said about causality. that the participants of both conditions had no trouble with the controls of the game. The second scale indicates that the Some shortcomings need to be mentioned as this could have participants of both condition knew what they were suppose had a significant impact on the study. First, the number of to do to progress and win the game. In other words, par- participant was too low. As the power analysis suggested, ticipants stated that they understood the gameplay. To see at least 100 participants were needed. This was not reached whether this is really the case, that the participants under- due to time constraints and unforeseen circumstances. The stand the gameplay, the open question was included. study was done online using online Minecraft communities. It was expected that the participation would be much higher. For the open question a number of participants were ex- Both versions combined were downloaded several hundred cluded based on either not answering it or giving an answer times, however the people who filled in the questionnaire was that was not related to the question. Only 10 participants much lower. This was beyond expectations. An explanation for the simple condition and 13 participants for the complex for this is that people were hesitant to click the link that led condition were used. The participant was marked as forgot to the online questionnaire. Whenever the link was clicked if they failed to state an activity crucial for the progression in the post or in-game, a pop-up warning message showed to of the game. If all activities were stated, then the partic- be cautious of external links. This could have caused people ipant was marked as complete. The dependent variable is to withdraw from the experiment. categorical. The independent variable is the experimental condition, which is categorical. To do this analysis a Pear- An alternative explanation for the low participation is that son Chi-Square was performed. The analysis shows that the people were not able to finish the game. Because of the participants of the complex conditions forgot the mechanics online nature of the study there was no direct feedback to significantly more than the participants of the simple condi- the experimentor. The game was play tested before it was tion, chi-square = 4.20, p = 0.04. put online. However, without the direct feedback it cannot be said for sure whether the users who downloaded Numbers 6. DISCUSSION Prison were also able to finish the game. This might explain the high download rate, but the low participation rate. The results do provide a number of insights that help answer the research question. First, the CEGEQ scales show that In the light of the amount of participants, the number of the players of the complex version enjoyed the game more participants that filled in the open question was much lower. than the players of the simple version. Also, the results indi- The Minecraft community consist of a significantly young cate that the players of the complex version found the game player base. A poll on a Minecraft forum showed that 46 more challenging and difficult. This can be linked to the percent of the users were between 15 and 21 years old, and enjoyment scale. Enjoyment increases with difficulty up to 20 percent was younger that 15 12. In the current study the a point, where the level becomes to difficult to enjoy [33]. average age is 18 years old where the youngest participant As the participants were acquired from online Minecraft communities, it can be assumed that the both version of 12http://www.minecraftforum.net/forums/minecraft- Numbers Prison were not to difficult to be enjoyable. Also, discussion/discussion/129318-demographics-age-poll is 10. As the questionnaire was filled online and possibly Players who played Numbers Prison with complex mechan- alone, the participants might have not taken it seriously. As ics indicated that the gameplay was more difficult and chal- some of the answers were either blank or not related to the lenging. Also, these players failed to state all the activi- question. ties they performed in the game. At first, it can be said that complex mechanics interferes with the understanding Another point to address is that the target audience is well of gameplay in Serious Games. However, the study cannot experienced with Minecraft. This was to ensure that the par- give an explanation to why this is the case as the study does ticipants were familiar with the controls of the game. There- not reach that far. fore, the controls of the game did not needed to be explained and the instructions could be focused on the gameplay of All in all, the current study provides some indication that Numbers Prison. However, the complex version consisted of the complexity of the game mechanics affects the under- the crafting and trading mechanics, which Minecraft is pop- standing of the gameplay. How the complexity causes this ular for. For the experienced player this could be considered effect is still yet to be answered. More research is needed as not complex, as the player knows what to do and what to get to that answer. Hopefully, future studies will be to expect. conducted to research this topic and help improve Serious Games. Final point to discuss is that serious games have an educa- tional goal [31]. With a serious game it is aimed to teach or train the player with learning material and through play. 8. REFERENCES The current study, however, does not measure if the players [1] Pieter Wouters, Christof Van Nimwegen, Herre understood the learning material. To pass Numbers Prison Van Oostendorp, and Erik D Van Der Spek. A as a serious game, it is necessary to see whether the play- meta-analysis of the cognitive and motivational effects ers actually understood the goal of the game, and if they of serious games., 2013. learned the basic principles of Digital Root. [2] Margarida Romero, Mireia Usart, and Michela Ott. Can serious games contribute to developing and Future studies should refrain from doing an online study. sustaining 21st century skills? Games and Culture, Too much control is lost when doing such study. Having di- 10(2):148–177, 2015. rect feedback whether the game is playable or not is crucial. [3] Brian Burke. Gamification 2020: What is the future of Even after play testing, the game can still be too difficult gamification. Gartner, Inc., Nov, 5, 2012. or too easy for the target audience. Having a game that is [4] Ernest Adams and Joris Dormans. Game mechanics: too easy or too difficult can lead to different results. Also, advanced game design. New Riders, 2012. a much higher amount of participants is needed to ensure [5] Coralie Girard, Jean Ecalle, and Annie Magnan. statistical sound results. It is also interesting to test play- Serious games as new educational tools: how effective ers that have no experience with Minecraft and compare are they? a meta-analysis of recent studies. Journal of them to the experienced players. Experienced players could Computer Assisted Learning, 29(3):207–219, 2013. have a better understanding of the gameplay as they are [6] Miguel Sicart. Play matters. MIT Press, 2014. playing in a familiar in-game environment. Finally, future studies should measure if the goal of the serious game is [7] Carlo Fabricatore. Gameplay and game mechanics reached. Measurement and testing of the acquired knowl- design: A key to quality in videogames (2007), 2007. edge and skills from the learning material in serious games [8] Johan Huizinga. Homo Ludens; a study of needs to be incorporated. play-elements in culture. Boston: Beacon Press, 1955. [9] Ludwig Wittgenstein. Ludwig wittgenstein (1997 [1953]): Philosophische untersuchungen/philosophical 7. CONCLUSION investigations, selected remarks (bemerkungen) 1-6, This paper started out with the aim to shed more light onto 10, 11, 23-26, 29, 30, 43. oxford: Blackwell. 2014. why Serious Games are failing. The study tries to look from [10] Roger Caillois. Les jeux et les hommes. Le masque et the aspect of the game instead of the aspect of the problem le vertige. Editions Gallimard, 2015. that the Serious Game tries to tackle. The reason for this is [11] Katie Salen and Eric Zimmerman. Rules of Play: that it is believed that games can be considered as a problem Game Design Fundamentals. The MIT Press, 2003. space. The player need to figure out and understand what the rules are of the game (mechanics), and how the system [12] Chris Crawford. Chris Crawford on Game Design. reacts to the input of the player (dynamics) in order to make New Riders Publishing, Thousand Oaks, CA, USA, progress. In Serious Games another problem space is added 2003. (e.g. learning math). Therefore, the research question that [13] Jesse Schell. The art of game design: A book of lenses, came forth was how complex mechanics in serious games af- 2008. fects the understanding of the gameplay. In this paper it [14] Isabela Granic, Adam Lobel, and Rutger CME Engels. is assumed that the player faces these two problem spaces The benefits of playing video games. American at the same time making the Serious Game more difficult Psychologist, 69(1):66, 2014. to play. Enjoyment of the game does increase with diffi- [15] Warren Viant, Jon Purdy, and Jason Wood. Serious culty, however stops at the point when the game becomes to games for fire and rescue training. In Computer difficult [33]. Science and Electronic Engineering (CEEC), 2016 8th, pages 136–139. IEEE, 2016. The results of the study do show that complex mechanics [16] Daniel Rodr´ıguez-Cerezo, Antonio Sarasa-Cabezuelo, affect the understanding of the gameplay to some degree. Mercedes G´omez-Albarr´an, and Jos´e-Luis Sierra. Serious games in tertiary education: A case study computation for procedural content generation in concerning the comprehension of basic concepts in platform games. In Computational intelligence and computer language implementation courses. games (cig), 2015 ieee conference on, pages 99–106. Computers in Human Behavior, 31:558–570, 2014. IEEE, 2015. [17] Mathijs FG Lucassen, Sally N Merry, Simon Hatcher, and Christopher MA Frampton. Rainbow sparx: A novel approach to addressing depression in sexual minority youth. Cognitive and Behavioral Practice, 22(2):203–216, 2015. [18] Per Backlund and Maurice Hendrix. Educational games-are they worth the effort? a literature survey of the effectiveness of serious games. In Games and virtual worlds for serious applications (VS-GAMES), 2013 5th international conference on, pages 1–8. IEEE, 2013. [19] Mojang. Minecraft, 2017. [20] Mojang. Minecraft educational edition, 2017. [21] Aaron J Hultstrand. Minecraft in the math classroom. 2015. [22] H Maas, S Klinkenberg, M Straatemeier, et al. Rekentuin. nl: Combinatie van oefenen en toetsen. Examens, (4):10–14, 2010. [23] Kristian Juha Mikael Kiili, Keith Devlin, and Jari Multisilta. Editorial: is game-based math learning finally coming of age. Intl. J. Serious Games, 2(4), 2015. [24] Marjoke Bakker. Using mini-games for learning multiplication and division: A longitudinal effect study. PhD thesis, Utrecht University, 2014. [25] Robin Hunicke, Marc LeBlanc, and Robert Zubek. Mda: A formal approach to game design and game research. In Proceedings of the AAAI Workshop on Challenges in Game AI, volume 4, 2004. [26] Miguel Sicart. Defining game mechanics. Game Studies, 8(2):1–14, 2008. [27] Ant´onio Coelho, Enrique Kato, Jo˜ao Xavier, and Ricardo Gon¸calves. Serious game for introductory programming. In International Conference on Serious Games Development and Applications, pages 61–71. Springer, 2011. [28] Sylvester Arnab, Theodore Lim, Maira B Carvalho, Francesco Bellotti, Sara Freitas, Sandy Louchart, Neil Suttie, Riccardo Berta, and Alessandro De Gloria. Mapping learning and game mechanics for serious games analysis. British Journal of Educational Technology, 46(2):391–411, 2015. [29] Mark EJ Newman. Complex systems: A survey. arXiv preprint arXiv:1112.1440, 2011. [30] Aura Hern´andez-Sabat´e, Meritxell Joanpere, N´uria Gorgori´o, and Llu´ısAlbarrac´ın. Mathematics learning opportunities when playing a tower defense game. International Journal of Serious Games, 2(4):57–71, 2015. [31] Nergiz Ercil Cagiltay, Erol Ozcelik, and Nese Sahin Ozcelik. The effect of competition on learning in games. Computers & Education, 87:35–41, 2015. [32] Eduardo H Calvillo-G´amez, Paul Cairns, and Anna L Cox. Assessing the core elements of the gaming experience. In Game User Experience Evaluation, pages 37–62. Springer, 2015. [33] Willian MP Reis, Levi HS Lelis, et al. Human