Impact of Textual Feedback on Player Retention in Endless Runner Mobile Games
A thesis presented to the academic faculty in partial fulfillment of the requirement for the Degree
Masters of Science in Game Science and Design in the College of Arts, Media and Design
by
Riddhi Padte
Thesis Committee Chair: Dr. Casper Harteveld
Thesis Advisor: Dr. Giovanni Troiano
Northeastern University
Boston, Massachusetts
December 2019
1
2 Impact of Textual Feedback on Player Retention in Endless Runner Mobile Games
by Riddhi Padte
Abstract
Player retention is key to free-to-play mobile games as their success relies heavily on players actively engaged in the game at any instant. Free-to-play games are widely played on mobile devices and endless runner mobile games, a type of game genre has been in the game industry since 2009. Endless runner games allow players to play endlessly while the game mechanic progressively becomes difficult. While such free-to-play business models aim to incentivize player retention, to date retaining players in a game for a long time remains challenging. This study investigates the use of textual feedback as a possible enhancer of player retention. I study textual feedback in free-to-play games under three different conditions, namely high-positive, low-positive, and no-textual-feedback. A study with 20 participants revealed that high-positive textual feedback impact player retention in free-to-play endless runner games, while low-positive has no impact. I conclude by discussing what the results mean to player retention and provide design recommendations for designers of free-to-play mobile games.
Keywords: Endless Runner Games, Free-to-play, Sentiment Analysis, Survey, Player Retention
Northeastern University Boston, Massachusetts December 2019
3 TABLE OF CONTENTS
Acknowledgements 7
1. Introduction 8
2. Background 9
3. Methodology 15
4. Results 19
5. Discussion 25
6. Conclusion 28
References 29
Appendix A 35
4 LIST OF TABLES
Table 3.1: List of words used in both versions of the endless runner game. 17
Table 4.1: Average session duration and number of players for each version 22
Table 4.2: Wilcoxon rank sum test results for each combination
23
5
LIST OF FIGURES
Figure 2.1: Screenshots of endless runner games: Temple Run, JetPack Joyride 11
Figure 2.1: Screenshot of game-play with textual feedback (Game: Candy Crush by
King) 13
Figure 3.1: Screenshots of game-play with higher positive score words 16
Figure 4.1: Bar chart of the responses to the questions “What part of free-to-play mobile games do you enjoy?” 19
Figure 4.2: Bar chart of the responses to the questions “Why do you play free-to-play mobile games?” 20
Figure 4.3: Bar chart of the responses to the questions “Do you like playing free-to-play mobile games?” 21
Figure 4.4: Bar chart of the responses to the questions “How often do you play free-to-play mobile games” 22
6
ACKNOWLEDGMENTS
This thesis would not have been possible without the constant guidance from my committee members. Thank you to Giovanni Troiano for supporting me and suggesting me better ways to write my thesis. Thank you to Casper Harteveld for making sure I meet my deadlines on time and keep up with the thesis. Thank you Jennifer Gradecki for helping me shape the idea of this thesis from the very start. I would not have been able to finish my thesis without the regular feedback and guidance from all of you.
Lastly, thank you Tejashree Sawadkar for being a constant encouragement throughout this process. A big thank you to my friends and family for being supportive and making sure I do this efficiently.
7
1. INTRODUCTION Player Retention (PR) is key to successful video games and is being increasingly investigated in various gaming contexts. Previous work investigated how game design influences PR in first-person shooters (FPS) (Allart et al., 2016), how the aggressive attitude of gamers can compromise PR of other gamers in online video games (Shores et al. 2014), and how music, sound effects, and animations affect PR in online casual games (Andersen et al. 2011). Recently, Koski (Koski 2019) showed that PR is important in free-to-play mobile games, and that these have higher chances to retain their players if they make them wait a while before resuming for in-game rewards. Free-to-play games do not charge players money upfront before downloading the game and hence, the players that engage in micro-transactions heavily contribute to generate revenue. More number of active players increase the chances of making profit for the game studio and hence, retaining players becomes important for such a business model. However, to date, finding in-game elements that may increase PR remains challenging.
Feedback has been an effective way to benefit learning (Azevedo and Bernard 1995). Textual instructions (or feedback) is often used in games to communicate status and help gamers progress through the gameplay (Green et al., 2018), and the benefits of text feedback in serious games have been demonstrated (for a review, see Johnson et al., 2015). As such, I argue that textual feedback may benefit PR in free-to-play mobile games too. Yet, the use of textual
8 feedback in free-to-play mobile games has received little attention and remains mostly under-investigated.
My approach to explore the effect of textual feedback (i.e., words like “great” or “well-done” popping-up on screen during gameplay, to provide players with feedback on their gaming performances) PR involves an endless runner free-to-play mobile game. The textual feedback that I investigate in this thesis are words meant to encourage players while they are playing the game. In other words, the feedback aims to promote players to continue playing the game. The game will have three different versions with different words. The first two versions will have words with different semantic scores (e.g., “delicious” food = 5- higher positive score, “good” taste = 3- lower positive score). The third version will have no words. Using Semantic
Orientation Calculator (SOCal) I categorize words into two lists: (1) higher positive score words (e.g., excellent ) and (2) lower positive score words (e.g., sweet) (Taboada et al., 2016). Then, I compare the session durations of each version and identify which version retains players the longest, to show the effect of different words on session-level retention.
This thesis will benefit game companies that develop free-to-play games. Designers will also get insight into how textual feedback can be used to enhance engagement and player-retention in free-to-play games. As I have three versions of textual feedback, I use multivariate testing.
Finally, I discuss how the results can be generalized to game design at-large and ethical implications to consider when designing for player retention.
2. BACKGROUND
9 This thesis investigates the use of textual feedback and its influence on PR in free-to-play mobile games. I investigate the above using sentiment analysis. Hence, I position my work in the areas of PR in games, particularly free-to-play mobile games, and sentiment analysis for textual feedback. Next, we provide background for previous work in those areas.
Prominence of Games
Games are widely accessible media and represent one of the most popular forms of entertainment. Jesse Schell defines games as problem-solving with a playful approach (Schell 2019). Games are often used as educational tools for a variety of topics, including the learning of social skills (Ducheneaut and Moore 2005), or engaging and training people in specific tasks like developing new knowledge or learning new skills (Breuer and Bente 2010). Through games, users can participate in immersive experiences and train their decision-making for real life situations. (Breuer and Bente 2010). For instance, “That Dragon Cancer” is a game that makes the player go through the experience of losing a family member to a terminal illness, and help users understand how to cope with their loss. Games can sometimes have lasting impact on players like in the case of “That Dragon Cancer”. In the same way, there are fleeting experiences that make a player feel good for the time its engaged in any casual game due to the game mechanics or graphics. There are a variety of games that need inference through data analytics to enhance game design or answer other game-play questions. In my thesis, I collect game data
(i.e., score, time spent playing the game), to analyse player retention in endless runner free-to-play mobile games. “Endless runner game is a genre of platform game in which the player character is continuously moving forward through an endless procedurally generated
10 world.” (D. Medeiros and D. Medeiros 2014) Examples of some endless runner games include Canabalt (2009), Temple Run (2011) and JetPack Joyride (2011).
Figure 2.1: Screenshots of endless runner games: Temple Run, JetPack Joyride
Further, we will learn why Player Retention is important in free-to-play games and how it has been measured.
Player Retention in Games
Free-to-play games are gaining more and more popularity in the industry. The game Fortnite by
Epic Games, for instance, has single-handedly generated $2.4 billion in revenue for the company
(Forbes). As such, understanding and enhancing player retention is becoming key to design successful free-to-play mobile games. The success of Free-to-Play mobile games heavily rely on how many players are actively playing the game (Drachen et al., 2015 ). The more time players spend playing a game, the higher the chances that players will generate revenue for game
11 companies. Therefore, player retention can be used to directly measure or predict how successful a free-to-play game is or will be.
Koster proposes two ways to measure PR (1) through session level retention (Koster 2013), namely measuring for how long a game holds a player in a gaming session, and (2) through classic retention (Koster 2013), namely measuring the number of players that return to the game after a given amount of time. One game-play session is measured as the time duration of the player playing the game on their device (e.g., Android smartphone) Therefore, session-level retention can be considered a direct measure of the player’s commitment to the game (Drachen et al., 2015). There can be several factors why a player chooses to play the game for a longer time. In other words, there are several elements in the game which can retain different players. This thesis investigates session-level player retention due to textual feedback in the game.
Importance of Textual Feedback in Games:
A game is made of several elements like art, music, interactive mechanics and also textual feedback. The positive feedback in games is meant to increase player engagement and enhance player experience. (Sivaji et al. 2017) looks at textual feedback in User Experience (UX) as a way to improve user efficiency of the application/tool under study . Their study finds out that textual feedback helps users evaluate UX in a better way. (Gleaves and Walker 2013) have studied aural and textual feedback in education. Their research suggests that feedback, both textual and auditory may reinforce and influence the students’ attitudes towards academic
12 progress in general. I take inspiration from these studies to understand how textual feedback was beneficial in interactive applications and potentially extend to games.
However, according to a study done by (Rieber et al. 1996) suggests that animated and graphical feedback is much more effective textual feedback. They investigated the effect of feedback in a computer generated simulation. Nonetheless, textual feedback are present in several games.
Figure 2.2: Screenshot of game-play with textual feedback (Game: Candy Crush by King)
When one talks about words, one of the ways to look at it is by categorizing them using their semantic scores and orientation. I will further discuss how sentiment analysis was used to categorize different types of words and their sematinc scores.
Sentiment Analysis
Sentiment Analysis is a technique used to label words with a semantic orientation and a semantic score; it entails extracting selected information from a text and determining the attitude (i.e.,
13 opinion) that the writer is trying to express through language (Thompson et al. 2017). For instance, in the context of food recipes, the word “good” and “delicious” can both be used as positive adjectives. However, while both words have a positive meaning, and their semantic orientation can be categorized as positive (Thompson et al. 2017), their semantic scores will be different. For instance, the word “good” has a semantic score of 2, while “delicious” has a semantic score of 5. Such semantic scoring is defined as a lexicon-based approach to categorise text (Thompson et al. 2017), and it is among the most efficient approaches to sentiment analysis.
Sentiment analysis can also be used to automatically detect the opinion (or gist) contained in a piece of text (Hailong et al. 2014). In this thesis, I use the Semantic Orientation Calculator (SO-CAL) for my sentiment analysis (Taboada et al., 2016 ). The SO-CAL is a general-purpose system for sentiment analysis, which is lexicon-based and available online for researchers. The system can be accessed by the Simon Fraser University site and it is open-source.
Application of Sentiment Analysis
Sentiment Analysis has been used in the industry for text analysis. With the growing popularity of big data, sentiment analysis is a powerful tool in the analysis of social media. Sentiment
Analysis is done by political parties and campaign managers, and by customer service representatives attempting to assess the overall opinion of customers towards a company
(Thompson et al. 2017). Twitter is one of the most popular user generated content social networking sites. The large amount of information contained in microblogging web-sites (like
Twitter) make them an attractive source of data for opinion mining and sentiment analysis (Pak
14 and Paroubek ). Even in games, Thompson et al. 2017 used sentiment analysis to predict the negativity in chat boxes in the game Starcraft. Sentiment Analysis establishes the categorization of texts. I use the categorized text to investigate how textual feedback impact player retention.
3. METHODOLOGY
I investigate how textual feedback impacts session-level player retention in free-to-play mobile game. The independent variable is textual feedback, while the dependent variable is session-level player retention. I use three versions of the game each with words which belong to a particular semantic score category (lower positive, higher positive and no words). These words are classified by using lexicon-based sentiment analysis technique. I also ask participants to fill out a pre-test and post-test survey on their previous experience with endless runner free-to-play games and for demographics .
Participants:
I recruited participants via social media advertisements through convenience and snowball sampling. I advertised the study on Facebook and included the link to the Android game in the text for the invitation.
I recruited a total of 20 participants. The average age of participants was 24.7 (SD = 1.27). Participants mentioned on the pre-survey that they play mobile games on a daily basis, while eight of them mentioned they play mobile games 2-4 times a week.
The free-to-play game
The game “Boat Endless Runner Mobile Template” (FlatTutorials, 2017) was a simple endless runner game where the player has to control a boat and try to avoid obstacles (Boat Endless
15 Runner Mobile). The game uses the android device’s gyroscope for control mechanics. The game is controlled by twisting the phone for left and right directions in order to avoid the boat from colliding with the obstacles (stones, crocodiles). At every 10 points the player scores, textual feedback pops up on the screen. If the player hits obstacles like stones or crocodiles, the game is over and has to be restarted to play again. The game was downloaded from the Unity
Asset Store and was edited to have three different versions of words on Unity 3D. The game was accessed by participants as a mobile application, which was downloadable through a link on
Google Drive. The participants were invited to download the app through the Google Drive link and install the game on their android devices.
Figure 3.1: Screenshots of game-play with lower and higher positive score words
16
Task
The game had 3 versions: (1) a versions with high-positive textual feedback, (2) a version with low-positive feedback, and (3) a version without textual feedback. The two versions with textual feedback had two different sets of words. The first set was words with lower positive scores
(e.g., “good” or “well done”) and the second was words with higher positive scores (e.g.,
“excellent” or “brilliant”). These words were picked from other mobile games and categorized using SOCal. I asked participants to play the game for 4 days. The list of the words presented to participants in the form of textual feedback are listed in the table below:
17 Table 3.1: List of words used in both versions of the endless runner game.
Lower Positive Score Words Higher Positive Score Words sweet excellent good fabulous nice remarkable positive impressive wow superb
Well done splendid cool extra-ordinary
What a delight Great job super awesome
The version in the game are as follows:
World 1: Textual feedback with lower-positive score words World 2: Textual feedback with higher-positive score words. World 3: No textual feedback
Procedure
I included the link to the game and asked people interested in participating in the study to fill out a brief pre- and post-test survey (Appendix). The pre-test survey was a multiple-choice survey to know their reasons and frequency to play free-to-play games. The post-test survey was to understand player demographics. The surveys were created using Qualtrics - a professional platform to create and assess surveys.
18
Data Analysis
The data is logged online on www.gameanalytics.com. Game Analytics is an online platform for collecting and analysing game data online. A plugin in Unity 3D with a short code enabled us to collect the data online on their website.
The data was logged in online every time the players played the game. The data logged was as follows:
1) The time duration of every gameplay session
2) Number of sessions for each user
4. RESULTS
Survey Results
I analysed the pre-experiment and post-experiment survey responses. The pre-experiment survey was filled by all participants, while the post-experiment survey was filled by 17 participants.
The pre-experiment survey gathered insights into players’ motivation for playing free-to-play games. The results are summarized below:
Figure 4.1: Bar chart of the responses to the questions “What part of free-to-play mobile games do you enjoy?”
19
The responses of the question “What part of free-to-play games do you enjoy the most?” shows that game experience and the variety of free-to-play games are a major factor for participants’ motivation to play games. It is also noteworthy that 5 participants answered that they enjoy playing free-to-play games because they are free to download.
Figure 4.2: Bar chart of the responses to the questions “Why do you play free-to-play mobile games?”
20
13 out of 20 participants responded that they like to play free-to-play games to pass the time. 7 participants indicated that they play games to become better gamers. Mobile games are played on a more casual level than other games. Hence, participants indicating that their motivation to play such games is to “pass the time” is not so surprising. None of the participants indicated that they played free-to-play games “to earn rewards”, thus emphasizing their intrinsic motivation to play games above their extrinsic motivation. This also hints at how the participants were not extrinsically motivated by any rewards. Finally, 16 out of 20 participants indicated that they enjoy playing free-to-play games.
Figure 4.3: Bar chart of the responses to the questions “Do you like playing free-to-play mobile games?”
21
Figure 4.4: Bar chart of the responses to the questions “How often do you play free-to-play mobile games”
22
The above bar graph shows the frequency at which the participants play free-to-play games in their daily lives. Seven participants play free-to-play games on a daily basis, while 8 participants answered they play free-to-play mobile games 2-4 times a week. Only 2 participants answered that they never play these games.
Game Results
Table 4.1: Average session duration and number of players for each version
World Number of players Mean session SD of session duration duration
1 6 7.74 10.24
2 7 21.74 15.83
3 7 14.68 12.86
23 Here, I notice that the version with higher positive score words (World 2) has an average game-play session time higher (M = 21.74; SD = 15.83) than the other versions. The version 2 2 with lower positive score textual feedback has the lowest average game-play session time (M = 1 7.74; SD = 10.24). However, the version with no textual feedback has an average game-play 1 session higher than the lower positive score textual feedback version (M3 = 14.68; SD3 = 15.83). As the dataset is not normally distributed, I cannot perform parametric statistics such as the t-tests. Therefore, I analyze the data using non-parametric tests.
Kruskal-Wallis test is a non-parametric test used for comparing two or more independent samples of data. I first use the Kruskal-Wallis rank sum test to establish if there is significance between the three different variables. The Kruskal-Wallis test gives us the below results:
Kruskal-Wallis chi-squared = 17.482, df = 2, p-value = 0.0001599
As the above p-value < 0.05, I infer from the test, that there is a significance between the three versions of games. To understand the significance between each of the three versions, I further do a post-hoc test with Wilcoxon signed-rank test using bonferroni correction.
The results for Wilcoxon tests are summarized in the Table below:
Table 4.2: Wilcoxon rank sum test results for each combination
Worlds for tests Wilcoxon Rank Sum Test Results
World 1 and World 2 p < 0.05 World 2 and World 3 p > 0.05 World 1 and World 3 p = 0.05
A Wilcoxon Signed-Ranks test indicated that the “lower positive-score textual feedback” when compared with the “higher positive-score textual feedback” was rated more favorably (p =
24 0.00011). The version with “lower positive-score textual feedback” when compared with the “no textual feedback” version also is almost significant according to Wilcoxon Signed Rank test which shows p = 0.05. The version with “higher positive-score textual feedback” when compared with “no textual feedback” version the significance is p = 0.19.
According to the Wilcoxon Signed-Ranks test the higher positive score words retain players significantly more compared to the lower positive score textual feedback. According to the test, the lower positive score textual feedback retains players significantly than the one with no textual feedback.
DISCUSSION
Player Retention is an important aspect in terms of free-to-play games. The success of
Free-to-Play mobile games heavily rely on how many players are actively playing the game
(Drachen et al. ). Besides trying to increase Player Retention, I also investigate textual feedback to analyze player engagement.
Impact of Textual Feedback on Player Retention
Textual feedback has proven useful in contexts like education and user experience more broadly.
(Sivaji et al. 2017) In our study, the average session duration of higher positive score is higher than the other two versions. The statistical test also indicates that the higher positive score words retain more players than using no words. Hence, there is empirical evidence supporting the assumption that textual feedback can be effective at retaining players in free-to-play mobile
25 games, when using higher positive score words (e.g., “fantastic”). However, I also found that the average duration of a session using lower positive score words was lower than the no textual feedback version. Hence, lower positive words do not seem to impact player retention.
I also consider several different genres of games like first-person shooter games, simulation games or platformer games. These genres immerse players on a different level. For example, first-person shooter games are much more immersive than endless runner mobile games in terms of experience. It might not have the same results for a first-person-shooter games as players usually don’t play those games casually like they play mobile games. In that case, the effect of textual feedback will vary depending on the genre the players are playing.
Looking at textual feedback in terms of game design will let us design games in a much more efficient way.
Limitations
Endless runner games are popular on mobile devices. However, they aren’t very popular on other gaming devices like consoles or computers (Paul Acevedo, 2015). Hence, this study remains significant to mobile games and can’t be extended to games on other devices. Mechanics of endless runner games are specific and are not necessarily used by other genres of games. The genre of game in this thesis in itself could be a limitation as endless runner games differ significantly from other games in terms of mechanics.
The textual feedback in the study had a pop-up mechanisms wherever the player scored 10 points. As the text popped up on the screen while playing, it could lead to distractions. If this
26 study was to extend to other genres of games, we should consider using the textual feedback in some other way that wouldn't be a disturbance in the experience of the game.
The way my thesis categorizes textual feedback is by using semantic analysis, which is one of the many ways to categorize texts. Using just one type of classification can also restrict the scope of our thesis.
Future Work
This thesis could be used to draw inspiration for future work regarding textual feedback in games. Every game genre arouses a different experience for the player. An adventure game might provide the players a sense of exploration. On the other hand, a simulation-based game might make a player feel control and achievement. As the experiences vary, the effect of textual feedback may also vary on different kinds of games. If the game is a puzzle game, the textual feedback might hinder the experience of the game or make the puzzles too easy. Further studies can allow us to understand how to use textual feedback to enhance game design and experience.
We can apply a detailed study with different words and look at how different words affect gameplay and engagement. By using a different kind of text classification technique, we can discover more ways of using effective textual feedback. In other words, by using jargon which matches the theme of the game, one may achieve better results.
Using similar classification of textual feedback, we can further analyze the impact of textual feedback in applications other than games. We can extend this categorizing method to other mobile applications as well.
27 Ethical implications on Player Retention
Game companies are constantly trying to reach out to more players and generate revenue from them. Even free-to-play business models want players to start paying which lets them make the game really engaging. There are instances where players have spent money to buy in-game currencies to continue playing. Game companies try to create certain situations which will lead to players spending money.
When we talk about Player Retention, it is done with the intention of making players spend more time in the game. However, we should also ask the question of how ethically appropriate that is.
In UX, the designs that mislead users and manipulates them into taking wrong decisions are called Dark Designs (Fansher et al. 2018). They are highly criticised in the field on UX and good designers try to stay clear off them. It is about time, we look at games with the lens of dark designs as well. Businesses run on money and games need to generate revenue but we have to take a step back and ask ourselves if everything we do to keep players in the game is ethically correct.
CONCLUSION
In this thesis, I showed that using higher positive score words may impact session-level retention in endless runner free-to-play mobile games. The results emerging from the study support the assumption that positive textual feedback may encourage players to engage in an endless runner free-to-play game for longer compared to the same version of the game with no textual feedback.
However, results from the study also show that textual feedback with lower positive scores have no observable impact on session level retention.
28
In short, this thesis demonstrates that textual feedback has potential for designing games that retain players for longer and enhance their gaming experience. However, I see potential for applying the same strategy beyond endless runner free-to-play mobile games. Future work should further investigate the impact of textual feedback in other games, as well as in other interactive applications. I also discussed how ethics have implications in player retention and that they should be carefully considered in future work.
29 REFERENCES
Aggarwal, P., Vaidyanathan, R., & Venkatesh, A. (2009). Using Lexical Semantic Analysis to
Derive Online Brand Positions: An Application to Retail Marketing Research. Journal of Retailing, 85(2), 145–158.
Allart, T., Levieux, G., Pierfitte, M., Guilloux, A., & Natkin, S. (2016). Design influence on
player retention: A method based on time varying survival analysis. 2016 IEEE Conference on Computational Intelligence and Games (CIG), 1–8.
Allen, J. F., Swift, M., & de Beaumont, W. (2008). Deep semantic analysis of text. Proceedings of the 2008 Conference on Semantics in Text Processing - STEP ’08. https://doi.org/10.3115/1626481.1626508
Andersen, E., Liu, Y.-E., Snider, R., Szeto, R., & Popović, Z. (2011). Placing a Value on
Aesthetics in Online Casual Games. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1275–1278. New York, NY, USA: ACM.
Andersen, E., O’Rourke, E., Liu, Y.-E., Snider, R., Lowdermilk, J., Truong, D., … Popovic, Z.
(2012). The Impact of Tutorials on Games of Varying Complexity. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 59–68. New York, NY, USA: ACM.
30 Ašeriškis, D. (n.d.). Computational Evaluation of the Effects of Motivation Reinforcement on Player Retention.
Azevedo, R., & Bernard, R. M. (1995). A Meta-Analysis of the Effects of Feedback in
Computer-Based Instruction. Journal of Educational Computing Research, 13(2), 111–127.
Boat Endless Runner Mobile Game Template (2017). FlatTutorials.
https://assetstore.unity.com/packages/templates/systems/boat-endless-runner-mobile-game-te
mplate-72470
Breuer, J., & Bente, G. (2010). Why so serious? On the relation of serious games and learning.
Journal for Computer Game Culture, 4, 7–24.
Burger, T. (n.d.). The Need for Formalizing Media Semantics in the Games and Entertainment Industry.
Candy Crush. King. 2012 d. Medeiros, R. J. V., & d. Medeiros, T. F. V. (2014). Procedural Level Balancing in Runner
Games. 2014 Brazilian Symposium on Computer Games and Digital Entertainment, 109–114.
Debeauvais, T., Schiano, D. J., & Ducheneaut, N. (n.d.). If You Build It They Might Stay: Retention Mechanisms in World of Warcraft.
31 Demirbilek, O., & Sener, B. (2003). Product design, semantics and emotional response.
Ergonomics, 46(13-14), 1346–1360.
Drachen, A., & Canossa, A. (2009). Analyzing spatial user behavior in computer games using
geographic information systems. Proceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous Era on - MindTrek ’09. https://doi.org/10.1145/1621841.1621875
Drachen, A., Lundquist, E. T., Kung, Y., & Rao, P. (n.d.). Rapid Prediction of Player Retention in Free-to-Play Mobile Games.
Drachen, A., & Schubert, M. (2013). Spatial game analytics and visualization. 2013 IEEE Conference on Computational Intelligence in Games (CIG), 1–8.
Ducheneaut, N., & Moore, R. J. (2005). More than just “XP”: learning social skills in massively
multiplayer online games. Interactive Technology and Smart Education, Vol. 2, pp. 89–100. https://doi.org/10.1108/1741565058000003
Fansher, M., Chivukula, S. S., & Gray, C. M. (2018). # darkpatterns: UX Practitioner
Conversations About Ethical Design. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, LBW082. ACM.
Fortnite. Epic Games. 2017.
Gleaves, A., & Walker, C. (2013). Richness, redundancy or relational salience? A comparison of
the effect of textual and aural feedback modes on knowledge elaboration in higher education
32 students’ work. Computers & Education, Vol. 62, pp. 249–261. https://doi.org/10.1016/j.compedu.2012.11.004
Hailong, Z., Wenyan, G., & Bo, J. (2014). Machine Learning and Lexicon Based Methods for
Sentiment Classification: A Survey. 2014 11th Web Information System and Application Conference, 262–265.
JetPack Joyride. Halfbrick Studios. 2011.
Koster, R. (2013). Theory of Fun for Game Design. “O’Reilly Media, Inc.”
Pak, A., & Paroubek, P. (n.d.). Twitter as a Corpus for Sentiment Analysis and Opinion Mining.
Rettie, R., Grandcolas, U., & Deakins, B. (2005). Text message advertising: Response rates and
branding effects. Journal of Targeting, Measurement and Analysis for Marketing, Vol. 13, pp. 304–312. https://doi.org/10.1057/palgrave.jt.5740158
Rieber, L. P., Smith, M., Al-Ghafry, S., Strickland, B., Chu, G., & Spahi, F. (1996). The role of
meaning in interpreting graphical and textual feedback during a computer-based simulation.
Computers & Education, 27(1), 45–58.
Schell, J. (2019). The Art of Game Design: A book of lenses. AK Peters/CRC Press.
Shores, K. B., He, Y., Swanenburg, K. L., Kraut, R., & Riedl, J. (2014). The Identification of
Deviance and Its Impact on Retention in a Multiplayer Game. Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, 1356–1365. New York, NY, USA: ACM.
33 Sivaji, A., Nielsen, S. F., & Clemmensen, T. (2017). A Textual Feedback Tool for Empowering
Participants in Usability and UX Evaluations. International Journal of Human–Computer Interaction, 33(5), 357–370
Taboada, M., Anthony, C., & Voll, K. (n.d.). Methods for Creating Semantic Orientation Dictionaries.
Temple Run. Imangi Studios. 2011.
Thompson, J. J., Leung, B. H. M., Blair, M. R., & Taboada, M. (2017). Sentiment analysis of
player chat messaging in the video game StarCraft 2: Extending a lexicon-based model.
Knowledge-Based Systems, 137, 149–162.
APPENDIX A:
The link to the game: https://drive.google.com/file/d/1UV_BoSFRnmj7QJp9hzuiC_E4CRfoFkF4/view?usp=sharing
Pre-experiment Survey:
34 1. How often do you play free-to-play mobile games?
● Daily
● 4-6 times a week
● 2-4 times a week
● Once a week
● Few times a month
● Never
2. What part of free-to-play mobile games do you enjoy?
● The fact that it is free
● The variety of free-to-play games in app store
● The game's experience
3. Why do you play free-to-play games?
● To pass the time
● To get better at the game
● To earn rewards
35 4. Do you like playing free-to-play mobile games?
● Yes
● Maybe
● No
The link to the survey: https://neu.co1.qualtrics.com/jfe/form/SV_4Zq9UelAjUQuQsJ
Post Experiment Survey:
1. I enjoyed playing the game
1. Strongly Disagree, 2. Disagree, 3. I don’t know, 4. Agree, 5. Strongly Agree
2. The textual feedback was clear and easy to see
1. Strongly Disagree, 2. Disagree, 3. I don’t know, 4. Agree, 5. Strongly Agree
3. The textual feedback helped me understand my performance in the game
1. Strongly Disagree, 2. Disagree, 3. I don’t know, 4. Agree, 5. Strongly Agree
1. Please provide your date of birth.
2. Is English your first language?
36 ● Yes
● No
The link to the post-experiment survey: https://neu.co1.qualtrics.com/jfe/form/SV_3WunzVktvCSs6Ed
Histogram with normality curve for world 1,2 and 3:
37
38