STUDYINGTHE DISTRIBUTION PLATFORMS FOR GAME MODSTO

CREATE ACTIVE MODDING COMMUNITIES.

by

DANIEL LEE

A thesis submitted to the

School of Computing

in conformity with the requirements for

the degree of Master of Science

Queen’s University

Kingston, Ontario, Canada

September 2019

Copyright © Daniel Lee, 2019 Abstract

AMER loyalty is important for the sales of a game. However, the gaming in-

dustry and the needs of gamers are rapidly increasing, making it difficult G for game developers to address these needs. Game mods are modifications of an original game, which can help game developers save cost and time. Hence, it is crucial for game developers to build and maintain an active modding community.

In this thesis, we study mods to provide insights to game and developers on building an active modding community. An active modding community consists of two dimensions: (1) continuous growth of mods from mod developers, and (2) contin- uous adoption of mods from gamers. We investigated these two dimensions respec- tively:

To help game developers understand how to help mod developers continuously deliver high quality mods, we investigate the release schedule and post-release sup- port of highly endorsed mods from the top 20 games on the Nexus Mods distribution platform. We observe that supporting the same modding tool within a game franchise

i is associated with faster release of mods. We also observe that bug reports (i.e., post- release support) of mods is poor in quality, making it difficult to address many of them.

However, solved bug reports are often resolved within one day.

To help mod developers understand how to build popular mods based on down- load count, we investigate the impact of 33 features across the following 5 dimensions of popular and unpopular mods in the CurseForge mod distribution platform: mod category, mod documentation, environmental context of mod, remuneration for the mod and community contribution for the mod. We observed that popular mods tend to have a high quality description and promote community contribution. Also, mods that simplify mod development are more likely to be popular.

Based on our findings, we suggest game developers who desire an active modding community to provide an officially-supported modding tool. Also, Nexus Mods should improve the quality of their bug reporting system to achieve higher quality bug reports.

Furthermore, we suggest mod developers to help improve the support for mod devel- opment, to improve the description of their mods, and to welcome community con- tributions.

ii Acknowledgments

First of all, I would like to give my biggest thanks to my supervisor Prof. Ahmed E.

Hassan for believing in me and guiding my research. He gave me the opportunity to learn and experience research in computer science, which I can utilize to pursue my own goals. If someone told me 5 years ago that I, who originally graduated in Political

Science would be doing their Master’s in Computer Science today, I would have never believed them.

I would also like to thank my dear friend, Dr. Dayi Lin who encouraged and guided me through challenging times. He also taught me how to expand my horizons and a series of skills that I could use in my research. I would also like to extend my gratitude to my dear friend Gopi Krishnan Rajbahadur who provided me support and guidance through his kindness and cornucopia of knowledge in machine learning. In addition, I would like to thank Prof. Cor-Paul Bezemer and Dr. Mohammed Sayagh in supporting me throughout my research.

iii I would also like to give thanks to my fellow labmates at SAIL (Software Analysis and Intelligence Lab): Aaditya Bhatia, Filipe Roseiro Cogo, Prof. Shaowei Wang, Hasan

Ibrahim, Abdullah Ahmad Zarir, Jiayuan Zhou, Dr. Heng Li, Kundi Yao, Dr. Safwat

Mohamed Ibrahim Hassan, and Dr. Gustavo Ansaldi Oliva.

I would like to thank my mom, dad, and sister for being there in my life. Life is filled with hard times, but as people we grow and learn to persevere. Family will always be there for you. Last, but not least, I would also like to thank my friends back at home for supporting me: Mark Molina, Ling Iv, Alex L’un,Macy Yim, Robert Nafrada, Jenica Bue- naventura, Matthew Chung, Luke Chung, Kenny Li, Izzy Sandiford, and Justin Chung.

iv Table of Contents

Abstract i

Acknowledgments iii

List of Tables vii

List of Figures viii

1 Introduction1 1.1 Thesis Statement...... 3 1.2 Thesis Overview...... 3 1.3 Thesis Contributions ...... 5

2 Background7 2.1 The Nexus Mods Distribution Platform...... 7 2.2 The Game...... 9 2.3 The CurseForge Mod Distribution Platform...... 10

3 Related Work 12 3.1 Game Modding ...... 12 3.2 Games and Software Engineering...... 13 3.3 Studies of the Minecraft Game ...... 15 3.4 Mining Online Distribution Platforms...... 15

4 Continuous Growth of Mods by Mod Developers 17 4.1 Introduction ...... 18 4.2 Methodology...... 22 4.3 Preliminary study of mods from the Nexus Mods distribution platform . 31 4.4 RQ1: What is the release schedule of mods?...... 36 4.5 RQ2: How well is the post-release support of mods?...... 44 4.6 Implications of our Findings...... 53 4.7 Research Challenges...... 57

v 4.8 Threats to Validity ...... 60 4.9 Chapter Summary ...... 62

5 Continuous Adoption of Mods by Gamers 65 5.1 Introduction ...... 66 5.2 Data Collection ...... 69 5.3 Characteristics of Popular and Unpopular Minecraft Mods...... 73 5.4 Implications ...... 90 5.5 Threats to Validity ...... 91 5.6 Chapter Summary ...... 93

6 Conclusions and Future Work 95

Bibliography 99

vi List of Tables

2.1 Online Mod Distribution Platforms Overview ...... 8 2.2 An overview of Minecraft mod distribution platforms ...... 10

4.1 An overview of the studied mods...... 23 4.2 Our studied games (sorted by the number of mods)...... 24 4.3 An overview of the official modding tools of the studied games, sorted by the total number of mods...... 26 4.4 An overview of the file categories and the included file types (sorted al- phabetically)...... 27 4.5 The definitions of the mod categories (sorted alphabetically)...... 31 4.6 The median number of days between an official game update and a mod release for each mod category, along with the median number of days between adjacent mod releases, sorted by the median number of days before an initial mod release...... 40 4.7 The definitions of the bug statuses (sorted alphabetically)...... 45

5.1 An overview of the CurseForge mod distribution platform dataset. . . . . 70 5.2 Dimensions and their features describing the characteristics of popular and unpopular Minecraft mods...... 71 5.3 An overview of the statistics of each dimension and its features. The larger the ∆LRχ 2, the larger the impact of a studied dimension. Simi- larly, the larger the Wald χ 2, the larger the explanatory power of a fea- ture in the combined model (The percentages and p-value are rounded to two decimal places). The feature is statistically significant if the p- 2 value <= 0.05. Sorted by the Wald χ per studied dimension...... 79

vii List of Figures

4.1 An overview of our methodology...... 22 4.2 The percentage of media files in each studied mod. The black vertical line shows the median value...... 28 4.3 An overview of the creation of our balanced sample of mods...... 32 4.4 The distribution of the endorsement ratios per studied mod with and without official modding support. The black vertical lines are the me- dian values...... 34 4.5 The distributions of the number of releases per mod across each studied dimension. The -axis is logged...... 38 4.6 The distributions of the median number of days before a release for the three types of mod releases. The x-axis is logged...... 39 4.7 The accumulated percentage of the number of released mods per day after the initial game release for the four studied game franchises. . . . . 42 4.8 The distribution of the number of bug reports per studied non-media and media mod. The x-axis is logged...... 46 4.9 The distribution of the number of bug reports per studied mod for each high level mod category. The x-axis is logged...... 47 4.10 The distribution of the percentage of bug reports per studied mod for each bug status. Sorted by the median percentage of bug reports per studied mod...... 48 4.11 The distribution of the number of comments per bug report...... 50 4.12 The distribution of the number of unique commenters per bug report. . 51 4.13 The distribution of the median length of the bug report comments per studied mod for non-media and media mods. The x-axis is logged. . . . 52

5.1 An overview of our data collection...... 69 5.2 The distribution of AUCs of models constructed with individual studied dimension, and with all studied dimensions combined. Sorted by the median AUC of models...... 75 5.3 An overview of the process that we used to build, evaluate and analyze the combined model...... 78

viii 5.4 The nomogram visualizes the role of each feature in distinguishing a mod’s popularity. The line against each feature in the figure, varies be- tween the range of values for that given feature. The “points” line at the top of the figure, is used to calculate the magnitude of impact that each feature has and “Total Points” at the bottom of the figure gives the to- tal points generated by all the features for a given instance (i.e., for a given mod). For instance, if for a given mod, the feature “is_cat_fabric” has a value of 1, then it contributes 100 points. Finally, the line against ‘‘Popularity” shows how the probability of a mod being popular varies with the total point (which is computed by summing up all the individ- ual points contributed by each feature). For instance, if all the features for a given mod, contribute a total of 260 points, then the probability of a mod being popular is 99% and similarly, if the total points given by all the features for a particular mod is less than 110, then there is no chance for that mod to be popular. Also, the model used to generate this nomo- gram achieved a median AUC of 0.92 on 100 out-of-sample bootstrap iterations. The values of the numeric features are logged...... 83 5.5 The impact of features on the outcome probability when another feature is held constant (features are held constant at the median for numeric features and at the mode for boolean features). The grey area shows a confidence interval at 95%. The values of the numeric features are logged...... 85

ix CHAPTER 1

Introduction

HE game industry is growing at an alarming rate, with the global games mar-

ket revenue projected at $152.1 billion USD for 2019, which is about a 10% T increase from 2018 (Wijman, 2019). Thus, game developers are under im- mense pressure to meet the needs of their users (Chambers et al., 2005), as game de- velopment can be complex and expensive. In addition, gamer loyalty is an important aspect for game developers, as long-time players of a game are more likely to purchase the next game of a developer (Peterson, 2014). Prior work (Poretski and Arazy, 2017) shows that game mods can contribute to the increased sales of the original game.

Game mods can help relieve the pressure on game developers of having to develop new content for the game. There has even been cases where mods have become their

1 CHAPTER 1. INTRODUCTION 2

own standalone games. For example, the Defense of the ancients (Dota) game1 was

originally a mod for the Warcraft 3 game,2 and then became a standalone game due to its popularity. The Dota 2 game still has an average of approximately 460k players per day, more than 6 years after its initial release and more than 17 years after the release of the Warcraft 3 game (Steamcharts.com, 2018).

However, game developers face challenges in meeting the needs of gamers because there is a tradeoff in balancing available resources to either extend the longevity of an old game or to work on a new project. Thus, game developers should leverage the available resources of the gaming community. Game modders are third party develop- ers or enthusiasts who participate in developing modifications for the original game to further enjoy or improve it. In addition, game mods are modifications of an original game created by game modders, which can help the original game remain replayable for longer than intended, especially since mods provide new content and updates to the original game. For example, the Skyrim game still receives a median of 86 new mods per day even 8 years after its original release (2011) (Scott, 2007). Mods can help game developers save cost and time, while providing gamers with a better gaming experi- ence. Hence, building and maintaining an active modding community is important for game developers.

1https://www.dota2.com/play/ 2http://us.blizzard.com/en-us/games/war3/ CHAPTER 1. INTRODUCTION 3

1.1 Thesis Statement

Mod distribution platforms can provide game and mod developers, along with

researchers an understanding on building and maintaining an active modding

community, which in turn can help improve the longevity of games and the loy-

alty of gamers, ultimately saving game developers cost and time. In our study, an

active modding community consists of two dimensions: the continuous growth

of mods by mod developers and the continuous adoption of mods by gamers. We

thoroughly investigate these dimensions in this thesis.

1.2 Thesis Overview

In this thesis, we focus on mining popular online distribution platforms for game mods.

We primarily focus on the Nexus Mods and CurseForge mod distribution platforms because Nexus Mods hosts one of the largest number of mods (Dey et al., 2016), and

CurseForge hosts one of the largest number of mods for the Minecraft game, which has one of the most active and largest modding communities (Mills, Aaron, 2015). We

first provide background about the Nexus Mods distribution platform, the Minecraft game, and the CurseForge mod distribution platform (Chapter2). Then, we discuss related literature on (1) game modding, (2) games and software engineering, (3) stud- ies on the Minecraft game, and (4) mining online distribution platforms (Chapter3).

We then provide details on our analysis of game mods from two mod distribution plat- forms (e.g., Nexus Mods and CurseForge) based on the two dimensions of an active modding community: 1) continuous growth of mods by mod developers 2) continu- ous adoption of mods by gamers. Finally, we conclude our thesis (Chapter6). Below, CHAPTER 1. INTRODUCTION 4

we summarize the two chapters of our thesis on game mods in mod distribution plat-

forms based on the two dimensions of an active modding community.

1.2.1 Building the Perfect Game – An Empirical Study of Game Mod-

ifications

In this chapter, our goal is to investigate how to increase the continuous growth of mods by mod developers by studying the release schedule and post-release support of mods, which can help build and maintain an active modding community. Our study can provide insights to the modding community of the Nexus Mods distribution plat- form from a quantitative perspective, and provide researchers with a solid foundation for future explorations of game mods.

We observe that providing official support for mods can be beneficial for the per- ceived quality of the mods of a game: games for which a modding tool is provided by the original game developer have a higher median endorsement ratio than mods for games that do not have such a tool. In addition, mod users are willing to submit bug reports for a mod. However, they often fail to submit such reports this in a systematic manner using the bug reporting tool of the Nexus Mods platform, resulting in low- quality bug reports which are difficult to resolve. Our findings give the first insights into the characteristics, release schedule and post-release support of game mods. In addition, our findings show that some games have a very active modding community, which actively contributes to extending the life expectancy of those games through the mods. Our findings offer valuable insights for game developers who want to build and maintain an active modding community. CHAPTER 1. INTRODUCTION 5

1.2.2 An Empirical Study of the Characteristics of Popular Minecraft

Mods

In this chapter, our goal is to investigate how to increase the active adoption of mods by gamers, from studying the characteristics of popular and unpopular Minecraft mods, which can help build an active modding community. Our study can provide mod de- velopers insights in understanding the important characteristics of popular mods.

We observe that the community contribution for the mod dimension has the strongest explanatory power of the popularity of mods. Popular mods tend to promote commu- nity contribution with a source code URL and an issue tracking URL. We also observe mods that simplify mod development are more likely to be popular. In addition, pop- ular mods tend to have a high quality description. Lastly, we observe that mods that accept donation tend to be more active and more popular. Our findings offer valuable insights for mod developers who want to support an active modding community.

1.3 Thesis Contributions

In this thesis, we study the continuous growth of mods by mod developers and the continuous adoption of mods by gamers. We identify useful suggestions for game and mod developers to build and maintain an active modding community. In particular, our primary contributions are:

1. We are the first to conduct a large-scale study of the release schedule and bug re-

ports of game mods. Our analysis of mods provides suggestions to game and mod

developers to build and maintain active modding communities. CHAPTER 1. INTRODUCTION 6

2. Our study on the continuous growth of mods shows that game developers who desire an active modding community and developers of third-party mod distribution platforms should provide an officially-supported modding tool, and support mod de- velopers with a better bug reporting system, respectively.

3. We are the first to investigate the characteristics of mods from the Minecraft game, which hosts one of the largest and most active modding communities. We provide useful insights to mod developers for building an active modding community.

4. Our study on the continuous adoption of mods by gamers shows that popular mods tend to have a high quality description and promote community contribution. In addition, mods that simplify mod development are more likely to be popular. CHAPTER 2

Background

N this chapter, we provide a brief background on the Nexus Mods distribution I platform, the Minecraft game, and the CurseForge mod distribution platform. 2.1 The Nexus Mods Distribution Platform

The Nexus Mods distribution platform is one of the largest distribution platforms for

game mods with over 230,000 downloadable mods (Dey et al., 2016). Table 2.2 shows a

comparison of the number of mods on other popular distribution platforms for mods.

All mods on the Nexus Mods distribution platform are free to download. Each mod has

a publisher and a creator: the mod publisher is the user who uploaded or distributed the mod and the mod creator is a self-defined name by the mod publisher. Hence,

7 CHAPTER 2. BACKGROUND 8

Table 2.1: Online Mod Distribution Platforms Overview

Mod Distribution Platforms # of Mods Nexus Mods 230,581 Game Modding1 71,262 Moddb2 18,293 Mods Online3 4,766 1 http://www.gamemodding.net/ 2 https://www.moddb.com/ 3 http://modsonline.com/ the creator data does not necessarily reflect the actual creator of the mod. Based on our observation, the mod creator data is noisy since it is self-defined; e.g., the mod creator name “me" is frequently specified as the mod creator. Therefore, we refer to the mod publisher as the modder in the remainder of this chapter. In some cases, the mod was published by the original game developer. For example, the CD Projekt Red game studio published their MODKit for the Witcher 3 game through the Nexus Mods platform (BlindJudge, 2018).

The Nexus Mods distribution platform has a community of over 14 million regis- tered users. Mod developers can distribute their mods through Nexus Mods, but also upload new releases of their mods (along with release notes). In addition, the Nexus

Mods distribution platform offers its own simple bug reporting system,1 which allows users who downloaded a mod to report a bug for that mod. Mod developers have the option to enable or disable the bug reporting system on their mod’s webpage. Each user has the option to add a title and a description, and the choice of making the report private. In addition, there are four bug report statuses on the Nexus Mods distribution platform that we consider to be open (i.e., new issue, being looked at, known issue, and

1https://www.nexusmods.com/news/12474 CHAPTER 2. BACKGROUND 9

needs more info), whereas we consider the other statuses (i.e., fixed, not a bug, won’t

fix, and duplicate) as resolved statuses.

Each mod has a dedicated mod page, which displays metadata such as the mod ti- tle, mod publisher, mod category, the number of endorsements, the number of unique downloads, and the files of the mod. In addition, the mod page can include a detailed description, images or videos, forum posts, mod statistics and bug reports. Endorse- ments2 are the Nexus Mods distribution platform’s version of a rating and represents the user’s appreciation of the mod. An endorsement can be considered an ‘upvote’ of the associated mod. Nexus Mods does not provide a mechanism for giving a negative endorsement (other than removing a prior endorsement of the mod). Endorsements help other users find useful or enjoyable mods, but does not necessarily reflect the quality of the mod. In order to provide an endorsement, the mod must be downloaded

first.

2.2 The Minecraft Game

The Minecraft game is an open-ended game, initially developed in the Java3 program- ming language, where gamers can use a limitless number of resources (e.g., blocks) to create their own world or explore worlds to gather resources and defend themselves (Mo- jang, 2019). The Minecraft game, developed by the Mojang4 game studio, is one of the best selling video games of all time in 2019, with over 176 million copies sold since its release in 2011 (Blake, Vikki, 2019). Mods are considered one of the most pop- ular aspects of the Minecraft game, and is credited for the success of the Minecraft

2https://help.nexusmods.com/article/45-what-are-file-endorsements 3https://www.java.com/en/ 4https://mojang.com/ CHAPTER 2. BACKGROUND 10

Table 2.2: An overview of Minecraft mod distribution platforms

Minecraft mod # of mods distribution platform CurseForge1 12,710 Planet Minecraft2 9,159 Minecraft Six3 3,880 Minecraft Mods4 532 1 https://minecraft.curseforge.com 2 https://www.planetminecraft.com 3 http://minecraftsix.com 4 https://www.minecraftmods.com game (Geere and Copeland, 2019; Finley, Klint, 2014; O’Brien, Chris, 2013).

2.3 The CurseForge Mod Distribution Platform

Minecraft Mods on CurseForge. The CurseForge mod distribution platform hosts one of the largest online Minecraft mod repositories with more than 12,000 downloadable mods (CurseForge, 2006). Table 2.2 shows a comparison to other Minecraft mod dis- tribution platforms with respect to the number of mods. The CurseForge mod distri- bution platform provides a dedicated page for each mod, which contains a list of mod developers, and at least one mod category. Furthermore, mod developers can provide a Paypal5 or Patreon6 URL on their mod’s page. Patreon is a crowdfunding platform where content creators such as mod developers can promote themselves, and receive monthly donations.

Mod contributors on CurseForge. A mod on the CurseForge mod distribution plat- form can have multiple contributors, and each contributor is assigned a role for the

5https://www.paypal.com/ca/home 6https://www.patreon.com/home CHAPTER 2. BACKGROUND 11

mod (i.e., artist, author, contributor, documenter, former author, maintainer, mascot, owner, tester, ticket manager, or translator). There can be multiple contributors of a mod with the same role, except for the “owner” role which is only assigned to the user that creates the mod on the platform. Unfortunately, the CurseForge mod dis- tribution platform does not provide any official definition for the roles. Furthermore, we observed that the number of mod developers in a mod does not always accurately represent the actual number of contributors. For example, the Fossils and Archeology

Revival mod7 shows 10 mod developers on the CurseForge page, but the mod has 17 contributors on Github. Hence, we do not use the mod developer roles or the number of mod developers in our study.

Mod releases and dependencies on CurseForge. In the CurseForge mod distribu- tion platform, mods provide a list of mod releases with a corresponding upload date and supported Minecraft, Java, and Bukkit8 version(s). Each mod release can provide information on the type of dependencies of a mod (i.e., incompatible, tool, required, embedded library, and optional dependencies). Mod dependencies are other mods on which a mod depends.

7https://minecraft.curseforge.com/projects/fossils 8https://getbukkit.org/download/craftbukkit CHAPTER 3

Related Work

N this chapter, we discuss work that is related to our study. We discuss work

on (1) game modding, (2) games and software engineering, (3) studies of the I Minecraft game, and (4) mining online distribution platforms.

3.1 Game Modding

The majority of prior work on game modding focused on the cultural, social and eco- nomical aspects of modding. For example, several studies focused on the types of mods that exist and the motivations of developers and players for modding and using mods (Laukkanen, 2005; Bostan and Kaplancali, 2010; Hirvonen, 2017; Postigo, 2007;

12 CHAPTER 3. RELATED WORK 13

Sotamaa, 2010). In addition, several studies investigated the relation and mutual ben- efits between a game and its mod developers (Jeppesen, 2004; Nieborg and Van der

Graaf, 2008; Arakji and Lang, 2007; Gallagher et al., 2017; Sotamaa, 2003). Poretski and

Arazy(2017) conducted an empirical study on 64 games from the Nexus Mods distri- bution platform and found value in mods with respect to an increase in sales of the original game.

There are relatively few studies on game modding from a software engineering per- spective. Dey et al.(2016) studied the mod-popularity of features in the 6 most-modded games on the Nexus Mods distribution platform by looking at the number of unique downloads and associated tags with each mod. They found that untagged mods are the least popular. Scacchi(2011a,b, 2010) gives an overview of the types of mods and the modding community, and describes modding as an open source approach to ex- tending closed source software.

3.2 Games and Software Engineering

Several studies focused on games and software engineering. Several studies mined data from game projects to study software engineering aspects. For example, Ahmed et al.(2017) and Pascarella et al.(2018) analyzed open source game projects. Graham and Roberts(2006) and Köhler et al.(2012) studied the development of their own game, whereas Guana et al.(2015) studied the development of their own game engine. Bé- cares et al.(2017) studied the gameplay of a small game called Time and Space. In addition, several studies (Lewis et al., 2010; Lin et al., 2019a) investigated what can be learned from bug videos. CHAPTER 3. RELATED WORK 14

Other studies investigated software engineering aspects by mining data from on- line game distribution platforms. Lin et al. studied the impact of the early access re- lease model (Lin et al., 2018), emergency updates (Lin et al., 2017) and reviews (Lin et al., 2019b) of games on the online distribution platform. Cheung et al.(2014) analyzed over 200 reviews for 360 games.

A few prior studies mined data from game literature (such as articles and maga- zines). For example, several studies (Lewis and Whitehead, 2011; Politowski et al., 2016;

Washburn Jr et al., 2016) mined post-mortems of games for insights about what went wrong and what went right during the game development. Scacchi and Cooper(2015) studied literature on computer games and software engineering. In addition, Ampat- zoglou and Stamelos(2010) systematically reviewed existing literature on software en- gineering for games.

A few prior studies studied game development through interviews (Daneva, 2017;

Murphy-Hill et al., 2014) and surveys (Kasurinen et al., 2017; Koleva et al., 2015).

The abovementioned studies focused on several aspects of games and software en- gineering, such as software engineering practices applied in game development (Pas- carella et al., 2018; Washburn Jr et al., 2016; Kasurinen et al., 2017; Graham and Roberts,

2006; Guana et al., 2015; Politowski et al., 2016), limitations of game development (Kol- eva et al., 2015; Murphy-Hill et al., 2014; Ahmed et al., 2017), game testing (Bécares et al., 2017; Köhler et al., 2012), empirical insights on game development for game de- velopers (Lewis et al., 2010; Lin et al., 2017, 2018, 2019b,a), user views of games (Cheung et al., 2014; Daneva, 2017), and potential software engineering research opportunities in games (Ampatzoglou and Stamelos, 2010; Lewis and Whitehead, 2011; Scacchi and

Cooper, 2015). CHAPTER 3. RELATED WORK 15

3.3 Studies of the Minecraft Game

Several prior studies have been conducted on the Minecraft game for pedagogical uses (Nebel et al., 2016; Stone et al., 2019; Lenig and Caporusso, 2018; Al-Washmi et al., 2014; Bayliss,

2012; Bebbington, 2014; Brand and Kinash, 2013; Duncan, 2011; Ekaputra et al., 2013;

Hanghøj et al., 2014; Petrov, 2014; Short, 2012; Siko et al., 2011; Zorn et al., 2013). In addition, Nebel et al.(2016) conducted an extensive literature review on the usage of the Minecraft game in education.

A few prior studies primarily focused on using the Minecraft game to study the play- ers of the game (Canossa et al., 2013; Müller et al., 2015; Quiring, 2015). Furthermore, a few prior studies primarily focused on using the Minecraft game to streamline the development of software (Balogh and Beszédes, 2013; Saito et al., 2014).

3.4 Mining Online Distribution Platforms

Studies on mining large game distribution platforms focused mostly on the Steam dis- tribution platform. The work of Lin et al.(2018, 2019b, 2017) (described above) mined the Steam platform from a software engineering point of view. Sifa et al.(2014) ana- lyzed the playtime of 6 million users on the Steam platform and observed that there were fundamental principles regarding playtime.

Blackburn et al.(2014) studied cheaters in the Steam gaming community and ob- served that the player’s network of friends impacted the likelihood of a player becom- ing a cheater.

The large body of prior work on mobile app analysis focused mostly on the Google

Play store and Apple’s App Store (Mojica et al., 2013; Khalid et al., 2014; McIlroy et al., CHAPTER 3. RELATED WORK 16

2015; Syer et al., 2013; McIlroy et al., 2016b). As a complete overview of prior work in

this field is outside of the scope of this brief survey, we refer to the survey of Martin

et al.(2017) for such an overview.

Several studies focused on release schedules of mobile apps. Hassan et al.(2017)

studied the frequency of emergency updates, and found eight patterns of emergency

releases. In addition, Hassan et al.(2018) showed that it is important to study mobile

app reviews at the release-level rather than at the update-level. McIlroy et al.(2016a)

studied the update frequency in 10,713 mobile apps that were mined from the Google

Play store and observed that only a small subset of the apps were updated more than

once a week, and that 45% of the updates did not provide a rationale for the update.

: Summary of Related Work Prior works on games primarily focus on the development of the original game.

In addition, prior works on game mods primarily focus on the social and busi-

ness oriented benefits of a modding community. However, there is no prior work

done on mining mod distribution platforms and studying the release sched-

ule, post-release support, and characteristics of mods to provide suggestions to

game developers in building and maintaining active modding communities. CHAPTER 4

Continuous Growth of Mods by Mod Developers

AME mods are modifications of the original game. Modders are enthusiasts

who provide modifications or completely new content for a game. Prior G work has shown that gamer loyalty is important for the sales of a devel- oper’s future games. Therefore, it is important for game developers to increase the longevity of their games. However, game developers cannot always meet the growing and changing needs of the gaming community, due to the often already overloaded schedules of developers. So-called modders can potentially assist game developers with addressing gamers’ needs. By supporting modders, game developers can meet the rapidly growing and varying needs of their gamer base but also extend the life ex- pectancy of their game, saving them time and money, and leading to a better overall gaming experience for their gamer base. However, to extend the life expectancy of a

17 CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 18

game on an ongoing basis, it is necessary that the modding community of that game is and remains active.

In this chapter, we empirically study the metadata of 9,521 mods that were ex- tracted from the Nexus Mods distribution platform. The Nexus Mods distribution plat- form is one of the largest mod distribution platforms for PC games at the time of our study. The goal of this chapter is to provide useful insights about mods on the Nexus

Mods distribution platform from a quantitative perspective, and to provide researchers a solid foundation to further explore game mods. To better understand the potential of mods to extend the longevity of a game we study their characteristics, and we study their release schedules and post-release support (in terms of bug reports) as a proxy for the willingness of the modding community to contribute to a game. We find that pro- viding official support for mods can be beneficial for the perceived quality of the mods of a game: games for which a modding tool is provided by the original game developer have a higher median endorsement ratio than mods for games that do not have such a tool. In addition, mod users are willing to submit bug reports for a mod. However, they often fail to do this in a systematic manner using the bug reporting tool of the Nexus

Mods platform, resulting in low-quality bug reports which are difficult to resolve.

4.1 Introduction

The gaming industry keeps on growing at a rapid pace. The Global Games Market pro- jected that 2.3 billion gamers would spend $137.9 billion USD on digital games in 2018, representing a 13% increase from 2017 (Wijman, 2018). Because of the scale of the gam- ing industry, and the fact that gamers are extremely hard to please (Chambers et al.,

2005), it has become challenging, and often costly, to develop a successful game. One CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 19

strategy that has proven to be profitable for game developers is to increase gamer re- tention, e.g., by striving to keep current players playing a game. Prior work (Peterson,

2014) shows that long time-players of a game are more likely to buy the future games of the same developer. Hence, gamer loyalty is an important aspect for game developers.

However, keeping up with the changing needs of gamers is hard, which makes main- taining gamer loyalty difficult as well. The increasing gamer expectations and rising de- velopment cost are increasing the pressure on game development teams (Kanode and

Haddad, 2009). Developers need to balance their available resources between working on a new(er) game, maintenance updates of an older game, and updates (such as new game levels) that are desired by the players of that game. Hence, the tradeoff between increasing the longevity of an older game and working on a newer game is a challeng- ing tradeoff for game developers.

For some games the gaming community itself extends a helping hand with increas- ing the longevity of the game. Game modding is the practice of modifying an existing game through game mods (Scacchi, 2011b). Game modders are external developers or game enthusiasts who make modifications to an existing game, because they enjoy or want to improve the original game. Game mods allow an original game to remain re- playable for longer than it was originally intended to, as the mods add new or updated content to the game.

For example, the Skyrim game1 continues to receive new content six years after its initial release thanks to modders (Ash0821, 2017). Thus, game modding can help alleviate the pressure on game developers of having to provide new game content. In some extreme cases, such new game content can even lead to a new standalone game.

1https://elderscrolls.bethesda.net/en/skyrim CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 20

For example, the Defense of the ancients (Dota) game series2 originated as a mod for

the Warcraft 3 game,3 and then became a standalone game due to its popularity. The

Dota 2 game still has an average of approximately 450k players every day, more than 5

years after its initial release and more than 16 years after the release of the Warcraft 3

game (Steamcharts.com, 2018).

In this chapter, we study 9,521 mods of the 20 most-modded games on the Nexus

Mods distribution platform,4 one of the largest online distribution platforms for game

mods. Our goal is to provide insights into the modding community of the Nexus Mods

distribution platform from a quantitative perspective, and to provide researchers with

a solid foundation for future exploration of game mods. In doing so, game develop-

ers can potentially reduce development time and cost due to the increased replaya-

bility of their games through mods. We compare the mods on the Nexus Mods dis-

tribution platform along three dimensions: (1) media (e.g., visual improvements) and

non-media mods (e.g., complete overhauls), (2) mods with and without official mod-

ding support from the game developer, and (3) mods for Bethesda and non-Bethesda

games (as the Bethesda game studio is known to be very supportive towards modders).

Overall, we want to study the characteristics of popular mods, and we want to investi-

gate how these mods are maintained.

We first conduct a preliminary study to understand the type of mods that are cre-

ated by modders. We observed that games with a better modding support from the

original game developers (e.g., through an official modding tool or API) tend to have a

more active modding community. An important feature of software distribution plat-

forms (including those for game mods) is to ease the deployment of software releases. 2https://www.dota2.com/play/ 3http://us.blizzard.com/en-us/games/war3/ 4https://www.nexusmods.com/ CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 21

This chapter focuses on two aspects related to game mod releases: their release sched-

ules and the post-release support. Both these aspects are well-studied by other soft-

ware engineering researchers for other types of distribution platforms (such as mobile

app stores (Hassan et al., 2018, 2017; Villarroel et al., 2016) or game distribution plat-

forms (Lin et al., 2018, 2017)). We address the following two research questions (RQs):

RQ1: What is the release schedule of mods? A mod has a median of two releases,

which are often made closely after each other. Providing an officially-supported

modding tool for a game is associated with mods being released faster, in par-

ticular when that tool is available for several games within (or even across) the

game franchise.

RQ2: How well is the post-release support of mods? In general, bug reports for mods

are of low quality. A very small portion (approximately 2 to 4%) of the bug reports

contain a code sample or a stack trace. Hence, it is not surprising that 67% of the

bug reports still have the ‘New issue’ status. Most bug reports are discussed only

by the bug reporter and mod developer. However, we came across many informal

bug reports outside of the official bug reporting tool. Hence, users are willing to

submit bug reports, but they fail to do so in a systematic manner, which leads to

low quality bug reports.

The remainder of this chapter is outlined as follows. Section 4.2 discusses our method- ology. Section 4.3 discuss our preliminary study of the characteristics of mods. Sec- tion 4.4 and 4.5 discuss the results of our empirical study. Section 4.6 discusses the implications of our study. Section 4.7 discusses the research challenges that we iden- tified for future researchers of game mods. Section 4.8 outlines threats to the validity of our findings. Section 4.9 concludes the chapter. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 22

Collecting Bug Reports

35,678 Extract bug bug Preliminary Study of reports reports Mods (Section 5)

Collecting Basic Mod Information Filtering Mod Population

Filter out mods Filter by mods with less than in the top 20 Nexus 230,581 9,521 Release Schedule of Extract mods mods or equal to games based mods Mods 1,000 on the number Mods (Section 6.1) endorsements of mods

Downloading File Contents

Download Unzip main 1,836,802 Post-Release Support main files files files of Mods (Section 6.2)

Figure 4.1: An overview of our methodology.

4.2 Methodology

This section discusses the methodology of our empirical study of game mods. Fig- ure 4.1 presents the steps of our methodology. In addition, our dataset is available in the supplementary material of this chapter (Lee et al., 2019a).

4.2.1 Collecting Basic Mod Information

We extracted 230,581 mods from the Nexus Mods distribution platform using a cus- tomized crawler on January 29, 2018. The mod metadata that we used in our study consists of the mod title, mod category, mod file size, mod files, total number of en- dorsements, total number of unique downloads, and the total number of bug reports. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 23

Table 4.1: An overview of the studied mods.

Total number of mods 230,581 Number of studied mods 9,521 Total number of games 502 Number of studied games 20 Number of studied mods with files 9,370 Number of studied files 1,836,802 Number of studied bug reports 35,678 Number of studied bug comments 109,133

4.2.2 Filtering Unpopular Mods

We observed that there are many mods on the Nexus Mods platform which do not work or are not used by anyone. As such mods are not representative of the mods that are potentially helpful to game developers, we studied only a subset of the collected mods.

We had the following selection criteria for the studied mods:

• Quality: The mod is of reasonable quality.

• Popularity: The mod has a large community. Hence, such a mod would reflect

what gamers would like to see in a game.

The first criteria is satisfied by selecting mods that do not only have a large com- munity, but are also widely appreciated (i.e., at least 1,000 endorsements). The second criteria is satisfied by selecting only mods from the top 20 most-modded games on

Nexus Mods, which left us with a total of 9,521 mods. This criterion ensures that our

findings are not biased towards unpopular games with a small number of mods. Ta- ble 4.1 gives an overview of the studied mods. Table 4.2 gives an overview of the studied games and their number of mods. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 24

Table 4.2: Our studied games (sorted by the number of mods).

Game Title Year of Release # of Mods Game Genre1 Game Developer : Skyrim2 2011 4,793 Action role-playing 4 2015 1,675 Action role-playing Bethesda Game Studios Fallout: New Vegas 2010 974 Action role-playing Bethesda Game Studios The Elder Scrolls: Skyrim Special Edition2 2016 518 Action role-playing Bethesda Game Studios The Elder Scrolls: Oblivion2 2006 442 Action role-playing Bethesda Game Studios 2008 387 Action role-playing Bethesda Game Studios Dragon Age 2009 224 Action role-playing BioWare The Witcher 3 2015 140 Action role-playing CD Projekt Red Dragon Age: Inquisition 2014 76 Action role-playing BioWare The Elder Scrolls: Morrowind2 2002 65 Action role-playing Bethesda Game Studios Dragon Age 2 2011 35 Action role-playing BioWare Dark Souls 2011 33 Action role-playing FromSoftware The Witcher 2 2011 27 Action role-playing CD Projekt Red 2016 23 Simulation ConcernedApe Mass Effect 3 2012 23 Action role-playing BioWare Mount & Blade: Warband 2010 21 Action role-playing TaleWorlds Entertainment XCOM 2 2016 20 Turn-based tactics Firaxis Games 2013 16 Action-adventure The Witcher 2007 15 Action role-playing CD Projekt Red XCOM: Enemy Unknown 2012 14 Turn-based tactics Firaxis Games

The game genre was obtained from Wikipedia Skyrim/Skyrim SE/Oblivion/Morrowind hereafter

4.2.3 Collecting Bug Reports

As explained in Chapter2, the bug reporting system of the Nexus Mods distribution platform only supports bug reports. As Table 4.7 shows there exists a bug status “Not a bug” in the Nexus Mods bug reporting system. We assume that mod developers lever- age this bug status to filter out reports that do not discuss a bug but for example, re- quest a feature. Hence, we assume that all bug reports without this status discuss actual bugs. We collected the bug report metadata of each studied mod. In total, we collected

35,678 bug reports for 2,160 out of the 9,521 studied mods. At the time of crawling, the publishers of the remaining 7,361 mods explicitly disabled the bug reporting function- ality. In addition, we collected the comments on each studied bug report. The total number of collected bug comments is 109,133. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 25

4.2.4 Determining Whether a Game has Official Modding Support

We determined if the original game developers provided modding support for a game by manually searching various sources, such as the game’sofficial website, to find whether the original game developers released a modding tool. We identified an official mod- ding tool for 65% of the studied games.

4.2.5 Downloading File Contents and Identifying Release Dates

The Nexus Mods platform hosts files in several categories for each mod. We collected

files from the following categories:

• Main files: These files are the base files of the latest version of a mod and usually

come as a compressed archive. We used these files to study the file types that are

modified or added by a mod. After downloading and decompressing the main

files, we used the File5 program to automatically classify the file type of each

decompressed file. Note that it is not sufficient to simply do the file type classi-

fication based on the file extension, as we observed that different games might

use the same extension for different file types. We automatically classified the

file type of 1,836,802 files.

• Update files and old files: These files are older versions of the main files. We used

the release dates of these files to identify the release schedule of a mod.

4.2.6 The Studied Dimensions of Mods

We study game mods along the following three dimensions:

5https://linux.die.net/man/1/file CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 26

Table 4.3: An overview of the official modding tools of the studied games, sorted by the total number of mods.

Game Title Official Tool Official URL Skyrim Creation Kit https://www.creationkit.com/ Creation Kit https://www.creationkit.com/ Fallout: New Vegas GECK https://geck.bethsoft.com/ Skyrim Special Edition Creation Kit https://www.creationkit.com/ Oblivion Construction Set https://cs.elderscrolls.com Fallout 3 GECK https://geck.bethsoft.com/ Dragon Age Toolset http://www.datoolset.net/ The Witcher 3 MODKit https://www.nexusmods.com/witcher3/mods/3173 Dragon Age: Inquisition - - - Morrowind Construction Set https://cs.elderscrolls.com Dragon Age 2 Toolset http://www.datoolset.net/ Dark Souls - - The Witcher 2 REDKit https://redkitwiki.cdprojektred.com/REDkit Stardew Valley - - Mass Effect 3 - - Mount & Blade: Warband - - XCOM 2 Modbuddy - State of Decay - - The Witcher D’jinni - XCOM: Enemy Unknown - -

1. Games with and without official modding support. While some games try to

prevent modding (e.g., multiplayer games to avoid cheating), other games pro-

vide official modding support. Games with official modding support (e.g., through

a tool or an API that is provided by the game’sdeveloper) may make it easier to de-

velop and maintain mods, which could promote an active modding community.

Table 4.3 gives an overview of the official modding tools of the studied games.

We studied 9,315 mods of games with official modding support and 206 mods of

games without official modding support.

2. Media and non-media mods. Media mods (e.g., mods that change the look-and-

feel of a game) require less programming effort than non-media mods (e.g., mods

that change the gameplay of a game) (Ramadan, 2016). Our hypothesis is that CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 27

Table 4.4: An overview of the file categories and the included file types (sorted alpha- betically).

File Category File Types Archive RAR archive data Binary PE32 executable (console) Intel 80386 PE32 executable (DLL) (console) Intel 80386 PE32 executable (DLL) (console) Intel 80386 Mono/.Net assembly PE32 executable (DLL) (GUI) Intel 80386 PE32 executable (DLL) (GUI) Intel 80386 Mono/.Net assembly PE32 executable (GUI) Intel 80386 PE32 executable (GUI) Intel 80386 Mono/.Net assembly Font TrueType font data Game Data Data Gamebryo game engine file MSVC program database ver 7.00 NetImmerse game engine file Unreal Engine Package XML 1.0 document text Image JPEG image data DirectDraw (DDS) PC bitmap PNG image data Targa image data - RGB 8 x 8 x 24 Targa image data - RGB 8 x 8 x 24 Targa image data - RGBA - RLE 32 x 32 x 32 - 8-bit alpha Multimedia Audio file with ID3 version 2.3.0 Bink Video rev.i MPEG ADTS Ogg data RIFF (little-endian) data Source Code Algol 68 source text C program text C++ source text Python script text executable

Text/Documentation ASCII text HTML document text ISO-8859 text Little-endian UTF-16 Unicode text Unified diff output text UTF-8 Unicode (with BOM) text UTF-8 Unicode text CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 28

non-media mods are harder to create and maintain. The following explains our

approach to distinguish between media and non-media mods:

(a) We collected 140 distinct file types from the studied mods. We removed file

types that took up less than 1% of the total files to remove the bias of in-

frequently used file types, leaving 37 distinct file types. We calculated the

percentage of files of each file type within each studied game.

(b) The postdoc and I independently categorized the 37 file types into high-

level file categories. Table 4.4 shows the file types that reside in each file

category. The process of grouping the file types was straight-forward be-

cause each file type had a distinct functionality. However, we did merge the

audio and video file types together into a multimedia file category. After

consolidating their categorization and resolving the conflicts, we ended up

0 20 40 60 80 100 Percentage of media files per studied mod

Figure 4.2: The percentage of media files in each studied mod. The black vertical line shows the median value. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 29

with 8 high-level file categories (archive, font, binary, source code, text/doc- umentation, multimedia, image, and game data).

(c) We then categorized each file in the image, font, archive, text/documenta- tion, and multimedia file categories as a media file. We categorized each file

in the binary, source code, or game data file categories as a non-media file.

(d) Figure 4.2 shows the distribution of the percentage of media files in the stud-

ied mods. While there appears to be a natural cutoff at approximately 95%

media files, this cutoff would introduce bias for mods with a small number

of files. We calculated that 57% of the studied mods have atmost 20 files,

which shows that the number of files could vary depending on the mod.

Hence, we used empirically observed thresholds that are based on the num-

ber of files to accurately determine if a mod is a media or non-media mod.

If the percentage of media files in a mod is larger than the threshold for the

number of files in the mod, the mod is categorized as a media mod. Oth-

erwise, the mod is categorized as a non-media mod. We used the following

thresholds:

 > 95%, if # of files 20.   ≥ Media mod thresholds = > 90%, if 20 > # of files 10.  ≥  €# of files 1Š > if # of files < 10. # of files−

We studied 1,348 media mods and 8,145 non-media mods.

3. Mods of Bethesda and non-Bethesda games. Table 4.2 shows that 7 of the 20

studied games were produced by Bethesda Game Studios (hereafter Bethesda). CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 30

Comparing mods of Bethesda games with mods of non-Bethesda games can yield

interesting insights into why Bethesda games have such active modding commu-

nities. We studied 8,829 mods of Bethesda games and 664 mods of non-Bethesda

games.

Throughout our study, we used the Wilcoxon test to assess whether the two groups within each of the abovementioned dimensions differ significantly. The Wilcoxon rank- sum test is an unpaired, non-parametric statistical test, where the null hypothesis is that two distributions are identical, while the Wilcoxon signed-rank test is a paired non-parametric statistical test (Wilcoxon, 1945). If the p-value of the applied Wilcoxon test is less than 0.05, then the two distributions are significantly different under a sig- nificance threshold of α = 0.05. To calculate the magnitude of the difference between the two distributions, we compute Cliff’s delta d (Long et al., 2003) effect size. We use the following thresholds for d (Romano et al., 2006):

 negligible(N), if d 0.147.   | | ≤  small(S), if 0.147 < d 0.33. Effect size = | | ≤ medium(M), if 0.33 < d 0.474.   | | ≤  large(L), if 0.474 < d 1. | | ≤

In the next sections, we discuss the results of our study. First we discuss our pre-

liminary study. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 31

4.3 Preliminary study of mods from the Nexus Mods dis-

tribution platform

Prior research used the total number of unique downloads of a mod as a proxy for its popularity (Dey et al., 2016). However, a download does not necessarily mean that a gamer enjoyed or played the mod. An endorsement is a better indication that a gamer has played and appreciated the mod. Therefore, we studied the characteristics of highly endorsed mods to gain a better understanding of the mods that are appreci- ated by the modding community.

Table 4.5: The definitions of the mod categories (sorted alphabetically).

Mod Category Definition # of studied mods Appearance Alters the visual features, such as the body, face, 283 and hair of the character Audio Alters the game audio 138 Bug Fixes Fixes a bug in the game 240 Character Alters playable and non-playable 873 characters in the game Cheats Alters the gameplay in an illegal or unjust manner 202 Game Items Adds or alters objects that a player can collect 2,250 or use in the game Game Visuals Alters the observable experience of the game environment 1,749 Gameplay Alters the player experience and interaction 1,498 within the domain of the game rules Overhaul Total conversion of certain game features or the full game 140 Performance Optimizes the game to improve performance 11 Save Starts the player in a specific state in the game 15 Story Adds to or alters the storyline of the game 194 User Interface Adds to or alters the interface of the game 296 Utility Tool that helps modders 307 World Object Adds or alters physical objects with which primary characters can 770 interact inside the game environment CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 32

Approach: As the number of players can differ considerably for each game, we nor- malized the number of endorsements by calculating the endorsement ratio and use this ratio instead of the absolute number of endorsements: # of endorsements endorsement ratio = # of unique downloads To identify which type of mods are highly endorsed, we leveraged the publisher- defined mod categories. As the definition of these mod categories appears not to be moderated by Nexus Mods, the definitions contain a considerable amount of noise.

The postdoc and I independently manually created a high-level mod categorization of the mod categories to group similar ones. For example, the mod categories ‘armour’ and ‘weapons’ were categorized as ‘game items’. The categorizations were consoli- dated and conflicts were resolved through discussion. The cross-validation process had a Cohen’s Kappa agreement of 0.93 (which is considered very high, indicating a straightforward categorization task). Table 4.5 shows the high-level mod categories

Name Downsample random sample Game 1 100 mods

random random sample sample Balanced Game 2 100 mods 800 mods 1,252 mods sample of 95 mods

random sample Game 3 100 mods

total Remaining 36 452 mods games

Figure 4.3: An overview of the creation of our balanced sample of mods. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 33

along with their definitions.

As the mod categories are defined by the mod publishers, we also wanted to validate

their accuracy. Therefore, we performed a sanity check in which we manually verified

for a balanced sample of 95 mods that the high-level category that we assigned to each

of these mods fits their actual description. We validated a balanced sample to avoid

bias towards mods from games with a large number of mods. Figure 4.3 shows how we

created the balanced sample. Below we detail each step:

1. We select a random sample of 100 mods for each of the 8 games that have more

than 100 mods (800 mods in total).

2. We select all 452 mods of the remaining 36 studied games.

3. We combine the mods selected in step 1 and 2.

4. We randomly choose a statistically representative sample of 95 mods (which yields

a confidence level of 95% with a confidence interval of 10).

After validating the sample of 95 mods, we found that the mod categorization of 19 of the 95 mods was incorrect or unclear. In all cases, this error or unclarity was caused by the mod publisher. Hence, our high-level mod categorization is correct for approxi- mately 80% of the mods. We did not assign a high-level category to the categories of the

19 mods that were misclassified. In total, we assigned a high-level category to 9,207 of the mods. In the rest of the chapter, all analyses that were performed on the high-level categories used only these 9,207 mods.

We used the Kruskal-Wallis test to study the differences between the median en- dorsement ratios of the studied games across the high-level mod categories. The Kruskal-

Wallis test allows us to test whether the differences between more than 2 distributions CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 34

are statistically significant.

Findings: Role-Playing Games (RPGs) are the most popular games for modding. Ta- ble 4.2 gives an overview of the studied games, sorted by the number of mods on Nexus

Mods. 90% of the studied games belong to the Role-Playing Game genre. A possible explanation is that RPGs often allow a player to customize their game character (e.g., by earning new weapons or clothing for the game character). Hence, players of RPGs may be more interested in customizing their game character beyond the officially- supported options. Another possible explanation is that RPGs have a longer playing time compared to games of other genres, which in turn motivates players to customize their game character precisely as they wish. For example, the mod-popular Skyrim and

Fallout 4 games have an average playing time of respectively 244.9 and 186.2 hours, whereas a non-RPG game such as the State of Decay game has an average playing time of 25.7 hours (Game Lengths, 2009). A final observation is that the vast majority of the

No Modding Support Modding Support

0.1 0.2 0.3 0.4 Endorsement ratio per studied mod

Figure 4.4: The distribution of the endorsement ratios per studied mod with and with- out official modding support. The black vertical lines are the median values. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 35

studied games are mainly single-player games. A possible explanation is that as players can benefit more considerably from cheating in multiplayer games (Consalvo, 2007), the developers of such games take stricter measures to avoid modding.

Mods of games with official modding support have a higher endorsement ra- tio than mods of games without official modding support. Figure 4.4 shows that mods of games with official modding support have a median endorsement ratio of 0.08, whereas mods of games without official modding support have a median endorsement ratio of 0.07. The Wilcoxon rank-sum test shows that the two distributions are signifi- cantly different, with a small Cliff’s delta effect size. While the difference between the endorsement ratios seems small, it is important to keep in mind that in general, only a small fraction of the players of a game will actually provide a rating or review of that game. For example, prior work (Lin et al., 2018) showed that games on the Steam plat- form have an average review ratio of 0.01. Hence, the difference between endorsement ratios of 0.07 and 0.08 are relevant in practice. In addition, our data shows that mods on the Nexus Mods distribution platform are endorsed more often than that the original games are rated or reviewed on the Steam platform. We are unable to find a statisti- cally significant difference under the significance threshold of α = 0.05 between the en- dorsement ratios of non-media and media mods, or between mods for non-Bethesda and Bethesda games. In addition, the Kruskal-Wallis test shows that we are unable to

find a statistically significant difference in the endorsement ratio across mods from the different high-level categories.

The most-modded games are created by Bethesda. Table 4.2 shows that Bethesda created 7 games in the top 20 of most-modded games. A possible explanation is that

Bethesda has a very strong modding support, which is well known in the modding CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 36

community (Camycamera, 2016). For example, Bethesda offers the Creation Kit mod-

ding tool. Table 4.2 shows that 73% of the studied mods are for the Skyrim, Skyrim

Special Edition and Fallout 4 games. A possible explanation for the mod-popularity of

these games is the existence of the Creation Kit (Hirvonen, 2017). A possible reason for

the strong support from Bethesda for the modding community is that Bethesda uses

the modding commmunity as a breeding ground for hiring new developers. For ex-

ample, potential level designers were asked during the interviewing process to create

a mod using the official tooling (Gallagher et al., 2017).

: Summary of Preliminary Study Role-Playing Games (RPGs) are the most mod-popular games. Mods of games

with official modding support are endorsed by gamers more often than those of

games without such support. Officially-supported modding tools are leveraged

by game developers during the hiring process of new developers or designers.

4.4 RQ1: What is the release schedule of mods?

Motivation: Mods are a contribution to the original game by the modding community.

Depending on the nature of a mod, it may require only one or several releases. For

example, a mod that improves a texture in a game may require only one release. On

the other hand, a mod that does a complete overhaul may require one or more follow-

up releases whenever its original game releases a new version. In this section, we study

the release schedules of the mods on the Nexus Mods platform. Assuming that a mod

requires more than one release, the release schedule of a mod is an indication of the CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 37

effort that a mod developer is willing to put into a mod. In addition, we study how long

it takes mod developers to release a new version of the mod after the release of a new

version of the original game.

Approach: Tocapture the release schedule of a mod, we calculated the median number of days between adjacent mod releases for each studied mod. Toinvestigate the release schedule of mods after official game updates and between adjacent mod releases, we

first manually collected the official release dates for each version of the studied games from several sources, such as the original game’s website. We were able to find official release dates for 80% of the studied games (8,804 mods). We distinguish three release types for a mod:

• The initial mod release: The first release of the mod after the original game re-

lease.

• A back-to-back mod release: A mod release that is adjacent in time to another

mod release without a game release in between. We assume that such a release

is to improve the previous mod release.

• A game-triggered mod release: A mod release that is adjacent in time to an offi-

cial game release, without another mod release in between. We assume that such

a release is to respond to the official game release.

In addition, we studied the time it took to release a mod for each of the games from the The Witcher, Fallout, XCOM, Dragon Age and Skyrim game franchises. By com- paring the release times for mods from games from the same franchise, we can study how official modding support (or its removal) for a game, or the official modding tool is associated with the release time of a mod. We studied one game franchise in which CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 38

official modding support was added (the XCOM franchise), one in which official mod- ding support was removed (Dragon Age), two franchises (The Witcher and Fallout) in which the officially-supported modding tool changed throughout the franchise’s life- time, and one franchise (Skyrim) in which the officially-supported modding tool did not change. To control for the difference in age between the games within a game fran- chise, we focus on mods that were released within two years after the original game release. Because there were no mods in our data set for The Witcher game that were released within the two-years time frame, we did not include this game in our study of the release time of mods within a game franchise.

Media

Non−media

Bethesda

Non−Bethesda

Official modding support No official modding support

1 10 100 Number of releases per studied mod

Figure 4.5: The distributions of the number of releases per mod across each studied dimension. The x-axis is logged. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 39

Findings: A mod has a median of two releases. Figure 4.5 shows the number of re- leases per mod across each studied dimension. The median number of releases is two across each studied dimension except for the non-media mods and mods of games without official modding support, which have a median of 3 releases. The Wilcoxon rank-sum test shows that we are unable to find a statistically significant difference un- der the significance threshold of α = 0.05 between the number of releases of mods for games with and without official modding support. For the media and non-media mods, the difference is significant but of negligible effect size. The finding for media and non-media mods is surprising, as one would expect that non-media mods would require more maintenance due to their closer relation with the original game’s source code.

Initial mod release

Back−to−back release

Game−triggered release

0 9 99 999 Median number of days between releases

Figure 4.6: The distributions of the median number of days before a release for the three types of mod releases. The x-axis is logged.

There is a much longer time period between an official game release and a mod release, than between back-to-back mod releases. Figure 4.6 shows the distributions of the median number of days between releases for the three types of mod release.

We identified 9,478 initial mod releases, 32,207 back-to-back mod releases and 26,400 CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 40

Table 4.6: The median number of days between an official game update and a mod re- lease for each mod category, along with the median number of days between adjacent mod releases, sorted by the median number of days before an initial mod release.

Median number of days before a release Mod Category Initial mod release Back-to-back mod release Game-triggered mod release Save Games 638.5 14.0 216.0 Bug Fixes 614.0 4.0 182.5 Overhaul 588.5 6.0 90.0 Story 512.0 10.0 214.0 Character 476.5 3.5 127.0 World Object 442.0 6.0 81.3 Utility 428.0 8.0 95.5 Game Items 339.0 2.0 49.0 User Interface 334.0 4.0 41.0 Game Visuals 314.0 1.0 56.0 Appearance 309.0 2.0 67.5 Gameplay 274.5 5.0 50.0 Audio 194.5 6.0 37.0 Cheats 130.0 3.8 18.8 Performance 13.0 2.5 11.0 game-triggered mod releases. Back-to-back releases are by far the fastest releases, with a median of 3.5 days between releases. Since the median of the number of mod releases is two, a possible explanation for the fast back-to-back releases is that mod developers

fine-tune (or fix) their mods based on feedback from the community.

It takes a median of 345.5 days before the initial version of a mod is released.

Figure 4.6 shows that the majority of mods are released after more than 345.5 days.

These slow release times indicate that mod developers are either long-time players of a game, or have renewed interest in the game after a long time. Either way, such a lag in initial release indicates that mods are a way in which players of the game extend the longevity of a game.

Performance mods are released approximately ten times faster than other types of mods. Table 4.6 shows the median number of days before a release for the three CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 41

types of mod releases across the studied mod categories. Performance mods are re-

leased the fastest by far. However, we observed that performance mods are also the

least endorsed, with a median endorsement ratio of 0.06. We found that the 11 per-

formance mods in our data set are optimizations to improve the game’s frames per

second. However, several of these mods turned out to be counterproductive and actu-

ally decrease performance (Tinydanser, 2017). In addition, the optimizations appear

to target very specific combinations of hardware, which may leave users of the mod

disappointed when the mod does not work for their hardware combinations.

Save game mods take the longest to release after an official game release. Ta- ble 4.6 shows that the initial release of a save game mod takes a median of 638.5 days.

Save game mods consist often of saved games which allow the player to load a very ad- vanced state of the game. For example, the Legit Skyrim Vanilla 99-Percent Completed save game mod for the Skyrim game has completed and collected 99% of the quests and items in the game. The game was played for a long time (more than 110 hours) to reach this advanced state. We observed that the median number of releases for save game mods was one. Save game mods with more than one release offered saved games for various story lines. For example, each release of the 100 Percent Clean Save for Skyrim6 mod offers a save game for a different story line in the game.

6https://www.nexusmods.com/skyrim/mods/56086 CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 42 100 100 80 80 60 60 40 40

Fallout 3 Fallout: New Vegas 20 20 Fallout 4 Dragon Age 1 Skyrim Dragon Age 2 Accumulated percentage of mods per day Accumulated Dragon Age Inquisition percentage of mods per day Accumulated Skyrim Special Edition 0 0

0 73 146 219 292 365 438 511 584 657 730 0 73 146 219 292 365 438 511 584 657 730

Days after initial release Days after initial release

(a) The Dragon Age game franchise. (b) The Fallout and Skyrim game franchises 100 100 80 80 60 60 40 40 20 20

The Witcher 2 XCOM: Enemy Unknown

Accumulated percentage of mods per day Accumulated The Witcher 3 percentage of mods per day Accumulated XCOM 2 0 0

0 73 146 219 292 365 438 511 584 657 730 0 73 146 219 292 365 438 511 584 657 730

Days after initial release Days after initial release

(c) The Witcher game franchise (d) The XCOM game franchise

Figure 4.7: The accumulated percentage of the number of released mods per day after the initial game release for the four studied game franchises.

The mods of games with official modding support are released faster than those of games without such support. Figure 4.7 shows the accumulated percentage of mods that were released on a particular day after the initial game release for our four studied game franchises. Hence, a steep line indicates that mods are released relatively fast. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 43

Figure 4.7a shows that the mods for the Dragon Age 1 and Dragon Age 2 games are re-

leased faster than those of the Dragon Age: Inquisition game, which was released after

the Dragon Age 2 game. In particular, it took longer before the first mod of the Dragon

Age: Inquisition game was released: 32 days compared to 13 and 2 days for the Dragon

Age 1 and Dragon Age 2 games. In addition, Figure 4.7d shows that the mods for the

XCOM 2 game were released much faster than for the XCOM: Enemy Unknown game.

Both the Dragon Age: Inquisition and XCOM: Enemy Unknown games do not have an

officially-supported modding tool. Hence, Figures 4.7a and 4.7d suggest that the mods

of games with official modding support are released faster. A possible explanation is

that it is likely easier to create such mods.

Supporting the same modding tool across games helps mod developers release

their mods faster. Figure 4.7b shows that the mods of the Fallout: New Vegas and Fall-

out 4 games were both released faster than those of the Fallout 3 game. For the mods

of Fallout: New Vegas game, a possible explanation is that this game supports the same

modding tool as the Fallout 3 game. While the Fallout 4 game supports a different mod-

ding tool, this tool was already supported by the Skyrim game. Figure 4.7b also shows

that Fallout 4 mods were released much faster than Skyrim mods. Figure 4.7b suggests

that when mod developers have the opportunity to get used to a modding tool, it allows

them to release mods faster, or even convert mods to the later game. For example, the

Asharas Hair Conversions mod for the Fallout: New Vegas game was ported from a mod

for the Fallout 3 game. The impact of supporting the same modding tool across games

is also demonstrated by the Fallout 76 game, which was released after we collected the

data for our study. Although the Fallout 76 game would have official modding support

at some point, modders did not wait for the official support and started to release mods CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 44

for the game using the modding tool of the preceding Fallout 4 game (Hansen, 2018).

A similar observation as for the Fallout game franchise can be made in Figure 4.7a for the Dragon Age 1 and 2 games, which support the same modding tool. In particular, the first 10% of the Dragon Age 2 were released within four days which suggests that they could be converted with relatively low effort from existing Dragon Age 1 mods.

Finally, an outlier in Figure 4.7 is the Skyrim Special Edition game for which 42% of

the mods were released in the first week after its release, as the Skyrim Special Edition

game is mostly a graphical update of the Skyrim game and hence shares most of the code base, making the mods easy to convert (Hruska, 2016).

: Summary of RQ1 A mod has a median of two releases, which are made closely after each other.

Performance mods are by far the fastest released type of mod, but they often are

not received well. The mods of games with official modding support are released

faster than mods of games without such support. In particular, supporting the

same modding tool across games within a franchise, or even different franchises,

is associated with faster mod releases.

4.5 RQ2: How well is the post-release support of mods?

Motivation: In the previous section, we studied the release schedules of mods to learn about the willingness and speed of mod developers to release new versions of a mod.

In this section, we study the willingness of mod developers to provide post-release sup- port for their mods. In particular, we study how, and how fast mod developers deal with CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 45

Table 4.7: The definitions of the bug statuses (sorted alphabetically).

Bug Status Definition Being looked at The reported bug is currently under investigation. Duplicate The report is a duplicate of another bug report. Fixed The reported bug is fixed. Known issue The reported bug was already known. Needs more info The bug report does not have sufficient information to fix the bug. New issue The bug report is new. Not a bug The reported bug is not reproducible or not a bug. Won’t fix The reported bug will not be fixed. user-submitted bug reports.

Approach: To study how mod developers deal with bug reports, we studied the bug reports of the mods along several dimensions:

• The percentage of bug reports per bug status. Table 4.7 shows the bug report

statuses that are supported by Nexus Mods. We calculated the percentage of bug

reports per status for each studied mod. We filtered out 4,741 mods that had less

than 10 bug reports as these mods with a low number of bug reports could bias

our results in this dimension.

• The activity of a bug report. We use the number of comments on the bug report

to capture its activity. The number of comments on a bug report is an indication

that users and mod developers are actively involved in addressing bug reports.

• The quality of a bug report. We use the following features that were proposed

by Bettenburg et al. in their study on what makes a good bug report (Betten-

burg et al., 2008): the number of itemizations, the number of code samples, the

number of stack traces, and the number of screenshots. In addition, we stud-

ied the length (in characters) of the bug reports and their comments per studied CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 46

mod. To investigate the quality of the bug reports the postdoc and I indepen-

dently manually analyzed a sample of 96 non-media and 90 media bug reports

(which statistically represents the population with a 95% confidence level and a

10% confidence interval). There were conflicting observations for two reports.

One of the conflicts was due a confusion about what is a code sample, and one

conflict was due to a confusion about what is a stack trace. We decided to use a

broad definition of code sample and stack trace. Hence, we considered a snip-

pet of a configuration file as a code sample, and logging output as a stack trace.

After broadening our definition of code samples and stack traces, there were no

further conflicts between our classifications.

• The people working on the bug report. We studied how many and which people

Non−Media Media

1 2 5 10 20 50 100 200 500

Number of bug reports per studied mod

Figure 4.8: The distribution of the number of bug reports per studied non-media and media mod. The x-axis is logged. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 47

were involved in the bug report (i.e., the poster and the people commenting on

the report).

Findings: Non-media mods have a higher number of bug reports per studied mod than media mods. Figure 4.8 shows the distributions of the number of bug reports per studied non-media and media mod. The Wilcoxon rank-sum test shows a statistically significant difference between the number of bug reports per mod for non-media and media mods, with a medium Cliff’s delta effect size. Non-media mods have a median number of six bug reports per studied mod and media mods have a median number of two bug reports per studied mod. An intuitive explanation is that non-media mods

Overhaul Story User Interface Game Items World Object Gameplay Utility Character Performance Bug Fixes Game Visuals Appearance Cheats Audio Save 1 10 100

Number of bug reports per studied mod

Figure 4.9: The distribution of the number of bug reports per studied mod for each high level mod category. The x-axis is logged. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 48

are more complex than media mods, and are therefore more error-prone. We are un-

able to find a statistically significant difference under the significance threshold of α = 0.05 between the number of bug reports for mods with and without official modding support, and mods for Bethesda and non-Bethesda games.

Complete overhaul mods have the highest median number of bug reports per mod. Figure 4.9 shows the number of bug reports per studied mod for all mods across each high level mod category. The median number of bug reports per mod for over- hauls is 14. However, the number of bug reports per mod across overhauls varies greatly.

We manually sampled 39 overhaul mods and found that 26 of the 39 overhauls mods

(67%) with bug reports added new content to a game. The median number of bug re- ports for these 26 mods was 16.

New issue

Not a bug

Fixed

Duplicate

Won't fix

Needs more info

Known issue

Being looked at

0 20 40 60 80 100

Percentage of bug reports per studied mod

Figure 4.10: The distribution of the percentage of bug reports per studied mod for each bug status. Sorted by the median percentage of bug reports per studied mod.

A median of 67% of the bug reports of a mod have the ‘New issue’ status. Fig-

ure 4.10 shows the percentage of bug reports per studied mod for each bug status. For CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 49

most studied mods, the majority of the bug reports are labeled with the ‘New issue’ sta-

tus. Prior work (Anvik et al., 2005) on bug tracking systems of the Eclipse and Firefox

open source projects reported that 18-33% of the bug reports have a status that is equiv-

alent to the ‘New issue’ status on Nexus Mods. The percentage that we found is consid-

erably higher. There are several possible explanations for this difference. First, there is

a large difference in scale between the studied open source projects in prior work and

our studied mods. In particular, many of these mods were created by a single devel-

oper who may lack the time or simply the interest to fix all reported bugs. Second, we

observed during our manual study that the vast majority of the bug reports were of low

quality, making them difficult to interpret and address. A possible explanation is that

gamers find it difficult to report stack traces or code samples, since it is commonly not

trivial to retrieve them (e.g., in games where it is difficult to cut and paste a stack trace).

Third, we learned during our manual analysis that many users have several mods for

a game installed. When a new version of the game is released, often one or more of

these mods break. Unfortunately, it is hard for users to identify which of the mods are

broken and therefore, they report all their installed mods as broken. For example, the

developer of the popular 90000 weight limit mod for the The Witcher 3 game explains that while most game releases break the mod, the mod is always updated.7 The devel- oper explains that many of the reported bugs are not related to the mod and he gives several suggestions on how to rule out problems with other mods before submitting a bug report.

The vast majority of bug reports for mods are of low quality. Bettenburg et al.

(2008) showed that bug reports that contain stack traces get fixed sooner while bug reports that contain code samples have a higher likelihood of getting fixed. 7https://www.nexusmods.com/witcher3/mods/836?tab=posts CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 50

0 20 40 60 80

Number of comments per bug report

Figure 4.11: The distribution of the number of comments per bug report.

During our manual analysis we found that 3% of the bug reports for non-media mods contained a stack trace, 2% contained a code sample and 6% contained a screen- shot or video. In addition, 3% of the bug reports for media mods contained a stack trace, 4% contained a code sample and 4% contained a screenshot. Hence, according to Bettenburg et al.’s study it is not surprising that many of the bug reports on Nexus

Mods are not yet fixed. Interestingly, bug reports for media mods contained a code sample more often than bug reports for non-media mods. While the higher percentage of code samples for media mods is surprising, a manual inspection of the bug reports shows that most of these code samples are not actual source code, but a snippet of a configuration file. Such confirmation files are essential to activate mods.

If a bug report gets a response, the median number of days to get that response is

0. Likewise, if a bug report gets fixed, this takes a median of 1 day. Figure 4.11 shows CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 51

0 10 20 30 40 50

Number of commenters per bug report

Figure 4.12: The distribution of the number of unique commenters per bug report. the number of comments per bug report. The median number of comments per bug report is one. While many bug reports do not get a response at all, the ones that got a response often received it on the same day as the bug was reported. In addition, while many bug reports remain unsolved, the ones that are solved are often solved within one day. Hence, it seems that mod developers are willing to solve and respond to bug reports quickly. A possible explanation is that mod developers would quickly address trivial issues, and avoid working on the more difficult ones. However, as explained above, the low quality of bug reports often prevents mod developers from addressing issues at all.

In 51% of the bug reports, the first comment is by someone other than the mod developer. Figure 4.12 shows that the median number of commenters per bug report is one, which would suggest that the discussion is done between the bug reporter and CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 52

the mod developer. However, we found that in 51% of the bug reports with one com- ment, the discussion was actually between the bug reporter and another community member. This relatively high percentage indicates the engagement of the community of a mod. We did observe a difference between the percentage of fixed bug reports: of the 1,642 fixed bug reports with one comment, 66% of the reports that received a com- ment from the mod developer were fixed compared to 34% of the reports that received a comment from someone other than the developer. In the latter case, we observed several indications that the the engaged community member was a contributor of the mod in question, or that the bug report was fixed without a message from the mod developer.

Non−Media Media

100 200 500 1000 2000 5000 10000

Median length of bug comments per studied mod

Figure 4.13: The distribution of the median length of the bug report comments per studied mod for non-media and media mods. The x-axis is logged. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 53

The bug report comments of non-media mods have a greater median length in

characters than those of media mods. Figure 4.13 shows the median length in char- acters of bug report comments of non-media (400 characters) and media mods (360 characters). The Wilcoxon rank-sum test shows a significant difference between the median lengths, with a small Cliff’s delta effect size. This difference suggests that bug reports for non-media mods are slightly more complex than those of media mods.

: Summary of RQ2 In general, bug reports for mods are of low quality. As a result, many of them

remain unsolved. In addition, in many bug reports, community members other

than the mod developers are engaging with the bug reporter.

4.6 Implications of our Findings

In this section, we outline the most important implications of our findings for game

developers and developers of mod distribution platforms.

4.6.1 Implications for Game Developers

Game developers who desire a modding community for their game should consider

supporting an official modding tool. One of the goals of this chapter is to investi-

gate whether providing official modding support for a game contributes to a healthy

modding community. In our preliminary study we observed that 9 of the 10 most mod-

popular games on the Nexus Mods platform have an official modding tool. In addition,

we showed in RQ1 that the mods for games with official modding support are released CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 54

faster than the mods for games without such support. In particular, this release speed improves for games that use the same modding tool as other games, which often allows mod developers to easily convert mods for newer games. An example of a game devel- oper who does well at supporting the modding community is Bethesda. In fact, when

Bethesda released the Skyrim Special Edition game, it was compared by journalists not only to the original Skyrim game, but also to the modded version of that game (Hruska,

2016). As Bethesda uses the modding community of their games as a breeding ground for hiring new developers (Gallagher et al., 2017), they show that it can be beneficial for game developers to maintain a healthy modding community.

Game developers should not only provide support for creating a mod, but also for maintaining it. Most of the studied mods are not well maintained after their initial release. In RQ1, we observed that the median time between back-to-back mod releases is only 3.5 days. In addition, in RQ2 we observed that mod developers respond to and solve some bug reports very fast. Hence, there are several indications that mod de- velopers and the modding community are willing to provide continuous maintenance for mods. Hence, game developers who wish to support the modding community of their games should also consider providing better support for maintaining the mod.

Therefore, game developers should continuously improve modding tools after a game release in order to help drive the popularity of the original game. For example, DOTA,

Counter-Strike, and DOTA Auto Chess are mods within games who have helped drive the popularity of the original games. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 55

4.6.2 Implications for Developers of Mod Distribution Platforms

Mod distribution platforms should consider providing tools for detecting conflicts between mods. During our study, we often came across cases in which the mod de- veloper provides an informal disclaimer about the compatibility of a mod with other mods of the game. In most of these cases, the mod developer does not guarantee com- patibility. The main reason is that mods for the same game often modify the same files.

Hence, there may be conflicts between these modifications, especially because many users install several mods for a game. Because of the large number of mods for some games, it is not feasible to test a mod in all possible installation configurations. We envision mod distribution platforms acting in a supporting role when it comes to de- tecting conflicts between mods. For example, these platforms could inspect the files of a mod that is being submitted and report potential conflicts with other mods. The mod developer can then focus on testing these combinations of mods.

The Nexus Mod distribution platform should improve its bug reporting system, e.g., by guiding users better when submitting a bug report. We observed in our study that the vast majority of the bug reports for mods are of low quality. An important first step to improving the quality of the bug reports would be to better guide users when they are reporting a bug. For example, we observed that a very small portion of the bug reports contains a stack trace, code sample or screenshot. Explicitly asking the bug re- porter to add such clarifications for a bug report may increase its quality. In addition, future studies should investigate what a good bug report for mods looks like. For ex- ample, we observed that mod developers ask users to post the loading order of their mods when submitting a bug report to investigate possible conflicts between mods.8

8https://www.nexusmods.com/Core/Libs/Common/Widgets/ModBugReplyList?issue_ id=18914 CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 56

To identify the percentage of bug reports that contain a loading order, we searched all

bug reports for several variations of the “load order” keywords and found that only 4%

of the bug reports mentions these keywords. After manually studying a representative

sample of 89 of those bug reports (95% confidence level and 10% confidence interval),

we found that only 25 out of those 89 (28%) bug reports actually contain a loading or-

der. Hence, the total number of bug reports that contain an actual load order is very

small (approximately 1%). We envision that mod distribution platforms explicitly ask

bug reporters to include their mod loading order.

In addition, we came across many complaints about bugs of mods in other sections

than the bug reports. For example, in the ‘posts’ section of the 90000 weight limit mod, there are many mentions of bugs, while the mod only has two bug reports. Hence, the current bug reporting tool is either unknown by or unclear to mod users.

4.6.3 Implications for Mod Developers

Mod developers should make it easier for gamers to collect in-game stack traces or code samples. In our study, we observed that bug reports tend to be of low quality. A possible explanation is that gamers find it difficult to report stack traces or code sam- ples, since it is commonly not trivial to retrieve them, e.g., in games where it is difficult to simply cut and paste a stack trace. Hence, we suggest that mod developers provide gamers with an automated and simple mechanism for collecting in-game stack traces or error symptoms. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 57

4.7 Research Challenges

In this section, we highlight the most important research challenges that we encoun-

tered during our study. These challenges all require dedicated studies to be addressed.

4.7.1 Establishing a Ground Truth for the Impact of a Modding Com-

munity

A major challenge in research on game mods is that it is hard to establish a ground truth for the impact of a modding community on the original game. For example, it is difficult to estimate how the popularity of a mod-popular game would have evolved without its modding community. In our work, we made a first step towards estimat- ing the impact of official modding tools on game mods by studying games within the same game franchise. While definitely not perfect, studying games within the same game franchise mitigates some of the confounding factors that exist when studying completely different games. Further studies are necessary to determine how a ground truth for the impact of a modding community can be determined. For example, future studies should conduct a large scale and longitudinal investigation of players of games with and without mods.

4.7.2 Studying the Impact of Game Mods on the Life Expectancy of

the Original Game

Another potential benefit of game mods for game developers is that game mods in- crease the life expectancy of the original games. In our data, we have observed sev- eral “older” games (such as Skyrim) which are still very popular, possibly because of CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 58

the modding community. Future studies should investigate how the impact of mods

on the life expectancy of a game can be measured reliably. First, one should decide

how to measure life expectancy. For example, do we consider the number of total

playing hours on Steam as a proxy for a game’s life expectancy? The challenge of this

metric is that it would be hard to associate with the existence of a modding commu-

nity, as there is no way to verify whether a Steam gamer has modded the game, and

whether the mods actually impact a game’s life expectancy. In addition, there could be

many confounding factors that impact the life expectancy of a game. For example, the

life expectancy could be impacted by the type of the game, news coverage about the

game, the addition of official new levels/items, an inherently more loyal playerbase, etc. These confounding factors cannot be ignored yet they are currently impossible

to quantify in a reliable manner. A possible direction for future work is to collaborate

with developers of games with an active modding community and monitor how many

of the currently active users run their games with mods. In addition, such collabora-

tion could also yield further insights on which mods are being used.

4.7.3 Studying the Impact of Game Mods on the Ratings of the Orig-

inal Game

A research challenge that we uncovered in our study is the complexity of analyzing the relations between a modding community and ratings of its original game. Understand- ing how game mods (or the existence of a modding community) affect the rating of the original game would help understand the benefits of game mods for game developers.

Unfortunately, there are many confounding factors that impact the ratings of a game.

For example, the ratings can be impacted by good/bad updates (Hassan et al., 2018) CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 59

or a change to the price of the game. Hence, there currently is no reliable mechanism

to correlate the presence of a modding community with the original game’s rating. In

addition, the benefits of a modding community for game developers may be reflected

in other metrics than the rating. For example, a game could have more daily players

because of its mods. However, the impact of a modding community on such metrics

is again difficult to reliably quantify, as there are many confounding factors. Future

studies should investigate reliable methods of quantifying the impact of the presence

of a modding community on the ratings of a game to provide deeper insights on the

benefits of game mods for game developers.

4.7.4 Studying the Impact of Game Mods on the Time Required to De-

velop Features

Another potential benefit of game mods is that they can be integrated into the original game, thereby potentially saving the implementation time. Unfortunately, this benefit is difficult to study at a large scale, as we would need access to the development team or source code history of the original game (which are extremely hard to get for the studied games). However, even with such access, the time to develop a new feature would be hard to quantify. The vast majority of new features in a game is not based on mods, and likewise the vast majority of game mods will never end up in the original game. Future studies that have access to development team or source code history data should use qualitative methods such as interviews to observe if and how mods impact the development time of a feature. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 60

4.8 Threats to Validity

This section outlines the threats to the validity of our findings.

4.8.1 Internal Validity

A threat to the internal validity of our study is the way in which our studied dimensions were defined. For example, we do not know which portion of the mods for a game with official modding support were actually built using the officially-supported tool. How- ever, we use this dimension to indicate the mindset of a game developer rather than draw conclusions about the quality of the modding tool or the mods that are created with it. Future studies should further investigate whether the quality of mods that are created using an officially-supported tool is higher.

Another threat to the internal validity of our study is that we chose a threshold to distinguish media and non-media mods. This threshold was selected based on the distribution of the portion of media files in a mod, and the threshold takes into account the number of files in a mod. To mitigate this threat, we repeated our analysis after lowering the thresholds to 90% and 85%, and we found that our findings still hold.

Another threat to the internal validity of our study is that game developers are not required to publish official game release notes (Lin et al., 2017). Hence, we may have missed game releases which in turn affects our calculation of the release time of a mod.

In addition, as the data on the Nexus Mods platform is barely moderated, it is quite noisy and we have no guarantee that the publication date of a mod is its release date.

Finally, not all releases of a mod may end up on Nexus Mods. For example, game de- velopers may release a closed beta-version of their game. Hence the numbers that we report in our study are all low bound estimates. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 61

Another threat is that we use the release schedule of mods as a proxy for the will- ingness of mod developers to release new versions of a mod. It is possible that a mod requires only one release. For example, simpler mods, such as texture mods, could be perfect after their initial release (even though we observed in Section 4.4 that me- dia mods have a median of two releases as well). To verify the impact of this threat, we attempted to identify whether single-release mods were indeed perfect. First, we observed that 74% of the bug reports that were posted after the single release have the ‘New issue’ status. This 74% is higher than the overall percentage of mods with the ‘New issue’ status (67%). Hence, this higher percentage is an indication that of- ten mod developers of single-release mods abandoned the mod while it still required maintenance, rather than that the mod was perfect. In addition, we manually studied a sample of 50 bug reports that were posted after the release of a single-release mod, and we observed that 13 of those bug reports were confirmed by others. Hence, the reported bugs affects multiple modders, making it more likely that the mod requires a bug fix (and is not perfect).

Finally, it is important to keep in mind that the actions of the original game devel- opers may bias our findings. For example, the original game developers could decide to integrate a mod in the original game, or they could ensure the proper functioning of the mod across releases of the original game. As a result, the mod may appear unsup- ported on Nexus Mods. Future studies should further investigate how game developers react to mods. CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 62

4.8.2 External Validity

A threat to the external validity of our study is that we only studied mods from the 20

most mod-popular games on the Nexus Mods distribution platform. As a result, we did

not study mods for games that may be mod-popular, but do not have a strong presence

on the Nexus Mods platform. An example of such a game is the Minecraft game, which is popular amongst modders but of which the mods are distributed through other plat- forms, such as the Planet Minecraft platform.9 Future studies should investigate how our findings apply to games and mods that are available through other mod distribu- tion platforms.

Another threat concerning the generalizability of our study is that we only studied mods with at least 1,000 endorsements. We conducted a sensitivity analysis to see if our findings are biased by the 1,000 endorsement threshold. We repeated our anal- ysis using 750 and 1,250 as endorsement thresholds and observed that our findings still hold. Future studies should investigate whether our findings apply to less popular mods as well.

4.9 Chapter Summary

Game mods are community-driven contributions to a game that are made by players of the game who wish to improve existing or add new functionality to the game. For example, a mod can improve the textures of a game, add new weapons to the game, or even provide a complete overhaul of the game. In this chapter, we studied 9,521 pop- ular mods from the 20 most-modded games on the Nexus Mods distribution platform.

9https://www.planetminecraft.com/resources/mods/ CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 63

In particular, we studied the characteristics, release schedules and bug reports of the

mods, to get a better understanding of the willingness of the modding community of

a game to contribute and maintain mods. We studied mods along three dimensions:

(1) mods for games with and without official modding support, (2) media and non-

media mods, and (3) mods of Bethesda and non-Bethesda games. Below, we list our

most important findings:

1. Mods for games with official modding support have a higher median endorse-

ment ratio than mods for games with no official modding support. In addition,

mods for games with official modding support are released faster.

2. Supporting the same modding tool across games within a game franchise, or

even different franchises, is associated with faster mod releases.

3. A mod has a median of two releases, which are often released within four days

of each other. In addition, mod developers respond to and solve bug reports

quickly.

4. Unfortunately, bug reports for mods are of low quality, which make them hard to

address. In addition, many bugs are reported outside the bug reporting tool.

In this chapter, we found several indications that the modding communities of games are willing to contribute to mods. Many of these mods are being used by hun- dreds of thousands of gamers. Hence, while we are definitely not suggesting that game developers should rely on mod developers for the core functionality of their games, mods are a good and cost-effective way of keeping the players of a game engaged. As a result, it can be beneficial for game developers to provide official support for modding CHAPTER 4. CONTINUOUS GROWTH OF MODS BY MOD DEVELOPERS 64

their games. We suggest that game developers and developers of third-party mod dis- tribution platforms step in to help mod developers improve the post-release support of their mods. Game developers could offer incentives to mod developers who pro- vide excellent post-release support for their mods. For example, popular mods that are well-maintained could be included in an official release of the game. In addition, developers of mod distribution platforms could improve the infrastructure for report- ing and addressing bugs, thereby striving to improve the quality of the bug reports. CHAPTER 5

Continuous Adoption of Mods by Gamers

T is becoming increasingly difficult for game developers to manage the cost of

developing a game, while meeting the high expectations of gamers. One way to I balance the increasing gamer expectation and development stress is to build an active modding community around the game. There exist several examples of games with an extremely active and successful modding community, with the Minecraft game being one of the most notable ones. Popular mods can help increase the adoption of mods by gamers, which contributes to building an active modding community.

This chapter reports on an empirical study of 1,114 popular and 1,114 unpopu- lar Minecraft mods from the CurseForge mod distribution platform, one of the largest

Minecraft mod distribution platforms. We analyzed the impact of 33 features across

5 dimensions of mod characteristics on the popularity of mods (i.e., mod category,

65 CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 66

mod documentation, environmental context of mod, remuneration for the mod, and community contribution for the mod), to understand the characteristics of popular

Minecraft mods. The goal of our study is to provide insights for mod developers to improve the popularity of their mods. We firstly verified that the studied dimensions have strong explanatory power for the popularity of the studied mods. Then we eval- uated the contribution of each of the 33 features in the 5 dimensions. We observed that popular mods tend to have a high quality description and promote community contribution. In addition, mods that simplify mod development are more likely to be popular. Based on our findings, we encourage mod developers to: improve the support for mod development, improve the quality of their mod descriptions, and encourage community contribution to their mods by either open sourcing the mods or providing a channel for the community to report issues about the mods.

5.1 Introduction

The team size, cost and complexity in game development can grow exponentially as the user requirements increase (Shumovsky, 2018). Thus, it has become challenging to de- velop a successful game, and game developers are constantly under immense amount of stress (Phillips, 2018).

One approach to balance the increasing gamer expectation and development stress is to build an active modding community around the game. Skyrim and Minecraft are examples of games that have been successful in building active modding communi- ties (Hackman and Björkqvist, 2014; Zorn et al., 2013) to increase the longevity of the games. For example, the Skyrim game still has a median of 86 new mods released per CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 67

day 8 years after its initial game release in 2011, along with a number of 514,063,250 to- tal unique downloads of mods (Scott, 2007). Prior work also shows that an active mod- ding community can contribute to the increased sales of the original game (Poretski and Arazy, 2017).

There are two key components of an active modding community of a game: the ac- tive development of mods, and the active adoption of mods by gamers. In Chapter4, we looked at how game developers can help maintain the active development of mods, and observed that games with a consistent modding support within the same or dif- ferent game franchises, were associated with faster releases of mods (Lee et al., 2018).

In this chapter, we investigate the characteristics of popular mods to provide insights to mod developers on how to increase the adoption of their mods by gamers, i.e., the popularity of their mods. To do so, we study 33 characteristics along 5 dimensions of

1,114 popular and unpopular mods for the Minecraft game from the CurseForge mod distribution platform – one of the largest Minecraft mod distribution platforms. We focus on the mods from the Minecraft game because it has one of the largest and most active modding communities (Mills, Aaron, 2015). In particular, we answer the follow- ing 2 research questions (RQs):

RQ1: Do our studied dimensions have enough power to distinguish popular mods

from unpopular ones?

Motivation: The goal of this research question is to investigate how well each

studied dimension of characteristics (i.e., features) of mods can individually dis-

tinguish the popular mods from unpopular ones. We also investigate how well

can all the studied dimensions together distinguish popular mods from unpop-

ular ones. Prior work (Tian et al., 2015) used similar dimensions to identify the CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 68

characteristics that distinguish mobile apps with high ratings from ones with low

ratings. The results of this research question lay the foundation of further inves-

tigations of the characteristics of popular mods.

Findings: We observed that each studied dimension of characteristics of a mod

has explanatory power in distinguishing popular mods from unpopular ones.

Among the studied dimensions, the community contribution for the mod di-

mension has the largest explanatory power. However, our combined model which

uses all the features across the five dimensions outperforms the best model using

individual dimension by 10% (median).

RQ2: What features best characterize a popular mod?

Motivation: The goal of this research question is to investigate which features

of mods can best characterize popular mods. The results of RQ1 show that the

studied features have a strong explanatory power on the popularity of a mod. In

this RQ, we further investigate the characteristics of popular mods at a granular

level.

Findings: We observed that 18 of the 33 (54.5%) studied features have an impact

in distinguishing popular mods from unpopular ones. Mods that simplify mod

development are more likely to be popular. In addition, popular mods tend to

promote community contribution with source code URL and issue tracking URL,

and have a richer mod description. Finally, mods that accept donation tend to

be more active and more popular. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 69

Collecting Selecting Filtering Mods Data Features

Filter out mods Select mods 33 features 12,710 Dimensions of without mod created between among 5 popular mods mods releases 2014 and 2016 dimensions (Section 5)

2,228 studied mods 111,574 CurseForge mod releases Select top and Filter out bottom 20% releases and Normalize Features of mods (1,114) dependencies numeric popular mods based on total that are not the features (Section 6) 76,453 downloads latest dependencies

Figure 5.1: An overview of our data collection.

The remainder of this chapter is outlined as follows. Section 5.2 discusses our method- ology. Section 5.3.1 and 5.3.2 discuss the results of our empirical study. Section 5.4 dis- cusses the implications of our study. Section 5.5 outlines threats to the validity of our

findings. Section 5.6 concludes the chapter.

5.2 Data Collection

This section discusses the methodology of our empirical study on the characteristics of popular and unpopular Minecraft mods. Figure 5.1 gives an overview of our method- ology.

5.2.1 Collecting Data

We collected the dataset for our study from the CurseForge mod distribution platform on June 6, 2019, using a customized crawler. Table 5.1 shows an overview our Minecraft mod dataset. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 70

Table 5.1: An overview of the CurseForge mod distribution platform dataset.

Number of total mods 12,710 Number of studied mods 2,228 Number of studied dimensions 5 Number of studied features 33 Number of total mod releases 111,574 Number of total mod dependencies 76,453

Collecting Mods. We collected the information of 12,710 mods. In particular, we col- lected the name, categories, number of total comments, source code URL, issue track- ing URL, Paypal URL, and Patreon URL for each mod.

Collecting Mod Releases. We collected the information of 111,574 releases across all mods. In particular, we collected the type, upload date, size, number of downloads, and supported Minecraft, Java, and Bukkit versions for each mod release.

Collecting Dependencies. We collected 76,453 mod dependencies across all mod re- leases. In particular, we collected the type, mods, and the direction for each depen- dency.

5.2.2 Filtering Mods

To ensure the quality of the studied mods, we first removed 295 inactive mods that have no mod releases. Then, we removed 6,845 mods that were created before 2014 or after 2016 to ensure the studied mods all have an equal chance to obtain a high number of downloads. For the remaining 5,570 mods, we selected the top and bottom 20% of mods based on their number of total downloads for our study. We consider the top 20% of mods (1,114 mods) as popular mods, and the bottom 20% of mods (1,114 mods) as CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 71

Table 5.2: Dimensions and their features describing the characteristics of popular and unpopular Minecraft mods.

Feature Name Explanation Type Rationale Number of categories Number of total categories that a mod be- Numeric Mods that offer a variety of categories can (num_categories) longs to. A mod must belong to at least one attract users with more options. category. Miscellaneous Mods that do not belong to any of the ex- Boolean Certain categories may attract more users (is_cat_misc) isting categories. For example, the Open- to the mod. Blocks mod.1 Food Mods that provide changes to anything re- Boolean (is_cat_food) lated to food in-game. For example, the Ap- pleSkin mod.2 World generation Mods that provide changes related to the Boolean (is_cat_world_gen) world, such as new terrains. Magic Mods that provide changes related to Boolean (is_cat_magic) magic in the Minecraft game. For example, the Roots mod.3 API and library Mods that provide shared code for other Boolean (is_cat_library_api) mod developers to use. Fabric Mods that are created using the Fabric4 Boolean (is_cat_fabric) modding toolchain. Technology Mods that provide changes for any in-game Boolean Mod Category (is_cat_technology) technology. Armor, tools, and Mods that provide changes to in-game ar- Boolean weapons mor, weapons, and tools. (is_cat_armor_weapons_tools) Addons Mods that provide utilities for mod users to Boolean (is_cat_addons) easily extend in-game features. Adventure and RPG Mods that change the gameplay experience Boolean (is_cat_adventure_rpg) of the in-game adventure. Server utility Mods that provide changes to the server- Boolean (is_cat_server_utility) side of the Minecraft game. Redstone Mods that are provide changes related to Boolean (is_cat_redstone) the redstone resource in the Minecraft game. Map and information Mods that provide changes related to the Boolean (is_cat_map_info) location and information on items. Storage Mods that provide mod users blocks and Boolean (is_cat_storage) items, which improve the existing in-game storage. Twitch integration Mods that provide changes related to the Boolean (is_cat_twitch_integration) interaction between the mod and the Twitch platform. Cosmetic Mods that provide changes to the texture Boolean (is_cat_cosmetic) and aesthetic of the in-game models. Length of short description Number of words in the mod’s preview de- Numeric The longer the description, the better that (num_words_short_desc) scription. mod users will understand what the mod offers without downloading the mod. Length of long description Number of words in the modâA˘Zs´ main de- Numeric (num_words_long_desc) scription. Mod wiki URL An external link with the documentation of Boolean The presence and quality of a mod’s doc- (is_mod_wiki_url) a mod. umentation can help other mod users un- derstand how to utilize the mod to its full potential, which can give users a better ex- Mod Documentation perience. Number of images Number of in-game screenshots that a mod Numeric In-game screenshots can help promote (num_images) has. and visually explain the mod’s functional- ities, which may attract users, without try- ing the mod first. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 72

Latest number of incompatible de- Number of incompatible dependencies Numeric Dependencies provide mods more func- pendencies that are in the latest release of the mod, tionalities, which could make the mod ap- (num_incompatible_dep) which means another mod is not compati- peal to more users. ble with the mod. Latest number of tool dependencies Number of tool dependencies that are in Numeric (num_tool_dep) the latest release of the mod, which is a util- ity for the mod. Latest number of required depen- Number of required dependencies that are Numeric dencies in the latest release of the mod, which (num_required_dep) means another mod is required to make the mod function. Latest number of embedded library Number of embedded library dependen- Numeric dependencies cies that are in the latest release of the mod, (num_embedded_lib_dep) which provides shared code for the mod’s development. Latest number of optional depen- Number of optional dependencies that are Numeric dencies in the latest release of the mod, which

Environmental Context of Mod (num_optional_dep) means the dependency adding a certain functionality can be switched on and off. Latest number of supported Number of Minecraft versions supported Numeric A greater number of supported versions Minecraft version(s) by the latest mod release, which cor- could attract more users to a mod by pro- (latest_num_mc_versions) responds to a specific version of the viding more stability, and access to func- Minecraft game. Mods must support at tionalities from different versions. least one Minecraft version. Latest number of supported Java ver- Number of Java versions supported by the Numeric sion(s) latest mod release. Mod developers can op- (latest_num_java_versions) tionally provide this information. Latest number of supported Bukkit Number of Bukkit API5 versions supported Numeric version(s) by the latest mod release. The Bukkit (latest_num_bukkit_versions) API extends the multiplayer server of the Minecraft game for others to modify. Mod developers can optionally provide this in- formation Paypal URL An external link to Paypal for donations. Boolean Mod developers that ask for donations are (is_paypal_url) more likely to be dedicated to modding, which can attract more users.

Patreon URL An external link to Patreon for donations. Boolean (is_patreon_urls) Remuneration for the Mod

Source code URL An external link to the source code of a mod Boolean Mods that provide a link to their source (is_mod_source_code) (e.g., Github). code could invite more contributors, which could attract users with more content at a faster speed.

Issue tracking URL An external link to an external issue track- Boolean Mods that provide a link to an issue track- (is_mod_issues) ing system. ing system could indicate to a user that a mod is more stable, which may attract them to the mod. Community Contribution for the Mod

https://www.curseforge.com/minecraft/mc-mods/openblocks https://www.curseforge.com/minecraft/mc-mods/appleskin https://www.curseforge.com/minecraft/mc-mods/roots http://fabricmc.net/ https://dev.bukkit.org/ CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 73

unpopular mods. In result, we studied a total of 2,228 mods. Our selection approach

is similar to prior work (Tian et al., 2015) which selected the highest rated and lowest

rated mobile apps for study.

For each of the 2,228 mods, we used the information of the mod’s latest release and

dependencies in our study.

5.2.3 Selecting Features

To understand the differences between the characteristics of popular and unpopular

Minecraft mods, we extracted 33 features (characteristics) from the collected data for

each studied mod. We then empirically grouped the 33 features into 5 dimensions:

mod category, mod documentation, environmental context of mod, remuneration for

the mod, and community contribution for the mod. Table 5.2 shows an overview of the

33 features and their associated dimensions, along with their corresponding explana-

tion and rationale. In addition, we normalized all features with the type ‘numeric’ in

Table 5.2 using a log(1 + x) transformation to prevent the bias caused by the outliers.

5.3 Characteristics of Popular and Unpopular Minecraft

Mods

In this section, we present the results of our empirical study of the characteristics of popular and unpopular Minecraft mods. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 74

5.3.1 RQ1: Do our studied dimensions have enough power to distin-

guish popular mods from unpopular ones?

Motivation: In this research question, we investigate how well each studied dimension of characteristics (i.e., features) of mods can individually distinguish the popular mods from unpopular ones. We also investigate how well can all the studied dimensions together distinguish popular mods from unpopular ones. Prior work (Tian et al., 2015) used similar dimensions to identify the characteristics that distinguish mobile apps with high ratings from ones with low ratings. The results of this research question lay the foundation of further investigations of the characteristics of popular mods.

Approach: To investigate how well the individual dimensions can distinguish popular mods from unpopular ones, we built a logistic regression model for each dimension in Table 5.2. In particular, for each dimension’s model, we used the features in a di- mension as independent variables and whether the mod is popular as the dependent variable. We used the glm function1 from the stats package2 to create the logistic re-

gression models.

To validate the performance of our built models, we performed 100 out-of-sample

bootstrap iterations to compute the AUC (Area Under the receiver operator charac-

teristics Curve) for each model. Prior work (Tantithamthavorn et al., 2016) showed

that the out-of-sample bootstrap technique had the best balance between the bias and

variance of estimates. The out-of-sample bootstrap technique randomly samples data

with replacement for n iterations. The sampled data in an iteration is used as the train-

ing set for that iteration, while the data that was not sampled in that iteration is used

1https://www.rdocumentation.org/packages/stats/versions/3.6.1/topics/glm 2https://www.rdocumentation.org/packages/stats/versions/3.6.1 CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 75

as the testing set for that iteration. We then trained a model with the training set and calculated the AUC of the model with the testing set for each iteration.

To investigate how well can all studied dimensions together distinguish popular mods from unpopular mods, we built a logistic regression model using the 33 features from the 5 dimensions in Table 5.2. We then evaluated the performance of the model by calculating the AUC of the model with 100 out-of-sample bootstrap iterations.

In addition, we used the Scott-Knott effect size difference test to statistically sort and rank the distributions of AUC of all studied dimensions (Tantithamthavorn et al.,

Combined Community contribution for the mod Environmental context of mod Mod documentation

Mod category Remuneration for the mod

0.65 0.70 0.75 0.80 0.85 0.90

AUC

Figure 5.2: The distribution of AUCs of models constructed with individual studied dimension, and with all studied dimensions combined. Sorted by the median AUC of models. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 76

2016). We used the sk_esd function3 from the ScottKnottESD package4 for the Scott-

Knott effect size difference test.

Findings: Each individual dimension has explanatory power in distinguishing pop- ular mods from unpopular ones. Figure 5.2 shows the distribution of AUCs per stud- ied dimension. The lowest median AUC among the studied dimensions was 0.66, im- plying that every dimension has some explanatory power in distinguishing popular mods from unpopular ones. In addition, the Scott-Knott effect size difference test shows a statistical significant difference between each studied dimensions, with non- negligible effect sizes. Among the studied dimensions, the community contribution for the mod dimension is ranked as having the largest explanatory power, whereas the remuneration for the mod dimension is ranked as having the lowest explanatory power.

The combined model has a larger explanatory power than each of the studied dimension individually. Figure 5.2 shows the distribution of AUCs of the combined model that combines all studied dimensions together. The combined model has the largest median AUC of 0.91, outperforming every one of the studied dimensions on their own. The Scott-Knott effect size difference test confirms that the combined model has the highest ranking in explanatory power compared to the individual studied di- mensions.

In addition, Figure 5.2 shows that the combined model has a 10% higher median

AUC than the community contribution for the mod dimension (the dimension with

3https://www.rdocumentation.org/packages/ScottKnottESD/versions/1.2.2/ topics/%22sk_esd%22 4https://www.rdocumentation.org/packages/ScottKnottESD/versions/1.2.2 CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 77

the highest explanatory power among the studied dimensions), and a 38% higher me-

dian AUC than the remuneration for the mod dimension (the dimension with the low-

est explanatory power among the studied dimensions). Prior work (Tian et al., 2015)

also observed that a combined model with all the dimensions has a larger explanatory

power than models with individual dimensions in the context of distinguishing mobile

apps with high ratings from mobile apps with low ratings.

: Summary of RQ1 Each studied dimension of characteristics of a mod has explanatory power in

distinguishing popular mods from unpopular ones. Among the studied dimen-

sions, the community contribution for the mod dimension has the largest ex-

planatory power. However, our combined model which uses all the features

across the five dimensions outperforms the best model using individual dimen-

sion by 10% (median).

5.3.2 RQ2: Which features best characterize a popular mod?

Motivation: In this research question, we investigate which features of mods can best

characterize popular mods. The results of RQ1 show that the studied features have a

strong explanatory power on the popularity of a mod. In this RQ, we further investigate

the characteristics of popular mods at a granular level.

Approach: To investigate which features can best characterize popular mods, in this

research question we focus on analyzing the combined model with all dimensions of

features, as RQ1 shows that the combined model has the most explanatory power for

mod popularity. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 78

5 studied dimensions 1. Correlation analysis 5a. Backward variable selection Remove Remove 33 total features with n features features based n features features spearman > on AIC 0.7

2. Redundancy analysis 5b. Build final model 5d. Nomogram analysis

Remove Build model Final features that n features (with 100 combined Build nomogram are redundant bootstrap model nomogram replications)

3. Build combined model 5c. Performance evaluation 5e. Partial effects analysis compute bootstrap Compute new Build estimate for metrics with combined AUC partial effects partial effect set of 100 bootstrap model diagram regression replications coefficients

4a. Explanatory power of features

Compute for using each Anova feature

For each dimension

4b. Explanatory power of dimensions compute Combined bootstrap Compute log- for model without estimate for likelihood ratio dimensions dimension of set of using lrtest of interest interest regression coefficients

Figure 5.3: An overview of the process that we used to build, evaluate and analyze the combined model.

Figure 5.3 shows an overview of our approach to construct, evaluate and analyze the combined model. Below we explain each step in details:

1. Correlation analysis. We performed correlation analysis to reduce collinearity be- tween the features before we built the models, since correlated features can break the interpretation of the model (Midi et al., 2010; McIntosh et al., 2016). We used the varclus function5 from the Hmisc package6 in R to filter out highly correlated fea- tures. We calculated Spearman’s correlation coefficients among the studied features.

We consider a pair of features with a Spearman correlation coefficient >= 0.7 as highly correlated. We did not observe high correlations among our studied features.

5https://www.rdocumentation.org/packages/Hmisc/versions/4.2-0/topics/ varclus 6https://cran.r-project.org/web/packages/Hmisc/index.html CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 79

Table 5.3: An overview of the statistics of each dimension and its features. The larger the ∆LRχ 2, the larger the impact of a studied dimension. Similarly, the larger the Wald χ 2, the larger the explanatory power of a feature in the combined model (The per- centages and p-value are rounded to two decimal places). The feature is statistically 2 significant if the p-value <= 0.05. Sorted by the Wald χ per studied dimension.

Wald χ 2 (%) P-value Mod Category (∆LRχ 2: 23.51%) 0.00 Fabric 20.54 < 0.01 Armor, tools, and weapons 7.92 < 0.01 Addons 2.22 < 0.01 Food 2.17 < 0.01 World generation 1.64 < 0.01 API and library 1.47 < 0.01 Miscellaneous 1.39 < 0.01 Server utility 1.17 < 0.01 Storage 0.79 0.02 Redstone 0.50 0.06 Adventure and RPG 0.43 0.08 Cosmetic 0.27 0.17 Technology 0.10 0.39 Map and information 0.02 0.73 Magic 0.01 0.74 Twitch integration 0.00 0.97 Mod Documentation (∆LRχ 2: 13.03%) 0.00 Number of words in the long description 5.18 < 0.01 Number of images 2.06 < 0.01 Mod wiki URL 1.72 < 0.01 Number of words in the short description 0.46 0.07 Environmental Context of the Mod (∆LRχ 2: 18.63%) 0.00 Latest number of Bukkit versions 15.45 < 0.01 Latest number of required dependencies 9.27 < 0.01 Latest number of optional dependencies 0.81 0.02 Latest number of Minecraft versions 0.49 0.07 Latest number of Java versions 0.06 0.53 Latest number of tool dependencies 0.00 0.85 Latest number of incompatible dependencies 0.00 0.86 Latest number of embedded library dependencies 0.00 0.98 Remuneration for the Mod (∆LRχ 2: 10.42%) < 0.01 Paypal URL 6.21 < 0.01 Patreon URL 1.56 < 0.01 Community Contribution for the Mod (∆LRχ 2: 34.41%) 0.00 Issues URL 11.21 < 0.01 Source code URL 4.86 < 0.01 Total 100.00 CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 80

2. Redundancy analysis. Before building the models, we also performed redundancy analysis to eliminate redundant features that can interfere with the relationship be- tween the independent variables (i.e., features), which in turn may distort the rela- tionship the independent variables have with the dependent variable (i.e., popular- ity) (McIntosh et al., 2016). We used the redun function7 from the Hmisc package in R to filter out features that can be linearly predicted by other features. We removed the

‘number of categories’ feature as it is redundant, leaving 32 features for the remainder of the study.

3. Building the combined model. We used all the remaining features after step 2 to build a logistic regression model. We then used the bootcov function from the rms package to adjust the regression coefficients with bootstrap estimates, to ensure the non-arbitrariness of the estimated regressions co-efficients in the combined model (Har- rell Jr et al., 1984; Harrell Jr and Slaughter, 2001).

4a. Explanatory power of features. We used Wald χ 2 to measure the explanatory power of the features in the model from step 3. The larger the Wald χ 2, the larger the explanatory power of the feature (Harrell Jr et al., 1984). Prior work (Thongtanunam and Hassan, 2018) used the same approach to compute the explanatory power of fea- tures. We computed the Wald χ 2 with the Anova function8 from the car package9 in

R using the parameter test.statistic=‘Wald’. Table 5.3 shows the explanatory power of each features (Wald χ 2).

4b. Explanatory power of dimensions. Though in RQ1, we observed that each di- mension of characteristics of a mod has explanatory power, we are uncertain of the 7https://www.rdocumentation.org/packages/Hmisc/versions/4.2-0/topics/redun 8https://www.rdocumentation.org/packages/car/versions/3.0-3/topics/Anova 9https://www.rdocumentation.org/packages/car/versions/3.0-3 CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 81

unique explanatory power each of them contain in relation to the other dimensions.

Understanding the unique explanatory power of each dimension is critical to assert which of these dimensions matter the most for characterizing the popularity of a mod.

For example, from Figure 5.2 we observe that the environmental context of mod and mod documentation dimensions by itself can predict the popularity of a mod with a median AUC of 0.74. However, we are uncertain of how much unique power each of these dimensions contribute to the model built on all the studied dimensions, which had a median AUC of 0.92.

Therefore, we conducted a chunk test on each of the studied dimensions in the com- bined model from step 3, to quantify the explanatory power of each studied dimen- sion (Harrell Jr, 2001; McIntosh et al., 2016). For each of the studied dimension (given in Table 5.2), the chunk test estimates the difference in goodness of fit (by comput- ing the difference in log-likelihood) between the full model (i.e., the combined model from step 3) and the combined model that was built without one studied dimension

(whose explanatory power we are computing). The chunk test reports a Chi-square value (∆LRχ 2) (which is the difference in log-likelihood compared to the Chi-squared distribution) and a p-value. The Chi-squared value quantifies the unique explanatory power that was lost due to the removal of the given dimension (in relation to the other dimensions) and the p-value signifies the dimension’s impact. We used the lrtest function10 from the lmtest package11 in R to conduct the chunk test. Table 5.3 shows the explanatory power of each dimensions (∆LRχ 2).

10https://www.rdocumentation.org/packages/lmtest/versions/0.9-37/topics/ lrtest 11https://www.rdocumentation.org/packages/lmtest/versions/0.9-37 CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 82

5a. Backward variable selection. We do backward variable selection to ensure par- simony of the constructed model, as suggested by Harrell Jr et al.(1984). They sug- gested to use backward variable selection when the goal of the model is to interpret.

We used the fastbw function12 from the rms package in R to perform a backward elim- ination of features.We removed 14 of the 32 features (44%) using the fastbw function.

In result, we obtained a new combined model with 18 features.

5b. Build final model. With the reduced feature set from step 5a, we reconstructed the final combined model. Similar to step 3, we adjusted the regression coefficients with the bootstrap estimate, as outlined by Harrell Jr et al.(1984).

5c. Performance evaluation. To demonstrate the quality of the constructed model from 5b, we calculated the AUC of the model using 100 out-of sample bootstrap iter- ations to evaluate the performance of the model.

5d. Nomogram analysis. We used the final combined model from step 5b to cre- ate and analyze a nomogram using the nomogram function13 from the rms package in

R, which provides a way to measure the explanatory power of each feature in distin- guishing popular mods from unpopular ones. Although, the Wald χ 2 can provide in- sight to the explanatory power of each feature in the combined model, the nomogram provides us with an exact interpretation on how the variation in each feature affects the outcome probability. Figure 5.4 shows the results of the nomogram analysis.

5e. Partial effects analysis. We used the final combined model from step 5b and the nomogram analysis from step 5d to create partial effects plots, which shows how

12https://www.rdocumentation.org/packages/rms/versions/5.1-3.1/topics/fastbw 13https://www.rdocumentation.org/packages/rms/versions/5.1-3.1/topics/ nomogram CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 83

0 10 20 30 40 50 60 70 80 90 100 Points 0 is_cat_misc 0 1 is_cat_food 0 is_cat_world_gen 1 1 is_cat_library_api 1 1 0 is_cat_fabric 0 is_cat_armor_weapons_tools 0 1 1 is_cat_addons 0 is_cat_server_utility 0 1 num_words_short_desc num_words_long_desc 0.5 2 3.5 1 0 1 2 3 4 5 6 7 8 9 is_mod_wiki_url num_images 0 0 0 1 2 3 4 5 latest_num_bukkit_versions 0.69 1.39 1.79 num_required_dep 0.69 1 is_paypal_url 0 1.1 1.61 1 is_patreon_urls 0 1 0 is_mod_issues 1 0 is_mod_source_code 0 Total Points Popularity 0 50 100 150 200 250 300 350 400 0.001 0.1 0.6 0.999

Figure 5.4: The nomogram visualizes the role of each feature in distinguishing a mod’s popularity. The line against each feature in the figure, varies between the range of val- ues for that given feature. The “points” line at the top of the figure, is used to calculate the magnitude of impact that each feature has and “Total Points” at the bottom of the figure gives the total points generated by all the features for a given instance (i.e., for a given mod). For instance, if for a given mod, the feature “is_cat_fabric” has a value of 1, then it contributes 100 points. Finally, the line against ‘‘Popularity” shows how the probability of a mod being popular varies with the total point (which is computed by summing up all the individual points contributed by each feature). For instance, if all the features for a given mod, contribute a total of 260 points, then the probability of a mod being popular is 99% and similarly, if the total points given by all the features for a particular mod is less than 110, then there is no chance for that mod to be popular. Also, the model used to generate this nomogram achieved a median AUC of 0.92 on 100 out-of-sample bootstrap iterations. The values of the numeric features are logged. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 84

changes to the value of some of the individual feature impact the outcome probability when the other features are held constant at the median for numeric features and at the mode for the boolean features.

In addition, to measure if two distributions are significantly different, we used the

Wilcoxon tests. The Wilcoxon signed-rank test is a paired and non-parametric statis- tical test, whereas the Wilcoxon rank-sum test is an unpaired and non-parametric sta- tistical test, where the null hypothesis indicates that it is equally likely that a randomly selected value from one sample will be less than or greater than a randomly selected value from a second sample (Wilcoxon, 1945). If the p-value of the used Wilcoxon test on the two distributions is less than 0.05, we reject the null hypothesis, and conclude that the two distributions are significantly different. In addition, to calculate the mag- nitude of the difference we calculate the Cliff’s delta d effect size (Long et al., 2003), with the following thresholds (Romano et al., 2006):

 negligible(N), if d 0.147.   | | ≤  small(S), if 0.147 < d 0.33. Effect size = | | ≤ medium(M), if 0.33 < d 0.474.   | | ≤  large(L), if 0.474 < d 1. | | ≤ Findings: Mods that simplify mod development are more likely to be popular. Fig- ure 5.4 shows that mods in the “fabric”, “addons”, and “API and library” categories have a larger probability of being a popular mod. In addition, among all studied features, being in the “fabric” category has the largest positive impact on being a popular mod.

We further investigated the mods under each category and observed that all of the 16 CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 85

1.00 1.00

χ2 = < 1 43 P 0.001

0.75 0.75

is_mod_wiki_url is_paypal_url

0.50 0 0.50 0 1 1 Probability Probability

0.25 0.25

χ2 = < 1 18.5 P 0.001

0.00 0.00 2 4 6 0 1 2 3 num_words_long_desc num_images (a) Number of words in the long description w.r.t a (b) Number of images w.r.t a PayPal URL. mod wiki URL.

1.00 1.00

0.75 0.75

is_mod_wiki_url is_mod_issues

0.50 0 0.50 0 1 1 Probability Probability

0.25 0.25

χ2 = < 1 18.5 P 0.001

χ2 = < 1 43 P 0.001 0.00 0.00 0 1 2 3 2 4 6 num_images num_words_long_desc (c) Number of images w.r.t a mod wiki URL. (d) Number of words in the long description w.r.t an issue tracking URL.

Figure 5.5: The impact of features on the outcome probability when another feature is held constant (features are held constant at the median for numeric features and at the mode for boolean features). The grey area shows a confidence interval at 95%. The values of the numeric features are logged. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 86

collected “fabric” mods are popular mods, 73.3% of the studied “addons” mods are pop- ular mods, and 71.1% of the studied “API and library” category mods are popular mods.

The “Fabric” category is a type of mod created using the “Fabric” mod development toolchain, which provides a mod ecosystem that makes updating a mod simpler and provides modularity of the code (Fabric development team, 2018). The “API and li- brary” category is a type of mod that can be leveraged by others and mod developers to make mod development simpler. In addition, the “Addons” category is a type of mod that adds support and extends other mods, such as the TOP Addons mod.14

However, mods in the following mod categories: “miscellaneous”, “food”, “world generation”, “armor tools weapons”, and “server utility” have a larger probability of be- ing unpopular. In addition, the “armor tools weapons” category shows the largest im- pact on the probability of a mod being unpopular.

Mods that provide a source code URL and/or an issue tracking URL are more likely to be popular. Figure 5.4 shows that mods with an issue tracking URL or a source code URL have a larger probability of being a popular mod. We investigated the studied popular mods and observed that 77% of the popular mods have an issue tracking URL, and 71% of the popular mods have a source code URL. In addition, Figure 5.5d shows that the presence of an issue tracking URL increases the probability of a mod being popular. Furthermore, from Table 5.3, we also observe that the community contribu- tion dimension (which captures the presence/absence of source code URL and/or an issue tracking URL) has the highest explanatory power (34.41%) among all the other studied dimensions. Even though other individual features contribute towards char- acterizing the popularity of a mod, community contribution dimension as a whole is

14https://www.curseforge.com/minecraft/mc-mods/top-addons CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 87

more important. Hence, mod developers should provide the URLs for their mod’s is- sues and/or source code if they want to improve the popularity of their mods. Mods with a better description about the mod are more likely to be popular. Fig- ure 5.4 shows that the number of words in the long description has the largest explana- tory power among the features in the mod documentation dimension. The long de- scriptions of popular mods have a median of 157 words, whereas the long descriptions of unpopular mods have a median of 79 words. The Wilcoxon rank-sum test confirms that the number of words in the long description of popular mods and unpopular mods is statistically significantly different, with a medium Cliff’s delta effect size. Figure 5.5a confirms such observation. In Figure 5.5a, we held the mod wiki URL at a constant against the number of words in the long description because if a mod developer is will- ing to provide external documentation, they could be more willing to make an effort into providing a richer description for the mod. Prior work (Tian et al., 2015) showed that high-rated mobile apps had significantly longer app descriptions, which is con- sistent with our results.

In addition, Figure 5.4 shows that both having more images and having a wiki URL have a positive impact on a mod being popular. Therefore we posit that, mod devel- opers who make an effort to provide an external documentation could be more likely to further explain on how the mod works to users through the use of images and Fig- ure 5.5c confirms such observation. Prior work (Tian et al., 2015) showed that the num- ber of images was one of the top three influential factors in determining that a mobile app will be high-rated, which is consistent with our results.

The number of words in long description, the number of images, and having a wiki CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 88

URL are all features related to the mod documentation dimension, and all have a posi- tive impact on mod popularity. Hence, providing more elaborate descriptions is asso- ciated with increasing the popularity of a mod.

Mods that accept donations are more active (i.e., more releases and comments) and are likely to be popular. Figure 5.4 shows that having a Paypal URL or Patreon URL increases the probability of a mod being popular. Figure 5.5b confirms such observa- tion. We observed that mods with a PayPal URL have a median of 13 mod releases, whereas mods without a PayPal URL have a median of 2 mod releases; mods with a

Patreon URL had a median of 21 mod releases, whereas mods without a Patreon URL had a median of 3 mod releases. The Wilcoxon rank-sum test confirms that the differ- ences in the number of mod releases between mods with and without a PayPal URL or

Patreon URL are both statistically significant, with a medium Cliff’s delta effect size for a PayPal URL and a large Cliff’s delta effect size for a Patreon URL. Furthermore, mods with a Patreon URL have a median of 25 comments per mod, while mods without a

Patreon URL have a median of 1 comment per mod. The Wilcoxon rank-sum test con-

firms a statistically significant difference in the number of comments between mods with and without a Patreon URL, with a small Cliff’s delta effect size.

In total, we observed that 88 mod developers which advertise their Patreon URL on their mods’ pages. We manually investigated the motivation of them accepting dona- tions by looking at each of their Patreon profiles. 14% of these mod developers created a Patreon to support their living (e.g., pay bills), 32% of them created a Patreon for fun and did not expect profit, 32% of them created a Patreon to obtain motivation in con- tinuously releasing new content (e.g., faster release of content), and 23% of them either closed or did not finish setting up their Patreon profile. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 89

We further investigated the release frequency of mods (with more than 1 mod re- lease) that are created by the 32% of mod developers who use Patreon for motivation to release new content. However, the Wilcoxon rank-sum test shows no statistically significant difference in the release frequency between mods that are created by mod developers that accept donations for motivation to mod (a median mod release fre- quency of every 6 days) and mods that are created by other mod developers (a median mod release frequency of 7 days). The Wilcoxon rank-sum test did show statistically significant difference in the number of mod releases between mods that are created by mod developers that accept donations to mod (a median number of 23 mod releases) and mods that are created by other mod developers (a median number of 11 mod re- leases), with a medium Cliff’s delta effect size. Hence, mod developers who accept donations for motivation to mod do produce a larger number of mods than other mod developers, although not more frequently.

Interestingly, LexManos15 received the most donations at $2,157 per month. Lex-

Manos is the creator and primary developer of the popular Minecraft Forge API (Mor- rison, 2014), which is a mod loader API that is required to run most Minecraft mods.

However, other mod developers who have a valid Patreon URL only generates a median of $4 per month.

15https://www.patreon.com/lexmanos CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 90

: Summary of RQ2 18 of the 33 (54.5%) studied features have an impact in distinguishing popular

mods from unpopular ones. Mods that simplify mod development are more

likely to be popular. In addition, popular mods tend to promote community

contribution with source code URL and issue tracking URL, and have a richer

mod description. Finally, mods that accept donation tend to be more active and

more popular.

5.4 Implications

In this section, we outline the most important implications of our findings and provide

suggestions for mod developers to improve their mod popularity.

Mods that ease mod development are popular among modding community. In

Section 5.3.2 we observed that mods that belong to the “API and library” categories have a larger probability in being popular. The “API and library” mods can be used by mod developers to ease their development process. Such observation indicates that the modding community welcomes better development support. Our previous chap- ter (Chapter4) also shows that official modding support benefits the perceived quality of mods. Hence, both mod developers and game developers should contribute to im- prove the support for mod development.

Mod developers should improve the description of their mods. In Section 5.3.2, we observed that better mod descriptions are more prevalent in popular mods. Mods with more words in the long description and more images have a larger probability in being popular. In addition, mods with a wiki URL are more likely to be popular. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 91

Hence, mod developers should provide more detailed description, together with im- ages of their mods, if they wish to make their mod more popular. In addition, mod developers can leverage external wiki pages to better describe their mods to increase their popularity.

Mod developers should welcome community contributions to their mod. In Sec- tion 5.3.2, we observed that mods with a source code URL or an issue tracking URL are more likely to be popular. Hence, mod developers can encourage community contri- butions by either open sourcing their mod, or providing a channel for the community to report issues about their mods, to increase the popularity of their mods.

5.5 Threats to Validity

This section outlines the threats to the validity of our findings.

5.5.1 Internal Validity

A threat to the internal validity of our study is that we only studied the top and bottom

20% of mods. However, the top and bottom 20% of mods ensures that there is a clear distinction between popular and unpopular mods, as mods having close to the median number of total downloads can belong to either one. Such approach is also used in prior work (Tian et al., 2015).

Another threat to the internal validity of our study is that we only focused on the mods that were created between 2014 and 2016. However, such restriction is necessary to eliminate the bias introduced by the extreme short or long lifetime of a mod. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 92

Finally, it is important to realize that mod developers of the CurseForge mod dis- tribution platform could at anytime change the name of their mod, remove mod de- velopers or delete the mod. As a result, some older mods or mod developers may not exist at the time of our data collection. Future studies should investigate the life cycle of mods and mod developers on the CurseForge mod distribution platform.

5.5.2 External Validity

A threat to the external validity of our study is that we only studied mods from the

CurseForge mod distribution platform. However, the CurseForge mod distribution platform has the largest number of mods out of the other mod distribution platforms, as shown in Chapter2.

Another threat to the external validity of our study is that we only studied mods for the Minecraft game. Although the Minecraft game is one of the best selling games in

2019, and hosts one of most active and largest modding communities, future studies should compare our results with mods of different games.

5.5.3 Construct Validity

A threat to construct validity in our study is that we make suggestions to mod devel- opers about increasing the popularity of a mod based on studying the characteristics of popular and unpopular mods. However, we cannot ensure the causal nature of our observed insights. Because, every characteristic of the mod that we use in our study is not recorded for each release of the mod in the CurseForge mod distribution plat- form. Therefore, it is impossible for us to verify if the insights that we garner in our study are of a causal nature (i.e, if the following of the generated insights actually affect CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 93

the popularity of a mod). Hence, we provide insights based on the current character- istics exhibited by the popular and unpopular Minecraft mods that are available in the

CurseForge mod distribution platform. However, future studies should validate the causal nature of our findings.

5.6 Chapter Summary

An active modding community not only helps game developers meet the growing and changing needs of their gamer base, but also leads to a better overall gaming expe- rience. In this chapter, we studied the characteristics of popular mods with a large number of downloads by analyzing 2,228 Minecraft mods from the CurseForge mod distribution platform, along 5 dimensions of characteristics for a mod: mod category, mod documentation , environmental context of mod, remuneration for the mod, and community contribution for the mod. We firstly verified that the studied dimensions have strong explanatory power for the popularity of the studied mods. Then, we inves- tigated the contribution of each of the 33 features across these 5 dimensions of mod characteristics on the popularity of a mod. The most important findings of this chapter are:

1. The community contribution for the mod dimension has the strongest explana-

tory power of the popularity of mods. Popular mods tend to promote community

contribution with a source code URL and an issue tracking URL.

2. Mods that simplify mod development are more likely to be popular.

3. Popular mods tend to have a high quality description. CHAPTER 5. CONTINUOUS ADOPTION OF MODS BY GAMERS 94

4. Mods that accept donation tend to be more active and more popular.

Based on our findings, we suggest mod developers to contribute to improve the support for mod development, improve the quality of their mod description, and en- courage community contribution to their mods by either open sourcing the mods or providing a channel for the community to report issues about the mods. CHAPTER 6

Conclusions and Future Work

HIS chapter summarizes our work and discusses possible opportunities for T future work. Modding communities can be an integral complement to game development, saving game developers cost and time by strengthening gamer loyalty and improving game longevity. Hence, it is important that mod and game developers build and maintain an active modding community. We denote two dimensions of active modding community that should be fulfilled: (1) the continuous growth of mods by mod developers and (2) the continuous adoption of mods by gamers.

In this thesis, we study the modding community of the Nexus Mods distribution platform, one of the largest mod distribution platforms, to increase the growth of mods

95 CHAPTER 6. CONCLUSIONS AND FUTURE WORK 96

by mod developers. Then, we study an active and large modding community from the

CurseForge mod distribution platform, to increase the adoption of mods by gamers.

Our findings can help mod and game developers understand how to build and main- tain an active modding community.

6.0.1 Thesis Contributions

Our major contributions of this thesis are as follows:

1. Official support for modding is important for an active modding community. We are the first to quantitatively analyze the release schedule of mods, across multiple game franchises. Our findings show that game developers who implement the same officially-supported modding tool within the same game series tend to have a more active release of mods than mods from games without the same officially-supported modding tool or no officially-supported modding tool.

2. Although the post-release support of mods is low in quality, bug reports that have a response from the mod developer are solved quickly. We manually analyze the quality of bug reports of mods. In addition, we observe how quickly bugs are often solved. Our study shows that bugs tend to be of low quality. However, we observe that in 51% of bug reports, the first comment by someone other than the mod developer, and for bug reports that do get a response, it takes a median of 0 days. Thus, game developers and developers of third-party mod distribution platforms should step in to help mod developers improve the post-release support of their mods, which can help maintain an active modding community.

3. There are certain mod features that characterize popular mods. We analyze 33 CHAPTER 6. CONCLUSIONS AND FUTURE WORK 97

features among 5 dimensions of mods to distinguish popular from unpopular mods.

Our study shows that the dimensions of mods have larger explanatory power when to- gether than when isolated. With further analysis, we observe that certain features play a larger role in popular mods. For instance, we observe that the community contri- bution for the mod dimension has the strongest explanatory power of the popularity of mods. Popular mods tend to promote community contribution with a source code

URL and an issue tracking URL. We also observe that popular mods tend to have a high quality description. Our findings can help mod developers support the continu- ous adoption of mods by gamers, which can help build an active modding community

6.0.2 Future Work

In this section, we discuss some potential opportunities for extending our work.

1. We perform a large scale study on the release schedule and post-release support of the modding community, but we did not investigate the players of mods. Future re- search should conduct a large scale and longitudinal investigation of players of games with and without mods.

2. We study the effort and willingness that mod developers put into modding the original game by investigating the release schedule and post-release support of mods, but we did not measure if a modding community does extend the longevity of the original game. Future research should investigate how the impact of mods on the life expectancy of a game can be measured reliably.

3. In this thesis, we investigated the characteristics of popular and unpopular mods, CHAPTER 6. CONCLUSIONS AND FUTURE WORK 98

but we did not analyze how the characteristics contributed to making the mod pop- ular or not. Future research should validate the causal nature of our findings.

4. We study the characteristics of mods from the Minecraft game because it fosters one of the most active and largest modding communities. However, some character- istics may be unique for the Minecraft game. Future research should compare our results against mods from different games. Bibliography

Ahmed, F., Zia, M., Mahmood, H., and Al Kobaisi, S. (2017). Open source computer

game application: An empirical analysis of quality concerns. Entertainment Com-

puting, 21:1–10.

Al-Washmi, R., Bana, J., Knight, I., Benson, E., Kerr, O. A. A., Blanchfield, P., and Hop-

kins, G. (2014). Design of a math learning game using a Minecraft mod. In European

Conference on Games Based Learning, volume 1, page 10. Academic Conferences In-

ternational Limited.

Ampatzoglou, A. and Stamelos, I. (2010). Software engineering research for computer

games: A systematic review. Information and Software Technology, 52(9):888–901.

Anvik, J., Hiew, L., and Murphy, G. C. (2005). Coping with an open bug repository. In

OOPSLA workshop on Eclipse technology eXchange, pages 35–39. ACM.

Arakji, R. Y. and Lang, K. R. (2007). Digital consumer networks and producer-consumer 99 BIBLIOGRAPHY 100

collaboration: Innovation and product development in the industry.

Journal of Management Information Systems, 24(2):195–219.

Ash0821 (2017). The Essential Skyrim Mods You Should Have In-

stalled Right Now. https : / / www . gamepedia . com / blogs /

1574-the-essential-skyrim-mods-you-should-have. (last visited: Septem-

ber 2, 2019).

Balogh, G. and Beszédes, Á. (2013). Codemetrpolis–a Minecraft based collaboration

tool for developers. In First IEEE Working Conference on Software Visualization,

pages 1–4. IEEE.

Bayliss, J. D. (2012). Teaching game ai through Minecraft mods. In IEEE International

Games Innovation Conference, pages 1–4. IEEE.

Bebbington, S. (2014). A case study of the use of the game Minecraft and its affinity

spaces for information literacy development in teen gamers. PhD thesis, Université

d’Ottawa/University of Ottawa.

Bécares, J. H., Valero, L. C., and Martín, P. P. G. (2017). An approach to automated

videogame beta testing. Entertainment Computing, 18:79–92.

Bettenburg, N., Just, S., Schröter, A., Weiss, C., Premraj, R., and Zimmermann, T.(2008).

What makes a good bug report? In 16th ACM SIGSOFT International Symposium on

Foundations of Software Engineering, pages 308–318. ACM.

Blackburn, J., Kourtellis, N., Skvoretz, J., Ripeanu, M., and Iamnitchi, A. (2014). Cheat-

ing in online games: A social network perspective. ACM Transactions on Internet

Technology (TOIT), 13(3):9. BIBLIOGRAPHY 101

Blake, Vikki (2019). Minecraft might be the biggest-selling video

game of all time now. https : / / www . eurogamer . net / articles / 2019-05-19-minecraft-might-be-the-biggest-selling-video-game-of

-all-time-now. (last visited: September 2, 2019).

BlindJudge (2018). Witcher 3 modding tools released with NMM support. https://

www.nexusmods.com/witcher3/news/12550. (last visited: September 2, 2019).

Bostan, B. and Kaplancali, U. (2010). Explorations in player motivations: Game mods.

In GAMEON-ASIA, pages 1–9. EUROSIS.

Brand, J. and Kinash, S. (2013). Crafting minds in Minecraft. Education Technology

Solutions, 55:56–58.

Camycamera (2016). Why are Bethesda games so mod-popular? https://www. reddit.com/r/Fallout/comments/4454ig/why_are_bethesda_games_so_

modpopular/. (last visited: September 2, 2019).

Canossa, A., Martinez, J. B., and Togelius, J. (2013). Give me a reason to dig Minecraft

and psychology of motivation. In IEEE Conference on Computational Inteligence in

Games (CIG), pages 1–8. IEEE.

Chambers, C., Feng, W.-c., Sahu, S., and Saha, D. (2005). Measurement-based char-

acterization of a collection of on-line games. In 5th ACM SIGCOMM conference on

Internet Measurement, pages 1–14. USENIX Association.

Cheung, G. K., Zimmermann, T., and Nagappan, N. (2014). The first hour experience:

How the initial play can engage (or lose) new players. In First ACM SIGCHI Annual

Symposium on Computer-Human Interaction in Play, pages 57–66. ACM. BIBLIOGRAPHY 102

Consalvo, M. (2007). Cheating : Gaining Advantage in Videogames. The MIT Press.

CurseForge (2006). CurseForge. https://minecraft.curseforge.com/. (last vis-

ited: September 2, 2019).

Daneva, M. (2017). Striving for balance: A look at gameplay requirements of massively

multiplayer online role-playing games. Journal of Systems and Software, 134:54–75.

Dey, T., Massengill, J. L., and Mockus, A. (2016). Analysis of popularity of game mods:

A case study. In Annual Symposium on Computer-Human Interaction in Play Com-

panion Extended Abstracts, pages 133–139. ACM.

Duncan, S. C. (2011). Minecraft, beyond construction and survival. Well Played: a

journal on video games, value and meaning, 1(1):1–22.

Ekaputra, G., Lim, C., and Eng, K. I. (2013). Minecraft: A game as an education and

scientific learning tool. ISICO 2013, 2013.

Fabric development team (2018). Fabric Announcement. http://fabricmc.net/

2018/12/10/announcement.html. (last visited: September 2, 2019).

Finley, Klint (2014). NEW MINECRAFT MOD TEACHES YOU CODE AS YOU PLAY.

https://www.wired.com/2014/08/learntomod/. (last visited: September 2,

2019).

Gallagher, R., Jong, C., and Sinervo, K. A. (2017). Who Wrote the Elder scrolls?: Mod-

ders, Developers, and the Mythology of . Loading..., 10(16).

Game Lengths (2009). Game Lengths. http://www.gamelengths.com/. (last visited:

September 2, 2019). BIBLIOGRAPHY 103

Geere, D. and Copeland, W. (2019). The best Minecraft mods. https://www.

pcgamer.com/best-minecraft-mods/. (last visited: September 2, 2019).

Graham, T. N. and Roberts, W. (2006). Toward quality-driven development of 3D com-

puter games. In International Workshop on Design, Specification, and Verification of

Interactive Systems, pages 248–261. Springer.

Guana, V.,Stroulia, E., and Nguyen, V.(2015). Building a game engine: A tale of modern

model-driven engineering. In IEEE/ACM 4th International Workshop on Games and Software Engineering, pages 15–21. IEEE.

Hackman, E. and Björkqvist, U. (2014). Modders of Skyrim: Motivations and Modifi-

cations: A qualitative study of what motivations and modifications the modders of

Elder Scrolls: Skyrim exhibit. Master’s thesis, Södertörn University.

Hanghøj, T., Hautopp, H., Jessen, C., and Denning, R. C. (2014). Redesigning and re-

framing educational scenarios for Minecraft within mother tongue education. In

European Conference on Games Based Learning, volume 1, page 182. Academic Con-

ferences International Limited.

Hansen, M. (2018). Fallout 76 is already being fixed

by modders. https : / / www . techradar . com / news /

fallout-76-is-already-getting-fixed-by-modders. (last visited: Septem-

ber 2, 2019).

Harrell Jr, F. E. (2001). Lecture notes in Biostatistical modeling. URL: http://

biostat.mc.vanderbilt.edu/wiki/pub/Main/BioMod/notes.pdf. Last vis-

ited: September 2, 2019. BIBLIOGRAPHY 104

Harrell Jr, F. E., Lee, K. L., Califf, R. M., Pryor, D. B., and Rosati, R. A. (1984). Regres-

sion modelling strategies for improved prognostic prediction. Statistics in medicine,

3(2):143–152.

Harrell Jr, F. E. and Slaughter, J. C. (2001). Introduction to Biostatistics for Biomedical

Research. URL: http://hbiostat.org/doc/bbr.pdf. Last visited: September 2,

2019.

Hassan, S., Bezemer, C.-P.,and Hassan, A. E. (2018). Studying Bad Updates of Top Free-

to-Download Apps in the Store. IEEE Transactions on Software Engineer-

ing. (In press).

Hassan, S., Shang, W., and Hassan, A. E. (2017). An empirical study of emergency up-

dates for top android mobile apps. Empirical Software Engineering, 22(1):505–546.

Hirvonen, E. (2017). Improving the game with user generated content: an overview of

Skyrim mod users. Master’s thesis, University of Jyväskylä, Finland.

Hruska, J. (2016). Comparing Skyrim: Enhanced Edition with the

classic version. https : / / www . extremetech . com / gaming /

238558-comparing-skyrim-enhanced-edition-classic-version. (last

visited: September 2, 2019).

Jeppesen, L. B. (2004). Profiting from innovative user communities: How firms orga-

nize the production of user modifications in the computer games industry. Techni-

cal report, Department of Industrial Economics and Strategy, Copenhagen Business

School. BIBLIOGRAPHY 105

Kanode, C. M. and Haddad, H. M. (2009). Software engineering challenges in game

development. In Sixth International Conference on Information Technology: New

Generations., pages 260–265. IEEE.

Kasurinen, J., Palacin-Silva, M., and Vanhala, E. (2017). What concerns game devel-

opers?: A study on game development processes, sustainability and metrics. In 8th

Workshop on Emerging Trends in Software Metrics, pages 15–21. IEEE.

Khalid, H., Shihab, E., Nagappan, M., and Hassan, A. E. (2014). What do mobile app

users complain about? IEEE Software, 32(3):70–77.

Köhler, B., Haladjian, J., Simeonova, B., and Ismailovic,´ D. (2012). Feedback in low

vs. high fidelity visuals for game prototypes. In Second International Workshop on

Games and Software Engineering: Realizing User Engagement with Game Engineer-

ing Techniques, pages 42–47. IEEE.

Koleva, B., Tolmie, P.,Brundell, P.,Benford, S., and Rennick Egglestone, S. (2015). From

front-end to back-end and everything in-between: work practice in game develop-

ment. In Annual Symposium on Computer-Human Interaction in Play, pages 141–

150. ACM.

Laukkanen, T. (2005). Modding Scenes-Introduction to user-created content in com-

puter gaming.

Lee, D., Lin, D., Bezemer, C.-P., and Hassan, A. E. (2018). Building the Perfect Game -

An Empirical Study of Game Modifications. Empirical Software Engineering, Under

review:1–23. BIBLIOGRAPHY 106

Lee, D., Lin, D., Bezemer, C.-P., and Hassan, A. E. (2019a). Supple-

mentary material. https : / / gitlab . com / reviewer . nexus . mods /

suppmaterial-19-danlee-nexusmods. (last visited: September 2, 2019 –

the repository will be public after the acceptance).

Lee, D., Rajbahadur, G. K., Lin, D., Sayagh, M., Bezemer, C.-P.,and Hassan, A. E. (2019b).

An Empirical Study of the Characteristics of Popular Minecraft Mods. Empirical Soft-

ware Engineering, Under review:1–23.

Lenig, S. and Caporusso, N. (2018). Minecrafting virtual education. In International

Conference on Applied Human Factors and Ergonomics, pages 275–282. Springer.

Lewis, C. and Whitehead, J. (2011). The whats and the whys of games and software en-

gineering. In 1st International Workshop on Games and Software Engineering, pages

1–4. ACM.

Lewis, C., Whitehead, J., and Wardrip-Fruin, N. (2010). What went wrong: A taxonomy

of video game bugs. In Fifth International Conference on the Foundations of Digital

Games, pages 108–115. ACM.

Lin, D., Bezemer, C.-P.,and Hassan, A. E. (2017). Studying the urgent updates of popular

games on the Steam platform. Empirical Software Engineering, 22(4):2095–2126.

Lin, D., Bezemer, C.-P., and Hassan, A. E. (2018). An empirical study of early access

games on the Steam platform. Empirical Software Engineering, 23(2):771–799.

Lin, D., Bezemer, C.-P., and Hassan, A. E. (2019a). Identifying gameplay videos that

exhibit bugs in computer games. Empirical Software Engineering, pages 1–28. BIBLIOGRAPHY 107

Lin, D., Bezemer, C.-P.,Zou, Y., and Hassan, A. E. (2019b). An empirical study of game

reviews on the Steam platform. Empirical Software Engineering, 24(1):170–207.

Long, J. D., Feng, D., and Cliff, N. (2003). Ordinal analysis of behavioral data. Handbook

of psychology, pages 635–661.

Martin, W., Sarro, F., Jia, Y., Zhang, Y., and Harman, M. (2017). A survey of app

store analysis for software engineering. IEEE Transactions on Software Engineering,

43(9):817–847.

McIlroy, S., Ali, N., and Hassan, A. E. (2016a). Fresh apps: an empirical study of

frequently-updated mobile apps in the Google play store. Empirical Software En-

gineering, 21(3):1346–1370.

McIlroy, S., Ali, N., Khalid, H., and Hassan, A. E. (2016b). Analyzing and automatically

labelling the types of user issues that are raised in mobile app reviews. Empirical

Software Engineering, 21(3):1067–1106.

McIlroy, S., Shang, W., Ali, N., and Hassan, A. E. (2015). Is it worth responding to re-

views? Studying the top free apps in google play. IEEE Software, 34(3):64–71.

McIntosh, S., Kamei, Y., Adams, B., and Hassan, A. E. (2016). An empirical study of

the impact of modern code review practices on software quality. Empirical Software

Engineering, 21(5):2146–2189.

Midi, H., Sarkar, S. K., and Rana, S. (2010). Collinearity diagnostics of binary logistic

regression model. Journal of Interdisciplinary Mathematics, 13(3):253–267.

Mills, Aaron (2015). A Brief History of Minecraft Modding. https://hub.packtpub.

com/brief-history-minecraft-modding/. (last visited: September 2, 2019). BIBLIOGRAPHY 108

Mojang (2019). WHAT IS MINECRAFT? https://www.minecraft.net/en-us/

what-is-minecraft/. (last visited: September 2, 2019).

Mojica, I. J., Adams, B., Nagappan, M., Dienst, S., Berger, T., and Hassan, A. E. (2013).

A large-scale empirical study on software reuse in mobile apps. IEEE software,

31(2):78–86.

Morrison, A. (2014). How To Make Minecraft A Survival Game. https://www.

rockpapershotgun.com/2014/10/24/minecraft-survival-mods/. (last vis-

ited: September 2, 2019).

Müller, S., Kapadia, M., Frey, S., Klinger, S., Mann, R. P.,Solenthaler, B., Sumner, R. W.,

and Gross, M. (2015). Statistical analysis of player behavior in Minecraft. In 10th

International Conference on the Foundations of Digital Games. Society for the Ad-

vancement of the Science of Digital Games.

Murphy-Hill, E., Zimmermann, T., and Nagappan, N. (2014). Cowboys, Ankle Sprains,

and Keepers of Quality: How is Video Game Development Different from Software

Development? In 36th International Conference on Software Engineering, pages 1–

11, New York, NY, USA. ACM.

Nebel, S., Schneider, S., and Rey, G. D. (2016). Mining learning and crafting scientific

experiments: A literature review on the use of minecraft in education and research.

Journal of Educational Technology & Society, 19(2):355–366.

Nieborg, D. B. and Van der Graaf, S. (2008). The mod industries? the industrial logic of

non-market game production. European Journal of Cultural Studies, 11(2):177–195. BIBLIOGRAPHY 109

O’Brien, Chris (2013). How Minecraft became one of the biggest

video games in history. https : / / www . latimes . com / business / la-xpm-2013-sep-03-la-fi-tn-how-minecraft-video-games-20130822-

story.html. (last visited: September 2, 2019).

Pascarella, L., Palomba, F.,Di Penta, M., and Bacchelli, A. (2018). How is video game de-

velopment different from software development in open source? In IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), pages 392–402.

Peterson, S. (2014). Gamer loyalty plays into profits. https://www.gamesindustry.

biz/articles/2014-08-29-gamer-loyalty-plays-into-profits. (last vis-

ited: September 2, 2019).

Petrov, A. (2014). Using Minecraft in education: A qualitative study on benefits and

challenges of Game-based Education. Unpublished master’s thesis, University of

Toronto, Ontario, Canada.

Phillips, T. (2018). The human cost of Red Dead Redemp-

tion 2. https : / / www . eurogamer . net / articles /

2018-10-25-the-human-cost-of-red-dead-redemption-2. (last visited:

September 2, 2019).

Politowski, C., Fontoura, L., Petrillo, F., and Guéhéneuc, Y.-G. (2016). Are the old days

gone?: A survey on actual software engineering processes in . In

5th International Workshop on Games and Software Engineering, pages 22–28. ACM.

Poretski, L. and Arazy, O. (2017). Placing Value on Community Co-creations: A Study of BIBLIOGRAPHY 110

a Video Game ‘Modding’ Community. In ACM Conference on Computer Supported

Cooperative Work and Social Computing, pages 480–491. ACM.

Postigo, H. (2007). Of mods and modders: Chasing down the value of fan-based digital

game modifications. Games and Culture, 2(4):300–313.

Quiring, T. (2015). From voxel vistas: Place-making in Minecraft. Journal for virtual

worlds research, 8(1).

Ramadan, R. (2016). Does game modding require programming? https://

www.quora.com/Does-game-modding-require-programming. (last visited:

September 2, 2019).

Romano, J., Kromrey, J. D., Coraggio, J., Skowronek, J., and Devine, L. (2006). Exploring

methods for evaluating group differences on the nsse and other surveys: Are the t-

test and cohen’s d indices the most appropriate choices. In Annual Meeting of the

Southern Association for Institutional Research. Citeseer.

Saito, D., Takebayashi, A., and Yamaura, T. (2014). Minecraft-based preparatory train-

ing for software development project. In IEEE International Professional Communi-

cation Conference (IPCC), pages 1–9. IEEE.

Scacchi, W. (2010). Computer game mods, modders, modding, and the mod scene.

First Monday, 15(5).

Scacchi, W. (2011a). Modding as a basis for developing game systems. In 1st Interna-

tional Workshop on Games and Software Engineering, pages 5–8. ACM.

Scacchi, W. (2011b). Modding as an open source approach to extending computer BIBLIOGRAPHY 111

game systems. In IFIP International Conference on Open Source Systems, pages 62–

74. Springer.

Scacchi, W. and Cooper, K. M. (2015). Research challenges at the intersection of com-

puter games and software engineering. In Conference on Foundations of Digital

Games.

Scott, R. (2007). Nexus Mods. https://www.nexusmods.com/. (last visited: Septem-

ber 2, 2019).

Short, D. (2012). Teachingscientific concepts using a virtual world–Minecraft. Teaching

Science-the Journal of the Australian Science Teachers Association, 58(3):55.

Shumovsky, Y. (2018). How much does it cost to make a video game? https://

vironit.com/how-much-does-it-cost-to-make-a-video-game/. (last vis-

ited: September 2, 2019).

Sifa, R., Bauckhage, C., and Drachen, A. (2014). The Playtime Principle: Large-scale

cross-games interest modeling. In IEEE Conference on Computational Intelligence

and Games, pages 1–8. IEEE.

Siko, J., Barbour, M., and Toker, S. (2011). Beyond Jeopardy and lectures: Using Mi-

crosoft PowerPoint as a game design tool to teach science. Journal of Computers in

Mathematics and Science Teaching, 30(3):303–320.

Sotamaa, O. (2003). Computer game modding, intermediality and participatory cul-

ture. New Media, pages 1–5.

Sotamaa, O. (2010). When the game is not enough: Motivations and practices among

computer game modding culture. Games and Culture, 5(3):239–255. BIBLIOGRAPHY 112

Steamcharts.com (2018). Dota 2: Steam Charts. https://steamcharts.com/app/

570. (last visited: September 2, 2019).

Stone, B. G., Mills, K. A., and Saggers, B. (2019). Online multiplayer games for the so-

cial interactions of children with autism spectrum disorder: a resource for inclusive

education. International Journal of Inclusive Education, 23(2):209–228.

Syer, M. D., Nagappan, M., Hassan, A. E., and Adams, B. (2013). Revisiting prior em-

pirical findings for mobile apps: An empirical case study on the 15 most popular

open-source android apps. In Conference of the Center for Advanced Studies on Col-

laborative Research, pages 283–297. IBM Corp.

Tantithamthavorn, C., McIntosh, S., Hassan, A. E., and Matsumoto, K. (2016). An

empirical comparison of model validation techniques for defect prediction models.

IEEE Transactions on Software Engineering, 43(1):1–18.

Thongtanunam, P.and Hassan, A. E. (2018). Review Dynamics and Its Impact on Soft-

ware Quality. IEEE Transactions on Software Engineering, pages 1–13.

Tian, Y., Nagappan, M., Lo, D., and Hassan, A. E. (2015). What are the characteristics of

high-rated apps? a case study on free Android applications. In Software Maintenance

and Evolution (ICSME), IEEE International Conference on, pages 301–310. IEEE.

Tinydanser (2017). List of performance mods that do more harm

than good? https : / / bethesda . net / community / topic / 72261 / list-of-performance-mods-that-do-more-harm-than-good / 2 ? lang =

en-US&page=1. (last visited: September 2, 2019). BIBLIOGRAPHY 113

Villarroel, L., Bavota, G., Russo, B., Oliveto, R., and Di Penta, M. (2016). Release plan-

ning of mobile apps based on user reviews. In IEEE/ACM 38th International Confer- ence on Software Engineering (ICSE), pages 14–24. IEEE.

Washburn Jr, M., Sathiyanarayanan, P., Nagappan, M., Zimmermann, T., and Bird, C.

(2016). What went right and what went wrong: an analysis of 155 postmortems from

game development. In 38th International Conference on Software Engineering Com-

panion, pages 280–289. ACM.

Wijman, T. (2018). Mobile Revenues Account for More Than

50% of the Global Games Market as It Reaches $137.9 Bil-

lion in 2018. https : / / newzoo . com / insights / articles / global-games-market-reaches-137-9-billion-in-2018-mobile-games-

take-half/. (last visited: September 2, 2019).

Wijman, T. (2019). The Global Games Market Will Gener-

ate $152.1 Billion in 2019 as the U.S. Overtakes China as the

Biggest Market. https : / / newzoo . com / insights / articles / the-global-games-market-will-generate-152-1-billion-in-2019-as-

the-u-s-overtakes-china-as-the-biggest-market/. (last visited: Septem-

ber 2, 2019).

Wilcoxon, F.(1945). Individual comparisons by ranking methods. Biometrics bulletin,

1(6):80–83.

Zorn, C., Wingrave, C. A., Charbonneau, E., and LaViola Jr, J. J. (2013). Exploring

Minecraft as a conduit for increasing interest in programming. In FDG, pages 352–

359. Citeseer.