Rating of Players in a Game

Sander Johannes Cornelis van Riel

ANR: 640828

SNR: 1247320

THESIS SUBMITTED IN PARTIAL FULFILLMENT

OF THE REQUIREMENTS FOR THE DEGREE OF

MASTER OF SCIENCE IN DATA SCIENCE AND SOCIETY,

AT THE SCHOOL OF HUMANITIES AND DIGITAL SCIENCES

OF TILBURG UNIVERSITY

Supervisor: prof. dr. ir. P.H.M. Spronck

Second Reader: dr. Ç. Güven

Tilburg University

School of Humanities and Digital Sciences

Department of Cognitive Science & Artificial Intelligence

Tilburg, The Netherlands

May 2019

Abstract

In this research, the goal is to investigate if the rating of a player can be predicted for the Player Unknown’s Battleground (PUBG). The dataset consisted of player statistics for approximately 85,000 top ranked PUBG players with 50 features for each game mode. First, to investigate which features are most important to predict the rating of a PUBG player an exploratory analysis was done. This resulted in the same features for the solo, duos and squads game modes. Second, this research showed that prediction of the rating of PUBG players was possible with the use of multiple classification models, where the results for all three game modes were better compared to their baseline accuracy scores.

2

Table of Contents

1. Introduction ...... 5 1.1 Context ...... 5 1.2 Problem Statement and Research Questions ...... 6 1.3 Outline ...... 6 2. Theoretical Framework ...... 7 2.1 Prior Literature ...... 7 2.1.1 Outcome Prediction ...... 7 2.1.2 Rating of Players ...... 8 2.2 Classification Models ...... 9 2.2.1 Decision Tree and Random Forest ...... 9 2.2.2 Logistic Regression ...... 10 2.2.3 k-Nearest Neighbor ...... 10 2.3 Battle Royale ...... 11 2.4 Player Unknown’s Battleground (PUBG) ...... 11 3. Experimental Setup ...... 13 3.1 Dataset ...... 13 3.2 Pre-processing ...... 14 3.3 Setup for RQ1 ...... 16 3.4 Setup for RQ2 ...... 17 4. Results ...... 18 4.1 Exploratory Analysis ...... 18 4.2 Classification Models ...... 20 4.2.1 Decision Tree and Random Forest ...... 21 4.2.2 Logistic Regression ...... 23 4.2.3 k-Nearest Neighbor ...... 24 4.2.4 Performance on Test Set ...... 25 5. Discussion and Conclusion ...... 27 5.1 Discussion ...... 27 5.2 Limitations and Future Research ...... 28 5.3 Conclusion ...... 28 References ...... 30 Appendix A ...... 32

3

Appendix B ...... 33 Appendix C ...... 34 Appendix D ...... 36

4

1. Introduction

In this chapter the goal of this research will be introduced. Section 1.1 will give an overview of the context for this research. In section 1.2 the problem statement and the research questions will be formulated. Finally, section 1.3 contains an outline of the rest of this research.

1.1 Context

Over the past couple of years two trends in the gaming industry were notable. Firstly, a new gaming genre has become very popular, namely Battle Royale. In this online multiplayer genre 100 players are dropped on an island with the purpose to be the last man or team standing. Player Unknown’s Battlegrounds (PUBG) was one of the first games that that introduced this gaming genre. PUBG quickly became one of the best-selling games of all time. Since its release the Battle Royale genre has rapidly grown, and is still growing. In 2018 the Battle Royale game broke the records of number of players and number of spectators who watched play the game on the streaming website . Other franchises like and Battlefield have also implemented the Battle Royale mode into their new games.

The second trend in the gaming industry is the rise of electronic sports (). In 2018 the prize money pools and number of viewers for eSports have outranked some of the greatest sports events, like the Tour de France and Wimbledon (The Washington Post, 2018). To further illustrate the rise of competitive gaming, eSports could be added to the Olympic program as an official medal sport in 2024 (The Guardian, 2017). For the relatively new Battle Royale gaming genre, both PUBG and Fortnite are in the top-5 of the 2018 eSports tournaments prize money pool (Statista, 2019).

Despite the economic significance of the aforementioned trends in gaming industry, little to no research has been done on the Battle Royale game mode. This research will explore which features will influence the rating of a Battle Royale player in PUBG and tries to predict the rating based on their pre-match statistics. From a practical point of view, this research could benefit the eSports gaming industry, especially for players trying to compete in eSports. They could focus on the features that mostly influence the rating of a player and therefore reach and stay in the highest rating of the game. From a scientific point of view, this research may add knowledge to past rating and outcome prediction research in the field of game analytics.

When looking at prior research for rating and outcome prediction, there has not been any scientific research done for the Battle Royale genre. Therefore, for outcome prediction this research will review studies that have been done on other gaming genres, such as Real-Time Strategy (RTS) games (Erickson & Buro, 2014; Ravari, Bakkes & Spronck, 2016) and Multiplayer Online Battle Arena (MOBA) games (Yang, Harrison & Roberts, 2014; Yang, Qin, & Lei, 2016). These studies were

5 mostly done with only during-match data or post-match data. The study from Yang, Qin, and Lei (2016) showed that including pre-match data will improve prediction models. For outcome prediction in First-Person Shooters (FPS) there has been limited to no work, except for the work of Ravari, Spronck, Sifa, and Drachen (2017).

There are not many studies that investigated rating systems for players of video games. Arpad Elo introduced a rating system for chess in 1959, which is still the most used rating system in video games. The studies of Myślak and Deja (2014) and Avontuur, Spronck, and Van Zaanen (2013) showed that prediction of a player’s rating is possible based on the skill of these players.

The dataset that will be used for this research consists of player statistics for approximately 85,000 top ranked PUBG players with 152 features. These features are split for 3 different game modes: solo, duos and squads. With the use of this dataset, this research will try to predict the rating of players based on their statistics.

1.2 Problem Statement and Research Questions

The problem statement of this research is as follows:

To what extent can we predict the rating of a PUBG player?

To address this problem statement, the following research questions will be investigated:

1. Is there a difference in factors that influence the rating of a player for the 3 different game modes? 2. Which model best predicts the rating of a player?

In order to answer RQ 1, an exploratory analysis will be done using correlation. For RQ 2 various classification models will be used to identify which model best predicts the rating of a player.

1.3 Outline

The outline of this research is as follows. In the next chapter the theoretical framework will be discussed, which contains related work and relevant concepts. Chapter 3 will give the experimental setup of this research. This includes the data collection, the data description, the pre-processing of the data and the methods used to answer the research questions. Chapter 4 will show the results of these methods. Finally, Chapter 5 will consist of the discussion and conclusion of this research. Here answers to the research questions will be given and some suggestions for future research will be discussed.

6

2. Theoretical Framework

The theoretical framework of this research is described in this chapter. In section 2.1 prior literature on rating and outcome prediction in game analytics will be discussed. Section 2.2 will specify the classification models that are used for this research. In section 2.3 the gaming genre Battle Royale will be explained. Finally, the Battle Royale game PUBG is briefly described in section 2.4.

2.1 Prior Literature

As stated in the introduction of this research, there is no prior literature based on the Battle Royale gaming genre. Therefore, section 2.1.1 will discuss prior literature on outcome prediction for three other gaming genres, namely First-Person Shooters (FPS), Real-Time Strategy (RTS) and Multiplayer Online Battle Arena (MOBA) games. Section 2.1.2 will review literature on assigning ratings to players of video games.

2.1.1 Outcome Prediction

When trying to compare Battle Royale games with other gaming genres, FPS games come closest. PUBG can be played in both first-person and third-person perspective. The perspective for video games can be defined by the point-of-view by which the player interacts with the overall area and the internal game environment (Taylor, 2002). In first-person perspective video games, the player will look through the point-of-view of the in-game character. The point-of-view of a player is behind the character in third-person perspective video games.

For outcome prediction in FPS games there has been almost no research, except for the work of Ravari et al. (2017). Ravari et al. (2017) used three different models, a ranking model, a win-loss model and a binary ranking model for the combined models. Using Random Forest the ranking model resulted in 68% average precision, while the win-loss model and the binary ranking model showed better performing average precisions, respectively 84% and 94%. This is because the ranking model used a multi class Random Forest. The highest and lowest ranked players showed better predictions than the middle ranked players in their ranking model. The results of Ravari et al. (2017) showed that some features are predictive for all game modes, but also that there are some features that were only important for specific game modes. Ravari et al. (2017) concluded that match outcome prediction should be done on specific game modes.

Most studies on outcome prediction in video games were focused on RTS games. These studies are between human and AI players (Erickson & Buro, 2014) and between human and human players (PvP) (Ravari et al., 2016). A RTS game is a where players collect resources to build structures and recruit armies to secure areas of the map or to destroy other players’ assets.

7

Using Logistic Regression Erickson and Buro (2014) achieved an accuracy of 72.59% for the prediction after 15 minutes of observed , but were noisy in the early game. While Erickson and Buro (2014) only investigated match outcome prediction on one match-up, Ravari et al. (2016) studied all six different match-ups. Ravari et al. (2016) used Random Forest and Gradient Boosting Regression Trees and their general model achieved a performance above 63% accuracy. They also showed that rank and the importance rate of the features differs for specific models.

MOBA games are a subgenre of RTS games in which each player controls only one character. A MOBA match typically consists of two teams of five players each. The difference with RTS games is that in MOBA matches players do not need to construct buildings or units. The strategy of MOBA games revolves around cooperative play within teams and around improvement of the players’ characters during a match. In most MOBA games the goal is to destroy the opposing team’s base. This game genre is popular with widely-played games such as League of Legends and .

To predict match outcome for the MOBA game Dota 2 Yang et al. (2014) used a Decision Tree based on common patterns of winning teams in combat tactics. Their results showed that extracted patterns achieve 80% accuracy. Yang et al. (2016) concluded that including pre-match features from individual players will improve prediction models to predict the winning team of a match in Dota 2. With the use of Logistic Regression, their experiments showed that including more pre-match features will improve the accuracy from 58.69% to 71.49%.

2.1.2 Rating of Players

There are not many studies that investigated rating systems for players of video games. The first commonly used rating system is the Elo rating system (Elo, 1978). This system was developed by Arpad Elo for a rating system of chess players. Most multiplayer video games are based on this Elo system, for example games such as World of Warcraft and Dota 2. Gaming professionals state that the Elo system is also used in the rating of players in PUBG, but this is not officially stated by the PUBG developers.

Myślak and Deja (2014) discussed the rating and matchmaking system of the game League of Legends. In their study they proposed a new rating system and stated that this rating system can easily be adopted by other online video games. Myślak and Deja (2014) used k-means clustering to identify different groups of players based on their level of skill. Their study predicted a match outcome with a result significantly better than 50% using skill estimated on a base of role’s performance with the use of a Logistic Regression model.

Avontuur et al. (2013) used a Support Vector Machine classification model to predict the league of a player in the RTS game StarCraft II. The StarCraft leagues can be viewed as the rating of a player’s skill level in the game. Their model, with an accuracy of 47.3% achieved better than the majority class

8 baseline, which had an accuracy of 25.5%. Avontuur et al. (2013) concluded that their model can predict a player’s skill early in the game.

2.2 Classification Models

The rating of a PUBG player can be learned with the use of different machine learning models. For this research classification models are used to predict the rating of a player. In this section the following classification models are discussed. Firstly, the Decision Tree and Random Forest is discussed in section 2.2.1. Section 2.2.2 explains the Logistic Regression model. Finally, k-Nearest Neighbor will be discussed in section 2.2.3.

2.2.1 Decision Tree and Random Forest

The goal of a Decision Tree is to predict the value of a target variable based on a number of input variables. This method is commonly used due to its simple interpretation. It is visually represented in a tree-like structure. For this research the classification tree is used to classify instances into a pre- defined set of classes based on the values of their input variables. The classification of an instance starts by navigating from the root node down to leaf node of a tree based on the value of interior nodes along the path. Each input variable is represented as an interior node in the tree, with an edge to each possible value of that input variable. The leaf nodes of the tree represent the values of the target variable.

There are different metrics for measuring the input variable that best splits the set of items. To provide a measure of the quality of the splits, the metric Gini impurity will be used in this research. Gini impurity measures the likelihood of an incorrect classification of a new instance, if that new instance was randomly classified according to the distribution of class labels from the data set. The split is then made on the input variable that minimizes this incorrect classification.

Breiman (2001) stated that Random Forests are a combination of tree predictors such that each tree depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest. In other words, a Random Forest is a collection of different Decision Trees, where at each interior node a random input variable is chosen instead of choosing an input variable that explains most variance. Decision Trees that grow very deep mostly overfit their training sets. Random Forests are a way to reduce this variance (Friedman, Hastie & Tibshirani, 2001).

2.2.2 Logistic Regression

Logistic Regression is built from Linear Regression where the dependent variable is binomial. This binary Logistic Regression model can be extended to Multinomial Logistic Regression where the dependent variable is categorical with more than two levels (Dayton, 1992). While Logistic Regression only models the probability of the output based on the input, it does not by itself perform

9 as a classification model. However, Logistic Regression is widely used as a classification model with the use of cut-off values, where inputs can be classified as classes above, between or below these cut- off values.

To regularize logistic regression two types of penalties can be applied. First, Ridge regression, where coefficients are penalized if they are too far from zero. In Ridge regression the model complexity will be decreased while keeping all the variables in the model. Second, Lasso regression is quite similar to Ridge regression, but it penalizes the sum of squared coefficients, where Ridge regression penalizes the sum of their absolute values. This can lead to coefficients shrinking to exactly zero, which is dissimilar to Ridge regression. The penalty within both regressions is regularized by the tuning parameter lambda (λ).

2.2.3 k-Nearest Neighbor

The k-Nearest Neighbor (k-NN) model is a non-parametric method, which can be used for both classification and regression problems (Cover & Hart, 1967). For this research, k-NN is used as a classification method. The input of the model is the k closest training instances in the feature space. The output consists of the class label, wherein an instance is classified. The instance is classified by the majority vote of its neighbors, with the instance being classified to the class which is most common among the k nearest neighbors, where k is a positive integer. When the value of k is high, the decision boundary is simple, which leads to underfitting. However, a low k can lead to overfitting, as the decision boundary is more complex (Thorhallsson & Singh).

To test which neighbors are most common to an instance, the distance between that instance and the training examples is measured, using a distance function. There are several distance functions, with the

Euclidean distance that is most widely used. If p = (p1, p2,…, pn) and q = (q1, q2,…, qn) are two points where n is the dimensionality of the feature space, then the distance d between p and q is given by:

2.3 Battle Royale

A Battle Royale game is a survival based video game with last man standing gameplay. Most Battle Royale games can be played individually (solo), with pairs of two players (duos), or with a small group of players, usually 3-4 players (squads). Almost every Battle Royale game is played online with around one hundred other players each match. At the start of a Battle Royale match all players skydive

10 from the air onto a large map. All players land without equipment, armor and weapons, giving no player an advantage at the start of the match. This loot, which is needed for combat and survival, is placed randomly across the whole map. Players will need to search this loot across the map while they avoid being killed by other players. When a player gets killed the loot from this eliminated player can be looted as well. Most Battle Royale games included a mechanic that pushes players closer together as the game progresses. The concept of most Battle Royale games is that each player has only one , where players who die will not be respawned during the match. However, when playing with a team member or a small group, players are allowed to revive a team member before they are finished off by an opponent. When only one player or one team remains, the Battle Royale match is over.

For 2018 it is estimated that Battle Royale games will generate $12.6 billion, which is a significant rise from 2017 where the games earned $1.7 billion. With shooter game franchises like Call of Duty and Battlefield taking advantage of this trend, it is expected that the revenues of Battle Royale games will be over $20 billion for 2019 (Superdata Research, 2018).

2.4 Player Unknown’s Battleground (PUBG)

Player Unknown’s Battleground (PUBG) is an online multiplayer Battle Royale game, which is developed by PUBG Corporation. PUBG Corporation is owned by video game company Bluehole from South Korea. PUBG is based on mods created by developer Brendan Greene, who expanded these mods into a standalone game, inspired by the Japanese film Battle Royale from 2000. PUBG was first released for Microsoft Windows in March 2017 for Steam's beta program. The game was officially released on December 20, 2017.

PUBG is a shooter game with each match up to one hundred players, with the goal to be the last man or team standing. In this game players can choose to play alone (solo), with another player (duo) or with a small group up to four players (squad). PUBG can be played in both first-person and third- person perspective. Figure 2.1 shows the view from the third-person perspective.

Each match starts when players parachute from an airplane onto the 8 x 8 kilometers sized map. The path of the flight is random for each match. All players start without gear. Therefore, the first thing to do for players once they land is to search for weapons, vehicles, armor, and other equipment, which are randomly distributed throughout the map at the start of a match, with some high-risk zones with better equipment. Killed players drop all of their equipment and can be looted by all players.

11

Figure 2.1. Third-person gameplay in PUBG.

In PUBG the playable area of the map shrinks down every few minutes, where players outside of this zone are constantly taking damage and could eventually be eliminated if the safe area is not reached within time. Players can use vehicles like cars, quad bikes and motorbikes to navigate around the map or can choose to stay on foot. During the match a number of airplanes fly over the playable area of the map and drop a package, which contains in most cases better equipment than found in the beginning of the match. Typically, a match in PUBG takes no more than 30 minutes.

After playing 10 ranked matches players will be assigned to a rating based on the PUBG rating system. Gaming professionals state this system is based on the Elo rating system, but this is not officially stated by the PUBG developers. By earning rank points players can get to a higher rating. These rank points are accumulated by simply playing and performing well in games (PUBG, 2018). Match placement and kills will contribute to the players’ rank points. Players can get a different rating for each of the three game modes. Matchmaking is not affected by the rating of a player. However, players will be assigned to a match based on their overall performance in previous matches.

With their official release in December 2017, PUBG Corporation reported that there were more than 30 million players already. By June 2018, the number of players was around 400 million. PUBG was the most played game of 2018 on Steam, almost doubling the hourly number of players of the number 2 of the list, DOTA 2 (Statista, 2019). In 4 months since the launch of the mobile version of the game, PUBG Corporation announced that the app had reached over 100 million downloads in August 2018. The next chapter will describe the experimental setup for this research.

12

3. Experimental Setup

In this chapter the experimental setup of this research will be described. Section 3.1 will describe the dataset and how it has been collected. In section 3.2 the pre-processing of the data will be discussed. Finally, section 3.3 and 3.4 will give the experimental setup for research questions 1 and 2, respectively.

3.1 Dataset

The dataset that is being investigated for this research contains detailed information on the performance of PUBG players. It is retrieved from the website Kaggle and has been published by Justin Moore in July 2017. The dataset includes 152 features for 87989 PUBG players. The complete list of all 152 features can be found in Appendix A. The PUBG players were selected as they were the top ranked players at the moment of data collection, July 2017. The 152 features are split in four different groups. First, the data includes 2 identifying features to identify each different player. The other 3 groups of features all include 50 features for each of the three game modes: solo, duos and squads. Not all of these 152 features will be used for this research. The 2 identifying features are not used, while they don’t include any information. Most features were both as a total number as well as a number per game or ratio included in the dataset. For this research the “per game” and ratio features are used. To investigate to what extent the rating of a player can be predicted, the following features will be used:

 Rating – Overall rating of the player.  Kill Death Ratio – Number of kills divided by the number of death over all games played.  Win Ratio – Number of wins divided by the number of games played.  Top 10 Ratio – Number of times the player reached the top 10 divided by the number of games played.  Damage per Game – Average damage done per game.  Heals per Game – Average number of heal items used per game. When a player is damaged, the player can use health items to restore health.  Kills per Game – Average number of kills per game.  Move Distance per Game – Average distance travelled per game.  Revives per Game – Average number of revives per game. In duos and squads a player is DBNO (down but not out) when the player’s health goes down to zero. A team member can revive the downed player before he bleeds out.  Time Survived per Game – Average time survived per game.

13

3.2 Pre-processing

First, to compare the different game modes, the data is split into three groups: solo, duos and squads. Furthermore, players with at least 50 games for each game mode were selected, as the rating of players cannot be computed accurately from only a small number of games. In PUBG players will be ranked after playing 10 ranked games of a single game mode, but for better understanding of the rating and the factors it is influenced by, this research uses a minimum of 50 games. This results in Table 3.1, which presents the distribution of the data for each game mode. The features used for the solo game mode is only 10 as the feature Revives per Game is excluded, since there is no team member to revive while playing solo.

Table 3.1.

Distribution of the data for the 3 different game modes.

Game mode Players Features Solo 42639 10 Duos 65163 11 Squads 76510 11

Second, the features used for this research, except for Rating and Kill Death Ratio, were already averaged per game by the creator of the dataset. These features are calculated by the total number of a statistic divided by the number of games played.

Last, for each of the game modes an extra feature is added to the data, to use the classification models described in Chapter 2. The feature Category is created for all three game modes, based on the feature Rating. While the dataset includes only the top ranked players, the rating system used by PUBG cannot be used for this research. In line with the research of Myślak and Deja (2014), the feature Category is created with k-means clustering, where each cluster corresponds to one class. In k-means clustering the data is partitioned into k number of clusters, where the within cluster distance in minimized and the between cluster distance is maximized. To determine the optimal number of clusters, the Elbow method is used. Here the percentage of variance explained is visualized as a function of the number of clusters. Figure 3.1 is showing that the optimal number of clusters for the solo game mode is 4. From the plots of the duos and squads game modes it can also be concluded that k equal to 4 is the optimal number of clusters. The plots for the duos and squads game mode are given in Appendix B.

14

Figure 3.1. Elbow method for solo game mode.

The feature Category can now be categorized for each of the game modes based on the feature Rating. K-means clustering gives the categorization of the feature Category shown in Table 3.2 for solo, duos and squads, with class label values ranging from 1 to 4 and cut-off values are rounded to tens. The distribution of the Category feature for the solo game mode is shown in Figure 3.2. The bar plots with the distribution of the Category feature for the duos and squads game mode are given in Appendix C. To further illustrate the clustering of this feature, Figure 3.3 shows the different classes in a scatterplot for the solo game. The scatterplots of the clustering for the duos and squads game mode are illustrated in Appendix C.

Table 3.2.

Categorization of the feature Category based on the feature Rating using k-means clustering.

Class Label Solo Rating Duos Rating Squads Rating 1 < 1750 < 1700 < 1850 2 1750 – 2000 1700 – 1900 1850 – 2100 3 2000 – 2250 1900 – 2100 2100 – 2350 4 > 2250 > 2100 > 2350

15

Figure 3.2. Distribution of the feature Category for solo game mode.

Figure 3.3. Clustering of the feature Category for solo game mode.

3.3 Setup for RQ1

An exploratory analysis is done to answer RQ1: whether there is a difference in factors that influence the rating of players in the three different game modes. Correlation matrices are used to see which features will correlate with the feature Rating for all three game modes. This method is applied with the use of the features specified in section 3.1, where the feature Revives per Game is excluded for the solo game mode. The exploratory analysis is done using the programming language R.

16

3.4 Setup for RQ2

In order to answer RQ2, i.e. which model best predicts the rating of a player, several classification models were built. The models were trained to predict the feature Category for all three game modes into classes with labels 1-4, specified in Table 3.2. The classification models that will be used for this research are: Decision Tree and Random Forest, Logistic Regression and k-Nearest Neighbor, as described in Chapter 2. These models were built and implemented in R using various packages. The dataset was split into a train set (80%) and a test set (20%). The tuning of the parameters of the train set is done using 5-fold cross-validation. These parameters were evaluated using the metric accuracy. Finally, the parameters are evaluated on the test set and the accuracy scores are compared to the baseline, which predicts the majority class. Chapter 4 will provide more detailed information on how the classification models are used, as well as the results for RQ1 and RQ2.

17

4. Results

This chapter will give the results for the models described in Chapter 3. Section 4.1 shows the results from the exploratory analysis. In section 4.2 the results from the classification models will be provided.

4.1 Exploratory Analysis

The first goal of this research is to investigate whether there are differences in features that affect the rating of a player for the solo, duos and squads game modes. As described in Chapter 3, this will be investigated with an exploratory analysis, namely correlation. This analysis is done with the original feature Rating, while the feature Category will not be used, as correlation works best with numerical features.

The correlations between the features for the solo game mode are shown in Table 4.1. This table shows that the feature Rating is strongest correlated with the features Move Distance per Game, Time Survived per Game and Top 10 Ratio, with correlations of 0.435, 0.433 and 0.427, respectively. Noticeable is the strong relationship between some of the features, which will be explained in section 5.1.

Table 4.1.

Correlation matrix for solo game mode.

KillDeathRatio WinRatio Top10Ratio DamagePg HealsPg KillsPg MoveDistancePg TimeSurvivedPg

Rating 0.275 0.342 0.427 0.295 0.307 0.269 0.435 0.433 KillDeathRatio 0.744 0.455 0.982 0.241 0.993 0.219 0.197 WinRatio 0.683 0.687 0.280 0.691 0.459 0.480 Top10Ratio 0.436 0.392 0.423 0.724 0.837 DamagePg 0.246 0.991 0.213 0.183 HealsPg 0.239 0.399 0.389 KillsPg 0.191 0.162 MoveDistancePg 0.791

Table 4.2 shows the correlations between the features for the duos game mode. Different from the solo game mode is the feature Revives per Game that has been added to the selection of features for the analyses of the duos game mode. The table shows that the feature Rating is strongest correlated with the features Top 10 Ratio, Time Survived per Game, Move Distance per Game and Win Ratio. The difference from the solo game mode correlations with these features is the correlation with the Win Ratio feature. Table 4.1 showed a correlation of 0.342, while the correlation between the features Rating and Win Ratio for the duos game mode is equal to 0.437.

18

Table 4.2.

Correlation matrix for duos game mode.

KillDeathRatio WinRatio Top10Ratio DamagePg HealsPg KillsPg MoveDistancePg RevivesPg TimeSurvivedPg

Rating 0.371 0.437 0.496 0.379 0.353 0.355 0.437 0.266 0.455 KillDeathRatio 0.714 0.446 0.971 0.364 0.989 0.245 0.507 0.309 WinRatio 0.682 0.629 0.374 0.636 0.495 0.367 0.562 Top10Ratio 0.413 0.416 0.395 0.743 0.254 0.892 DamagePg 0.365 0.981 0.226 0.514 0.289 HealsPg 0.354 0.367 0.301 0.378 KillsPg 0.199 0.515 0.260 MoveDistancePg 0.161 0.813 RevivesPg 0.195

The correlations between the features for the squads game mode are given in Table 4.3. As for the duos game mode, the feature Revives per Game has been added to the selection of features for this analysis. As can be seen in Table 4.3, the feature Rating is strongest correlated with the features Top 10 Ratio, Move Distance per Game and Time Survived per Game. These are the same features as for the solo and duos game mode. Similar to the solo and duos game mode, there is high correlation between some of the features for the squads game mode. This will be discussed in section 5.1, which represents the discussion of this research.

Table 4.3.

Correlation matrix for squads game mode.

KillDeathRatio WinRatio Top10Ratio DamagePg HealsPg KillsPg MoveDistancePg RevivesPg TimeSurvivedPg

Rating 0.248 0.373 0.459 0.229 0.262 0.218 0.437 0.185 0.415 KillDeathRatio 0.661 0.357 0.957 0.363 0.980 0.278 0.385 0.346 WinRatio 0.673 0.541 0.393 0.552 0.556 0.376 0.632 Top10Ratio 0.301 0.409 0.285 0.778 0.277 0.910 DamagePg 0.361 0.978 0.242 0.374 0.306 HealsPg 0.353 0.372 0.342 0.395 KillsPg 0.217 0.380 0.281 MoveDistancePg 0.231 0.829 RevivesPg 0.277

19

4.2 Classification Models

In this section the results of the analyses will be provided to answer the second research question: which model best predicts the rating of a player. In contrast with the previous section, these analyses are done with the Category feature, while the feature Rating will not be used. As the results in section 4.1 suggest that none of the features are extremely weak or extremely strong correlated with the feature Rating, all other features described in Chapter 3 will be used in this section. The data for all three game modes will be split into a train set (80%) and a test set (20%). The data is divided with the use of the R package Caret, which uses random stratified sampling on the feature Category to ensure two homogenous subsets of the data.

To evaluate the classification models used for this research, the metric accuracy will be used. The accuracy is the percentage of correctly classifying the input samples. The baseline will be set to the model that predicts the majority class of the dataset. In Table 4.4 the baseline accuracy scores of the train and test sets for all three game modes will be given. The baseline accuracy scores for the train test sets are similar, as the partitioning of these sets was done with random stratified sampling. In the solo game mode the class with label 3 is the majority class, however for the duos and squads game mode the majority class is class 2. As many players were categorized into one class for the squads game mode, this baseline accuracy is higher than the baseline accuracy scores found for the solo and duos game modes.

Table 4.4.

Majority baseline accuracy (%) for train and test sets for each game mode.

Game mode Train Test Solo 33.39 33.39 Duos 33.00 33.00 Squads 38.31 38.31

For the classification models used in this chapter, 5-fold cross-validation is used. Here each dataset is randomly partitioned into 5 equal sized subsets, where one subset is used for validating the model and the remaining subsets are used for training the model. After repeating this 5 times, the 5 results will be averaged to produce a single accuracy score.

After finding the optimal parameters for each of the classification models, the performance of these models will be evaluated on the test set, which includes unseen data. In order to compare performances of the classification models, the same subset of the data will be used for training and testing over all classification models.

20

As described in Chapter 3, this research will use different classification models. In section 4.2.1 the use of a Decision Tree and Random Forest will be illustrated. Section 4.2.2 describes a Logistic Regression model. In section 4.2.3 the use of k-NN will be explained. Finally, the performance of these classification models on the test set will be described in section 4.2.4.

4.2.1 Decision Tree and Random Forest

To predict the rating of a PUBG player a Decision Tree is trained on the train set using the R package party. Figure 4.1 shows the Decision Tree for the squads game mode, with a maximal depth of three levels. Similar trees for the solo and duos game mode can be found in Appendix D. In Figure 4.1 the first feature that is used is the Top 10 Ratio feature. This is in line with the findings in the exploratory analysis of this research, which stated that the feature Rating had the strongest correlation with the Top 10 Ratio feature for the squads game mode. The tree also uses the features Move Distance per Game and Win Ratio at the second and third level. Table 4.3 also showed that the correlation between these features and the Rating feature were relatively high. So, some of the features with strong correlation are also most indicative for the prediction of the rating of players.

Figure 4.1. Decision Tree for squads game mode, with maximal depth is 3.

Each leaf node states the number of players and the distribution of classes among these players. Figure 4.1 illustrates that most leaf nodes contain a majority class. For example node 15, which includes 1016 players and a majority class with label 4. From this node the following decision rule could be inferred: IF Top 10 Ratio is larger than 37.5 AND Move Distance per Game is larger than 4053.83 AND Top 10 Ratio is larger than 64.9, then the player will be classified to class with label 4. To evaluate a Decision Tree model, no restriction on the maximal depth will be computed and no parameters had to be optimized.

Different Random Forest models will be computed to predict the rating of PUBG players, using the R package Caret and its backend RandomForest package. To optimize the Random Forest models, two parameters will be tuned on the train set with 5-fold cross-validation. First, the number of trees (ntree)

21 to grow in the forest is considered. The values that will be used to optimize the Random Forest models for the number of trees are: 1, 10, 100 and 500. For computational and capacity reasons the maximum number of trees is 500, while Random Forest will perform better with 1000 or 2000 trees (Breiman, 2001). Second, the number of features randomly sampled as candidates at each split (mtry), where values of 2, 5 and 8 are considered. This results in 12 different Random Forest models for each game mode that will be considered for this research.

Table 4.5 shows the optimal tuned parameters for each of the game modes with the performance for each number of trees. The table shows that all models are performing better than the baseline accuracy, except for the Random Forest model with 1 tree for the squads game mode. These results also show that for each game mode the Decision Tree performs better than the Random Forest model with 1, 10, 100 trees, except for the solo game mode with 100 trees, where accuracy scores are approximately the same. As expected the Decision Tree performs better than Random Forest models with a small number of trees, as the Decision Tree will choose the features that are most effective, while Random Forest randomly chooses features. But, when choosing a lot of trees in the Random Forest model, the accuracy increases, while the number of features randomly sampled as candidates at each split decreases. For all three game modes the best performing Random Forest model is the model with 500 trees and 2 features as candidates at each split. However, for the duos and squads game mode the Decision Tree performs better.

Table 4.5.

Results of parameter tuning of Random Forest for classifying Category feature for each game mode.

Accuracy (%) ntree mtry(Solo) Solo mtry(Duos) Duos mtry(Squads) Squads Baseline 33.39 33.00 38.31 Decision Tree 43.05 43.81 44.76 1 8 35.10 8 34.69 5 34.89 10 5 39.98 5 39.38 2 39.68 100 2 43.16 2 42.41 2 43.12 500 2 43.60 2 42.83 2 43.61

4.2.2 Logistic Regression

Another classification model to predict the rating of PUBG players is Logistic Regression. For this research Multinomial Logistic Regression with regularization is used with the R package glmnet. Before tuning the Logistic Regression models, the data needs to be normalized, when using

22 regularization (Friedman, Hastie & Tibshirani, 2001). For this model, two parameters will be tuned on the train set with 5-fold cross-validation. The first parameter is the type of regularization penalty, where Lasso and Ridge are considered. Second, the weight of the regularization parameter (λ) will be tuned, by examining the values: 0.0001, 0.001, 0.01, 0.1. So, there will be 8 different Logistic Regression models considered for each game mode.

Table 4.6 shows the optimization of the parameters for classifying the rating of a player. All models performed better than the baseline scores. For all three game modes, the Lasso penalty model performed better than the Ridge penalty model, where for the duos game mode the accuracy scores are approximately the same. The Ridge penalty models results showed that for each game mode values of λ equal to 0.0001, 0.001 and 0.01 gave exactly the same results, while the Lasso penalty models showed different accuracy scores for each value of λ. For the duos and squads game mode the Lasso penalty model with λ equal to 0.001 performed best, with accuracy scores of 43.60% and 43.90%, respectively. For the solo game mode the Lasso penalty model with λ equal to 0.0001 performed best with an accuracy of 41.57%.

Table 4.6.

Results of parameter tuning of Logistic Regression for classifying Category feature for each game mode.

Accuracy (%) Penalty λ(Solo) Solo λ(Duos) Duos λ(Squads) Squads Baseline 33.39 33.00 38.31 Ridge 0.01 41.12 0.01 43.54 0.01 43.61 Lasso 0.0001 41.57 0.001 43.60 0.001 43.90

4.2.3 k-Nearest Neighbor

The final classification model that will be used for this research is k-Nearest Neighbor (k-NN). The k- NN model is implemented in R using the package Caret, which uses the metric Euclidean distance. Before tuning the k-NN models, the data needs to be normalized. This is done within the function of the R package Caret. To optimize the k-NN models, the parameter k, which represents the number of neighbors considered, will be tuned on the train set with 5-fold cross-validation. The values that will be used to optimize the k-NN models for k are: 1, 5, 11, 21, 31, 41, 51, 61, 71 and 81. Thus, for each game mode 10 different k-NN models will be considered.

Table 4.7 gives the accuracy of for different values of k for each game mode, compared to the baseline. The accuracy increases with the number of neighbors k for each game mode, with k is equal

23 to 81 being the best performing k-NN model for all three game modes. For the solo and squads game mode, the k-NN model with k is equal to 81, results in accuracy scores of 41.65% and 43.24%, respectively. For the duos game mode the k-NN model with k is equal to 71 performed best with an accuracy of 42.40%.These results are better than the baseline scores for all three game modes.

Table 4.7.

Results of parameter tuning of k-NN for classifying Category feature for each game mode.

Accuracy (%) k Solo Duos Squads Baseline 33.39 33.00 38.31 1 34.42 34.66 34.96 5 36.99 37.49 38.11 11 38.73 39.47 39.82 21 39.72 40.78 40.98 31 40.41 41.34 42.01 41 40.96 41.50 42.31 51 41.26 41.73 42.71 61 41.51 41.93 43.02 71 41.61 42.40 43.22 81 41.65 42.35 43.24

4.2.4 Performance on Test Set

In this section the tuned classification models that performed best on the train set will be evaluated on the test set to predict the rating of PUBG players. Table 4.8 shows the accuracy of the following models: Decision Tree, Random Forest, Logistic Regression and k-Nearest Neighbor for each of the game modes. The best performing model for each game mode is in bold text. The table suggests that there was no overfitting for all classification models, except for the Decision Tree where for the solo game mode the decrease in performance is largest and equal to 2.08%. The reason for this phenomenon could be that there was no pruning done and there was no restriction on the maximal depth. For the three other classification models most train and test set accuracy scores were approximately equal, with the largest decrease of 0.60% in performance for the Logistic Regression model for the duos game mode. The largest increase in performance is an increase of 0.36% for the Logistic Regression model for the solo game mode.

24

Table 4.8 shows that Random Forest is the best performing model for the solo and duos game mode, with accuracy scores of 43.89% and 43.14%, respectively. For the squads game mode the model that has the best performance is the Logistic Regression model, where accuracy is equal to 44.17%. However, for the duos and squads game modes differences in model accuracy scores are small for all models. The Random Forest model stands out for the solo game mode. Finally, these results are better compared to their baseline accuracy scores.

Table 4.8.

Accuracy (%) of all models on train and test set for each game mode.

Game Decision Tree Random Forest Logistic Regression k-NN

Mode Baseline Train Test Train Test Train Test Train Test

Solo 33.39 43.05 40.97 43.60 43.89 41.57 41.93 41.65 41.20

Duos 33.00 43.81 42.95 42.83 43.14 43.60 43.00 42.40 42.34

Squads 38.31 44.76 43.76 43.61 43.88 43.90 44.17 43.24 43.41

For each of the game modes the confusion matrix based on predictions of the test set for the Random Forest model is illustrated in Table 4.9. The Random Forest is used here, as this model performs best for 2 out of 3 games modes. Table 4.9 shows that for the solo game mode the model is best at predicting class with label 3. For the duos and squads game mode class with label 2 performed best. These results are in line with the majority baseline, where the majority class for the solo game mode is the class with label 3, while for duos and squads the majority class is the class with label 2.

Table 4.9.

Confusion matrix Random Forest model of test set for each game mode.

Solo Duos Squads

Actual / Class Class Class Class Class Class Class Class Class Class Class Class Predicted 1 2 3 4 1 2 3 4 1 2 3 4 Class 1 666 392 209 25 743 466 183 24 1405 970 379 50

Class 2 536 904 639 212 1130 2248 1606 528 1890 3748 2374 703

Class 3 330 1034 1606 805 308 1321 1726 1013 279 1059 1278 630

Class 4 21 188 393 566 20 265 576 874 6 85 161 283

25

To conclude, this chapter showed which features are most influencing a player’s rating for each game mode, and showed results that the rating of a PUBG player can be predicted for all three game modes, where the best performing models performed better than the baseline accuracy scores.

26

5. Discussion and Conclusion

This chapter will provide the discussion and conclusion of this research. Section 5.1 will discuss the results. In section 5.2 the limitations of this research and insights for future research will be given. Finally, section 5.3 describes the conclusion, where the research questions and the problem statement are answered.

5.1 Discussion

The first goal of this research was to investigate whether there are differences in features that influence the rating of a player for the solo, duos and squads game modes in the Battle Royale game PUBG. The results showed that the most important features to predict the rating of players were the same for all three game modes. These results are in line with Ravari et al. (2017), where they showed that some features are predictive for all game modes. There were also two relationships found between the features for all three game modes. First, the high correlation between the Kills per Game feature and the two features, Kill Death Ratio and Damage per Game. Since more damage will lead to more kills and more kills will lead to a higher kill/death ratio, these correlations are no surprise. Second, the high correlation between the most predictive features to predict the rating of PUBG players; Top 10 Ratio, Time Survived per Game and Move Distance per Game. This suggests that players that survive long and move far during a match will be more likely to place themselves within the top 10 players of that match.

The second part of this research investigated which model was best at predicting the rating of a PUBG player. The results showed that Random Forest was the best performing model for the solo and duos game mode, with accuracy scores of 43.89% and 43.14%, respectively. For the squads game mode the model that had the best performance was the Logistic Regression model, where accuracy was equal to 44.17%. These results were better compared to their baseline accuracy scores, but did not show a large increase in performance. One reason for this relatively low increase of performance could be that a large proportion of the players are close to the boundaries of two classes, where these players will be constantly switching between two classes. A second reason could be the character of the dataset, which will be described in section 5.2.

5.2 Limitations and Future Research

One of the limitations of this research was the character of the dataset. The dataset only included the top ranked players in the Battle Royale game PUBG. Therefore, bad and mediocre players were not investigated. This could make the classification models less accurate, as high ranked players are more difficult to compare to each other. Hence, a more complete dataset could lead to better results, when classifying the rating of players in a Battle Royale game.

27

A second weakness is the clustering of the different groups of players based on their rating. As stated in the experimental setup, the rating system of the game PUBG could not be used for the same reason that the dataset only included top ranked players. This would lead to lower ratings with no players in it, while the higher ratings are full of players. Therefore, k-means clustering is used to divide the players into different groups based on a single feature. However, k- means clustering usually works better with multiple features.

Thus, for future research a more complete dataset could be used, which includes players from all ratings. Then, the players could be divided into groups based on the rating system of the Battle Royale game itself.

5.3 Conclusion

The goal of this research was to investigate if the rating of a player could be predicted for the Battle Royale game PUBG. To address this problem, the following research questions were investigated:

1. Is there a difference in factors that influence the rating of a player for the 3 different game modes? 2. Which model best predicts the rating of a player?

To answer the first research question an exploratory analysis was done, using correlation. For each of the game modes the features that influenced the rating that were found were the same, namely Top 10 Ratio, Move Distance per Game and Time Survived per Game, which is in line with the findings of Ravari et al. (2017).

The second research question looked at the results of different classification models. This research investigated the following models: Decision Tree and Random Forest, Logistic Regression and k- Nearest Neighbor. The results showed that Random Forest was the best performing model for the solo and duos game model. For the squads game mode the model that had the best performance was the Logistic Regression model. However, for the duos and squads game mode small differences in accuracy scores for all models were found. For the solo game mode the Random Forest model outperformed the other models.

The problem statement of this research was:

To what extent can we predict the rating of a PUBG player?

The results showed that prediction of the rating of a PUBG player is possible. All in all, for all three game modes the results were significantly better compared to their majority baseline accuracy scores. Since there has been little to no prior research on the rating of players for a Battle Royale game, this research could lay the groundwork for future researches based on this rising gaming genre.

28

References

Avontuur, T., Spronck, P., & Van Zaanen, M. (2013). Player skill modeling in Starcraft II. In Ninth

Artificial Intelligence and Interactive Digital Entertainment Conference.

Breiman, L. (2001). Random forests. Machine learning, 45(1), 5-32.

Cover, T. M., & Hart, P. E. (1967). Nearest neighbor pattern classification. IEEE transactions on

information theory, 13(1), 21-27.

Dayton, C. M. (1992). Logistic regression analysis. Stat, 474-574.

Elo, A. E. (1978). The rating of chessplayers, past and present. Arco Pub.

Erickson, G. K. S., & Buro, M. (2014). Global state evaluation in StarCraft. In Tenth Artificial

Intelligence and Interactive Digital Entertainment Conference.

Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, No. 10).

New York: Springer series in statistics.

Myślak, M., & Deja, D. (2014). Developing game-structure sensitive matchmaking system for

massive-multiplayer online games. In International Conference on Social Informatics (200-

208). Springer, Cham.

PUBG. (2018). Dev Letter: Rank System Beta. Retrieved from:

https://www.pubg.com/2018/10/03/dev-letter-rank-system-beta/

Ravari, Y. N., Bakkes, S., & Spronck, P. (2016). Starcraft winner prediction. In Twelfth Artificial

Intelligence and Interactive Digital Entertainment Conference.

Ravari, Y. N., Spronck, P., Sifa, R., & Drachen, A. (2017). Predicting Victory in a Hybrid Online

Competitive Game: The Case of Destiny. In Thirteenth Artificial Intelligence and Interactive

Digital Entertainment Conference.

29

Statista. (2019). Leading esports games worldwide 2018, by tournament prize pool. Retrieved from:

https://www.statista.com/statistics/501853/leading-esports-games-worldwide-total-prize-pool/

Statista. (2019). Most played games on Steam 2018, by average players per hour. Retrieved from:

https://www.statista.com/statistics/656319/steam-most-played-games-average-player-per-

hour/

Superdata Research. (2018). Battle Royale. Retrieved from:

https://www.superdataresearch.com/reports/battle-royale-report/

Taylor, L. N. (2002). Video games: Perspective, point-of-view, and immersion. Retrieved from:

https://www.researchgate.net/profile/Laurie_Taylor2/publication/35486642_Video_games_ele

ctronic_resource_perspective_point-of-

view_and_immersion/links/551e9f450cf2a2d9e13c6f53/Video- games-electronic-resource-

perspective-point-of-view-and-immersion.pdf

The Guardian. (2017). eSports could be medal event at 2024 Olympics, Paris bid team says. Retrieved

from: https://www.theguardian.com/sport/2017/aug/09/esports-2024-olympics-medal-event-

paris-bid-committee

The Washington Post. (2018). The massive popularity of eSports, in charts. Retrieved from:

https://www.washingtonpost.com/business/2018/08/27/massive-popularity-esports-

charts/?noredirect=on&utm_term=.c818381877b9

Thorhallsson, H., & Singh, G. Visualizing the Bias-Variance Tradeoff.

Yang, P., Harrison, B. E., & Roberts, D. L. (2014). Identifying patterns in combat that are predictive

of success in MOBA games. In FDG.

Yang, Y., Qin, T., & Lei, Y. H. (2016). Real-time esports match result prediction. arXiv preprint

arXiv:1701.03162.

30

Appendix A

List of all features in dataset

player_name tracker_id solo_KillDeathRatio solo_WinRatio solo_TimeSurvived solo_RoundsPlayed solo_Wins solo_WinTop10Ratio solo_Top10s solo_Top10Ratio solo_Losses solo_Rating solo_BestRating solo_DamagePg solo_HeadshotKillsPg solo_HealsPg solo_KillsPg solo_MoveDistancePg solo_RevivesPg solo_RoadKillsPg solo_TeamKillsPg solo_TimeSurvivedPg solo_Top10sPg solo_Kills solo_Assists solo_Suicides solo_TeamKills solo_HeadshotKills solo_HeadshotKillRatio solo_VehicleDestroys solo_RoadKills solo_DailyKills solo_WeeklyKills solo_RoundMostKills solo_MaxKillStreaks solo_WeaponAcquired solo_Days solo_LongestTimeSurvived solo_MostSurvivalTime solo_AvgSurvivalTime solo_WinPoints solo_WalkDistance solo_RideDistance solo_MoveDistance solo_AvgWalkDistance solo_AvgRideDistance solo_LongestKill solo_Heals solo_Revives solo_Boosts solo_DamageDealt solo_DBNOs duo_KillDeathRatio duo_WinRatio duo_TimeSurvived duo_RoundsPlayed duo_Wins duo_WinTop10Ratio duo_Top10s duo_Top10Ratio duo_Losses duo_Rating duo_BestRating duo_DamagePg duo_HeadshotKillsPg duo_HealsPg duo_KillsPg duo_MoveDistancePg duo_RevivesPg duo_RoadKillsPg duo_TeamKillsPg duo_TimeSurvivedPg duo_Top10sPg duo_Kills duo_Assists duo_Suicides duo_TeamKills duo_HeadshotKills duo_HeadshotKillRatio duo_VehicleDestroys duo_RoadKills duo_DailyKills duo_WeeklyKills duo_RoundMostKills duo_MaxKillStreaks duo_WeaponAcquired duo_Days duo_LongestTimeSurvived duo_MostSurvivalTime duo_AvgSurvivalTime duo_WinPoints duo_WalkDistance duo_RideDistance duo_MoveDistance duo_AvgWalkDistance duo_AvgRideDistance duo_LongestKill duo_Heals duo_Revives duo_Boosts duo_DamageDealt duo_DBNOs squad_KillDeathRatio squad_WinRatio squad_TimeSurvived squad_RoundsPlayed squad_Wins squad_WinTop10Ratio squad_Top10s squad_Top10Ratio squad_Losses squad_Rating squad_BestRating squad_DamagePg squad_HeadshotKillsPg squad_HealsPg squad_KillsPg squad_MoveDistancePg squad_RevivesPg squad_RoadKillsPg squad_TeamKillsPg squad_TimeSurvivedPg squad_Top10sPg squad_Kills squad_Assists squad_Suicides squad_TeamKills squad_HeadshotKills squad_HeadshotKillRatio squad_VehicleDestroys squad_RoadKills squad_DailyKills squad_WeeklyKills squad_RoundMostKills squad_MaxKillStreaks squad_WeaponAcquired squad_Days squad_LongestTimeSurvived squad_MostSurvivalTime squad_AvgSurvivalTime squad_WinPoints squad_WalkDistance squad_RideDistance squad_MoveDistance squad_AvgWalkDistance squad_AvgRideDistance squad_LongestKill squad_Heals squad_Revives squad_Boosts squad_DamageDealt squad_DBNOs

31

Appendix B

Elbow method plot for duos game mode

Elbow method plot for squads game mode

32

Appendix C

Distribution duos game mode

Clustering duos game mode

33

Distribution squads game mode

Clustering squads game mode

34

Appendix D

Decision Tree for solo game mode

Decision Tree for duos game mode

35