ELO REGRESSION EXTENDING THE ELO RATING SYSTEM A Thesis Presented to The Graduate Faculty of The University of Akron In Partial Fulfillment of the Requirements for the Degree Master of Science Jonathan Dorsey May, 2019 ELO REGRESSION EXTENDING THE ELO RATING SYSTEM Jonathan Dorsey Thesis Approved: Accepted: Advisor Dean of the College Dr. Sujay Datta Dr. Linda Subich Faculty Reader Dean of the Graduate School Dr. Mark Fridline Dr. Chand Midha Department Chair Date Dr. Timothy O'Neil ii ABSTRACT The Elo rating system has long been a mainstay for comparing the strengths of chess players. The utility of the system lies in its computational simplicity and its prospective nature: the ratings for any given time period are calculated using only information from previous time periods. By calculating ratings for each time period of interest, Elo can be used to model the relative changes in players' skills over time in an ad hoc fashion. The main weakness of this approach is that the prospective nature of Elo means not all information is being used. The main purpose of this research is to display a modified version of the Elo rating system that better handles retrospective ratings. We achieve this by extending Elo to explicitly incorporate time as a covariate. We call our modified system \Elo regression." We find that Elo regression outperforms Elo, in terms of predicting the outcomes of held out games, on two different data sets, and performs equivalently on a third. A secondary purpose of this work is to display the benefits of reinterpreting pre- existing algorithms as particular methods for fitting a probabilistic model. Once an algorithm has been understood this way, principled modification of the algorithm is possible via changing the underlying model, the method of parameter estimation, and the method of calculating or approximating those estimates. iii TABLE OF CONTENTS LIST OF FIGURES . vi LIST OF TABLES . vii I. INTRODUCTION . 1 II. THE ELO RATING SYSTEM . 5 The Technical Details of Elo . 5 A Probabilistic View of Elo . 8 III. ELO REGRESSION . 13 Basis Expansion . 13 Penalization . 15 Minibatch Gradient Ascent . 15 Overview of Modifications . 16 Comparison to Elo . 17 IV. DATA..................................... 18 The Kaggle Chess Data . 18 Synthetic Data Set 1 . 20 Synthetic Data Set 2 . 23 V. METHODS . 26 Models Tested . 26 Measuring Performance . 26 iv Fitting the Models . 30 VI. RESULTS . 32 The Kaggle Chess Data . 33 Synthetic Data Set 1 . 33 Synthetic Data Set 2 . 33 VII. DISCUSSION . 34 General Comments . 34 The Kaggle Chess Data . 35 Synthetic Data Set 1 . 37 Synthetic Data Set 2 . 37 VIII. CONCLUSION . 39 REFERENCES . 42 v LIST OF FIGURES 1 Left: Gaussian radial basis functions with uniformly spaced centers plotted individually. Right: Random linear combinations of the basis functions on the left. 14 2 The number of games per month for the Kaggle data. 20 3 A histogram of the distribution of games per player. 21 4 Number of players entering the player pool each month. 21 5 Example skill curves from Synthetic Data Set 1. 22 6 Example skill curves from Synthetic Data Set 2. Blue curves are from group 1, while red curves are from group 2. 24 7 Some example rating curves for the Kaggle data. Blue is Elo, green is Elo regression, and yellow is constant. 36 8 Some example rating curves for the first synthetic data set. Blue is Elo, green is Elo regression, and yellow is constant. 37 9 Some example rating curves for the second synthetic data set. Blue is Elo, green is Elo regression, and yellow is constant. 38 vi LIST OF TABLES 1 Results of each model on each data set in terms of binomial deviance. 32 2 Results of each model on each data set in terms of 3-way classification accuracy. 33 vii CHAPTER I INTRODUCTION The Elo rating system, or just \Elo," is a method of assigning ratings to players based only on the outcomes of games, in order to assess their relative skills. Elo was invented by its namesake, physicist Arpad Elo, in the 1950s [9]. Since its adoption by the World Chess Federation (FIDE) in the 1970s, it has been the primary metric for measuring the strength of chess players. Chess ratings are very important to players, as FIDE uses ratings to determine invitations to tournaments, pairings at tournaments, and titles such as Grandmaster [5]. Elo is also applied to other zero-sum games, such as soccer and basketball [7, 15]. The fundamental idea behind Elo is that ratings can be used to predict the ex- pected outcomes of future games. After each game, the players' ratings can be up- dated based on the discrepancy between the actual outcome and the expected out- come. This way, the ratings evolve over time and correct themselves based on the observed outcomes. The main strength of the system is its computational simplicity, which allows chess federations to maintain the ratings of hundreds of thousands of players with modest computational resources [4]. Another benefit of Elo's simplicity is that players can in principle compute their rating updates themselves by consulting tables and using a pocket calculator. This gives the ratings a sense of fairness since the details of the 1 system are not hidden from the average player behind complicated mathematics and algorithms. Until now, we have been focusing on prospective uses of Elo ratings. That is, looking to the future to decide who should be invited to a tournament, or who will be worthy of the title of Grandmaster. This is what chess federations use Elo ratings for. Elo is also used to retrospectively or historically to analyze players' abilities over time [9, 1, 2]. This exercise is of general historical interest to chess players and fans, since they want to compare chess players across history and find the players with the greatest legacies. This is a very different way to use ratings. As an analogy, consider the differences between singling out the best and brightest current scientists and deciding whether, in hindsight, a scientist's work has been historically important enough to warrant a lifetime achievement award. What seems important in the moment may not pan out in the future. This is similar to the situation in chess ratings, where the players who seem the best right now turn out, with the benefit of hindsight, to be overrated. Thus, a rating system must decide to what extent new information informs only new ratings, or whether to use this new information to try to retroactively correct old ratings. We call any system that uses new information to retroactively correct old ratings a retrospective rating system. A system which never changes its past ratings, that is, it only uses information from the past to predict the future, we call a prospective rating system. The decision about how much to retroactively adjust ratings is implicitly an assumption about how quickly skill can change over time. If skill can change arbitrarily quickly, then there is no justification for revising old ratings. However, if we believe there are constraints on how quickly skill can change, then when a player's rating quickly drops, we may infer that it must have been too high to begin with. As we mentioned above, Elo can be used to model skill over time: since a player's rating changes after each game, keeping track of his or her rating over time provides 2 some measure of how that player's skill is changing. Here and throughout the rest of the paper we use skill to refer to the \true" underlying ability of the player, and rating to refer to an estimate of skill. We put \true" in quotes because it is doubtful that any single number accurately reflects all aspects of chess ability. The fact that Elo is a prospective rating system, which, to reiterate, means that at each time period it only uses data from the past to predict the future, limits its accuracy for historical analysis, where we are concerned with finding the best ratings for each time period using data from all time periods. Moreover, the Elo system does not explicitly attempt to model the dependence of skill on time. The relationship is the ad hoc result of the Elo updating method. In this paper, we propose a way to combine the core underlying model of Elo with linear regression, so that we may explicitly regress skill on time. We call the resulting model \Elo regression." Using an appropriate basis expansion with linear regression allows for flexible modeling of any relationship between skill and time, not merely straight line relationships. By making the relationship between skill and time more formal, we can get better predictive performance on held out data, which we use as a proxy for learning the underlying skills of the players. We also gain more control over the assumptions of the model, for example, through the basis expansion we choose, which lets us incorporate domain knowledge and common sense. Another benefit is that the model is conceptually very simple, since it is the combination of the simple Elo system with the well-understood technique of linear regression. The main downside of this approach is the heavy computational burden, which will be discussed below. Several other chess rating systems have been proposed as improvements over the Elo system. Perhaps the most well-known are the Glicko and Glicko2 rating systems [10, 11]. There is also the TrueSkill system, which was originally designed for rat- ing Xbox players [12]. Both the Glicko rating systems and TrueSkill are based on a Bayesian model that accounts for uncertainty in the player's current rating.