A Move in the Security Measurement Stalemate: Elo-Style Ratings to Quantify Vulnerability Wolter Pieters Sanne H.G. van der Ven Christian W. Probst Delft University of Technology University of Amsterdam Technical University of Technology, Policy and Social and Behavioural Denmark Management Sciences Informatics and Mathematical Infrastructure Systems & Psychological Methods Modelling Services Weesperplein 4 Richard Petersens Plads P.O. Box 5015 NL-1018 XA Amsterdam DK-2800 Kongens Lyngby NL-2600 GA Delft The Netherlands Denmark The Netherlands [email protected] [email protected] [email protected] ABSTRACT 1. INTRODUCTION One of the big problems of risk assessment in information security is the quantification of risk-related properties, such 1.1 Problem Statement as vulnerability. Vulnerability expresses the likelihood that One of the big problems of risk assessment in information a threat agent acting against an asset will cause impact, for security is the quantification of risk-related properties, such example, the likelihood that an attacker will be able to crack as the expected frequency of threat events, vulnerability of a password or break into a system. This likelihood depends a system to such events, and impact on the system. In on the capabilities of the threat agent and the strength of this research area, the goal is not finding new threats or the controls in place. In this paper, we provide a framework attack scenarios (which is another big problem), but rather for estimating these three variables based on the Elo rating quantifying the risk of known ones. Simply counting incident used for chess players. This framework re-interprets secu- numbers is not very adequate though, especially if they are rity from the field of Item Response Theory. By observing self-reported [12]. Therefore, we need more detailed models the success of threat agents against assets, one can rate the of what constitutes a risk and how much damage to expect strength of threats and controls, and predict the vulnerabil- from it [22]. ity of systems to particular threats. The application of Item Among the different components of risk, the notion of Response Theory to the field of risk is new, but analogous vulnerability is particularly problematic. There are at least to its application to children solving math problems. It pro- three issues with the use of the term: vides an innovative and sound way to quantify vulnerability 1. The term is used in both countable and uncountable in models of (information) security. forms, to express two different things: (a) A particular security weakness in a system design: Categories and Subject Descriptors \This software has two buffer overflow vulnerabil- K.6.5 [Management of Computing and Information ities", or Systems]: Security and Protection (b) The ease with which a system can be damaged by threats: \This system has a higher vulnerability General Terms than that one"; Management, Measurement, Security 2. If we assume the second meaning, it is unclear how to define the vulnerability (level) of a system to a partic- ular threat; Keywords 3. Even if we know the vulnerability levels of a system to control strength, Elo, Item Response Theory, rating systems, specific threats, it is unclear how to define an aggregate risk assessment, security metrics, threat capability, vulner- vulnerability level of a system, to provide justification ability for the statement that one system is more vulnerable than another. In this paper, we deal with the second problem, namely Permission to make digital or hard copies of all or part of this work for the definition of the level of vulnerability of a system with personal or classroom use is granted without fee provided that copies are respect to a particular threat. This means that we will use not made or distributed for profit or commercial advantage and that copies vulnerability in the uncountable sense, i.e., the ease with bear this notice and the full citation on the first page. To copy otherwise, to which damage is caused by a threat. The solution we envi- republish, to post on servers or to redistribute to lists, requires prior specific sion is based on the psychometric approach of Item Response permission and/or a fee. NSPW’12, September 18–21, 2012, Bertinoro, Italy. Theory, related to the widely known Elo ratings used to rank Copyright 2012 ACM 978-1-4503-1794-8/12/09 ...$10.00. chess players. 1 1.2 Ingredients a method for estimating vulnerability, as a component of To define our concepts, we use the risk definitions pro- quantification of risk. Besides, the Elo approach has the ad- vided by The Open Group [15]. They assume the existence vantage that it also yields ratings of the different threat and of \threat agents" in the environment of a system that may control types, which will give information about the vulnera- cause damage to system assets. According to The Open bility of other assets to the same threat, or the vulnerability Group, vulnerability expresses the likelihood that a threat of the same asset to other threats. agent acting against an asset will cause impact. Vulnerabil- 1.4 Paper Outline ity is based on the \skills" of the threat (Threat Capability) and the \skills" of the defender (Control Strength). In Section 2, we discuss related work, including the use Thus, in a non-malicious context, a storm with a certain of Elo style ratings as well as information security mod- wind speed will have a certain likelihood of causing a power els. In Section 3, we provide the definitions of our concepts, line breakdown. This likelihood will increase with the wind based on the Risk Taxonomy of The Open Group [15]. In speed. On the other hand, different power lines will have dif- Section 4, we define the relation between threat capability, ferent resistance against storms, and the likelihood of break- control strength, and vulnerability. In Section 5, we discuss down will decrease with the resistance. how to use this model to estimate relevant variables in a risk There are three variables involved here: wind speed, re- context. In Section 6, we investigate the necessary infras- sistance, and breakdown. For each particular case of the tructure to support such applications. We end with open threat agent storm acting against the asset power line, we questions in Section 7 and conclusions in Section 8. may be able to obtain certain information. In particular, we may know the wind speed, the type of power line, and/or 2. RELATED WORK whether the \attack" results in a breakdown or not. For malicious threats, we may know the attacker capability, the 2.1 Rating Systems type of system, and/or the \success" of an attack. Given To enable the quantification of vulnerability, threat capa- this information, we would like to predict the likelihood of bility, and control strength, we make use of existing rating breakdown for future events (i.e., the vulnerability). How systems. Several existing rating systems enable rating of the to do this? capabilities of entities, and we will discuss their properties in In a very similar setting, we have the following problem. this section. Throughout the section, we use θ to represent We have a population of children who attempt so solve in- a rating where both entities have the same type, e.g., the stances from a population of math problems. Initially we Elo system for chess players. When the entities have two may know neither the capability of the children nor the dif- (different) types, we use δ for the difficulty of the problem ficulty of the problems. Can we, based on observing which or asset and β for the ability of the solver or the attacker. children are able to solve which problems, determine the This may not match the notation in the original papers. children's capabilities, and the difficulty of the problems? If Furthermore, the outcome of a \match" between two en- we can, then we can also predict the success of a particu- tities is 1 for the entity that \wins", 0 for the entity that lar child in solving a particular (previously unencountered) \loses", and 0.5 for each in case of a \draw". Other possible problem. outcome scales will be discussed later. Klinkenberg, Straatemeier, and Van der Maas [19] devel- A central assumption of the rating systems discussed is the oped just such a system for math problems, called Math following: if the ratings of two entities are equal, then the Garden, which is based on the Elo rating. The Elo rating expected result is 0.5 for each of them. If there is no draw was originally conceived to rank players in chess [8], but is option (the outcome is always 0 or 1), then we can state now also regularly applied outside the chess domain [9, 10]. equivalently that the likelihood of winning is 0.5, when the Each player has a rating that can be used to estimate the ex- ratings of the entities are equal. pected outcome of a particular match between two players. This rating is updated after each match: players' ratings 2.1.1 One Type, Dynamic Rating: The Elo System increase if they win, and decrease if they lose. The rating The Elo Rating System [8] is used for rating players in increases more if they win against a highly ranked player. chess. The system consists of two independent sets of equa- When applied to math problems, Elo works analogously, al- tions. The first set of equations is the update rule: after a though we now have two different types of entities: children match between two players, the ratings of these players are and problems.