Understanding Cyber Risk Quantification A Buyer’s Guide Created for the FAIR Institute by Jack Jones, Chairman Sponsored by: Table of Contents Overview 3 Questions related to data 14 Questions related to analytics 16 Risk management program needs 3 Questions related to reporting 17 What is CRQ and how does it help? 4 Red flags 18 CRQ in a nutshell 4 “Risks” that aren’t loss event scenarios 18 CRQ value propositions 4 Scope overlap 18 Risk remediation cost-benefit analysis 5 Precise inputs and outputs 19 Other CRQ value propositions 5 Use of control standards 19 Common concerns regarding CRQ 6 Weighted values 20 Reliability 6 “Industry data” 20 Level of effort 9 “Eliminate guessing” 21 Other risk-related measurement approaches 9 “Know your organization’s risk in a very short time frame” 21 Numerically expressed ordinal risk measurement 10 Proprietary algorithms 21 Controls-focused assessments 11 Simplistic Aggregation 22 Vulnerability assessments 12 Spreadsheets 22 Credit-like scoring 12 Summary 22 Maturity model assessments 13 Addendum 24 Threat analysis 13 Questions checklist 24 Important questions 13 Red Flags checklist 26 Questions regarding utility 13 Overview Others who may benefit from this paper include industry analysts, consultants, regulators, cyber risk solution providers, and cyber risk The cyber risk landscape is increasingly impactful, complex and dynamic, and technology investors. organizations have limited resources to apply to the problem. Furthermore, every dollar spent on cyber risk management is a dollar that can’t be spent on other business or mission imperatives. This means that cyber risk Risk management program needs management programs must be able to effectively prioritize an organization’s Risk management programs exist to help their organizations cost-effectively cyber risk concerns and choose cost-effective solutions. Both of these require achieve and maintain an acceptable level of exposure to loss. Accomplishing the ability to measure cyber risk well. this — particularly within a complex and dynamic landscape like cyber risk — requires being able to do a number of things well: Unfortunately, there is a lot of confusion about cyber risk measurement methods — their inherent benefits and challenges, and what the qualities are 1. Identify/define the potential loss event scenarios that an of “good” cyber risk measurement. As a result, misperception is rampant and organization is exposed to it’s easy for organizations to leverage methods and solutions that may not fit their needs, or that may be severely flawed. This is true as well for cyber risk quantification (CRQ). 2. Understand the factors (assets, threats, and controls) that affect the probability and impact of those loss event scenarios, and how those factors interact The primary goal of this paper is to help organizations make better-informed decisions about whether CRQ is a good fit and, if so, choose a CRQ solution they can rely on. It will also provide a high-level overview of other cyber risk- 3. Continually monitor risk factor conditions related measurement approaches to provide some contrast between them and CRQ. 4. Given current and projected risk factor conditions, accurately estimate1 the probability of various loss event scenarios occurring, and This paper is organized to approximately align with the process an their likely impact if they do occur organization might go through when evaluating and selecting CRQ solutions. Specifically, it: 5. Compare current loss exposure levels against desired states ■ Discusses where cyber risk measurement fits within the cyber 6. Accurately identify opportunities to reduce risk when or where risk risk management landscape exceeds the desired state, or opportunities to increase risk when or where doing so supports other organization imperatives ■ Defines CRQ and describes its value proposition 7. Accurately and clearly communicate all of the above to appropriate ■ Reviews common concerns regarding CRQ stakeholders so that well-informed decisions can be made ■ Provides an overview of other risk-related measurement categories that are often confused with CRQ 8. Reliably execute the risk management decisions made by executive stakeholders ■ Proposes questions that should be asked of any CRQ solution provider Although this process may appear to be linear, in reality a risk management program will be doing many of these things constantly and in parallel. ■ Describes red flags that should be looked for in CRQ solutions Furthermore, because business needs evolve and the risk landscape is always in flux, the monitoring, measurement and reporting aspects of a 1 Because all measurements related to potential future events have some degree of uncertainty they are in fact estimates, regardless of how much or how little empirical data are used in the analysis Understanding Cyber Risk Quantification - A Buyer’s Guide 3 Copyright (c) 2019 FAIR Institute | All rights reserved program operate as a feedback mechanism. This enables risk management What is CRQ and how does it help? adjustments in response to changes that have occurred to the landscape, or that are expected to occur. Figure 1 below provides a high-level illustration of CRQ in a nutshell the risk management landscape. Well-established risk domains (e.g., credit risk, market risk, operational risk, and various insurance domains such as life, health, and property) have existed for some time to help organizations and individuals manage their exposure to loss — i.e., their risk. In these domains, risk is measured in terms of the probability and magnitude of loss from various scenarios (credit defaults, natural disasters, etc.). In the cyber risk domain there’s rarely any meaningful disagreement regarding risk measurement being a function of loss probability and magnitude. The more common question is what “quantification” looks like and the characteristics of good versus poor CRQ. Later in this paper we’ll discuss some of the measurement approaches that are often confused with CRQ. For now, simply recognize that loss event probability2 is expressed as a percentage (e.g., 10% probability of occurrence in the next 12 months) and magnitude is expressed as a loss of monetary value (e.g., $1.5M). When desired, these values can be combined to express risk as an annualized amount (e.g, $150,000)3. It’s important to recognize however, that measuring risk quantitatively shouldn’t be a goal in itself. What is most important is ensuring well-informed decisions through reliable and meaningful risk measurements (whether qualitative or quantitative). CRQ value propositions Prioritization Given the nature of this landscape, it is clear that the quality of risk management decision-making relies on both the Monitoring and Testing Being able to recognize which parts of the (data) and Analysis & Reporting (analytics) elements of a risk management risk landscape need to be addressed first is program. fundamental to managing loss exposure well. Historically, cybersecurity has leveraged qualitative measurements to categorize the significance of risk- Although the primary focus of this paper is on CRQ as an analysis and related concerns. However, even assuming that a reporting approach, questions related to data also will be discussed. qualitative measurement is accurate4, there remain some fairly obvious questions. For example, imagine that your organization has identified a set of risk- related concerns (Figure 2): 2 Loss event frequency is often a more useful term than probability, particularly with scenarios that can occur multiple times in a year. 3 In reality, to appropriately reflect measurement uncertainty, probability and impact should be expressed as ranges or distributions. 4 Many qualitative cyber risk measurements lack the rigor to be considered reliable. Understanding Cyber Risk Quantification - A Buyer’s Guide 4 Copyright (c) 2019 FAIR Institute | All rights reserved ■ How much more risk does the highest “high” (#1) represent than the probability or magnitude of the loss event scenarios an organization is lowest “high” (#2)? exposed to. Understanding which program elements affect which scenarios, and by how much, is foundational to effective risk management and ■ In a bucket of “high” risk-related concerns, how do you determine responsible resource stewardship. It is also instrumental in helping executive which one is highest and how would you defend that judgment? stakeholders understand the value to be gained (in terms of risk reduction) from cybersecurity investments. This information can only be achieved ■ How much more risk does the lowest “high” (#2) represent than the using CRQ. highest “medium” (#3). Other CRQ value propositions ■ What prevented #3 from landing in the “high” category, and how The ability to prioritize more effectively and choose risk remediation solutions would you defend that judgment? based on cost-benefit analyses is likely to be strong enough justification for many organizations to adopt CRQ. There are, however, other advantages as ■ If resources are applied to #1, how much less risk will result and how well, including: would you defend those results? ■ How much risk is there in total? ■ Senior executives are better able to understand the significance of cyber-related concerns when those concerns are expressed in terms that are similar to other dimensions of their problem space (e.g., ■ Why were the lines drawn where they were between high, medium, economic expressions of revenue, value, operational costs, and other and low risk? forms