Towards Security Climate Forecasts1

Towards the Future Internet 41 G. Tselentis et al. (Eds.) IOS Press, 2010 © 2010 The authors and IOS Press. All rights reserved. doi:10.3233/978-1-60750-539-6-41 Towards Security Climate Forecasts1 Stephan NEUHAUS 2 and Fabio MASSACCI Università degli Studi di Trento, Trento, Italy Abstract. The complexity and interdependencies of deployed software systems has grown to the point where we can no longer make confident predictions about the security properties of those systems from first principles alone. Also, it is very complex to state correctly all relevant assumptions that underlie proofs of security, so that attackers constantly seek to undermine these assumptions. Complexity metrics generally do not correlate with vulnerabilities and security best practices are usu- ally founded on anecdotes rather than empirical data. In this paper, we argue that we will therefore need to embrace empirical methods from other sciences that also face this problem, such as physics, meteorology, or medicine. Building on previous promising work, we suggest a system that can deliver security forecasts just like climate forecasts. Keywords. Large System Security, Empirical research, Climate forecasts Introduction Software systems have become so complex that we can no longer effectively understand the behaviour of the entire system from the behaviour of its smallest parts alone: in order to find out about the security of a complex system, it is no longer sufficient to know the precise semantics of an if or while statement. This is not because such knowledge is useless, but rather because the amount of interaction between program statements, interpreters, compilers, libraries, operating systems and the run-time environment creates an explosion in interconnection that makes predicting system behaviour uncertain at best and impossible at worst. This situation is worsening as we progress towards the Future Internet, where applications will consist of large numbers of interacting components, written by different organisations according to different development standards. Physicists face a similar problem when asked to describe the behaviour of an ideal gas: from Newtonian mechanics, it should be possible to compute the temperature or pressure of a gas by calculating the movements of individual gas moecules, but in prac- tice there are so many that this is impossible. Thus, high-level gas laws describing the behaviour of a gas as a whole were invented. In a similar way, medical researchers and meteorologists use models based on differential equations to forecast the spread of in- fectious diseases and to make climate forecasts, respectively. We propose to use a similar approach, where we no longer infer the macroscopic behaviour of a system from properties of its microscopic constituents such as library com- 1Research supported by the EU under the project EU-IST-IP-MASTER (FP7-216917) 2Corresponding author 42 S. Neuhaus and F. Massacci / Towards Security Climate Forecasts ponents, programming language statements or even machine instructions, but where we make statistical predictions about the system’s behaviour from simpler, higher-level metrics. Note that our point is not that it is only complexity, as understood by software engi- neers and expressed in complexity metrics, that is causing the inability to predict system behaviour. If that were the case, adding more computing power would make predictions possible again. Rather, it is that causes and effects of vulnerabilities are non-local (i.e., can occur at widely separated points in the same source code, in different source codes, between code in one system and code in a library, in the interaction between some code and the operating system, etc.) and are hence not accessible to source code complexity metrics. Vulnerability is a systemic issue. In the rest of this paper, we will first show case studies of two very large and widely deployed software systems (Section 1) in order to make the point that the sheer size of software makes it important to make reasonably precise predictions about the location of vulnerabilities: otherwise, quality assurance teams won’t know where to look. Next, we briefly review the state of the art in software security (Section 2), and then propose our solution to the problems that we identified (Section 3). We finish with the outline of a research program that could address these points, and conclude (Section 4). 1. Case Studies 1.1. Mozilla Mozilla3 is a large software system: As of 4 January 2007, Mozilla contained 1,799 directories and 13,111 C/C++ files, which can be combined into 10,452 components, which are files whose names only differ in their suffixes. These files have a large degree of interconnection: there are over ten thousand unique #include statements, and over ninety thousand unique function calls. At the same time, the Mozilla Foundation has published 134 Mozilla Foundation Security Advisories (MFSAs), which caused 302 bug reports. Of all 10,452 components, 424 or 4.05% were found to be vulnerable. [16] Studies by Shin and Williams [19] have looked at a particularly vulnerability-ridden part of Mozilla: the JavaScript engine. JavaScript is a way to execute code from within Web pages; it is executed inside a sandbox, which regulates all access attempts by the JavaScript program to the outside world. Still, attacks using JavaScript have shown an remarkable ability to break out of their sandboxes into the execution environment. Shin and Williams have looked at the correlation between code complexity metrics and vulnerabilities inside Mozilla’s JavaScript engine and found only low correlations; such correlations would not be enough to predict which components inside the JavaScript engine had as yet undiscovered vulnerabilities. This supports our point, since code complexity metrics operate under the assumption that vulnerabilities are local phenomena; after all, code complexity metrics are, by definition, computed at the source code level. Let us now put ourselves in the shoes of a Mozilla QA engineer. The new release is days away and we have the suspicion that there are components with undetected vulnerabilities. Yet, even with a large staff, we cannot hope to inspect comprehensively all 10,000 components: we have to make a choice, since our QA resources are finite. Out of 3http://www.mozilla.org/ S. Neuhaus and F. Massacci / Towards Security Climate Forecasts 43 Figure 1. Location of vulnerabilities in Mozilla source code. those 10,000 components we can perhaps choose ten for a thorough inspection. Which ten are going to be the lucky ones? Going only by personal experience, we would probably tend to use the anecdotal knowledge floating around in our QA team and look closely at those modules that already have reported vulnerabilities and are therefore known troublemakers. How good would such an approach be? In our own study of Mozilla [16], we we first mapped Mozilla Foundation Secu- rity Advisories (MFSAs) back to those source files that were fixed as a consequence of these MFSAs. From this purely empirical approach, we got two main results. The first is shown in Figure 1, which contains a graphical depiction of the distribution of vulnerabilities in Mozilla’s source code. There, named rectangles represent directories, unnamed rectangles represent components. The size of a rectangle is proportional to the size of the component in bytes, and a rectangle is shaded the darker the more vulnerabilities it has had. The JavaScript engine, the object of Shin and Williams’s study above, occu- pies the lower left-hand corner of the picture, and it is apparent that JavaScript is indeed vulnerability-ridden. But what can also be seen from the picture is that there are subsys- tems that are almost as bad, such as the “layout” subsystem (above the JavaScript rectangle), which is concerned with cascading style sheets and the like. Overall, the distribution of vulnerabilities is very uneven: there are also large parts that have no vulnerabilities. The second result is shown in Figure 2 (left). In this figure, we counted how many MFSAs applied to a vulnerable component. The most striking feature is that there are more than twice as many components with one fix than there are components with two or more fixes. This is apparent from the tall spike at the left, and the subsequent series of bars that decrease so rapidly in height that there is be no discernible height difference for 6 and 14 MFSAs. To return to our QA perspective, components with many vulnerabilities 44 S. Neuhaus and F. Massacci / Towards Security Climate Forecasts Distribution of MFSAs Distribution of RHSAs Number of Packages Number of Components 0 50 100 200 300 0 100 200 300 400 500 600 135791113 1 9 19 30 41 73 88 112 129 Number of MFSAs Number of RHSAs Figure 2. The number of components versus the number of MFSAs in Mozilla (left), and distribution of Red Har Security Advisories (right) are rare whereas components with only one vulnerability are common. Therefore, if we looked only at components with an established history of vulnerabilities, we would waste effort: past history of vulnerabilities is not a good predictor for future vulnerabilities. 1.2. Red Hat We also looked at the packages in Red Hat Linux [15]. It showed that out of a total of 3,241 packages, 1,133 or 35% had vulnerabilities reported in 1,646 Red Hat Security Advisories (RHSAs) between January 2000 and August 2008, inclusive. Again, there is no evidence that the developers of those packages were negligent, and again, the study shows that most vulnerable packages have one or two vulnerabilties, so again, finding promising candidates for QA measures will be difficult; see Figure 2 (right). For Red Hat, the situation is even more dire than for Mozilla. Mozilla is at least developed in a single language, C++.

Load more