First Principles Vulnerability Assessment ∗ James A. Kupsch Barton P. Miller Eduardo Cesar´ Elisa Heymann University of Wisconsin Universitat Autonoma` de Barcelona {kupsch,bart}@cs.wisc.edu {Eduardo.Cesar,Elisa.Heymann}@uab.es September 2009 Abstract Vulnerability assessment is a key part of deploying secure software. As part of a our effort to secure critical middleware, we have developed a new approach called First Principles Vulnerability Assessment (FPVA). FPVA is a primarily analyst-centric (manual) approach to assessment, whose aim is to focus the analyst’s attention on the parts of the software system and its resources that are mostly likely to contain vulnerabilities that would provide access to high-value assets. In addition, our approach is designed to find new threats to a system, and is not dependent on a list of known threats. We have applied FPVA to several large and widely-used middleware systems, including the Con- dor high-throughput scheduling system, the Storage Resource Broker (SRB) data grid management system from San Diego Supercomputer Center, and the MyProxy public key credential management system. The FPVA studies of these software systems produced a significant list of serious vulnerabil- ities (and fixes), which were then fed back to the development teams to remediate the vulnerabilities. In addition, we used this opportunity to help improve the development teams’ coding strategies and develop vulnerability reporting, response, patch and release strategies. Manual assessment is a labor-intensive activity, and therefore the development and use of auto- mated assessment tools is quite attractive. Unfortunately, while these tools can find common prob- lems in a program’s source code, they miss a significant number of serious vulnerabilities found by FPVA. When we compared the results of our FPVA to that of the top commercial tools, we found that these tools miss most of the serious vulnerabilities found by FPVA. As a future effort, we will use the results of this comparison study to guide our future research into improving the automated software assessment. In summary, the key findings from our study include: • A new architecture- and resource-based analysis that is not dependent on known vulnerabilities. • A clear reminder that assessment must be an independent activity, done by analysts separate from the software development team. In addition, assessment must be part of the normal software development lifecycle. • A demonstration of the strengths and weaknesses of automated assessment tools by comparing their results against those of a thorough FPVA study (and not just comparing the automated tools against each other). • Several assessment activities of key middleware systems, resulting in significant improvements in the security of these systems, and similar improvements by creating a security-aware atmo- sphere among the software developers. • A foundation and new approach for future research into improved automated vulnerability assessment tools. • Tutorial materials to help train a new generation of software security analysts. ∗This research funded in part by National Science Foundation grant OCI-0844219, NATO grant CLG 983049, the National Science Foundation under contract with San Diego Supercomputing Center, and National Science Foundation grants CNS- 0627501 and CNS-0716460. 1 Table of Contents 1 Introduction 3 2 Methodology 4 3 Integration with Software Development and Release 6 3.1 The Assessment Activity . 6 3.2 Integration into the Release Cycle . 7 3.3 Notification . 7 4 Manual Assessment Test Cases 7 4.1 Condor . 8 4.1.1 Architectural Analysis . 8 4.1.2 Controlled/Accessed resources . 11 4.1.3 Trust and Privileges . 12 4.1.4 Component Analysis . 12 4.2 SRB . 15 4.2.1 Architectural Analysis . 15 4.2.2 Controlled/Accessed resources . 17 4.2.3 Trust and Privileges . 18 4.2.4 Component Analysis . 18 4.3 MyProxy . 20 4.3.1 Architectural Analysis . 20 4.3.2 Controlled/Accessed resources . 22 4.3.3 Trust and Privileges . 23 4.3.4 Component Analysis . 24 4.4 GLExec . 25 4.4.1 Architectural Analysis . 25 4.4.2 Controlled/Accessed resources . 27 4.4.3 Trust and Privileges . 27 4.4.4 Component Analysis . 27 4.5 Quill . 28 4.5.1 Architectural Analysis . 29 4.5.2 Controlled/Accessed resources . 31 4.5.3 Trust and Privileges . 32 4.5.4 Component Analysis . 32 5 Comparison with Automatic Tools 33 5.1 Methodology . 33 5.2 Coverity and Fortify . 34 5.3 Analysis of Results . 36 6 Agenda 37 6.1 Research . 37 6.2 Transition and Dissemination Agenda . 38 7 Related Work 38 7.1 Microsoft Methodology . 38 7.2 Related Projects . 39 7.3 Analysis Tools . 40 8 Conclusions 42 9 Acknowledgments 43 References 43 Appendix A Condor Vulnerability Reports 48 Appendix B SRB Vulnerability Reports 80 Appendix C MyProxy Vulnerability Reports 112 Appendix D GLExec Vulnerability Reports 122 2 1 Introduction Middleware ties together the critical Internet computing resources of commerce, government, and sci- ence. The components of this middleware communicate over high speed networks and span many or- ganizations, forming complex distributed systems. This middleware has a diverse and potentially large number of components, often with a myriad of interactions. The security of middleware directly effects the safety and integrity of these organizations and their computer systems. Assessment of the middleware is a key part of securing these systems. In this paper, we describe a new approach to manual (analyst cen- tric) vulnerability assessment, called First Principles Vulnerability Assessment (FPVA). This technique allows us to evaluate the security of a system in depth. We assume access to the source code, docu- mentation, and, when available, the developers. While FPVA is certainly a labor intensive approach to vulnerability assessment, it has been shown to be effective in real systems, finding serious vulnerabilities that were not discoverable through the use of the best automated tools of today [56]. Rather than working from known vulnerabilities, the starting point for FPVA is to identify high value assets in a system, i.e., those components (for example, processes or parts of processes that run with high privilege) and resources (for example, configuration files, databases, connections, and devices) whose exploitation offer the greatest potential for damage by an intruder. From these components and resources, we work outward to discover execution paths through the code that might exploit them. This approach has a couple of advantages. First, it allows us to find new vulnerabilities, not just exploits based on those that were previously discovered. Second, when a vulnerability is discovered, it is likely to be a serious one whose remediation is of high priority. FPVA starts with an architectural analysis of the source code, identifying the key components in a distributed system. It then goes on to identify the resources associated with each component, the privilege level of each component, the value of each resource, how the components interact, and how trust is delegated. The results of these steps are documented in clear diagrams that provide a roadmap for the last stage of the analysis, the manual code inspection. In addition, the results of this step can also form the basis for a risk assessment of the system, identifying which parts of the system are most immediately in need of evaluation. After these steps, we then use code inspection techniques on the critical parts of the code. Our analysis strategy targets the high value assets in a system and focuses attention on the parts of the system that are vulnerable to not only unauthorized entry, but unauthorized entry that can be exploited. We describe the FPVA methodology in more detail in Section 2. In addition to the analysis task, there needs to be a way to integrate the detection and repair of security flaws into the software development and release process. The software development process must now include vulnerability reporting, release integration, and a policy on public dissemination of the vulnerabilities. Open source software offers specials challenges for this last item. We discuss the effect of vulnerability assessment on the development process in Section 3. We have applied FPVA to several major systems, including the Condor high-throughput scheduling system [15, 59, 88], the Storage Resource Broker (SRB) data grid management system from San Diego Supercomputer Center [85, 8], and the MyProxy public key credential management system [66, 69]. In these analyses, significant vulnerabilities were found, software development teams were educated to the key issues, and the resulting software was made more resistant to attack. The details of these assessments and a summary of current ongoing assessments are reported in Section 4. A major question when evaluating a technique such as FPVA is: why not use an automated code analysis tool? To address this question, we surveyed security practitioners in academia, industry, and government laboratories to identify which tools were consider “best of breed” in automated vulnerability assessment. Repeatedly, two commercials software packages were named, Coverity Prevent [17] and Fortify Source Code Analyzer (SCA) [32]. To evaluate the power of these tools and better understand our new FPVA, we compared the results of our largest assessment activity (on Condor) to the results gathered 3 from applying these automated tools. As a result of these studies, we found that the automated tools found few of the vulnerabilities that we had identified in Condor (i.e., had significant false negatives) and identified many problems that were either not exploitable or led to minor vulnerabilities (i.e., had many false positives). The results from these studies are presented in Section 5. Our current work is just the start of a longer term effort to develop more effective assessment tech- niques. We are using our experiences with these techniques to help design tools that will simplify the task of manual assessment.