Automatic Discovery of API-Level Vulnerabilities Vinod Ganapathyy, Sanjit A. Seshiaz, Somesh Jhay, Thomas W. Repsy, Randal E. Bryantz yComputer Sciences Department, zSchool of Computer Science, University of Wisconsin-Madison Carnegie Mellon University fvg|jha|
[email protected] fsanjit|
[email protected] UW-MADISON COMPUTER SCIENCES TECHNICAL REPORT: UW-CS-TR-1512, JULY 2004. Abstract A system is vulnerable to an API-level attack if its security can be compromised by invoking an allowed sequence of operations from its API. We present a formal framework to model and analyze APIs, and develop an automatic technique based upon bounded model checking to discover API-level vulnerabilities. If a vulnerability exists, our technique produces a trace of API operations demonstrating an attack. Two case studies show the efficacy of our technique. In the first study we present a novel way to analyze printf-family format-string attacks as API-level attacks, and implement a tool to discover them automatically. In the second study, we model a subset of the IBM Common Cryptographic Architecture API, a popular cryptographic key-management API, and automatically detect a previously known vulnerability. 1 Introduction Software modules communicate through application programming interfaces (APIs). Failure to respect an API's usage conventions or failure to understand the semantics of an API may lead to security vulnerabilities. For instance, Chen et al. [13] demonstrated a security vulnerability in sendmail-8.10.1that was due to a misunderstanding of the semantics of UNIX user-id setting commands. A programmer mistakenly assumed that setuid(getuid()) would always drop all privileges.