Testing Network-based Intrusion Detection Signatures Using Mutant Exploits Giovanni Vigna William Robertson Davide Balzarotti [email protected] [email protected] [email protected] Reliable Software Group University of California, Santa Barbara Santa Barbara, CA 93106 ABSTRACT 1. INTRODUCTION Misuse-based intrusion detection systems rely on models of Intrusion detection systems analyze one or more streams of attacks to identify the manifestation of intrusive behavior. events looking for the manifestations of attacks. For example, Therefore, the ability of these systems to reliably detect at- network-based intrusion detection systems (NIDSs) analyze tacks is strongly affected by the quality of their models, which network packets, while host-based intrusion detection sys- are often called “signatures.” A perfect model would be able tems (HIDSs) analyze audit data collected by an operating to detect all the instances of an attack without making mis- system about the actions performed by users and applica- takes, that is, it would produce a 100% detection rate with tions. 0 false alarms. Unfortunately, writing good models (or good The analysis of the event streams can be performed accord- signatures) is hard. Attacks that exploit a specific vulner- ing to different approaches. A common classification of ap- ability may do so in completely different ways, and writing proaches divides them into misuse detection approaches and models that take into account all possible variations is very anomaly detection approaches. In the first case, the analysis difficult. For this reason, it would be beneficial to have test- relies on models that represent intrusive behavior. The anal- ing tools that are able to evaluate the “goodness” of detection ysis process tries to identify series of events that conform to signatures. This work describes a technique to test and eval- these models and, therefore, represent an intrusion. In the uate misuse detection models in the case of network-based second case, the analysis uses models that characterize the intrusion detection systems. The testing technique is based normal behavior of a system and aims at identifying events on a mechanism that generates a large number of variations that do not fit the established models, in the assumption that of an exploit by applying mutant operators to an exploit tem- anomalous behavior is often evidence of malicious intent. plate. These mutant exploits are then run against a victim Network-based intrusion detection systems based on mis- host protected by a network-based intrusion detection system. use detection approaches are the most widely-deployed type The results of the systems in detecting these variations pro- of intrusion detection systems. For example, Snort [28] and vide a quantitative basis for the evaluation of the quality of ISS’s RealSecure [11], which represent the leading products the corresponding detection model. in the open-source and commercial worlds, respectively, are both network-based misuse detection systems. One problem with misuse detection systems is that their Categories and Subject Descriptors ability to reliably detect attacks is strongly affected by the C.2.0 [Computer-Communication Network]: Security quality of their models, which are often called “signatures.” and Protection A perfect model would be able to detect all the instances of the modeled attack without making mistakes. In technical terms, a perfect model would produce a 100% detection rate General Terms with 0 false alarms (also called false positives). Security Unfortunately, writing good models (or good signatures) is hard and resource-intensive. Attacks that exploit a certain vulnerability may do so in completely different ways. This Keywords problem could easily be solved by writing a model for each Security Testing, Intrusion Detection, Quality Metrics way in which the vulnerability can be exploited, or, even bet- ter, by creating a model which is abstract enough to capture all the different variations of an attack. For example, the work in [10] suggests that good models should consider only Permission to make digital or hard copies of all or part of this work for those events that if removed from the attack would make the personal or classroom use is granted without fee provided that copies are attack unsuccessful. Unfortunately, this is not always pos- not made or distributed for profit or commercial advantage and that copies sible, and sometimes by creating an abstract signature it is bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific possible to undermine its detection precision (i.e., the model permission and/or a fee. would also flag as intrusive perfectly normal activity). Also, CCS'04, October 25–29, 2003, Washington, DC, USA. the security expertise of the signature developer may have Copyright 2004 ACM 1-58113-961-6/04/0010 ...$5.00. a notable impact on the ability of the model to characterize 2. RELATED WORK the attack correctly. In the past few years, the problem of systematically testing If the models used in intrusion detection are known, it is intrusion detection systems has attracted increasing interest possible to examine them to identify possible “blind spots” from both industry and academia. that could be exploited by an attacker to perform the attack A class of intrusion detection evaluation efforts have sought while avoiding detection. Unfortunately, few commercial sys- to quantify the relative performance of heterogeneous intru- tems (if any) provide access to the models they use to detect sion detection systems by establishing large testbed networks intrusions. Even in the cases when these models are available, equipped with different types of IDSs, where a variety of ac- it is extremely time consuming to devise testing procedures tual attacks are launched against hosts in the testbed [14, 5, that analyze the models and identify blind spots. 4, 8]. These large-scale experiments have been a significant This work describes a technique to test and evaluate mis- benefit to the intrusion detection community. Practitioners use detection models in the case of network-based intrusion have gained quantitative insights concerning the capabilities detection systems. The testing technique is based on an au- and limitations of their systems (e.g., in terms of the rates of tomated mechanism to generate a large number of variations false positive and false negative errors) in a test environment of an exploit by applying mutant operators to an exploit intended to be an unbiased reproduction of a modern com- template. The mutant exploits are then run against a victim puter network. While generally competitive in flavor, these system where the vulnerable applications and/or operating evaluations have precipitated valuable intellectual exchanges systems are installed. The attacks are analyzed by a network- between intrusion detection practitioners [15]. based intrusion detection system. The intrusion alerts pro- Unfortunately, testing and comparing intrusion detection duced by the NIDS are then correlated with the execution of systems is difficult because different systems have different the mutant exploits. By evaluating the number of successful operational environments and may employ a variety of tech- attacks that were correctly detected, it is possible to get a niques for producing alerts corresponding to attacks [23, 26]. better understanding of the effectiveness of the models used For example, comparing a network-based IDS with a host- for detection. based IDS may be very difficult because the event streams Obviously, this technique does not provide a formal eval- they operate on are different and the classes of attacks they uation of the “goodness” of an attack model. Nonetheless, detect may have only a small intersection. For these reasons, we claim that this is a valid way to improve one’s confidence IDS testing and comparison is usually applied to homoge- in the generality of a detection model. Note that the tech- neous categories of IDSs (e.g., host-based IDSs). nique could be easily extended to host-based intrusion de- In this paper we are concerned with the black-box evalu- tection systems and to systems that use anomaly detection ation of the signatures of network-based intrusion detection approaches. Nonetheless, hereinafter we will limit the scope systems. This is a complementary approach with respect to of our analysis to network-based misuse detection systems. our previous research on using IDS stimulators (e.g., Mu- The mutation process is deterministic and guided by a seed cus [17], Snot [30], Stick [7], and IDSwakeup [2]) to per- value, which makes the mutations reproducible. The mutant form cross-testing of network-based signatures. In partic- operators are supposed to preserve the “effectiveness” of the ular, in [17] we used the set of signatures of a network- attack, that is, all the generated mutants are supposed to be based intrusion detection system to drive an IDS stimula- functional exploits. Unfortunately, both the exploits and the tor and generate test cases (i.e., traffic patterns that match attack targets may be very complex. Therefore, it is possible the signatures). These test cases were then analyzed using that a variant of an exploit becomes ineffective because of a different network-based intrusion detection system. This some condition that may be difficult (or impossible) to model. cross-testing technique provided valuable insights about how To address this issue, the technique relies on an oracle to network-based sensors detect attacks. However, its applica- determine if an attack has been successful or not. In most bility was limited by the lack of publicly available signature cases, the oracle mechanism can be embedded in the exploit sets. In fact, developers of closed-source systems believe that itself, for example by crafting an exploit so that it will gen- keeping their signatures undisclosed is an effective way to erate side effects that can be used to determine if the exploit protect the system from evasion techniques, over-stimulation was successful.