A Machine Learning Evaluation of an Artificial Immune System Matthew Glickman [email protected] Department of Computer Science, University of New Mexico, Albuquerque, NM 87131-1386, USA Justin Balthrop [email protected] Department of Computer Science, University of New Mexico, Albuquerque, NM 87131-1386, USA Stephanie Forrest [email protected] Department of Computer Science, University of New Mexico, Albuquerque, NM 87131-1386, USA and Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM 87501, USA Abstract ARTIS is an artificial immune system framework which contains several adaptive mechanisms. LISYS is a version of ARTIS specialized for the problem of network in- trusion detection. The adaptive mechanisms of LISYS are characterized in terms of their machine-learning counterparts, and a series of experiments is described, each of which isolates a different mechanism of LISYS and studies its contribution to the sys- tem’s overall performance. The experiments were conducted on a new data set, which is more recent and realistic than earlier data sets. The network intrusion detection problem is challenging because it requires one-class learning in an on-line setting with concept drift. The experiments confirm earlier experimental results with LISYS, and they study in detail how LISYS achieves success on the new data set. Keywords Anomaly detection, artificial immune systems, machine learning, immune system, net- work intrusion detection, computer security. 1 Introduction The natural immune system uses a variety of evolutionary and adaptive mechanisms to protect organisms from foreign pathogens and misbehaving cells in the body. Artifi- cial immune systems (AIS) seek to capture some aspects of the natural immune system in a computational framework, either for the purpose of modeling the natural immune system or for solving engineering problems. In either form, a fundamental problem solved by most AIS can be thought of as learning to discriminate between “self” (the normally occurring patterns in the system being protected, e.g., the body) and “non- self” (foreign pathogens, such as bacteria or viruses, or components of self that are no longer functioning normally). Almost any set of patterns that can be expressed as strings of symbols can be placed into this framework, for example, the set of normally occurring TCP connections in a local area network (LAN) and the set of TCP connec- tions observed during a network attack. This is the example on which we will focus in this paper. c 2003 by the Massachusetts Institute of Technology Evolutionary Computation x(x): xxx-xxx LISYS Hofmeyr introduced an artificial immune system framework called ARTIS, which he specialized for the problem of network intrusion detection in a system known as LISYS (Lightweight Intrusion detection SYStem) (Hofmeyr, 1999; Hofmeyr and Forrest, 2000). In that work, he focused on explaining the analogy between real immunology and the LISYS artifact and reported early experiments demonstrating how well the system performed. Hofmeyr’s results from this immune-inspired architecture were encouraging, and there are several related projects in in network intrusion detection based on AIS (Das- gupta, 1999; Williams et al., 2001; Kim and Bentley, 1999a). However, LISYS is a com- plex architecture, similar in complexity to learning classifier systems (Holland et al., 1986), and the task of understanding and predicting its behavior is challenging. The research reported in this paper is focused up three complementary goals: (1) to connect LISYS with the broader context of Machine Learning, (2) to further the empirical eval- uation of LISYS’ performance, and (3) to deepen understanding of the contributions made by LISYS’ many components to its overall performance. Machine Learning: LISYS sends an alert to an operator when it detects anomalous network packets, potentially indicating intrusion attempts. LISYS’ detection mecha- nism does not rely on preprogrammed knowledge. Rather, it observes an initial sam- ple of network traffic, builds a model of normal traffic patterns, and then compares the model against subsequent traffic. Because new connections frequently occur on a LAN, LISYS must generalize from previously observed traffic to assess whether newly observed packets are anomalous. This capacity—that of an artificial system to cope with novelty via generalization from experience—is the domain of Machine Learning (ML). Examining LISYS in the context of machine learning is important for emerging fields such as artificial immune systems, so that a common language can be identified and results communicated between fields. Here, we are interested in discovering what might be novel about LISYS from an ML point of view, and in what insights ML can give us about LISYS. In particular, one objective of this paper is to isolate the LISYS mechanisms so that they could potentially be used in other more traditional ML frame- works. LISYS introduces several novel learning mechanisms, and we show that some of them improve performance dramatically. At its core, LISYS resembles a simple memory-based (also called “instance-based”) learning algorithm. We examine each of LISYS’ mechanisms as extensions to this simple learning algorithm, and assess how each mechanism addresses issues of interest in the more general machine learning com- munity and computer security domain. Evaluation: A preliminary study was conducted in the context of a small and well controlled data set (Balthrop et al., 2002b; Balthrop et al., 2002a), which focused on a limited number of LISYS mechanisms. Here, we extend that work to a more realistic and larger data set, and and we consider the complete LISYS architecture. Our new data set represents data collected over a 62-day period from a subnet behind a firewall with 17 active machines. The scale of this network and its placement behind a masquerading firewall are typical of small enterprise networks. This setting, which is more controlled than the original data collected by Hofmeyr, allows us to isolate specific network events which affect LISYS performance and to be more certain that our core training data are free of attacks Experiments assessing individual mechanisms: Although the essential idea of LISYS is quite simple, there are a variety of mechanisms incorporated into the full sys- 2 Evolutionary Computation Volume x, Number x LISYS tem, each of which was originally inspired by some aspect of the biological immune system. The contributions to overall system performance made by each mechanism have until now been unclear. We report a number of new experiments designed to assess the relative contribution of LISYS’ many components. We begin by comparing the performance of LISYS with that of a “null algorithm” that performs no generalization from experience. We then conduct experiments with a set of variant algorithms covering the spectrum between the null algorithm and LISYS. In some cases, we incrementally add individual features into the null algorithm, while in others we subtract features from full LISYS (known as ablation experiments). As an ML algorithm, the performance of LISYS reflects the validity of its assump- tions about inference in its domain of application. Thus, the aggregation and ablation experiments reported here reveal the validity of the assumptions represented by each of LISYS’ individual mechanisms in the given network intrusion-detection domain. The remainder of the paper is organized as follows. Section 2 gives some back- ground material on intrusion-detection systems, machine learning used for intrusion detection, LISYS, and other immunologically inspired approaches to intrusion detec- tion. Section 3 examines LISYS from the perspective of machine learning; Section 4 describes the data set used for our experiments, and section 5 presents the experiments themselves. In section 6, we discuss the experimental results, LISYS’ current limita- tions, and ideas for extension and improvement. We summarize our conclusions in section 7. 2 Background Several areas of research are relevant to the work reported here. In this section, we briefly review earlier work on intrusion detection, discuss machine-learning ap- proaches to this application domain, give an overview of LISYS, and review other AIS approaches to network intrusion-detection. 2.1 Intrusion Detection Intrusion detection systems (IDS) vary widely, but they all seek to protect an informa- tion system (e.g., a single computer, a database, a server, or a network of computers) from violations of the system’s security policy. Debar et al. (Debar et al., 1999) defined a taxonomy of IDS, distinguishing between behavior-based (often called anomaly- based) and knowledge-based (also known as signature) IDS. LISYS is an example of an anomaly IDS in which a model is constructed of the normal (legal) operation of the system and discrepancies from the model are labeled anomalous. The model of normal behavior can be based on any observable behavior of the system. Audit logs, patterns of network traffic, user commands, and system-calls are all common choices. Such an ap- proach can work well if the anomalies in normal operation are correlated with security violations. The extent to which this is true is a topic of debate in the intrusion-detection community. A second relevant distinction is whether an IDS defines the set of allowable behav- iors (known as positive detection) or the set of disallowed behaviors (known as nega- tive detection) (Forrest et al., 1994; Esponda et al., 2004). Because knowledge-based sys- tems use knowledge of specific attacks or classes of attacks to identify intrusion events, we refer to them as negative-detection schemes. The knowledge can be represented many different ways. One common approach is to define a set of rules in a produc- tion system which is used to deduce the state of the monitored system. Examples of this approach include EMERALD (Porras and Neumann, 1997) and NSM (Heberlein Evolutionary Computation Volume x, Number x 3 LISYS et al., 1990; Mukherjee et al., 1994). Alternatively, knowledge of attacks is often com- piled into simple “signatures” or syntactic patterns which can be readily recognized in a data stream.