Adaptive Real-Time Anomaly Detection with Improved Index and Ability to Forget∗
Kalle Burbeck and Simin Nadjm-Tehrani Department of Computer and Information Science Linkoping¨ university SE–581 83 Linkoping,¨ Sweden
E-mail: {kalbu, simin}@ida.liu.se
Abstract improve information quality. By correlation the strength of different types of detection schemes may be combined, and Anomaly detection in IP networks, detection of devia- weaknesses compensated for. tions from what is considered normal, is an important com- The main detection scheme of most commercial intru- plement to misuse detection based on known attack descrip- sion detection systems is misuse detection, where known tions. Performing anomaly detection in real-time places bad behaviours (attacks) are encoded into signatures. In hard requirements on the algorithms used. First, to deal anomaly detection normal behaviour of users or the pro- with the massive data volumes one needs to have efficient tected system is modelled, often using machine learning or data structures and indexing mechanisms. Secondly, the data mining techniques rather than given signatures. Dur- dynamic nature of today’s information networks makes the ing detection new data is matched against the normality characterization of normal requests and services difficult. model, and deviations are marked as anomalies. Since no What is considered as normal during some time interval knowledge of attacks is needed to train the normality model, may be classified as abnormal in a new context, and vice anomaly detection may detect previously unknown attacks. versa. These factors make many proposed data mining tech- Anomaly detection still faces many challenges, where niques less suitable for real-time intrusion detection. In this one of the most important is the relatively high rate of false paper we extend ADWICE, Anomaly Detection With fast In- alarms (false positives). The problem of capturing a com- cremental Clustering. Accuracy of ADWICE classifications plex normality makes the high rate of false positives intrin- is improved by introducing a new grid-based index, and its sic to anomaly detection except for simple problems. We ar- ability to build models incrementally is extended by intro- gue that the usefulness of anomaly detection is increased if ducing forgetting. We evaluate the technique on the KDD combined with further aggregation, correlation and analysis data set as well as on data from a real (telecom) IP test of alarms, thereby minimizing the number of false alarms network. The experiments show good detection quality and propagated to the administrator (or automated response sys- illustrate the usefulness of adapting to normality. tem) that further diagnoses the scenario. The fact that nor- mality changes constantly makes the false alarm problem even worse. When normality changes over time, a model 1. Introduction with acceptable rates of false alarms may rapidly deterio- rate in quality. To minimize this problem and the resulting Intrusion detection is an important part of computer sys- additional false alarms, the anomaly detection model needs tem defence. Due to increasing complexity of the intru- to be adaptable. sion detection task, the use of many IDS sensors to increase The training of the normality model for anomaly detec- coverage, and the need for improved usability of intrusion tion may be performed by a variety of different techniques detection, a recent trend is alert or event correlation [3, 6]. and many approaches have been evaluated. One impor- Correlation combines information from multiple sources to tant technique is clustering, where similar data points are ∗This work was supported by the European project Safeguard [13] IST– grouped together into clusters using a distance function. In 2001–32685. We would like to thank Thomas Dagonnier, Tomas Lingvall this paper we develop a new indexing mechanism for AD- and Stefan Burschka at Swisscom for fruitful discussions and their many WICE (Anomaly Detection With fast Incremental Cluster- months of work with the test network. The work is currently supported by a grant from CENIIT, Center for Industrial Information Technology at ing), an adaptive anomaly detection scheme inspired by the Linkoping¨ university. BIRCH clustering algorithm [15]. We improve an earlier version of the algorithm used for anomaly detection [2] in anomaly detection in this paper, training data is assumed two ways: (1) a novel search index that increases precision to consist only of normal data. Munson and Wimer [10] detection quality, and (2) new means to handle adaptation used a cluster-based model (Watcher) to protect a real web of the model (forgetting). This paper does not attempt to server, proving anomaly detection based on clustering to be justify the quality of clustering within anomaly detection useful in real life. ADWICE uses pure anomaly detection to or compare performance with other machine learning work. reduce the training data requirement of classification-based The interested reader is referred to the original report for methods and to avoid the attack volume assumption of un- that purpose [2]. supervised anomaly detection. By including only normal The paper is organised as follows. In section 2 the moti- data in the detection model the low accuracy of pure anom- vation of this work is presented together with related work. aly detection can be significantly improved. Section 3 describes the indexing technique, and evaluation In a real live network with connection to Internet, data is presented in section 4. The final section discusses the re- can never be assumed to be free of attacks. Pure anomaly sults and describes opportunities and challenges for future detection also works when some attacks are included in the work. training data, but those attacks will be considered normal during detection and therefore not detected. To increase de- 2 Motivation and Related Work tection coverage, attacks should be removed from the train- ing data to as large an extent as possible, with a trade-off between coverage and data cleaning effort. Attack data can One fundamental problem of intrusion detection re- be filtered away from training data using updated misuse de- search is the limited availability of good data to be used for tectors, or multiple anomaly detection models may be com- evaluation. Producing intrusion detection data is a labour bined by voting to reduce costly human effort. intensive and complex task involving generation of normal One of the inherent problems of anomaly detection in system data as well as attacks, and labelling the data to general is the false positives rate. In most realistic settings make evaluation possible. If a real network is used, the normality is hard to capture and even worse, is changing problem of producing good normal data is reduced, but then over time. Rather than making use of extensive periodi- the data may be too sensitive to be released to other re- cal retraining sessions on stored off-line data to handle this searchers publicly. Learning-based methods require data change, ADWICE is fully incremental making very flexible not only for testing and comparison but also for training, on-line adaptation of the model possible without destroying resulting in even higher data requirements. The data used what is already learnt. for training needs to be representative for the network to An intrusion detection system in a real-time environment which the learning-based method will be applied, possibly needs to be fast enough to cope with the information flow, requiring generation of new data for each deployment. have explicit limits on resource usage, and adapt to changes Classification-based methods [4, 9] require training data in the protected network in real-time. Many proposed clus- that contains normal data as well as good representatives of tering techniques require quadratic time for training [7], those attacks that should be detected, to be able to separate making real-time adaptation of a cluster-based model hard. attacks from normality. Producing a good coverage of the They may also not be scalable, requiring all training data to very large attack space (including unknown attacks) is not be kept in main memory during training, limiting the size of practical for any network. Also the data needs to be labelled the trained model. We argue that it is important to consider and attacks to be marked. One advantage of clustering- scalability and performance in parallel to detection quality based methods [5,10,12,14] is that they require no labelled when evaluating algorithms for intrusion detection. Most training data set containing attacks, significantly reducing work on applications of data mining to intrusion detection the data requirement. There exist at least two approaches. consider those issues to a very limited degree or not at all. When doing unsupervised anomaly detection [5, 12, 14] ADWICE is scalable since compact cluster summaries are a model based on clusters of data is trained using unlabelled kept in memory rather than individual data. ADWICE also data, normal as well as attacks. If the underlying assump- has better performance compared to similar approaches due tion holds (i.e. attacks are sparse in data) attacks may be to the use of local clustering with an integrated improved detected based on cluster sizes, where small clusters cor- search index. respond to attack data. Unsupervised anomaly detection is a very attractive idea, but unfortunately the experiences so far indicate that acceptable accuracy is very hard to obtain. 3 The Anomaly Detection algorithm Also, the assumption of unsupervised anomaly detection is not always fulfilled making the approach unsuitable for at- This section describes how ADWICE handles training tacks such as denial of service (DoS) and scanning. and detection, more details on the old index may be found In the second approach, which we simply denote (pure) in the original paper [2]. The present implementation re- quires data to be numeric. Data is therefore assumed to be Case 2: If the size (number of clusters) of the model has transformed into numeric format by pre-processing. reached the maximum M we need to reduce the model size by allowing clusters to grow. The threshold T is increased, 3.1 Basic concepts relaxing the threshold requirement and the model is rebuilt. When rebuilding the model, the old cluster features are re- moved and reinserted into the model. Due to the increase The basic clustering concepts are presented in the orig- of T , previously close (similiar) clusters may be merged, inal BIRCH paper [15] and the relevant parts are summa- thereby reducing the size of the model. The new data point rized here. An ADWICE model consists of a number of v is then inserted into the new model. clusters, a number of parameters, and a tree index in which the leaves contain the clusters. A central idea of BIRCH Rebuilding the model requires much less effort than the inherited by ADWICE is to store only condensed informa- initial insertion of data since only cluster features rather tion (cluster feature) instead of all data points of a cluster. than individual data points are inserted. If the increase of A cluster feature is a triple CF =(n, S,SS) where n is T is too small, a new rebuild of the tree may be needed to the number of data points in the cluster, S is the linear sum reduce the size below M again. A heuristic described in of the n data points and SS is the square sum of all data the original BIRCH paper [15] may be used for increasing points. From now on we represent clusters by cluster fea- the threshold to minimize the number of rebuilds, but in this tures (CF). work we use a simple constant to increase T conservatively (to avoid influencing the result by the heuristic). Given nd-dimensional data vectors vi|i =1 ...n in a cluster CF the centroid v0 is defined as v0 = i vi/n, Below is an algorithmic description of the training phase which may be computed given only the summary CF.The of ADWICE, in which only the main points of the algorithm distance between a data point v and a cluster CF is the are presented and some simplifications made to facilitate Euclidian distance between v and the centroid, denoted presentation. Note also that the index is abstracted away at D(v,CF) while the distance between two clusters CFi and this point. CFj is the Euclidian distance between their centroids, de- 1: procedure TRAIN(v,model) noted D(CFi,CFj). If two clusters CFi =(ni, Si,SSi) 2: closestCF = findClosestCF() and CFj =(nj , Sj ,SSj) are merged, the CF of the result- 3: if thresholdRequirementOK(v,closestCF ) then ing cluster may be computed as (ni + nj, Si + Sj ,SSi + 4: merge(v,closestCF ) SSj). This also holds if one of the CFs is only based on one 5: else data point making incremental update of CFs possible. 6: if size(model) Dimension d1slices Non-leaf 1 3 0 0.33 0.67 1 node 0 d1 1 3 b Leaf node a 1 0.25 v v Data point c a b d 2 d2 2 4 e Cluster 0.5 3 e CF Slice number Slice 0.75 Hash Dimension d2 slices c d e e 4 mapping 1 1 2 3 Slice number Figure 1. Basic notions of the grid and grid tree approach that our current implementation adopts. An alter- 3.5 Adaptation of the normality model native or complementary approach to handle this situation is to rebuild the tree using a smaller width of the intervals Below we describe two scenarios in which it is very use- for each dimension. ful to have an incremental algorithm in order to adapt to changing normality. 3.4 Detection Scenario 1 New cases of normality require the model to The basic detection procedure is the same for any choice adapt incrementally. In some settings, it may be useful to of index and thus similar to the original index [2]. During let the normality model relearn autonomously. If normal- the detection the model is searched for the closest cluster ity drifts slowly, an incremental clustering algorithm may (v ) CFi (using the index). Then the distance D ,CFi from handle this in real-time during detection by incorporating v the centroid of the cluster to the new data point is com- every test data classified as normal with a certain confidence puted. Informally, if D is small, i.e. lower than a limit, into the normality model. If slower drift of normality is re- v v is similar to data included in the normality model and quired, a subset of those data based on sampling could be v should therefore be considered normal. If D is large, is incorporated into the normality model. Even if autonomous an anomaly. relearning is not allowed in a specific network there is need Let the threshold T be the limit (L) used for detection. for model adaptation. Imagine that the ADWICE normal- Using two parameters E1 and E2, MaxL = E1 ∗ L and ity model has been trained, and is producing good detec- MinL = E2 ∗ L may be computed. Then we compute the tion results for a specific network during some time inter- belief b that v is anomalous using the formula below: val. However, in an extension of the interval the admin- istrator recognizes that normality has changed and a new class of data needs to be included as normal. Otherwise this 0 if D ≤ MinL new normality class produces false positives. Due to the b = 1 if D ≥ MaxL (1) D−MinL incremental property, the administrator can incorporate this if MinL 4 Evaluation 1 0,95 In all following experiments ADWICE with a grid index 0,9 is used unless otherwise stated. 0,85 0,8 We choose to start evaluation of detection quality using Detection rate Detection 0,75 the KDDCUP99 intrusion detection data set [11] to be able 0,015 0,02 0,025 0,03 0,035 to compare the results with earlier evaluations [2]. Despite False positives rate the shortcomings of the DARPA related data sets [8] they have been used in at least twenty research papers and are un- Figure 3. Detection quality of ADWICE on fortunately currently the only openly available network data KDD data set commonly used for comparison purposes. The purpose of this first evaluation is a proof of concept for anomaly de- tection with ADWICE using a large feature set in real-time, testing the grid index. days data is used for training an initial model, while the fol- Since ADWICE needs only normal data for training at- lowing seven days of data are used for testing. The features tacks in the KDD training data set are removed resulting include source and destination IP addresses and port num- in 972 781 session records (41 fields summarizing a TCP bers, time of day, connection length, bytes transferred and a session) used to train the ADWICE model. This model is flag indicating a parsing error. evaluated on the KDD testing set consisting of 311 029 ses- sion records of normal data and many instances from 37 4.1 Detection quality of ADWICE different attack types. With a model size of 12 000 clusters, mean processing In all we have evaluated three different versions of AD- time per testing instance was 0.5-0.6 ms in off-line experi- WICE as shown in Figure 3. The trade-off between detec- ments on 1.8 GHz P3 computer with 512 MB RAM illus- tion rate and false positives rate are realised by changing the trating the good performance of the old ADWICE model. detection parameter E. Figure 2 shows that the primitive search operation of the ADWICE-TRR [2] is base-line with a training phase new grid-index (hashing) is 6-10 times faster than the prim- very similar to the original clustering algorithm BIRCH. itive operation of the old index (Euclidian distance, D) de- ADWICE-TRD [2] improves the first implementation while pending on the number of dimensions (60-20 in Figure 2). still using the original index. ADWICE-Grid uses the new Full performance evaluation of the new index depends on grid index rather than the original BIRCH index, elimi- the parameters and the function getSplitDimension() and nating the index-misses and thereby further improving the is left as future work. The use of 972 781 elements of 41 result. A comparison with a classification based method dimensional data in the normality model illustrates the scal- and unsupervised anomaly detection is present in earlier ability of ADWICE. work [2]. To illustrate forgetting and incremental training in a more realistic setting, a data set generated by the tele- 4.2 Incremental training com management test network in the European Safeguard project [13] was used, at the time of evaluation consisting An important property of ADWICE is that the original of 50 real machines (now more than 100). A period of three model, known to truly reflect recent normality, does not original model incremental 1 incremental 2 forgetting no forgetting 12 5 10 4 8 3 6 2 4 1 2 0 Number of alarms of Number Number of alarms of Number 0 0 20 40 60 80 100 120 0 20 40 60 80 100 120 Number of 2 hour periods from start of testing Number of 2 hour periods from start of testing Figure 4. Adapting to changing normality Figure 5. Adapting to changing normality us- with incremental training ing forgetting need to be retrained as soon as new cases of normality are 4.3 Forgetting encountered. We evaluate this on data generated in the Safe- guard test network. The three days of training data is used In this section we illustrate the use of forgetting. A to train an initial model which is then used for detection on model is trained on data from three days of data and is then the seven days of testing data. When certain types of traf- used for detection with and without forgetting on the fol- fic (new cases of normality) start producing false alarms the lowing seven days of data. Figure 5 shows alarms for one administrator may tell ADWICE to incrementally learn the instance of traffic (host x.x.202.73, port 137) that ceases to data causing those alarms to avoid similar false alarms in be present in the (normal) testing data, making that kind the future. Figure 4 shows alarms for host x.x.202.183 in of traffic anomalous. With forgetting this fact is reflected three scenarios. Period number 1 starts at time 2004-03- in the normality model. In this experiment a CheckPeriod 11 00:00 (start of testing data) and each two-hour period of 12 hours and RememberPeriod of 3 days (72 hours) are presents the sum of alarms related to host x.x.202.183 dur- used. ing the corresponding time interval. At 2004-03-14 12:00, When traffic from host x.x.202.73 on port 137 is again corresponding to period number 43, the host is connected to visible in data (periods 55-66) the traffic is detected as the network. anomalous. Without forgetting these anomalies would go In the first scenario, no incremental training is used, and undetected. the testing is performed on the original model. This corre- sponds to the first curve of Figure 4. We see that when the 5 Discussion and future work host connects, ADWICE starts to produce alarms and this continues until the testing ends at period 102. We have developed and preliminary evaluated an adap- In the second scenario the administrator recognizes the tive network anomaly detector. In related work, real-time new class of alarms as false positives. She tells ADWICE detection and indexes for fast matching against the normal- to learn the data resulting in those false alarms at time 2004- ity model are seldom considered together with the basic de- 03-14 17:55 (end of period 45).The second curve shows that tection approach. We consider the index in the detection many of the original false alarms are no longer produced. scheme from the start, since the index may influence not However, at regular intervals there are still many alarms. only performance, but also other properties such as adap- Those intervals correspond to non-working hours. tiveness. In the third scenario incremental training is done in two Preliminary evaluations show that there may be a need steps. After the first incremental training 2004-03-14 a sec- for the administrator to tell the anomaly detector to learn ond incremental training is initiated at 2004-03-15 07:55 not only the data producing the alarm, but also a gener- (end of period 52) when the administrator notices that false alisation of data. For example, the two step incremental alarms related to host x.x.202.183 are still produced. Fig- training in section 4.2 would not have been necessary if the ure 4 shows how almost all alarms now disappear after the administrator could have told the system to learn that the second incremental training period. data producing alarms was normal both during working and The need for the two-step incremental learning arouse non-working hours. Those experiences call for more intel- since the model differs between working hours and non- ligent tools, such as sophisticated model and data visual- working hours. The alarms the administrator used for ini- ization mechanisms, to help the administrator use anomaly tial incremental training were all produced during working detection in a real environment. Also systematic methods to hours (2004-03-14 12:00 - 2004-03-14 17:55). decide on suitable parameter settings (e.g. M, numSlices) is needed. Full evaluation of how sensitive ADWICE is to for Collaborative Enterprises (WETICE 2004), pages 217– minor changes of parameters remains to be done. 222, Modena, Italy, 2004. IEEE Computer Society. Extensions of the work will consider cluster size and fre- [2] K. Burbeck and S. Nadjm-Tehrani. Adwice - anomaly detec- quency of usage as criteria to decide whether a cluster ought tion with real-time incremental clustering. In Proceedings to be forgotten. Large clusters, corresponding to very com- of the 7th International Conference on Information Security and Cryptology, Seoul, Korea, 2004. Springer Verlag. mon normality in the past, should be very resistant against [3] T. Chyssler, S. Burschak, M. Semling, T. Lingvall, and forgetting. With this approach also outliers resulting in K. Burbeck. Alarm reduction and correlation in intrusion de- small clusters in the model could be handled by forgetting tection systems. In Proceedings of Detection of Intrusions them. Another improvement would be to gradually decrease and Malware & Vulnerability Assessment, GI SIG SIDAR the influence of clusters over time, rather then forgetting Workshop (DIMVA 2004), volume 46 of LNI, pages 9–24, them completely. Dortmund, Germany, July 2004. GI. Full evaluation of forgetting and incremental training re- [4] C. Elkan. Results of the kdd’99 classifier learning. ACM quires a long period of real data collection. We see the cur- SIGKDD Explorations, 1(2):63 – 64, 2000. rent work as a proof of concept. It is improbable that parts [5] Y. Guan, A. A. Ghorbani, and N. Belacel. Y-means a cluster- of normality should be forgotten already after a few days ing method for intrusion detection. In Canadian Conference on AI, volume 2671 of Lecture Notes in Computer Science, in a real network and suitable configuration of forgetting pages 616–617, Montreal, Canada, 2003. Springer. is network and policy specific. This is a trade-off between [6] J. Haines, D. Kewley Ryder, L. Tinnel, and S. Taylor. Valida- (1) performance (decreasing model size), (2) detection of tion of sensor alert correlators. IEEE Security and Privacy, (very) old normality as anomalies and (3) false positives if 1(1):46–56, 2003. the normality is forgotten too soon. [7] J. Han and M. Kamber. Data Mining - Concepts and Tech- Producing good public data for intrusion detection evalu- niques. Morgan Kaufmann Publishers Inc., San Francisco, ation including anomaly detection and correlation is still an CA, USA, 2001. important task for the intrusion detection community and [8] M. V. Mahoney and P. K. Chan. An analysis of the 1999 we think collaboration on this task is very important. Two darpa/lincoln laboratory evaluation data for network anom- aly detection. In Recent Advances in Intrusion Detection, approaches exist: volume 2820 of Lecture Notes in Computer Science, pages • A test network or simulator can be used for genera- 220–237, Pittsburgh, PA, USA, 2003. Springer. tion of data, thereby realising a fully controlled envi- [9] S. Mukkamala, G. Janoski, and A. Sung. Intrusion detec- ronment. Generation of more extensive data sets in tion using neural networks and support vector machines. In Proceedings of the 2002 International Joint Conference on the Safeguard test network is ongoing work but re- Neural Networks (IJCNN ’02), pages 1702–1707, Honolulu, quires more resources, primarily to generate good nor- HI, 2002. Institute of Electrical and Electronics Engineers mal data, but also to perform attacks. Inc. • Data can be collected from real live networks. Here, [10] J. Munson and S. Wimer. Watcher the missing piece of the security puzzle. In Proceedings of the 17th Annual Com- normality is no longer a problem, but privacy is, and so puter Security Applications Conference, pages 230–9, New is the issue of attack coverage. Emerging tools [1] can Orleans, LA, USA, 2001. IEEE Comput. Soc. be used to sanitize data with some limitations. San- [11] U. of California Irvine. The uci kdd archive, 2003. itizing data while keeping relevant properties of data http//kdd.ics.uci.edu Acc. February 2004. so that intrusion detection analysis is still valid is not [12] L. Portnoy, E. Eskin, and S. Stolfo. Intrusion detection with easy, especially for unstructured information, for ex- unlabeled data using clustering. In ACM Workshop on Data ample, network packet content. Attacks may have to Mining Applied to Security, 2001. be inserted into data off-line if the real attack present [13] Safeguard. The safeguard project, 2003. http//www.ist- in data is not enough, since performing attacks in a real safeguard.org/ Acc. May 2004. [14] K. Sequeira and M. Zaki. Admit anomaly-based data min- network is typically not a viable option. ing for intrusions. In Proceedings of the 8th ACM SIGKDD We would like to explore both paths for evaluation of fu- international conference on Knowledge discovery and data ture work and possibly include not only network data, but mining, pages 386–395, Edmonton, Alberta, Canada, 2002. also explore other types of data. This is possible since AD- ACM Press. WICE is a general and flexible anomaly detection scheme. [15] T. Zhang, R. Ramakrishnan, and M. Livny. Birch an efficient data clustering method for very large databases. SIGMOD Record 1996 ACM SIGMOD International Conference on References Management of Data, 25(2):103–14, 1996. [1] M. Bishop, B. Bhumiratana, R. Crawford, and K. N. Levitt. How to sanitize data. In Proceedings of 13th IEEE Interna- tional Workshops on Enabling Technologies, Infrastructure