IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 13, NO. 5, OCTOBER 2005 961 The Monitoring and Early Detection of Internet Worms Cliff C. Zou, Member, IEEE, Weibo Gong, Fellow, IEEE, Don Towsley, Fellow, IEEE, and Lixin Gao, Member, IEEE Abstract—After many Internet-scale worm incidents in re- scale worm-monitoring and early-warning system. (The U.S. cent years, it is clear that a simple self-propagating worm can Department of Homeland Security launched a “Cybersecurity quickly spread across the Internet and cause severe damage to Monitoring Project” in October 2003 [40]). our society. Facing this great security threat, we need to build an early detection system that can detect the presence of a worm A straightforward way to detect an unknown (zero-day) in the Internet as quickly as possible in order to give people worm is to use various anomaly detection systems. There are accurate early warning information and possible reaction time many well-studied methods or systems in the anomaly “intru- for counteractions. This paper first presents an Internet worm sion detection” research area, for example, the “IDES” [13], monitoring system. Then, based on the idea of “detecting the trend, not the burst” of monitored illegitimate traffic, we present “NIDES” [5] and “eBayes” [39] from SRI International; the a “trend detection” methodology to detect a worm at its early anomaly intrusion detection method [15] based on “sequences propagation stage by using Kalman filter estimation, which is of system calls”; the automatic model-construction intrusion robust to background noise in the monitored data. In addition, for detection system based on data-mining of audit data [24], etc. uniform-scan worms such as Code Red, we can effectively predict Anomaly intrusion-detection systems usually concentrate on the overall vulnerable population size, and estimate accurately how many computers are really infected in the global Internet detecting attacks initiated by hackers. In the case of Internet based on the biased monitored data. For monitoring a nonuniform worm detection, we find that we can take advantage of the dif- scan worm, especially a sequential-scan worm such as Blaster, we ference between a worm’s propagation and a hacker’s intrusion show that it is crucial for the address space covered by the worm attack. A worm code exhibits simple attack behaviors; all com- monitoring system to be as distributed as possible. puters infected by a worm send out infection traffic that has Index Terms—Computer network security, early detection, In- similar statistical characteristics. Moreover, a worm’s propaga- ternet worm, network monitoring. tion in the Internet usually follows some dynamic models be- cause of its large-scale distributed infection. On the other hand, I. INTRODUCTION a hacker’s intrusion attack, which is more complicated, usually targets one or a set of specific computers and does not follow INCE the Morris worm in 1988 [33], the security threat any well-defined dynamic model in most cases. S posed by worms has steadily increased, especially in the last several years. Code Red appeared on July 19, 2001 [27], Based on this observation, we present a new detection which began the new wave of Internet-scale worm attacks. methodology, “trend detection,” by using the principle “de- After that, Code Red II, Nimda, Slammer, Blaster, Sasser, and tecting monitored traffic trend, not burst” [45]. Our “trend Witty have repeatedly attacked the Internet [9] and caused great detection” system attempts to detect the dynamic trend of mon- damage to our society. itored traffic based on the fact that, at the early stage, a worm Currently, some organizations and security companies, such propagates exponentially with a constant, positive exponential as the CERT, CAIDA, and SANS Institute [7], [8], [32], are rate. The “trend” we try to detect is the exponential growth monitoring the Internet and paying close attention to any ab- trend of monitored traffic. normal traffic. When they observe abnormal network activi- Based on worm propagation dynamic models, we detect the ties, their security experts immediately analyze these incidents. presence of a worm in its early propagation stage by using the Given the fast-spreading nature of Internet worms and their se- Kalman filter estimation algorithm, which is robust to back- vere damage to our society, it is necessary to set up a nation- ground noise existing in the monitored data. The Kalman filter is activated when the monitoring system encounters a surge of illegitimate scan activities. If the infection rate estimated by the Manuscript received February 13, 2004; revised August 17, 2004; approved by IEEE/ACM TRANSACTIONS ON NETWORKING Editor V. Paxson. This Kalman filter, which is also the exponential growth rate of a work was supported in part by the Army Research Office under Contract worm’s propagation at its early stage, stabilizes and oscillates DAAD19-01-1-0610, the Defense Advanced Research Projects Agency under slightly around a constant positive value, we claim that the il- Contract DOD F30602-00-0554, the National Science Foundation under Grants EIA-0080119, ANI9980552, and ANI-0208116, and the Air Force Research legitimate scan activities are mainly caused by a worm, even if Laboratory. the estimated worm infection rate is still not well converged. C. C. Zou is with the School of Computer Science, University of Central If the monitored traffic is caused by nonworm noise, the traffic Florida, Orlando, FL 32816-2362 USA (e-mail: [email protected]). W. Gong and L. Gao are with the Department of Electrical and Computer will not have the exponential growth trend, and the estimated Engineering, University of Massachusetts, Amherst, MA 01003 USA (e-mail: value of the infection rate would converge to zero or oscillate [email protected]; [email protected]). around zero. In other words, the Kalman filter is used to detect D. Towsley is with the Department of Computer Science, University of Mass- achusetts, Amherst, MA 01003-9264 USA (e-mail: [email protected]). the presence of a worm by detecting the trend, not the burst,of Digital Object Identifier 10.1109/TNET.2005.857113 the observed illegitimate traffic. In this way, the noisy illegiti- 1063-6692/$20.00 © 2005 IEEE 962 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 13, NO. 5, OCTOBER 2005 mate traffic in the Internet we observe everyday will not cause the log data collected from intrusion detection sensors or fire- too many false alarms in our detection system. walls for current monitoring systems. In addition, we present a formula to predict a worm’s vulner- In the area of worm modeling, Kephart, White, and Chess able population size when the worm is still at its early propaga- of IBM performed a series of studies from 1991 to 1993 on tion stage. We also present a formula to correct the bias in the viral infection based on epidemiology models [21], [20], [22]. number of infected hosts observed by a monitoring system. This Staniford et al. [37] used the classical epidemic model to model bias has been mentioned in [10] and [29], but neither of them has the spread of Code Red right after the Code Red incident on presented methods to correct it. In this way, we can know how July 19, 2001; they also proposed several more vicious worms many computers in the global Internet are really infected based in the same paper. Zou et al. [46] presented a “two-factor” worm on local monitored data. Furthermore, we point out that in de- model that considered both the effect of human countermeasures signing a worm monitoring system, the address space covered and the effect of the congestion caused by extensive worm scan by a monitoring system should be as distributed as possible in traffic. Chen et al. [10] presented a discrete-time version worm order to monitor and detect nonuniform scan worms, especially model that considered the patching and cleaning effect during a a sequential scan worm such as Blaster. worm’s propagation. The rest of this paper is organized as follows. Section II sur- For a fast spreading worm such as Slammer, it is necessary veys related work. Section III introduces the worm-propagation to have automatic response and mitigation mechanisms. Moore models used in this paper. Section IV describes briefly the mon- et al. [28] discussed the effect of Internet quarantine for con- itoring system. Data collection and the bias correction formula taining the propagation of a worm. Williamson [42] proposed for monitored biased data are described in Section V.Section VI a general rate-limiting “throttling” method to greatly constrain presents the Kalman filters for early worm detection, and the infection traffic sent out by infected hosts while not affecting formula to predict the vulnerable population size. We conduct normal traffic. Zou et al. [47] presented a feedback dynamic extensive simulation experiments and show the major results in quarantine system for automatic mitigation by borrowing two Section VII. In Section VIII, we discuss limitations and possible principles used in the epidemic disease control in the real world: future work. Section IX concludes this paper. “preemptive quarantine” and “feedback adjustment.” Staniford [36] presented automatic worm quarantine for enterprise net- works by using CounterMalice devices to separate an enterprise II. RELATED WORK network into many isolated subnetworks. Weaver et al. [41] fur- ther improved the CounterMalice quarantine by designing hard- In recent years, people have paid attention to the necessity of ware-centered quarantine algorithms. Jung et al. [18], [19] pro- monitoring the Internet for malicious activities. Symantec Cor- posed a “threshold random walk” algorithm to quickly detect poration has an “enterprise early warning solution” [1], which and block worm scans based on the excessive illegal scans sent collects IDS and firewall attack data from the security systems out by worm-infected hosts.