Louisiana Tech University Louisiana Tech Digital Commons Doctoral Dissertations Graduate School Summer 2006 Stochastic propagation modeling and early detection of malicious mobile code Xin Xu Louisiana Tech University Follow this and additional works at: https://digitalcommons.latech.edu/dissertations Part of the Computer Sciences Commons Recommended Citation Xu, Xin, "" (2006). Dissertation. 538. https://digitalcommons.latech.edu/dissertations/538 This Dissertation is brought to you for free and open access by the Graduate School at Louisiana Tech Digital Commons. It has been accepted for inclusion in Doctoral Dissertations by an authorized administrator of Louisiana Tech Digital Commons. For more information, please contact [email protected]. STOCHASTIC PROPAGATION MODELING AND EARLY DETECTION OF MALICIOUS MOBILE CODE By Xin Xu, M.S. A Dissertation Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy COLLEGE OF ENGINEERING AND SCIENCE LOUISIANA TECH UNIVERSITY August 2006 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. UMI Number: 3259729 INFORMATION TO USERS The quality of this reproduction is dependent upon the quality of the copy submitted. Broken or indistinct print, colored or poor quality illustrations and photographs, print bleed-through, substandard margins, and improper alignment can adversely affect reproduction. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if unauthorized copyright material had to be removed, a note will indicate the deletion. ® UMI UMI Microform 3259729 Copyright 2007 by ProQuest Information and Learning Company. All rights reserved. This microform edition is protected against unauthorized copying under Title 17, United States Code. ProQuest Information and Learning Company 300 North Zeeb Road P.O. Box 1346 Ann Arbor, Ml 48106-1346 Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. LOUISIANA TECH UNIVERSITY THE GRADUATE SCHOOL May 17, 2006_______________________ Date We hereby recommend that the thesis prepared under our supervision by __________________ Xin Xu___________________________________________ entitled______________________________________ ________________________________________________ ________ Stochastic Propagation Modeling and Earlv Detection of Malicious Mobile Code ________ be accepted in partial fulfillment of the requirements for the Degree of Doctor of Philosophy in Computational Analysis and Modeling __________________________ \JaX u w / u a gupervisor of Thasis Research A J2J2-gU f Head of Department CAM__________ Department Recommendation concurred in: Advisory Committee Approved: Approved: Director of Graduate Studies Dean of the Graduate School , I w Dean of the College / / M GS Form 13 (5/03) Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. ABSTRACT Epidemic models are commonly used to model the propagation of malicious mobile code like a computer virus or a worm. In this dissertation, we introduce stochastic techniques to describe the propagation behavior of malicious mobile code. We propose a stochastic infection-immunization (INIM) model based on the standard Susceptible- Infected-Removed (SIR) epidemic model, and we get an explicit solution of this model using probability generating function (pgf.). Our experiments simulate the propagation of malicious mobile code with immunization. The simulation results match the theoretical results of the model, which indicates that it is reliable to use INIM model to predict the propagation of malicious mobile code at the early infection stage when immunization factor is considered. In this dissertation, we also propose a control system that could automatically detect and mitigate the propagation of malicious mobile programs at the early infection stage. The detection method is based on the observation that a worm always opens as many connections as possible in order to propagate as fast as possible. To develop the detection algorithm, we extend the traditional statistical process control technique by adding a sliding window. We do the experiment to demonstrate the training process and testing process of a control system using both real and simulation data set. The experiment results show that the control system detects the propagation of malicious mobile code with zero false negative rate and less than 6% false positive rate. Moreover, iii Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. we introduce risk analysis using Sequential Probability Ratio Test (SPRT) to limit the false positive rate. Examples of risk control using SPTR are presented. Furthermore, we analyze the network behavior using the propagation models we developed to evaluate the effect of the control system in a network environment. The theoretical analysis of the model shows that the propagation of malicious program is reduced when hosts in a network applied the control system. To verify the theoretical result, we also develop the experiment to simulate the propagation process in a network. The experiment results match the mathematical results. Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. APPROVAL FOR SCHOLARLY DISSEMINATION The author grants to the Prescott Memorial Library o f Louisiana Tech University the right to reproduce, by appropriate methods, upon request, any or all portions o f this Dissertation. It is understood that “proper request” consists o f the agreement, on the part o f the requesting party, that said reproduction is for his personal use and that subsequent reproduction will not occur without written approval o f the author o f this Dissertation. Further, any portions o f the Dissertation used in books, papers, and other works must be appropriately referenced to this Dissertation. Finally, the author o f this Dissertation reserves the right to publish freely, in the literature, at any time, any or all portions of this Dissertation. Author Date GS Form 14 (5/03) Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. TABLE OF CONTENTS ABSTRACT............................................................................................................................ iii LIST OF TABLES.....................................................................................................................x LIST OF FIGURES..................................................................................................................xi LIST OF NOTATIONS.........................................................................................................xiii ACKNOWLEDGMENTS ................................................................................................. xv CHAPTER 1 INTRODUCTION ......................................................................................1 1.1 Computer Security .............................................................................................1 1.2 Malicious Mobile Code ..................................................................................... 3 1.2.1 What Is Malicious Mobile Code? ....................................................... 3 1.2.2 Defense against Malicious Mobile Code ............................................4 1.2.3 Epidemic Modeling of Malicious Code ..............................................6 1.3 The Contributions of This Dissertation ............................................................7 1.3.1 Propagation Modeling of Malicious Code ..........................................7 1.3.2 Early Detection and Propagation Mitigation of Malicious Code 7 1.3.3 Risk Control and Network Performance Analysis .............................8 1.4 The Organization of This Dissertation ............................................................. 8 CHAPTER 2 BACKGROUND AND RELATED RESEARCH.......................................10 2.1 Epidemic Models .............................................................................................10 2.1.1 Deterministic Modeling ..................................................................... 11 vi Reproduced with permission of the copyright owner. Further reproduction prohibited without permission. 2.1.1.1 Deterministic SI mode ...........................................................11 2.1.1.2 Deterministic SIR model ...................................................... 15 2.1.2 Stochastic Modeling ......................................................................... ..16 2.1.2.1 Introduction to stochastic process .........................................16 2.1.2.2 Stochastic epidemic modeling .............................................. 16 2.2 Literature Review of Malicious Mobile Code Propagation Modeling 18 2.3 Literature Review of Early Detection of Malicious Mobile Code ............. 20 2.4 Statistic Process Control ................................................................................ 22 2.4.1 A Brief Review of Process Control ..................................................22 2.4.2 An Overview of Building the Control System Using Process Control Techniques ............................................................................ 24 2.5 Summary..........................................................................................................25