ON COUNTERMEASURES of WORM ATTACKS OVER the INTERNET a Dissertation by WEI YU Submitted to the Office of Graduate Studies Of
Total Page:16
File Type:pdf, Size:1020Kb
ON COUNTERMEASURES OF WORM ATTACKS OVER THE INTERNET A Dissertation by WEI YU Submitted to the Office of Graduate Studies of Texas A&M University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY May 2008 Major Subject: Computer Engineering ON COUNTERMEASURES OF WORM ATTACKS OVER THE INTERNET A Dissertation by WEI YU Submitted to the Office of Graduate Studies of Texas A&M University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Approved by: Co-Chairs of Committee: Wei Zhao Riccardo Bettati Committee Members: Narasimha Reddy Jennifer Welch Head of Department: Valerie E. Taylor May 2008 Major Subject: Computer Engineering iii ABSTRACT On Countermeasures of Worm Attacks over the Internet. (May 2008) Wei Yu, B.S., Nanjing University of Technology; M.S., Tongji University Co-Chairs of Advisory Committee: Dr. Wei Zhao Dr. Riccardo Bettati Worm attacks have always been considered dangerous threats to the Internet since they can infect a large number of computers and consequently cause large-scale service disruptions and damage. Thus, research on modeling worm attacks, and defenses against them, have become vital to the field of computer and network security. This dissertation intends to systematically study two classes of countermeasures against worm attacks, known as traffic-based countermeasure and non-traffic based countermeasure. Traffic-based countermeasures are those whose means are limited to monitoring, collecting, and analyzing the traffic generated by worm attacks. Non-traffic based countermeasures do not have such limitations. For the traffic-based countermeasures, we first consider the worm attack that adopts feedback loop-control mechanisms which make its overall propagation traffic behavior similar to background non-worm traffic and circumvent the detection. We also develop a novel spectrum- based scheme to achieve highly effective detection performance against such attacks. We then consider worm attacks that perform probing traffic in a stealthy manner to obtain the location iv infrastructure of a defense system and introduce an information-theoretic based framework to obtain the limitations of such attacks and develop corresponding countermeasures. For the non-traffic based countermeasures, we first consider new unseen worm attacks and develop the countermeasure based on mining the dynamic signature of worm programs’ run-time execution. We then consider a generic worm attack that dynamically changes its propagation patterns and develops integrated countermeasures based on the attacker’s contradicted objectives. Lastly, we consider the real-world system setting with multiple incoming worm attacks that collaborate by sharing the history of their interactions with the defender and develop a generic countermeasure based on establishing the defender’s reputation of toughness in its repeated interactions with multiple incoming attackers to optimize the long-term defense performance. This dissertation research has broad impacts on Internet worm research since this work is fundamental, practical and extensible. Our developed framework can be used by researchers to understand key features of other forms of new worm attacks and develop countermeasures against them. v DEDICATION To MY FAMILY. vi ACKNOWLEDGMENTS To reach this stage in my Ph.D. study and this point in my life, I am indebted to many great people for their wisdom, support, and love. I would like to express my deep appreciation to my advisor, Dr. Wei Zhao. I am greatly indebted to him for his constant inspiration, encouragement, and guidance throughout my Ph.D. studies, which were essential to the completion of this dissertation. His broad vision has always been the inspiration in my existing and future professional growth. I would like to thank my co- advisor, Dr. Riccardo Bettati, for his sound technical advice and encouragement of my research. I would like to thank Dr. A. L. Narasimha Reddy for his guidance as a member of my dissertation committee. His insightful advice of my research work has inspired me to achieve important results. Many thanks go to Dr. Jennifer Welch for providing great technical advice to my research and serving on my advisory committee. I would like to thank Dr. Donald Friesen for his kind advice relating to many academic issues. Many thanks also go to former and current graduate students in the Real-Time Systems group. It was a great fortune for me to work with Dr. Dong Xuan and his Ph.D students who have provided me with useful assistance during my research. I have benefited greatly from working with Dr. Xinwen Fu and Dr. Nan Zhang over the past few years. I have also had many helpful discussions with Dr. Yong Guan, Dr. Shengquan Wang, Dr. Shu Jiang, Dr. Chengzhi Li, and Dr. Ye Zhu. I would like to express my appreciation to Ms. Elena Catelena, Ms. Larisa Archer, and Ms. Valerie Ann Sorenson for their friendship and help during my Ph.D. studies. In particular, I would like to thank Ms. Archer for her selfless help in polishing many of my writings. I would like to express my appreciation to my Cisco colleagues, Mr. Man Loh, Dr. vii Ping Ni, Mr. Tony Zhu, Mr. Ed Chen, Mr. Lee Ji, Mr. Dagang Wang, Mr. Thang Nguyen, Mr. Clint Entrop, Mr. Chris Pearce and many others for their friendship and help. I am indebted to my parents, parents-in-law, my sister, and her husband, and their lovely son for their unconditional love and support. I would like to take this opportunity to express my greatest gratitude to my wife, Jian Wang, for her selfless love and support these years. It is this family that has made me strong and courageous to be the person I am today. viii TABLE OF CONTENTS Page ABSTRACT ……….......………………………………………………………………………...iii DEDICATION …………………………………………………………………………………..v ACKNOWLEDGMENTS ……………………………………………………………….............vi LIST OF FIGURES .......................................................................................................................xi LIST OF TABLES ……………………………………………………………………………..xiii CHAPTER I INTRODUCTION ..................................................................................................1 1. Worm Attacks Are Major Threats to the Internet .....................................................1 2. Overview of Dissertation Research...........................................................................2 2.a. Traffic-Based Countermeasures .................................................................2 2.b. Non-Traffic Based Countermeasures.........................................................4 3. Significance of Proposed Work ................................................................................6 4. Organization of This Dissertation.............................................................................7 CHAPTER II REVIEWS OF WORM ATTACKS AND COUNTERMEASURES ...................8 1. Worm Attacks ...........................................................................................................8 2. Countermeasures.....................................................................................................10 2.a. Traffic-Based Countermeasure.................................................................10 2.b. Non-Traffic Based Countermeasure.........................................................12 CHAPTER III COUNTERMEASURE BASED ON PROPAGATION TRAFFIC....................14 1. Overview.................................................................................................................14 2. C-Worm Propagation ..............................................................................................15 2.a. Overview ..................................................................................................15 2.b. Effectiveness ............................................................................................17 2.c. Discussion ................................................................................................19 3. Detection of the C-Worm........................................................................................21 3.a. Design Rationale ......................................................................................21 3.b. Spectrum-Based Detection Scheme .........................................................23 3.c. Analysis....................................................................................................26 4. Performance Evaluation..........................................................................................29 4.a. Evaluation Methodology ..........................................................................29 4.b. Evaluation Results of Traffic Volume-Based Detection Schemes...........31 4.c. Evaluation Results of Traffic Distribution-Based Detection Schemes.....34 5. Summary .................................................................................................................36 ix Page CHAPTER IV COUNTERMEASURE BASED ON PROBING TRAFFIC ...............................37 1. Overview.................................................................................................................37 2. Attack Model...........................................................................................................38 2.a. lLOC Attack .............................................................................................38 2.b. Attack Traffic Generation Stage ..............................................................42 2.c. Attack Traffic Decoding Stage.................................................................44