Disk-Level Behavioral Malware Detection A Dissertation Presented to the faculty of the School of Engineering and Applied Science University of Virginia In Partial Fulfillment of the requirements for the Degree Doctor of Philosophy Computer Science by Nathanael R. Paul May 2008 Approvals This dissertation is submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Computer Science Nathanael R. Paul Approved: David E. Evans (Advisor) Jack W. Davidson (Chair) Sudhanva Gurumurthi Nina Mishra Ronald D. Williams Accepted by the School of Engineering and Applied Science: James H. Aylor (Dean) May 2008 Abstract We present a new malware detection method that takes advantage of the processing power now available on disk drives. Our method uses the disk processor to monitor disk requests and identifies malicious programs based on characteristic properties of the disk requests they make. Disk-level behavioral detection offers several advantages over traditional ap- proaches since the disk processor can perform computation without burdening the host processor and can mediate disk accesses before they reach the physical medium. This dissertation describes and evaluates two instances of our approach: one uses a simple, general infection signature to reliably detect a class of file-infecting viruses; the other il- lustrates how our approach can be used with behavior-specific signatures to recognize known malware. By identifying a large class of common disk-level virus behavior, we develop simple rules that can be enforced by the disk processor. These rules are able to detect unknown viruses by recognizing their characteristic file-infecting behavior. Two of the rules are able to detect all but two types of the file-infecting viruses in our test set. From our testing based on traces of disk activity collected from eight different users, we identify a small set of activities that generate false positives. We present mechanisms to mitigate or avoid these false positives. Some malware performs other malicious actions besides the recognized file-infecting behavior. We develop a process for finding behavior-specific signatures to precisely iden- tify disk-based malware using a candidate set of three viruses and one worm. We can detect a family of malware (i.e., its variants) using a single disk-level behavior-specific iii iv signature. We present the design of a disk-level malware detector that can enforce these rules and signatures. The disk infers high-level file system activity from disk-level events and is able to map disk blocks to files. Based on this design, we implemented a prototype disk-level detector that demonstrates the feasibility of disk-level virus detection. Our detection is resilient to many traditional obfuscation techniques, because our ap- proach is behavior-based. It is more difficult to change a program’s behavior than it is to change a few bytes to generate a new variant. Further, by using the disk processor to recognize disk-based malware, a disk-level detector can use general behavioral rules to detect unknown malware and use behavior-specific signatures to precisely identify other more sophisticated malware. Since the detector is running below the host level, it cannot be circumvented even if the host is compromised. Acknowledgments Throughout graduate school, there have been many people that have helped me in ways both small and great. My advisor, Dave Evans, always encouraged me to pursue and study interesting topics. His mentorship was “handsoff” to let me learn from my mistakes, but he was also always there to steer me in the right direction. Other professors have also had a significant impact. Although I came to know Sud- hanva Gurumurthi later in graduate school, I learned much from our work together. I am grateful to Nina Mishra, Jack Davidson, and Ron Williams for serving on my thesis committee. From Clemson, I thank Harold Grossman for his friendship and guidance. I have been blessed with many friends along the way, and I am grateful for these friend- ships. Many of my friends and family back at home have helped me, and I am grateful for their prayers. While our distance to each other may have grown, they were still much appreciated. To my UVa friends, I thank Mark, John, and Carrie for being there from the beginning; Jeremy, Chris, and Shahrukh for sharing in my passion of food and sports (and an extra thank you to Jeremy for helping me get my thesis turned in 4000 miles away); and Adrienne for her help in the thesis work. I enjoyed and learned a lot from all of the many times I shared with all of you. I am indebted to the systems staff, especially Mark, for incredible systems support. The administrative staff was helpful in helping me get the necessary things done and on time. I thank Brian and Jolynn for their encouragement and many fun meals, Jim and Cindy for letting their daughter go around the world with me, and my family for their support. v vi I am fortunate to have such loving, supporting, and encouraging parents. Even while young, I was always encouraged to ask questions and learn about interesting things. My dads´ patience in teaching me, even in the face of strong jet blasts, helped me to have fun while learning. And last, while my wife has shared in many of my experiences, she has always shown her sacrificial love and support. My thesis and work are far better because of her. “Be still, and know that I am God...” Psalm 46:10 Contents 1 Introduction 1 1.1 Flaws in Current Malware Detection . 1 1.2 Overcoming Flaws in Current Detector Designs . 3 1.3 Contributions . 5 1.4 Roadmap . 6 2 Background 7 2.1 Current Approaches in Malware Detection . 8 2.2 Malware Authors’ Motivation . 12 2.3 Malware Signature Database Growth . 13 2.4 Related Work . 16 2.5 Summary . 25 3 Detector Design 27 3.1 Detector Design . 28 3.2 Smarter Disks . 29 3.3 The Semantic Gap . 31 3.4 Implementing a Prototype Detector . 38 3.5 Cost Analysis . 43 3.6 Updates . 48 3.7 Recovery and Response . 51 vii viii Contents 4 General Behavior Detection 55 4.1 Common Virus Behaviors . 55 4.2 Detection Rules . 57 4.3 General Behavioral Rules Costs . 59 4.4 Detection Rates . 60 4.5 False Positives . 70 4.6 Summary . 76 5 Behavior-Specific Signatures 79 5.1 Developing Signatures . 80 5.2 Example Signatures . 82 5.3 Costs . 93 6 Security Analysis 99 6.1 Behavior Obfuscators . 99 6.2 Circumvention . 101 6.3 Resource Exhaustion . 103 6.4 Exploiting False Positive Mechanisms . 104 6.5 Summary . 105 7 Conclusion 107 7.1 Contributions . 107 7.2 Limitations . 109 7.3 Future Work . 112 7.4 Final Thoughts . 121 Bibliography 123 List of Figures 2.1 Norton and RAV signature database size. 15 3.1 A Malware Disk Detector . 28 3.2 Parts of a Disk . 30 3.3 Prototype Setup . 39 3.4 Disk Detector Prototype Storage Setup . 42 3.5 File System Fragmentation . 45 3.6 Updating Signatures. 49 4.1 Windows PE File Format . 56 4.2 Program Installation and Updates. 74 5.1 Developing a Signature . 80 5.2 Efish Dropping Rule . 85 5.3 Ganda Signature . 87 5.4 Observed Disk-level Actions of Sality.L Instance. 89 5.5 Sality.L Signature . 90 5.6 Parite Signature . 92 5.7 Example Behavior-Specific Signature . 95 ix List of Tables 4.1 Virus Detection Results . 62 4.2 Virus Detection Results on Larger Sample . 65 4.3 False Positive Causes . 70 4.4 False Positive Results . 71 4.5 Installation Files Used for False Positive Testing . 73 xi Chapter 1 Introduction Current malware detection has limited success because of essential problems in its design. We discuss limitations with traditional anti-virus (AV) programs and then describe how disk-level behavioral detection can overcome these limitations. 1.1 Flaws in Current Malware Detection Prevailing malware detection approaches suffer from three fundamental flaws that give malicious software creators a seemingly insurmountable advantage in the arms race be- tween viruses and anti-virus software: host-level detectors can be circumvented, detectors can be easily evaded, and detectors cannot detect new threats because of reliance on black- lists. Host-level Detection is Circumventable. One problem with most malware detection is that monitoring and response are performed at the level of the host operating system. If malware can compromise the host operating system (OS), or worse, below the host OS, it can circumvent detection while causing arbitrary damage to the host. Rootkits are examples of malware that circumvent OS-based protection often by changing the OS itself (e.g., creating a backdoor in the system for future unauthorized access) [King and Chen, 2005; Halderman and Felten, 2006]. Recent research has proposed rootkits that attack a layer below the OS [King and Chen, 2005]. This presents a new and challenging problem. 1 2 Chapter 1. Introduction The OS must now detect if it is running within a malicious virtual environment. OS-based approaches offer no reliable solution to detect or recover from low-level rootkit infections, since the malware may be able to circumvent detection monitoring points by hiding its location below the detection software residing in the OS. Static Code Detection is Easily Evaded. A second problem is that current approaches are predominantly static: they attempt to detect malware by recognizing its code, not by observing its behavior. This is problematic, since changing the code of a program without disrupting its behavior is easy. There are infinitely many different instruction sequences that can be inserted in a program without changing its observed behavior.