Liu Dissertation 2011.Pdf
Total Page:16
File Type:pdf, Size:1020Kb
Multi-level Sandboxing Techniques for Execution-based Stealthy Malware Detection A dissertation submitted in partial ful¯llment of the requirements for the degree of Doctor of Philosophy at George Mason University By Lei Liu Master of Science Huazhong University of Science and Technology, 1999 Bachelor of Science Huazhong University of Science and Technology, 1996 Director: Songqing Chen, Associate Professor Department of Computer Science Spring Semester 2011 George Mason University Fairfax, VA Copyright °c 2011 by Lei Liu All Rights Reserved ii Dedication I dedicate this dissertation to my parents Zuoxun Liu and Dingfeng Luo. iii Acknowledgments I would like to thank the following people who made this possible. I would like to thank my advisor Dr. Songqing Chen, who has spent so much of his time directing this dissertation work. It would have been impossible without his guidance and support. My gratitude also goes to Dr. Sanjeev Setia, Dr. Brian L. Mark, Dr. Fei Li, and Dr. Yutao Zhong who served on my dissertation committee and gave me their invaluable input. I would also like to thank Dr. Hassan Gomaa for his advice on my dissertation presen- tation, as well as Drs. Xinwen Zhang, Guanhua Yan, Xinyuan Wang, and Zhao Zhang for their advice and collaboration in several research projects during my Ph.D. study. I would also express my appreciation to my lab mates Dongyu Liu and Yao Liu for their help and collaboration, and to my friends I met at George Mason University: Fayin Li, Haidong Lu, Yifan Liu, Bo Zhang, and Baoxian Zhao, just to name a few. As an international student I received a lot of help from people in George Mason Uni- versity and the local Fairfax community. Finally I would like to thank my family members who always stand by me. iv Table of Contents Page List of Tables . viii List of Figures . ix Abstract . x 1 Introduction . 1 1.1 Background . 1 1.2 Malware Detection . 4 1.2.1 Signatured-based Malware Detection Schemes . 4 1.2.2 Anomaly-based Malware Detection Schemes . 5 1.2.3 Sandboxing Techniques . 7 1.3 Problem Statement . 8 1.4 Our Contributions: Multi-level Sandboxing Techniques for Execution-based Detection . 9 1.5 Summary . 12 2 OS-level Sandboxing and BotTracer . 13 2.1 Introduction . 13 2.2 Principle and Overview of BotTracer . 17 2.2.1 Invariant Characteristics of Bot-like Malware . 17 2.2.2 BotTracer Design Rationale . 18 2.2.3 BotTracer Overview . 19 2.3 BotTracer Design . 21 2.3.1 Whitelist and Starting Point Set . 22 2.3.2 Command and Control Channel Event Model . 23 2.3.3 Command and Control Channel Detection . 26 2.3.4 Information Harvesting/Dispersion Behavior Analysis . 29 2.4 BotTracer Evaluation . 33 2.4.1 Prototype Implementation and Experimental Setup . 33 2.4.2 Channel Establishment Detection . 35 2.4.3 Information Harvesting/Dispersion Detection . 36 2.4.4 False Positive Experiments . 37 v 2.5 Discussion . 40 2.6 Summary . 41 3 Process-level Sandboxing and Malyzer . 42 3.1 Introduction . 42 3.2 Malyzer Design . 44 3.2.1 Startup Tracker . 45 3.2.2 Shadow Process Manager . 48 3.2.3 Shadow Process Monitor . 55 3.3 Malyzer Implementation . 58 3.4 Malyzer Evaluation . 60 3.4.1 Whether Malyzer Can Defeat Malware Anti-detections . 60 3.4.2 Whether a Shadow Process Functions Normally and Whether Its Mis- behavior Can Be Detected . 63 3.5 Discussion . 64 3.6 Summary . 66 4 Intra-process Sandboxing and sePlugin . 67 4.1 Introduction . 67 4.2 sePlugin Design . 71 4.2.1 Threat Model . 71 4.2.2 Overview of sePlugin . 72 4.2.3 Sandbox Manager . 73 4.2.4 Proxy . 73 4.2.5 Intra-process Sandbox . 74 4.3 sePlugin Implementation . 78 4.3.1 Sandbox Manager . 78 4.3.2 Proxy and COM interceptor . 79 4.3.3 System Call Interceptor . 82 4.3.4 .NET Interceptor . 84 4.3.5 Security Checker . 86 4.4 Securing IE with sePlugin . 87 4.4.1 Same Origin Policy for Content Extensions . 87 4.4.2 Policies for Browser Extensions . 88 4.5 sePlugin Evaluation . 92 4.5.1 Detecting Malicious Plugins . 92 4.5.2 False Positive Tests . 94 vi 4.5.3 Operational Overhead . 95 4.6 Summary . 97 5 Conclusion . 98 Bibliography . 102 vii List of Tables Table Page 1.1 Multi-level Sandboxing and Systems . 10 2.1 Command and Control Channel Detection . 35 2.2 Agobot HTTP DDoS Attack Packets . 36 2.3 API Called When Graybird Launches Attacks . 37 2.4 Outlook Express Connecting Packets . 38 3.1 Experiment Samples . 61 3.2 rBot Detection . 63 3.3 OS-level Sandbox and Process-level Sandbox Comparison . 64 4.1 Summary of How Plugins Are Secured in Existing Browser Architectures and Our Objectives . 69 4.2 Plugin Interfaces Implemented . 80 4.3 Malicious Plugins Detection . 93 4.4 False Positive Rate (Round 1) . 94 4.5 False Positive Rate (Round 2) . 94 4.6 False Positive Rate (Round 3) . 95 viii List of Figures Figure Page 1.1 Multi-level Sandboxing . 10 2.1 OS-level Sandbox . 16 2.2 BotTracer Detection Logic: Startup, Preparation, and Attack During a Bot Onset . 19 2.3 BotTracer Architecture and Flow of Control . 20 2.4 Tra±c Pattern of Command and Control Channel: Centralized and Decen- tralized . 25 2.5 Command and Control Channel Event Model . 27 2.6 The Dormant Pro¯le of Internet Explorer . 31 2.7 The Dormant Pro¯le of Outlook Express ................... 39 3.1 Malyzer System Architecture . 45 3.2 Dynamic Code Replacement of Graybird ................... 47 3.3 Shadow Process Manager . 51 3.4 A shadow process of reptile .......................... 57 3.5 A Shadow Process of Graybird ......................... 61 3.6 A Shadow Process of Agobot3 .......................... 62 4.1 sePlugin Overview . 72 4.2 Intra-process Sandbox . 75 4.3 Proxy Interfaces . 79 4.4 .NET Weaving with RAIL . 85 4.5 .NET-based Plugin Overview . 86 4.6 Execution Overhead of sePlugin ........................ 96 ix Abstract MULTI-LEVEL SANDBOXING TECHNIQUES FOR EXECUTION-BASED STEALTHY MALWARE DETECTION Lei Liu, PhD George Mason University, 2011 Dissertation Director: Songqing Chen These days all kinds of malware are pervasive on the Internet. Compared to their ances- tors that were commonly used for vandalism or demonstration of skills, modern malware, such as Bots, are driven by the underground economics. Often consisting of hundreds to thousands of bots, botnets are one of the most serious threats on the Internet, responsible for various attacks, such as spamming and distributed denial of service (DDoS). As web browsers are the main interface for the majority of Internet users to surf the Internet today, many of such stealthy malware seek to invade via web browsers in the form of browser helper objects (BHO) and browser toolbars. To defend against Internet malware, existing schemes mainly rely on either signature- based or anomaly-based detection approaches. Signature-based detection is e®ective for known malware if the malware signature has been generated. However, the e®ectiveness of signature-based schemes is challenged by polymorphism, metamorphism, obfuscation, en- cryption,.