EXFILD: A TOOL FOR THE DETECTION OF DATA EXFILTRATION USING ENTROPY AND ENCRYPTION CHARACTERISTICS OF NETWORK TRAFFIC by Tyrell William Fawcett A thesis submitted to the Faculty of the University of Delaware in partial fulfillment of the requirements for the degree of Masters of Science in Electrical and Computer Engineering Spring 2010 c 2010 Tyrell William Fawcett All Rights Reserved EXFILD: A TOOL FOR THE DETECTION OF DATA EXFILTRATION USING ENTROPY AND ENCRYPTION CHARACTERISTICS OF NETWORK TRAFFIC by Tyrell William Fawcett Approved: W. David Sincoskie, Ph.D. Professor in charge of thesis on behalf of the Advisory Committee Approved: Kenneth E. Barner, Ph.D. Chairman of the Department of Electrical and Computer Engineering Approved: Michael J. Chajes, Ph.D Dean of the College of Engineering Approved: Debra Hess Norris, M.S. Vice Provost for Graduate and Professional Education ACKNOWLEDGMENTS To my advisor, Dave Sincoskie: Dr. Sincoskie has given me the opportunity to further my education at the University of Delaware. He has provided me with the guidance needed to perform my research at a level higher than I believed I could. He has taught me not to blindly believe things without proving them myself. His vast experience in research and ability to see the importance in this area of research is what lead to my thesis. To Chase Cotton: Dr. Cotton is always around and willing to brainstorm and work through technical problems with anyone. His practical experience in the field and willingness to perform experiments all hours of the day has proven to be invaluable my research. To Fouad Kiamilev: Dr. K has been gracious enough to provide space in his lab and access to all of his equipment to me during my Graduate career. He provides an enthusiastic and upbeat lab environment for CVORG. His eagerness to learn and encouragement to do what you love is what keeps CVORG such a great place to work. To Charles Boncelet: Dr. Boncelet provided great advice and guidance re- lating to encryption and entropy calculations in my work. I'm thankful for him sharing his experience and saving me time and effort going down unnecessary paths in unfamiliar territory. To all the members of CVORG: CVORG is definitely a very eclectic group of people piled into labs together. The members are always there to help someone in need. The brainstorming sessions in this lab leads to a lot of great ideas and iii research. A lot of aspects of both my thesis work and other projects have resulted from these brainstorming sessions and has provided me with some paths forward in future work. I am proud to be a part of CVORG. To Larry and Karen Steenhoek: The Steenhoeks have provided a lot of sup- port over the last semester to my wife and I. It has made this transitional period and specifically the writing process a much easier time and for that I thank them. To my parent, Jim and Deby Fawcett, and my sister, Sammy Fawcett: I always know that no matter what happens I always have their support. I attribute my successes up to now to the work ethic and morals they instilled in me. I love them for everything they have done for me. To my wife, Valerie Fawcett: She has been my best friend for as long as most people remember. She has kept my life interesting with her goofy antics, which I'm sure will only get more entertaining in DC. She has made sacrifices allowing me to quench my thirst for knowledge and for that I can't thank her enough. She will be as happy as I am to see this thesis completed, because she calls it the day she gets her husband back! I couldn't ask for a more supportive wife to spend the rest of my life with. iv TABLE OF CONTENTS LIST OF FIGURES ::::::::::::::::::::::::::::::: viii LIST OF TABLES :::::::::::::::::::::::::::::::: xii ABSTRACT ::::::::::::::::::::::::::::::::::: xiii Chapter 1 BACKGROUND ::::::::::::::::::::::::::::::: 1 2 INTRODUCTION :::::::::::::::::::::::::::::: 8 2.1 Motivation ::::::::::::::::::::::::::::::::: 8 2.1.1 Scenario of Security Against Infiltration :::::::::::: 9 2.1.2 System Administrators :::::::::::::::::::::: 10 2.1.3 Encryption :::::::::::::::::::::::::::: 11 2.1.4 Out of the Box Firewall ::::::::::::::::::::: 11 2.1.5 Login Credentials ::::::::::::::::::::::::: 12 2.1.6 Antivirus Software / Malware Detection :::::::::::: 13 2.1.7 Properly Configured Firewall :::::::::::::::::: 13 2.1.8 Intrusion Detection System ::::::::::::::::::: 14 2.1.9 Intrusion Prevention System ::::::::::::::::::: 15 2.1.10 Outgoing Traffic ::::::::::::::::::::::::: 15 2.1.11 Consequences of Encryption ::::::::::::::::::: 16 2.2 Goals :::::::::::::::::::::::::::::::::::: 16 2.3 Related Work ::::::::::::::::::::::::::::::: 18 3 NETWORK TOOLS :::::::::::::::::::::::::::: 19 3.1 Network Sniffer :::::::::::::::::::::::::::::: 20 3.2 Corporate Watcher :::::::::::::::::::::::::::: 22 3.3 Network Top ::::::::::::::::::::::::::::::: 24 v 3.4 DNS Extractor :::::::::::::::::::::::::::::: 28 3.5 Session Extractor ::::::::::::::::::::::::::::: 29 3.6 IP Helper ::::::::::::::::::::::::::::::::: 31 4 EXFILD DESIGN AND IMPLEMENTATION :::::::::::: 33 4.1 Packet and Session Processing :::::::::::::::::::::: 33 4.1.1 Packet Decoding ::::::::::::::::::::::::: 34 4.1.2 Extract Sessions :::::::::::::::::::::::::: 35 4.1.3 Entropy Calculation ::::::::::::::::::::::: 36 4.1.3.1 Scaling by Initial Values :::::::::::::::: 41 4.1.3.2 Scaling by Size ::::::::::::::::::::: 42 4.1.4 Checking if Encryption is Expected ::::::::::::::: 44 4.1.5 Checking if Encryption is Present :::::::::::::::: 45 4.2 Tree :::::::::::::::::::::::::::::::::::: 48 4.2.1 Expected Unencrypted and Received Unencrypted ::::::: 49 4.2.2 Expected Encrypted and Received Encrypted ::::::::: 51 4.2.3 Expected Encrypted and Received Unencrypted :::::::: 52 4.2.4 Expected Unencrypted and Received Encrypted :::::::: 54 5 EXPERIMENTS, RESULTS, AND ANALYSIS ::::::::::: 57 5.1 Experiments :::::::::::::::::::::::::::::::: 57 5.1.1 Control Data Set ::::::::::::::::::::::::: 57 5.1.2 Data Set #1 :::::::::::::::::::::::::::: 60 5.1.3 Data Set #2 :::::::::::::::::::::::::::: 61 5.1.4 Malware Data Sets :::::::::::::::::::::::: 62 5.1.4.1 Kraken Botnet ::::::::::::::::::::: 63 5.1.4.2 Zeus Botnet ::::::::::::::::::::::: 63 5.1.4.3 Black Worm ::::::::::::::::::::::: 64 5.2 Results and Analysis ::::::::::::::::::::::::::: 65 5.2.1 Packet Versus Session Alerts ::::::::::::::::::: 65 vi 5.2.2 Control Data Set ::::::::::::::::::::::::: 66 5.2.3 Data Set #1 :::::::::::::::::::::::::::: 70 5.2.4 Data Set #2 :::::::::::::::::::::::::::: 71 5.2.5 Malware Data Sets :::::::::::::::::::::::: 72 5.2.5.1 Kraken ::::::::::::::::::::::::: 72 5.2.5.2 Zeus ::::::::::::::::::::::::::: 72 5.2.5.3 Blackworm ::::::::::::::::::::::: 73 5.2.6 Data Exfiltration Detection Performance :::::::::::: 74 6 CONCLUSIONS ::::::::::::::::::::::::::::::: 78 7 FUTURE WORK :::::::::::::::::::::::::::::: 80 7.1 Performance :::::::::::::::::::::::::::::::: 80 7.2 Handle A Network :::::::::::::::::::::::::::: 81 7.3 Application Layer Decoding of Packets ::::::::::::::::: 82 7.4 Comparison to packet and session entropy ::::::::::::::: 82 7.5 Compressed File Analysis :::::::::::::::::::::::: 83 7.6 Behavioral Analysis :::::::::::::::::::::::::::: 83 7.7 Additional Tools ::::::::::::::::::::::::::::: 84 BIBLIOGRAPHY :::::::::::::::::::::::::::::::: 85 Appendix A EXPERIMENTS ::::::::::::::::::::::::::::::: 89 B ENTROPY PLOTS FOR DATA SETS ::::::::::::::::: 99 C ALERTS FOR DATA SETS :::::::::::::::::::::::: 108 D VERIFICATION OF MALWARE PACKET CAPTURES ::::: 112 D.1 Kraken Packet Capture :::::::::::::::::::::::::: 112 D.2 Zeus Packet Captures ::::::::::::::::::::::::::: 114 D.2.1 Zeus #1 :::::::::::::::::::::::::::::: 114 D.2.2 Zeus #2 :::::::::::::::::::::::::::::: 116 D.2.3 Zeus #3 :::::::::::::::::::::::::::::: 119 D.3 Blackworm ::::::::::::::::::::::::::::::::: 121 vii LIST OF FIGURES 3.1 Example Output from the Network Sniffer :::::::::::::: 21 3.2 Example Output from the Corporate Watcher Program (Simple) :: 23 3.3 Example Output from the Corporate Watcher Program (Complex) 24 3.4 Example Output from the Network Top Program :::::::::: 27 3.5 Example Output from the DNS Extractor Program ::::::::: 29 3.6 Example Output from the Session Extractor Program :::::::: 30 3.7 GUI for the IP Helper Program :::::::::::::::::::: 32 4.1 The Flow of the Processing of Packets :::::::::::::::: 34 4.2 The Flow of the Processing of Sessions :::::::::::::::: 34 4.3 Maximum and Minimum Entropy vs. Size of a Packet's Payload :: 38 4.4 Maximum Entropy vs. Size of a Packet's Payload :::::::::: 39 4.5 Maximum and Minimum Entropy Values with Different Initial Values. :::::::::::::::::::::::::::::::::: 42 4.6 Scaled Maximum and Minimum Entropy Values ::::::::::: 44 4.7 HTTP and HTTPS Traffic ::::::::::::::::::::::: 47 4.8 The Four Branches of the Tree :::::::::::::::::::: 48 4.9 Flow for Expected Unencrypted and Received Unencrypted Branch 49 viii 4.10 Flow for the Expected Encrypted and Received Encrypted Branch : 51 4.11 Flow for Expected Encrypted and Received Unencrypted Branch : 53 4.12 Flow for Expected Unencrypted and Received Unencrypted Branch 55 5.1 Network Diagram for the Control Data Set :::::::::::::: 58 5.2 Network Diagram for Data Set #1 :::::::::::::::::: 61 5.3 Network Diagram for Data Set #2 :::::::::::::::::: 62 B.1 Packet Entropies for the Control Data Set :::::::::::::: 99 B.2 Session Entropies for the Control Data Set :::::::::::::: 100 B.3 Packet Entropies for Data Set #1 (First Plot) :::::::::::: 100 B.4 Packet Entropies for Data Set #1 (Second Plot) ::::::::::: 101