BOTNET REVERSE ENGINEERING AND CALL SEQUENCE RECOVERY PROSENJIT SINHA A THESIS IN THE DEPARTMENT OF COMPUTER SCIENCE PRESENTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF COMPUTER SCIENCE CONCORDIA UNIVERSITY MONTRÉAL,QUÉBEC,CANADA FEBRUARY 2011 c PROSENJIT SINHA, 2011 CONCORDIA UNIVERSITY School of Graduate Studies This is to certify that the thesis prepared By: Prosenjit Sinha Entitled: Botnet Reverse Engineering and Call Sequence Recovery and submitted in partial fulfillment of the requirements for the degree of Master of Computer Science complies with the regulations of the University and meets the accepted standards with respect to originality and quality. Signed by the final examining committee: ______________________________________Dr. Adam Krzyzak Chair ______________________________________Dr. Terry Fancott Examiner ______________________________________Dr. Benjamin Fung Examiner ______________________________________Dr. Mourad Debbabi Supervisor Approved by ________________________________________________ Chair of Department or Graduate Program Director ________________________________________________ Dean of Faculty Date ________________________________________________February 17, 2011 Abstract Botnet Reverse Engineering and Call Sequence Recovery Prosenjit Sinha The focus on computer security has increased due to the ubiquitous use of Internet. Criminals mistreat the anonymous and insidious traits of Internet to commit monetary on- line fraud, theft and extortion. Botnets are the prominent vehicle for committing online crimes. They provide platform for a botmaster to control a large group of infected Internet- connected computers. Botmaster exploits this large group of connected computers to send spam, commit click fraud, install adware/spyware, flood specific network from distributed locations, host phishing sites and steal personal credentials. All these activities pose se- rious threat for individuals and organizations. Furthermore, the situation demands more attention since the research and the development of underground criminal industry is faster than security research industry. To cope up against the ever growing botnet threats, security researchers as well as Internet-users need cognizance on the recent trends and techniques of botnets. In this thesis, we analyze in-depth by reverse engineering two prominent bot- nets namely, Mariposa and Zeus. The findings of the analysis may foster the knowledge of security researchers in multiple dimensions to deal with the botnet issue. To enhance the abstraction and visualization techniques of reverse engineering, we develop a tool which is used for detailed outlook of call sequences. iii Acknowledgments I would like to express my sincere gratitude to my supervisor Mourad Debbabi for his guidance and help throughout the research work. He encouraged me patiently to achieve my research successfully and develop creative thinking. I am proud to get the opportunity to work with him. I would also like to acknowledge all my co-researchers at the National Cyber-Forensics and Training Alliance (NCFTA) CANADA, for helping in my research with their technical expertise. Finally, I would like to thank my wife Arundhati Sinha and my parrents for their constant love, support and encouragement, without which this work would not be possible. iv Contents List of Figures x List of Tables xiii List of Acronyms xv 1 Introduction 1 1.1 Motivations .................................. 2 1.2 Objectives ................................... 5 1.3 Contributions ................................. 6 1.4 Thesis Organization .............................. 7 2 Malware and Malware Analysis 8 2.1 Overview of Malware ............................. 8 2.1.1 Viruses ................................ 9 2.1.2 Worms ................................ 10 2.1.3 Trojans ................................ 10 2.1.4 Rootkits ................................ 11 v 2.1.5 Botnets ................................ 11 2.2 Sophistication of Botnet Techniques ..................... 20 2.2.1 Encryption .............................. 21 2.2.2 Polymorphism ............................ 21 2.2.3 Metamorphism ............................ 21 2.2.4 Multithreading ............................ 22 2.2.5 Stealth Techniques .......................... 22 2.2.6 Anti-analysis Techniques ....................... 23 2.3 Reverse Engineering ............................. 25 2.3.1 Reversing Malicious Software .................... 26 2.3.2 Assembly Language ......................... 27 2.3.3 Basic x86 Architecture ........................ 27 2.4 Miscellaneous Analysis ............................ 29 2.4.1 File Fingerprinting .......................... 29 2.4.2 AV Testing .............................. 30 2.4.3 String Analysis ............................ 30 2.4.4 Packer Detection ........................... 30 2.5 Reverse Code Analysis ............................ 31 2.5.1 Static Code Analysis ......................... 31 2.5.2 Live Code Analysis .......................... 32 2.5.3 Disassembler ............................. 32 2.5.4 Decompiler .............................. 33 vi 2.5.5 Debugger ............................... 33 2.6 Behavioral Analysis .............................. 36 2.6.1 Registry Monitoring ......................... 37 2.6.2 Process Monitoring .......................... 38 2.6.3 File System Monitoring ....................... 39 2.6.4 InstallSpy ............................... 40 2.6.5 SysAnalyzer ............................. 40 2.6.6 Network Monitoring ......................... 41 2.6.7 Capture BAT ............................. 41 2.6.8 Sandboxes .............................. 42 2.7 Literature Review ............................... 43 2.8 Summary ................................... 45 3 Reversing Mariposa Botnet 46 3.1 Overview ................................... 47 3.2 Behavioral Analysis .............................. 49 3.2.1 Environment setup .......................... 49 3.2.2 Network Analysis ........................... 51 3.2.3 Sandbox Analysis .......................... 54 3.3 Dynamic Code Analysis ........................... 56 3.3.1 De-obfuscation and Decryption ................... 58 3.3.2 Anti-debugging traps in Mariposa .................. 59 vii 3.3.3 Second Layer Decryption ...................... 62 3.3.4 Code Injection ............................ 65 3.3.5 Injected Thread Activity ....................... 71 3.4 Modules .................................... 75 3.4.1 Spreader Module ........................... 75 3.4.2 Uploader and Downloader Modules ................. 79 3.5 Functional diagram .............................. 80 3.6 Summary ................................... 82 4 Zeus Crimeware Analysis 83 4.1 Overview ................................... 83 4.2 Zeus components ............................... 85 4.2.1 C&C Server .............................. 85 4.2.2 Bot Builder .............................. 86 4.3 Network Analysis ............................... 93 4.4 Reverse Engineering of Zeus ......................... 95 4.4.1 Revealing De-obfuscation ...................... 97 4.4.2 Code Injection and Installation ....................101 4.4.3 After-Injection Activity ........................103 4.5 Key Extraction ................................104 4.5.1 Automated Key Extraction ......................105 4.6 Summary ...................................106 viii 5 Control Flow Visualization 107 5.1 Low-level Program Comprehension .....................110 5.2 Tracks: The Sequence Viewer ........................111 5.2.1 Static Control Flow ..........................111 5.2.2 Dynamic Control Flow ........................112 5.2.3 Navigation History ..........................114 5.2.4 Diagram Features ...........................115 5.2.5 Design and Implementation .....................115 5.3 Case Study ..................................118 5.3.1 Obfuscation and Decryption .....................120 5.3.2 Injection Phase ............................122 5.3.3 Injection Preparation .........................123 5.3.4 Injection ...............................125 5.3.5 After Injection ............................125 5.4 Analysis ....................................126 5.5 Summary ...................................130 6 Conclusion 131 6.1 Summary ...................................131 6.2 Future Work ..................................133 Appendix 135 ix List of Figures 2.1 Simple Autorun File ............................. 20 2.2 Typical IDA Pro Screen ............................ 35 2.3 IDA Pro Graphical View ........................... 36 2.4 IDAStealth Interface ............................. 37 2.5 Regshot Interface ............................... 38 2.6 Process Explorer Interface .......................... 39 3.7 Confined Environment Structure ....................... 50 3.8 Mariposa Bot Initialization Protocol ..................... 52 3.9 Mariposa Bot Liveness Protocol ....................... 53 3.10 Mariposa Bot Action Protocol ........................ 54 3.11 Mariposa File System Activity By GFI Sandbox ............... 55 3.12 Mariposa Registry Activity By GFI Sandbox ................. 55 3.13 Mariposa Network Activity By GFI Sandbox ................ 56 3.14 Mariposa Decryption Phases ......................... 57 3.15 Unwanted Loop ................................ 58 3.16 First Decryption Routine ........................... 59 x 3.17 Stack Segment Register Trap in Mariposa .................. 61 3.18 OutputDebugString Trap in Mariposa .................... 62 3.19 Second Layer Decryption ..........................