Automated Analyses of Malicious Code Kris Mikael Krister Master of Science in Computer Science Submission date: June 2009 Supervisor: Torbjørn Skramstad, IDI Norwegian University of Science and Technology Department of Computer and Information Science Problem Description Malware (malicious software) is a general term for any malicious program such as a virus, worm, trojan, bot or rootkit. The prevalence of malware is rapidly growing on the Internet and poses an increasing threat to computer systems. Effective incident handling can be achieved by analysing malware in order to understand its functionality and capacity. This may involve analysing already compromised systems or assessing the level of risk for systems being exposed to such a threat. However, malware analysis is labour intensive and time consuming, and does not scale well with the ever increasing prevalence of malware. Automating parts of the analysis process can reduce the required amount of human intervention, and save precious time during the analysis. Techniques involving running malware in a controlled and secure environment whilst monitoring its behaviour is often referred to as dynamic analysis. This project should focus on studying dynamic malware analyses, and automate such a process. The student is free to automate any analysis process that is regarded dynamic, but it would be preferable if the information gained from the automated analysis is actionable. That is, the information gained is directly useful for handling the particular incident(s) where the malware is involved. The project may optionally look into integrating the final system with NorCERT's internal system for handling malware samples. Assignment given: 15. January 2009 Supervisor: Torbjørn Skramstad, IDI Preface This report is the result of a problem stated by the Norwegian Computer Emer- gency Response Team (NorCERT), a division in the Norwegian National Security Authority (NSM). The task was structured and written by Kris-Mikael Krister, at- tending the Norwegian University of Science and Technology (NTNU), Department of Computer and Information Science (IDI) and was accomplished as a master’s thesis in the late spring of 2009. I would like to thank my internal supervisor, Torbjørn Skramstad from IDI NTNU for guidelines and support when writing and structuring the thesis, and my external supervisor Lars Haukli from NorCERT for assistance regarding the thesis’ content, and facts about the malware topic. Oslo, June 2009 Kris-Mikael Krister Summary Sophisticated software with malicious intentions (malware) that can easily and aggressively spread to a large set of hosts is located all over the Internet. Such software struggles to avoid malware analysts to continue its malicious actions without interruption. It is difficult for analysts to find the locations of machines infected with unknown and alien malware. Likewise, it is hard to estimate the prevalence of the outbreak of the malware. Currently, the processes are done using resource demanding manual work, or simply rough guessing. Automating these tasks is one possible way to reduce the necessary resources. This thesis presents an in-depth study of which properties such a system should have. A system design is made based on the findings, and an implementation is carried out as a proof of concept system. The final system runs (malicious) software, and at the same time observes network traffic originating from the software. A signature for intrusion detection systems (IDSes) is generated using data from the observations. When loaded in an IDS, the signature localises hosts that are infected with the same malware type, making network administrators able to find and repair the hosts. The thesis also covers a deep introductory study of the malware problem and possible countermeasures, focusing on a malware analyst’s point of view. Contents List of Figures................................v List of Tables................................. vii Listings.................................... ix 1 Introduction1 1.1 Terminology and acronyms......................1 1.2 Result goals...............................3 1.3 Methodology..............................3 1.4 Related work..............................6 1.5 Document structure..........................6 2 Background and Rationale9 2.1 Malware propagation..........................9 2.2 Techniques for controlling malware.................. 12 2.3 Custom made malware......................... 15 2.4 The development of malware..................... 16 2.5 Threats from malware......................... 22 2.6 In need for more effective countermeasures.............. 28 2.7 Computer emergency response teams (CERTs)........... 30 2.8 Summary................................ 33 3 Phases of a Malware Analysis 35 3.1 Structure of a malware sample.................... 35 3.2 Surface scanning............................ 37 3.3 Dynamic malware analysis....................... 38 3.4 Static malware analysis........................ 38 3.5 Finalising the analysis......................... 39 4 Dynamic Analyses in Depth 41 4.1 Analysis methods............................ 41 4.2 Information prioritisation....................... 46 4.3 Available complete sandbox-solutions................. 47 4.4 In need for applications running locally............... 48 4.5 Available smaller dynamic analytical tools.............. 48 4.6 Summary of described tools and solutions.............. 49 4.7 Problems with dynamic analysis................... 49 i 5 A Malware Analysis Scenario 55 5.1 Initial setup and environment overview................ 55 5.2 Surface scanning............................ 56 5.3 Dynamic analysis............................ 59 5.4 Reflections................................ 61 6 Concretisation of the Task 63 6.1 Related work.............................. 63 6.2 Available approaches.......................... 65 6.3 Selecting an approach......................... 72 6.4 Selecting products as base....................... 73 6.5 Zero Wine in detail........................... 79 6.6 Summary of selections......................... 81 7 Requirements Specification 83 7.1 High level requirements........................ 83 7.2 Use case analysis............................ 84 7.3 Overall list of requirements...................... 88 7.4 Mapping of requirements........................ 90 8 Design 93 8.1 Modifying Zero Wine.......................... 93 8.2 Task automation............................ 94 8.3 Generating IDS signatures....................... 95 8.4 Other design choices.......................... 96 8.5 Choice of programming languages................... 96 8.6 System deployment overview..................... 97 8.7 Design limitations........................... 98 9 Implementation 101 9.1 Implementing vmcom lite....................... 101 9.2 Implementing Zero+One........................ 101 9.3 Code conventions............................ 106 9.4 Testing................................. 108 9.5 Software licenses............................ 112 9.6 Integration with NAAS........................ 112 10 Evaluation 115 10.1 Result goals achieved, system requirements fulfilled......... 115 10.2 Discussion................................ 119 10.3 Further work.............................. 119 11 Thesis Conclusion 123 11.1 Contributions.............................. 123 11.2 The future................................ 124 References 137 ii A User Manual 139 A.1 Configuring and using Zero+One................... 139 A.2 Configuring and using vmcom lite.................. 139 B Screen Captures 141 C Implementation Appendix 145 C.1 iptables script.............................. 145 C.2 vmcom lite source code........................ 146 C.3 Zero+One source code......................... 152 D Testing Appendix 163 D.1 Suspicious API calls returned..................... 163 D.2 Email correspondence......................... 163 iii iv List of Figures 2.1 Increase in the amount of discovered malware during the time span from 1997 to 2008............................ 10 2.2 Amount of discovered vulnerabilities in software code during the time span from 1997 to 2008...................... 11 2.3 Attacker in control of three networks of infected hosts........ 13 2.4 Control flow alteration in a sample to change its appearance.... 19 2.5 Oligomorphic, polymorphic and metamorphic malware mutations. 21 2.6 Attacker controlling an attack on a server through his or her network of infected hosts (botnet)........................ 24 3.1 Graphical representation of the different steps in a malware analysis. 36 4.1 Analysing malware in a separate network isolated from remote communication.............................. 43 4.2 Conditional if-clause giving two branches in the program flow... 52 6.1 Example of an IDS architecture.................... 69 6.2 Example of an IPS architecture.................... 70 6.3 Dependency tree for a unmodified version of Zero Wine....... 80 6.4 UML Sequence diagram displaying program flow from unmodified Zero Wine................................ 82 7.1 Use case diagram for “Generate Snort signature”........... 85 7.2 Use case diagram for “Gain information about sample”....... 86 8.1 A possible system set up shown as a UML deployment diagram... 97 8.2 Sequence diagram showing program flow in the final system..... 98 9.1 NAAS integration in a simplified UML class diagram......... 114 B.1 Screen capture from vmcom lite’s help screen............ 142 B.2 Screen capture from Zero+One’s help screen.............. 142 B.3 Screen capture from Wireshark capturing network