Master‟s Thesis Computer Science Thesis no: MCS-2011-07 January 2011 Runtime Analysis of Malware Muhammad Shahid Iqbal Muhammad Sohail School of Computing Blekinge Institute of Technology SE – 371 39 Karlskrona Sweden This thesis is submitted to the School of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Computer Science. The thesis is equivalent to 20 weeks of full time studies. Contact Information: Authors: Muhammad Sohail E-mail: [email protected] Muhammad Shahid Iqbal E-mail: [email protected] University advisor(s): Bengt Carlsson Martin Boldt Department of Systems and Software Engineering School of Computing Internet : www.bth.se/com Blekinge Institute of Technology Phone : +46 455 38 50 00 SE – 371 39 Karlskrona Fax : +46 455 38 50 57 Sweden ii ABSTRACT Context: Every day increasing number of malwares are spreading around the world and infecting not only end users but also large organizations. This results in massive security threat for private data and expensive computer resources. There is lot of research going on to cope up with this large amount of malicious software. Researchers and practitioners developed many new methods to deal with them. One of the most effective methods used to capture malicious software is dynamic malware analysis. Dynamic analysis methods used today are very time consuming and resource greedy. Normally it could take days or at least some hours to analyze a single instance of suspected software. This is not good enough especially if we look at amount of attacks occurring every day. Objective: To save time and expensive resources used to perform these analyses, AMA: an automated malware analysis system is developed to analyze large number of suspected software. Analysis of any software inside AMA, results in a detailed report of its behavior, which includes changes made to file system, registry, processes and network traffic consumed. Main focus of this study is to develop a model to automate the runtime analysis of software which provide detailed analysis report and evaluation of its effectiveness. Methods: A thorough background study is conducted to gain the knowledge about malicious software and their behavior. Further software analysis techniques are studied to come up with a model that will automate the runtime analysis of software. A prototype system is developed and quasi experiment performed on malicious and benign software to evaluate the accuracy of the newly developed system and generated reports are compared with Norman and Anubis. Results: Based on thorough background study an automated runtime analysis model is developed and quasi experiment performed using implemented prototype system on selected legitimate and benign software. The experiment results show AMA has captured more detailed software behavior then Norman and Anubis and it could be used to better classify software. Conclusions: We concluded that AMA could capture more detailed behavior of the software analyzed and it will give more accurate classification of the software. We also can see from experiment results that there is no concrete distinguishing factors between general behaviors of both types of software. However, by digging a bit deep into analysis report one could understand the intensions of the software. That means reports generated by AMA provide enough information about software behavior and can be used to draw correct conclusions. Keywords: Malware Analysis, Automated malware analysis, malicious software. ii ACKNOWLEDGMENTS In the name of Allah the most Merciful and Beneficent We are thankful to Almighty Allah who gave us this opportunity and strength to accomplish this study to best of our efforts. We would like to express our gratitude to our supervisor Dr. Bengt Carlsson, for both his continuous guidance throughout this period and for always finding the time. We are thankful to our co-supervisor Dr. Martin Boldt for his assistance and guidance in developing prototype of automated malware analyzer. We also like to thank Charlie Svahnberg for providing us with experiment environment and equipment. We are thankful to our friends who supported us throughout this time period and for being with us in all hard times. Last but not the least; we were forever being grateful to our parents for their countless prayers and unconditional support and for always being the best parents due to which we are able to stand here. iii TABLE OF CONTENTS RUNTIME ANALYSIS OF MALWARE ............................................................................................I ABSTRACT ...........................................................................................................................................I ACKNOWLEDGMENTS ................................................................................................................. III TABLE OF CONTENTS ..................................................................................................................... 1 LIST OF FIGURES .............................................................................................................................. 4 LIST OF TABLES ................................................................................................................................ 5 1 INTRODUCTION ....................................................................................................................... 6 1.1 CHALLENGES OF THE FIELD ................................................................................................... 7 1.2 STRUCTURE OF THE THESIS .................................................................................................... 8 2 BACKGROUND .......................................................................................................................... 9 2.1 RELATED WORK .................................................................................................................... 9 2.2 AVAILABLE SOLUTIONS ....................................................................................................... 11 2.2.1 Web based Analysis Tools ............................................................................................... 12 2.2.2 Open source Tools .......................................................................................................... 13 2.2.3 GUI Automation .............................................................................................................. 13 2.2.4 GUI Automation Tools .................................................................................................... 14 2.3 STATIC ANALYSIS VS. DYNAMIC ANALYSIS ........................................................................ 14 2.4 IDENTIFYING LEGITIMATE AND ILLEGITIMATE SOFTWARE .................................................. 15 2.4.1 Behavior of Legitimate Software .................................................................................... 15 2.4.2 Behavior of Illegitimate Software ................................................................................... 15 2.5 EMULATION ......................................................................................................................... 16 2.6 VIRTUALIZATION ................................................................................................................. 16 2.6.1 Classification of Virtualization ....................................................................................... 16 2.6.2 VMM Advantage ............................................................................................................. 17 2.7 DIFFERENCE BETWEEN VIRTUALIZATION AND EMULATION ................................................. 17 2.8 ORACLE VIRTUAL BOX ........................................................................................................ 17 3 RESEARCH DESIGN ............................................................................................................... 19 3.1 RESEARCH MOTIVATION ...................................................................................................... 19 3.2 AIMS AND OBJECTIVES ........................................................................................................ 19 3.3 RESEARCH QUESTION .......................................................................................................... 20 3.4 RESEARCH METHOD ............................................................................................................ 20 1 3.5 BACKGROUND STUDY .......................................................................................................... 22 3.5.1 Search Strategy ............................................................................................................... 22 4 CHARACTERISTICS OF MALICIOUS SOFTWARE ........................................................ 24 4.1 MALWARE SOURCES ............................................................................................................ 24 4.2 OBSERVED MALICIOUS BEHAVIOR ...................................................................................... 24 4.2.1 File System Activity ......................................................................................................... 24 4.2.2 Registry Activity .............................................................................................................. 25 4.2.3 Network Activity .............................................................................................................