Reverse Engineering of a Malware
Total Page:16
File Type:pdf, Size:1020Kb
REVERSE ENGINEERING OF A MALWARE EYEING THE FUTURE OF SECURITY A Thesis Presented to The Graduate Faculty of The University of Akron In Partial Fulfillment of the Requirements for the Degree Master of Science Supreeth Burji August, 2009 REVERSE ENGINEERING OF A MALWARE EYEING THE FUTURE OF SECURITY Supreeth Burji Thesis Approved: Accepted: ________________________________ ________________________________ Advisor Department Chair Dr. Kathy J. Liszka Dr. Chien-Chung Chan ________________________________ ________________________________ Faculty Reader Dean of the College Dr. Timothy W. O'Neil Dr. Chand Midha ________________________________ ________________________________ Faculty Reader Dean of the Graduate School Dr. Wolfgang Pelz Dr. George R. Newkome ________________________________ Date ii ABSTRACT Reverse engineering malware has been an integral part of the world of security. At best it has been employed for signature logging malware until now. Since the evolution of new age technologies, this is now being researched as a robust methodology which can lead to more reactive and proactive solutions to the modern security threats that are growing stronger and more sophisticated. This research in its entirety has been an attempt to understand the in and outs of reverse engineering pertaining to malware analysis, with an eye to the future trends in security. Reverse engineering of malware was done with Nugache P2P malware as the target showing that signature based malware identification is ineffective. Developing a proactive approach to quickly identifying malware was the objective that guided this research work. Innovative malware analysis techniques with data mining and rough sets methodologies have been employed in this research work in the quest of a proactive and feasible security solution. iii ACKNOWLEDGEMENTS I extend my warmest regards and appreciation to Dr. Kathy J. Liszka, a wonderful teacher and guide I have had in my life. Without her guidance, support and inputs this research work would not have been possible. I would like to thank Dr. Wolfgang Pelz, for giving me an opportunity to study the domain that interested me a lot, the world of Computer Security. The ideas for this research work, was in these Security courses where it all began. I thank Dr. Timothy W. O'Neil, whose valuable inputs during the research work, gave a new and thought provoking direction on more than one occasion. I would like thank Dr. Chan, for helping me throughout the Data Mining phase and with usage of Roughsets and guiding me in depth and detail about the subject. I would like to thank Adam, for knowingly or unknowingly helping me to go offensive against certain kind of malware strains called the Rogue Softwares. I thank Stephen Sciarini and Chuck Van Tilburg, for extending their help in the research labs and providing a workable environment there. I would like to thank Snehal, one of my very good friend and whom I look up to as my sister, for extending her help in one of the case studies of this research work. I thank my parents for the kind of patience they have shown with me all these years and provided support. Thank you mom and dad. iv Last, but not the least, I would like to convey my heartfelt appreciation to my girl friend Preetham, to whom I dedicate the work to, without whose love and support I would have never made it so far. v TABLE OF CONTENTS Page LIST OF TABLES..............................................................................................................ix LIST OF FIGURES.............................................................................................................x CHAPTER I. INTRODUCTION............................................................................................................1 II. REVERSE ENGINEERING NUGACHE......................................................................4 2.1 Overview............................................................................................................4 2.2 Experiments.......................................................................................................6 2.3 An Explanation of P2P.......................................................................................7 2.3.1 Implementation...................................................................................9 2.3.2 Infection Techniques.........................................................................10 2.3.3 Infection............................................................................................11 2.4 Network Flow Analysis...................................................................................14 2.5 Possible Security Solutions..............................................................................15 2.6 The Nugache Creator.......................................................................................15 III. THE LIFECYCLE OF MALWARE...........................................................................22 3.1 Overview..........................................................................................................22 3.2 Hardware Propagation.....................................................................................23 vi 3.3 Application Obfuscation ..................................................................................25 3.4 Key Loggers .....................................................................................................27 3.5 Conclusions ......................................................................................................29 IV. ROGUE SOFTWARE .................................................................................................30 4.1 Attack Trends ...................................................................................................31 4.2 Reverse Engineering the Rogue softwares ......................................................32 4.3 Features ............................................................................................................34 4.4 Virus Total Reports ..........................................................................................35 V. DATA MINING ............................................................................................................40 5.1 Patterns .............................................................................................................40 5.2 Dataset..............................................................................................................41 5.3 System Design .................................................................................................42 5.4 Implementation ................................................................................................43 5.5 Decision Trees .................................................................................................45 5.6 Naive Bayesian Classifier ................................................................................46 5.7 Pattern Analysis ...............................................................................................48 VI. ROUGH SETS .............................................................................................................51 6.1 Predictive Modeling .........................................................................................51 6.2 Implementation ................................................................................................52 6.3 Results and Evaluation .....................................................................................54 VII. CONCLUSIONS AND FUTURE WORK.................................................................57 REFERENCES ..................................................................................................................59 APPENDICES ...................................................................................................................62 vii APPENDIX A. THE MALWARE DATASET.....................................................63 APPENDIX B. DATA MINING...........................................................................73 APPENDIX C. ROUGH SETS.............................................................................76 LIST OF TABLES Table Page 4.1 Characteristics of similar Rogue malware strain.........................................................34 5.1 Attribute list of the malware dataset with type characteristics....................................41 5.2 Microsoft Clustering algorithm applied on the malware dataset.................................49 A.1 Attribute list used for creating the malware dataset....................................................63 A.2 Malware dataset with the file size and MD5 hash attributes......................................64 A.3 Malware dataset with the time/date and filename attributes.......................................65 A.4 Malware dataset with the unique strings, URL, registry and API reference attributes.............................................................................................................................67 A.5 Malware dataset the with the file packer attributes.....................................................68 A.6 Malware dataset the with the programming platform attributes.................................70 A.7 Malware dataset the with the DLL, directory, file, internet access and label attributes.............................................................................................................................71 ix LIST OF FIGURES