Analysis Avoidance Techniques of Malicious Software
Total Page:16
File Type:pdf, Size:1020Kb
Edith Cowan University Research Online Theses: Doctorates and Masters Theses 2010 Analysis avoidance techniques of malicious software Murray Brand Edith Cowan University Follow this and additional works at: https://ro.ecu.edu.au/theses Part of the Computer Sciences Commons Recommended Citation Brand, M. (2010). Analysis avoidance techniques of malicious software. https://ro.ecu.edu.au/theses/138 This Thesis is posted at Research Online. https://ro.ecu.edu.au/theses/138 Edith Cowan University Copyright Warning You may print or download ONE copy of this document for the purpose of your own research or study. The University does not authorize you to copy, communicate or otherwise make available electronically to any other person any copyright material contained on this site. You are reminded of the following: Copyright owners are entitled to take legal action against persons who infringe their copyright. A reproduction of material that is protected by copyright may be a copyright infringement. Where the reproduction of such material is done without attribution of authorship, with false attribution of authorship or the authorship is treated in a derogatory manner, this may be a breach of the author’s moral rights contained in Part IX of the Copyright Act 1968 (Cth). Courts have the power to impose a wide range of civil and criminal sanctions for infringement of copyright, infringement of moral rights and other offences under the Copyright Act 1968 (Cth). Higher penalties may apply, and higher damages may be awarded, for offences and infringements involving the conversion of material into digital or electronic form. Analysis Avoidance Techniques of Malicious Software Analysis Avoidance Techniques of Malicious Software Murray Brand BEng (Hons) (Electronics and Communications) MEngSc (Electrical Engineering) GradCert (Computer Security) A thesis submitted for the Award of Doctor of Philosophy At the Faculty of Computing, Health and Science Edith Cowan University, Mount Lawley Campus Principal Supervisor Professor Craig Valli Associate Supervisor Doctor Andrew Woodward Submitted 30 November 2010 USE OF THESIS The Use of Thesis statement is not included in this version of the thesis. Analysis Avoidance Techniques of Malicious Software PUBLICATIONS ARISING FROM THIS RESEARCH Brand, M. (2007). Forensic Analysis Avoidance Techniques of Malware. Paper presented at the 5th Australian Digital Forensics Conference, Edith Cowan University, Mount Lawley Campus, Western Australia. Szewczyk, P., Brand, M. (2008). Malware Detection and Removal: An Examination of Personal Anti-Virus Software. Paper presented at the 6th Australian Digital Forensics Conference, Edith Cowan University, Mount Lawley Campus, Western Australia. Valli, C., Brand, M. (2008). Malware Analysis Body of Knowledge. Paper presented at the 6th Australian Digital Forensics Conference, Edith Cowan University, Mount Lawley Campus, Western Australia. Brand, M., Valli, C., Woodward, A. (2010). Lessons Learned from an Investigation into the Analysis Avoidance Techniques of Malicious Software. Paper presented at the 8th Australian Digital Forensics Conference, Duxton Hotel, Perth Western Australia. Brand, M., Valli, C., Woodward, A. (2010). Malware Forensics: Discovery of the intent of Deception. Paper presented at the 8th Australian Digital Forensics Conference, Duxton Hotel, Perth Western Australia. Brand, M., Valli, C., Woodward, A. (2010). Malware Forensics: Discovery of the intent of Deception. The Journal of Digital Forensics, Security and Law, 5(4), 31-42. iii Analysis Avoidance Techniques of Malicious Software ABSTRACT Anti Virus (AV) software generally employs signature matching and heuristics to detect the presence of malicious software (malware). The generation of signatures and determination of heuristics is dependent upon an AV analyst having successfully determined the nature of the malware, not only for recognition purposes, but also for the determination of infected files and startup mechanisms that need to be removed as part of the disinfection process. If a specimen of malware has not been previously extensively analyzed, it is unlikely to be detected by AV software. In addition, malware is becoming increasingly profit driven and more likely to incorporate stealth and deception techniques to avoid detection and analysis to remain on infected systems for a myriad of nefarious purposes. Malware extends beyond the commonly thought of virus or worm, to customized malware that has been developed for specific and targeted miscreant purposes. Such customized malware is highly unlikely to be detected by AV software because it will not have been previously analyzed and a signature will not exist. Analysis in such a case will have to be conducted by a digital forensics analyst to determine the functionality of the malware. Malware can employ a plethora of techniques to hinder the analysis process conducted by AV and digital forensics analysts. The purpose of this research has been to answer three research questions directly related to the employment of these techniques as: 1. What techniques can malware use to avoid being analyzed? 2. How can the use of these techniques be detected? 3. How can the use of these techniques be mitigated? These questions were effectively answered by validating anti-analysis techniques, showing how the techniques can be effectively detected and mitigated as well as by analyzing malware collected from the internet. This research contributes to the knowledge of malware analysis and digital forensics by: iv Analysis Avoidance Techniques of Malicious Software • Demonstrating that anti-analysis techniques can be very effective at hindering analysis by the tools typically used by analysts. • Showing that the use of anti-analysis techniques can be effectively detected and mitigated by the use of appropriate analysis techniques, scripts and plugins. • Support of claims virus signature based detection by anti-virus software can be far less than ideal. • Showing that extensive use of packers and protectors are employed by network based malware collected from the internet to obstruct signature based detection and to hinder analysis. • Support of an alternate paradigm of malware detection that could use detection of deception and anti-analysis techniques to detect malicious software instead of using virus signatures and heuristics. • Identification of a Malware Analysis Body of Knowledge (MABOK) that incorporates anti-analysis techniques as a core component. • Identification of deficiencies in analysis tools given the extent of available anti-analysis techniques. • Determination of an appropriate analysis methodology tailored for dealing with anti-analysis techniques. • Development of a taxonomy of analysis avoidance techniques. v Analysis Avoidance Techniques of Malicious Software DECLARATION I certify that this thesis does not, to the best of my knowledge and belief: (i) incorporate without acknowledgement any material previously submitted for a degree or diploma in any institution of higher education; (ii) contain any material previously published or written by another person except where due reference is made in the text; or (iii) contain any defamatory material. I also grant permission for the Library at Edith Cowan University to make duplicate copies of my thesis as required. Signature ……………………………………………………………………. Date ……………………………………………………………………………. vi Analysis Avoidance Techniques of Malicious Software ACKNOWLEDGEMENTS This thesis would not have been possible without the guidance and encouragement provided by my Principal Supervisor Professor Craig Valli and Associate Supervior Doctor Andrew Woodward. vii Analysis Avoidance Techniques of Malicious Software TABLE OF CONTENTS PUBLICATIONS ARISING FROM THIS RESEARCH .................................................................... III ABSTRACT ............................................................................................................................................. IV DECLARATION ..................................................................................................................................... VI ACKNOWLEDGEMENTS ................................................................................................................... VII LIST OF FIGURES ............................................................................................................................. XVII LIST OF TABLES .................................................................................................................................. XX CHAPTER 1 INTRODUCTION .......................................................................................................... 1 1.1. OVERVIEW .............................................................................................................................. 1 1.2. A STATEMENT OF THE PROBLEM ...................................................................................... 2 1.3. RESEARCH QUESTIONS ....................................................................................................... 6 1.4. SIGNIFICANCE OF RESEARCH ............................................................................................ 6 1.5. STRUCTURE OF THIS THESIS .............................................................................................. 8 CHAPTER 2 LITERATURE REVIEW ............................................................................................ 10 2.1. CHARACTERISATION OF NETWORK BASED MALWARE ..........................................