Bypassing Modern Sandbox Technologies

Gustav Lundsgård & Victor Nedström Bypassing modern sandbox technologies technologies sandbox modern Bypassing Master’s Thesis Bypassing modern sandbox technologies An experiment on sandbox evasion techniques — Gustav Lundsgård An experiment on sandbox evasion techniques Victor Nedström Series of Master’s theses Department of Electrical and Information Technology LU/LTH-EIT 2016-517 Department of Electrical and Information Technology, http://www.eit.lth.se Faculty of Engineering, LTH, Lund University, 2016. Bypassing modern sandbox technologies An experiment on sandbox evasion techniques Gustav Lundsgård Victor Nedström Department of Electrical and Information Technology Lund University Advisor: Paul Stankovski June 21, 2016 Printed in Sweden E-huset, Lund, 2016 Abstract Malware (malicious software) is becoming an increasing problem, as it continu- ously grows both in numbers and complexity. Traditional, signature based anti- virus systems are often incapable of detecting new, sophisticated malware, which calls for more advanced tools. So called sandboxes are tools which automate the process of analyzing malware by actually running them in isolated environments and observing their behavior. Although this approach works very well in theory, some malware have recently begun deploying sandbox detection techniques. With the help of these techniques, malware may detect when they are being analyzed and manage to evade the sandbox by hiding their malicious behavior. The authors of this Master’s Thesis have developed and compared differ- ent types of sandbox detection techniques on five market leading products. It was shown that an average of roughly 43% of the detection techniques developed were capable of both detecting and bypassing the sandboxes, and that the best performing sandbox caught as much as 40% more of the techniques than the worst. Patterns of weaknesses were noticed in the sandboxes, affecting primar- ily the limited hardware and lack of user interaction - both of which are typical sandbox characteristics. Surprisingly, the time for which the sandbox vendors had been developing their sandboxing technology seemed to have no positive impact on the result of their product, but rather the other way around. Further- more, some detection techniques proved very efficient while being trivial to de- velop. The test results have been communicated to the sandbox vendors, and the authors are of the belief that the sandboxes could be quite significantly improved with these results as a guideline. i ii Acknowledgements This Master’s Thesis is a result not only of our own work but also of the efforts of people we have had the privilege to work with. Without you, this Master’s Thesis would not be what it is today. First and foremost, we would like to warmly thank our supervisor at the department of Electrical and Information Technology, Paul Stankovski, for his com- mitment and care throughout the entire process. Secondly, we want to express our sincere gratitude towards Coresec Systems AB for giving us the opportunity to write our Master’s Thesis on a topic close to our hearts. We would especially like to thank our two supervisors, Stefan Lager and David Olander, for their continuous support, technical advisory and encouragement. Furthermore, both Philip Jönsson and Holger Yström deserve many thanks for their efforts in helping and sharing their technical expertise. We are very grateful to all of you for everything we have learnt during the process and are looking forward to working with you in the future. Lastly, we would like to thank The Swedish National Board of Student Aid, Centrala Studiestödsnämnden (CSN), for their investment in our studies. iii iv Acronyms A API - Application Programming Interface C C&C - Command and Control (server) CPU - Central Processing Unit D DLL - Dynamic Link Library DMZ - Demilitarized Zone DNS - Domain Name Server G GPU - Graphics Processing Unit GUI - Graphical User Interface H HTTP - Hypertext Transfer Protocol I IAT - Import Address Table ICMP - Internet Control Message Protocol IRC - Internet Relay Chat M MAC - Media Access Control P PE - Portable Executable R RDP - Remote Desktop Protocol V v VLAN - Virtual Local Area Network VM - Virtual Machine VMM - Virtual Machine Monitor VPN - Virtual Private Network vi Glossary C Command and Control (C&C) server: a server to which malware, after infecting a target host, connects to download files or receive additional instructions. D Debugger: enables a program to be examined during execution. Most debuggers can run programs step by step. F False positive: a benign file causing a "false alarm" in an anti-malware system, commonly because its behavior resembles that of malicious files. G Golden image: an operating system image configured according to best effort. H Hypervisor: software that enables virtualization by creating and running virtual machines. M Malware sample: a single copy of malware being subject to analysis. R Ransomware: a type of malware that encrypts files on a file system and demands a ransom to be paid in exchange for the decryption key. S Snapshot: a preserved state in a virtual machine. Z Zero-day threat: a previously unseen threat. vii viii Table of Contents 1 Introduction 1 1.1 Background ................................ 1 1.2 Problem Description ............................ 1 1.3 Purpose .................................. 2 1.4 Method ................................... 2 1.5 Scope .................................... 3 1.6 Report Layout ............................... 4 2 Theory 5 2.1 Malware .................................. 5 2.1.1 Origin and purpose 5 2.1.2 Types of malware 6 2.2 Malware analysis .............................. 7 2.2.1 Static Analysis 7 2.2.2 Dynamic Analysis 8 2.3 Malware analysis detection ........................ 8 2.4 The Portable Executable file format ................... 9 2.5 Running executable files .......................... 11 2.6 Virtual machines .............................. 11 2.7 Sandboxes ................................. 12 2.7.1 Detection of virtual machines and sandboxes 13 2.8 DLL injection and API hooking ...................... 14 2.9 Sandbox analysis procedure ........................ 15 2.10 Related work ................................ 17 3 System Description 19 3.1 Introduction ................................ 19 3.2 Vendors and sandboxes .......................... 19 3.2.1 Vendor A 19 3.2.2 Vendor B 20 3.2.3 Vendor C 21 3.2.4 Vendor D 22 3.2.5 Vendor E 22 3.2.6 Sandbox specification summary 22 ix 4 Experiments 25 4.1 Introduction ................................ 25 4.2 Test case design .............................. 25 4.3 Test method, environment and configuration .............. 27 4.3.1 Inline deployment 29 4.3.2 Upload deployment 30 4.4 Test cases ................................. 31 4.4.1 Timing 32 4.4.2 Process 33 4.4.3 File 33 4.4.4 Environment 34 4.4.5 Hardware 35 4.4.6 Network 35 4.4.7 Interactive 36 4.5 Test criteria ................................ 37 4.6 Choice of malware ............................. 37 5 Results 39 5.1 Introduction ................................ 39 5.2 Results per category ............................ 39 5.2.1 Timing 40 5.2.2 Process 42 5.2.3 File 44 5.2.4 Environment 46 5.2.5 Hardware 48 5.2.6 Network 50 5.2.7 Interactive 52 5.3 Result of golden image configuration .................. 54 5.4 Result summary .............................. 55 6 Discussion 57 6.1 Sandbox technologies and result patterns ................ 57 6.2 Weakness patterns based on detection technique ............ 60 6.3 Simplicity contra efficiency in detection techniques ........... 61 6.4 System lifetime in relation to test results ................ 62 6.5 General discussion and sources of error ................. 63 7 Conclusions 67 7.1 Future work ................................ 68 References 69 A Test case simplicity-efficiency score 73 x List of Figures 2.1 Flow chart of an inline API hook. .................... 14 2.2 Communication between user mode and kernel mode components. .. 15 3.1 Relative vendor size measured in number of employees ......... 20 3.2 Relative vendor age ............................ 21 4.1 Test case components overview ..................... 26 4.2 Test case flow chart ............................ 26 4.3 A simplified overview of the test environment. ............. 28 4.4 Detailed view of the test environment including network interfaces. 30 4.5 A detailed view of how the malware is hidden as a resource in the test case [1]. .................................. 32 5.1 Timing - detected malware per vendor. ................. 41 5.2 Process - detected malware per vendor. ................. 43 5.3 Files - detected malware per vendor. ................... 45 5.4 Environment - detected malware per vendor. .............. 47 5.5 Hardware - detected malware per vendor. ................ 49 5.6 Network - detected malware per vendor. ................. 51 5.7 Interactive category - detected malware per vendor. .......... 53 5.8 Malware detection rate per vendor. ................... 55 5.9 Average malware detection rate per category and coefficient of variation. 56 6.1 Sandbox detection rate per vendor where the test cases detected the sandbox .................................. 59 6.2 Simplicity contra efficiency score per category. ............. 62 xi xii List of Tables 3.1 Sandbox specifications and features ................... 23 5.1 Timing results statistics. ......................... 40 5.2 Timing category results per vendor. ................... 40 5.3 Process results statistics. ........................

Load more