Architecture for Analyzing Potentially Unwanted Applications
Total Page:16
File Type:pdf, Size:1020Kb
Architecture for analyzing Potentially Unwanted Applications Alberto Geniola School of Science Thesis submitted for examination for the degree of Master of Science in Technology. Espoo 10.10.2016 Thesis supervisor: Prof. Tuomas Aura Thesis advisor: M.Sc. Markku Antikainen aalto university abstract of the school of science master’s thesis Author: Alberto Geniola Title: Architecture for analyzing Potentially Unwanted Applications Date: 10.10.2016 Language: English Number of pages: 8+149 Department of Computer Science Professorship: Information security Supervisor: Prof. Tuomas Aura Advisor: M.Sc. Markku Antikainen The spread of potentially unwanted programs (PUP) and its supporting pay par install (PPI) business model have become relevant issues in the IT security area. While PUPs may not be explicitly malicious, they still represent a security hazard. Boosted by PPI companies, PUP software evolves rapidly. Although manual analysis represents the best approach for distinguishing cleanware from PUPs, it is inapplicable to the large amount of PUP installers appearing each day. To challenge this fast evolving phenomenon, automatic analysis tools are required. However, current automated malware analisyis techniques suffer from a number of limitations, such as the inability to click through PUP installation processes. Moreover, many malware analysis automated sandboxes (MSASs) can be detected, by taking advantage of artifacts affecting their virtualization engine. In order to overcome those limitations, we present an architectural design for imple- menting a MSAS mainly targeting PUP analysis. We also provide a cross-platform implementation of the MSAS, capable of running PUP analysis in both virtual and bare metal environments. The developed prototype has proved to be working and was able to automatically analyze more that 480 freeware installers, collected by the three top most ranked freeware websites, such as cnet.com, filehippo.com and softonic.com. Eventually, we briefly analyze collected data and propose a first strategy for detecting PUPs by inspecting intercepted HTTP traffic. Keywords: PUP, PPI, MSAS, PUA, Potentially Unwanted Application, Potentially Unwanted Programs iv Preface This thesis work constitutes the final step of my master education in Computer Engineering. It has been written during my exchange year, spent at the Aalto University, in Helsinki. I wish to thank professor Tuomas Aura, who gave me the opportunity of working in the research area of the information security deparment. I had the opportunity to expand my education background, by working with IT security research experts, immerse in an intellectually challenging environment. Special thanks go to Markku Antikainen, whose valuable advices guided me while writing the thesis. I must also offer a very special word of thanks to all my researchers colleagues of the security department, especially to Blomqvist Tuomas, who demonstrated genuine interest about my work. Great thanks go to professor Fulvio Risso, for his constant (and valuable) availability and support. I would like to thank my two flatmates, Luigi and Raul, whose sympathy and cheerfulness have been a relief during hard times. Furthermore, I would like to thank all my closest friends Marina, Marta, Valentina, Valeria D. for their constant emotional support. Thanks to my girlfriend Elena, who always believed in me and supported my ideas. Special rewards go to Valeria Z., who inspired me with her IT-difficulties. I must spend special word of thanks for Francesco and Riccardo, two of my closest friends who shared hard times during these two years of master studies. Lastly, I would like to thank all family that has always stood by my side. Great thanks to Maria, whose generosity helped me in finishing my studies. Turin, Sep 26, 2016 Alberto Geniola This work was supported by TEKES as part of the Cyber Trust program of DIGILE (the Finnish Strategic Center for Science, Technology and Innovation in the field of ICT and digital business). Contents Abstract........................................ iii Preface......................................... iv Contents.........................................v Symbols and abbreviations.............................. viii 1 Introduction1 1.1 Motivation....................................1 1.2 Research goals..................................2 1.3 Structure.....................................3 2 Background5 2.1 Potentially Unwanted Programs.........................5 2.1.1 Definition.................................5 2.1.2 Potentially-unwanted ambiguity.....................6 2.1.3 PUP Classification............................7 2.1.4 Severity..................................8 2.1.5 Profitability................................9 2.1.6 The EULA trap............................. 10 2.1.7 User informed-consent to dicriminate PUP.............. 10 2.1.8 Habituation and blind approach.................... 10 2.2 Problem: PUP on Windows........................... 11 2.2.1 Windows installers............................ 11 2.2.2 Where do PUPs fit the most...................... 13 2.3 PUP detection state of the art......................... 14 2.3.1 How antivirus detect PUPs....................... 14 2.3.2 Suspicious behaviors........................... 15 2.4 Malware analysis techniques to the rescue................... 15 2.4.1 Static Analysis.............................. 16 2.4.2 Dynamic analysis............................. 21 2.5 Malware Sandbox Analysis Systems...................... 26 2.5.1 Limitations................................ 27 2.6 Automated GUI interaction........................... 27 2.6.1 Windows UI Architecture........................ 28 2.6.2 Low level UI interactions........................ 28 2.6.3 Basic Win32 UI messages........................ 29 2.6.4 Windows UI libraries........................... 30 2.6.5 Inspection tools for windows: Spy++ and Snoop........... 31 2.6.6 Inspection tools limitations....................... 32 v vi 2.6.7 Image recognition frameworks...................... 32 2.6.8 Optical character recognition techniques................ 33 2.6.9 UI automation frameworks....................... 33 2.6.10 Installers vs Applications, considerations................ 34 3 Related Work 39 3.1 Automated Malware analysis.......................... 39 3.1.1 Anti-spyware solutions.......................... 39 3.1.2 Sandbox Analysis............................. 40 3.2 Automated UI Interaction............................ 41 4 Problem Statement 43 4.1 Automating software installation........................ 44 4.2 Data collection and correlation......................... 44 4.3 Scalability and performances.......................... 44 4.4 Avoiding MSASs detection............................ 45 5 Design of a Windows PUP analysis infrastructure 47 5.1 Design goals.................................... 47 5.2 Architecture.................................... 48 5.2.1 Overview................................. 49 5.2.2 Crawlers.................................. 50 5.2.3 Central DB................................ 51 5.2.4 Host Controller.............................. 53 5.2.5 Sandbox Machine............................. 55 5.2.6 Networking Design............................ 58 5.2.7 Bare metal and virtual environments support............. 63 5.3 UI Interaction................................... 67 5.3.1 UI interaction engine basic architecture................ 67 5.3.2 Automated UI interaction challenges.................. 68 5.3.3 UI elements detection.......................... 71 5.3.4 UI element selection........................... 73 5.3.5 Interacting with UI elements...................... 76 6 Implementation details 79 6.1 Central database................................. 79 6.1.1 DB Schema................................ 80 6.1.2 Multiple Host Controller synchronization............... 80 6.2 Resource monitoring implementation...................... 81 6.2.1 Shared resource access in Windows NT................ 82 6.2.2 API Hooking: Overview......................... 84 6.2.3 Dll Injection & Injector......................... 87 6.2.4 Clooser look at HookingDLL...................... 89 6.2.5 Process hierarchy, dll injection and API hooking........... 94 6.2.6 Hooking Windows services........................ 97 6.3 Sniffer....................................... 101 6.3.1 Networking services........................... 101 6.3.2 Host controller interaction........................ 104 6.3.3 Capture file synthesis.......................... 105 vii 7 Test and evaluation of results 109 7.1 Test configuration................................ 109 7.1.1 Job selection............................... 109 7.1.2 Infrastructure configuration....................... 111 7.2 Evaluation of results............................... 115 7.2.1 Install automation results........................ 116 7.2.2 Looking for PUP installers....................... 119 8 Conclusions and future work 133 8.1 Contributions................................... 133 8.2 Limitations.................................... 134 8.3 Future work.................................... 135 A Database schema 137 B Hooked APIs 139 References........................................ 141 viii Symbols and abbreviations Abbreviations API Application Programming Interface CTPH Context Triggered Piecewise Hashing DNS Domain Name Service EULA End User License Agreement FS File System GUI Graphical User Interface HTTP Hyper Text Transfer Protocol HTTPS Hyper Text Transfer Protocol over Secure Socket Layer IO Input Output IT Information Technology JSON