ABSTRACT WU, CHIACHIH. Virtualization-Based Approaches for Mitigation of Malware Threats. (Under the direction of Xuxian Jiang.) Modern computer systems consist of a number of software layers to provide efficient resource management, secure isolation, and convenient environment for program development and ex- ecution. The hypervisor virtualizes multiple instances of hardware for guest operating systems or another layer of hypervisors with strong isolation between the virtualized hardware instances. Running directly on hardware, physical or virtualized, the operating systems (OSs) provide efficient resource management and a convenient set of interface for applications to access the hardware resource including CPU. In addition, the OS regulates each application with virtual memory, user identification, file permissions, etc. such that a non-privileged application cannot interact with other applications or access privileged files without gaining the corresponding permissions. One level above the OS, the runtime libraries and system daemons help applications to communicate with the OS in an easier manner. Each software layer has various software bugs and design flaws such that the secure isolation could be broken through. This motivates our research of securing different software layers with a series of virtualization-based approaches. In this dissertation, we firstly present an OS-level virtualization system, AirBag, to improve Android architecture in the aspect of malware defense and analysis for dealing with flaws in Android runtime environment. AirBag allows users to “test” untrusted apps in an isolated Android runtime environment without private data leakage, system files corruption, and more severe damages such as sending SMSs to premium numbers. Besides, users can “profile” untrusted apps in the instrumented isolated Android runtime, which improves the capabilities of dynamic analysis. However, such an OS-level approach is vulnerable to attacks that exploit the vulnerabilities inside the OS. When the OS is compromised, all private data such as the bank account and password could be leaked while the amount of an online payment could be changed by the attacker. Since building a bug-free OS is impossible, we present a tiny hypervisor, tHype, to provide trusted IO access to users when they input the sensitive data or perform critical operations. Compared to existing hypervisors, tHype outperforms by its code size since it only virtualizes critical IO on mobile devices, mainly touchscreen and framebuffer. Yet, in general, virtualizing computer systems is complicated such that most existing hypervisors have large code base which make them vulnerable. Even worse, the hosted (or Type-II) hypervisor is considered to include the host OSs in its trusted code base (TCB) that makes it have a wider attack surface compared to bare-metal (or Type-I) hypervisors. We present the DeHype system to reduce the TCB of the hosted hypervisor by deprivileging its execution to user mode. With DeHype, the hypervisor is executed in the context of an user mode process for each guest VM, which prevents a compromised hypervisor from attacking other guests. © Copyright 2015 by Chiachih Wu All Rights Reserved Virtualization-Based Approaches for Mitigation of Malware Threats by Chiachih Wu A dissertation submitted to the Graduate Faculty of North Carolina State University in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Computer Science Raleigh, North Carolina 2015 APPROVED BY: Douglas Reeves William Enck Huiyang Zhou Xuxian Jiang Chair of Advisory Committee DEDICATION To my parents. ii BIOGRAPHY The author was born in a small town ... iii ACKNOWLEDGEMENTS I would like to thank my advisor for his help. iv TABLE OF CONTENTS LIST OF TABLES ......................................................... vii LIST OF FIGURES ........................................................ viii Chapter 1 INTRODUCTION .............................................. 1 1.1 Problem Overview................................................. 1 1.2 Our Approach .................................................... 3 1.3 Dissertation Statement and Contributions ............................... 4 1.4 Dissertation Organization ........................................... 5 Chapter 2 RELATED WORK ............................................... 6 2.1 Securing Mobile Apps .............................................. 6 2.2 Securing I/O Access................................................ 8 2.3 Securing Hypervisors............................................... 10 Chapter 3 Isolating Android Malware with AirBag .............................. 13 3.1 Introduction ..................................................... 13 3.2 Design.......................................................... 16 3.2.1 Design Goals and Threat Model.................................. 16 3.2.2 Enabling Techniques ......................................... 17 3.2.3 Additional Capabilities........................................ 19 3.3 Implementation................................................... 21 3.3.1 Namespace/Filesystem Isolation................................. 21 3.3.2 Context-Aware Device Virtualization.............................. 22 3.3.3 Decoupled App Isolation Runtime................................ 26 3.3.4 Lessons Learned............................................. 28 3.4 Evaluation....................................................... 29 3.4.1 Effectiveness ............................................... 30 3.4.2 Performance Impact.......................................... 34 3.4.3 Power Consumption and Memory Usage........................... 35 3.5 Discussion....................................................... 36 3.6 Summary........................................................ 38 Chapter 4 Securing Critical I/O Operations with tHype .......................... 39 4.1 Introduction ..................................................... 39 4.2 Design.......................................................... 43 4.2.1 Critical I/O Virtualization...................................... 43 4.2.2 Capability-Based Memory Sharing and Alternate Memory View .......... 45 4.3 Implementation................................................... 46 4.3.1 Critical I/O Virtualization...................................... 46 4.3.2 Capability-Based Memory Sharing and Alternate Memory View .......... 47 4.4 Evaluation....................................................... 51 4.4.1 Performance Overhead of Memory View Switching ................... 51 v 4.4.2 Latency of Virtualized IO....................................... 52 4.5 Discussion....................................................... 52 4.6 Summary........................................................ 54 Chapter 5 Deprivileging Hosted Hypervisors with DeHype ....................... 55 5.1 Introduction ..................................................... 55 5.2 Design.......................................................... 57 5.2.1 Dependency Decoupling....................................... 59 5.2.2 Memory Rebasing............................................ 60 5.2.3 Optimizations .............................................. 61 5.3 Implementation................................................... 62 5.3.1 Dependency Decoupling....................................... 62 5.3.2 Memory Rebasing............................................ 63 5.3.3 Optimizations .............................................. 66 5.3.4 Lessons Learned............................................. 68 5.4 Evaluation....................................................... 69 5.4.1 Security Benefits............................................. 69 5.4.2 Other Benefits .............................................. 70 5.4.3 Performance ............................................... 73 5.5 Discussion....................................................... 73 5.6 Summary........................................................ 76 Chapter 6 Conclusion and Future Work ..................................... 77 BIBLIOGRAPHY ......................................................... 79 vi LIST OF TABLES Table 3.1 Supported Android Hardware Devices in AirBag ...................... 23 Table 3.2 Effectiveness of AirBag in Successfully Blocking 20 Representative Android Malware.................................................... 29 Table 3.3 Benchmarks Used in Our Evaluation............................... 34 Table 4.1 Hypercalls to Support Capability-Based Memory Sharing and Alternate Mem- ory View ................................................... 50 Table 5.1 Ten Privileged Services in DeHype................................. 63 Table 5.2 Cached VMCS Fields........................................... 67 Table 5.3 Software Packages used in Our Evaluation........................... 73 vii LIST OF FIGURES Figure 1.1 An Overview of the Dissertation.................................. 5 Figure 3.1 An Overview of AirBag to Confine Untrusted Apps .................... 17 Figure 3.2 Framebuffer Virtualization in AirBag (Nexus One) .................... 22 Figure 3.3 Telephony Virtualization in AirBag................................ 26 Figure 3.4 Seamless Integration of AirBag .................................. 28 Figure 3.5 GoldDream Analysis.......................................... 31 Figure 3.6 DKFBootKit Analysis.......................................... 32 Figure 3.7 HippoSMS Analysis........................................... 33 Figure 3.8 Performance Measurement of AirBag on Google