Workshop for Frontiers of Cloud Computing, Dec 1, 2011, IBM T.J. Watson Research Center, NY Process Out-Grafting: An Efficient “Out-of-VM” Approach for Fine-Grained Process Execution Monitoring Deepa Srinivasan, Zhi Wang, Xuxian Jiang, Dongyan Xu * North Carolina State University, Purdue University* Malware Infection Trend New malware samples collected by McAfee Labs, by month* *Figure source: McAfee Threats Report: Second Quarter 2011, McAfee Labs 2 Anti-Malware Isolation Traditional anti-malware tools are not well-isolated Virtual Machine (VM) introspection Isolate tool by placing it outside a VM Analyze states and events externally User-mode Applications Monitor Virtual Machine … OS Kernel Hypervisor 3 Anti-Malware Isolation Traditional anti-malware tools are not well-isolated Virtual Machine (VM) introspection Isolate tool by placing it outside a VM Analyze states and events externally User-mode Applications Monitor VM Virtual Introspection Machine … OS Kernel Hypervisor 4 Out-of-VM Solutions Livewire (Garfinkel et al. , NDSS ‘03) XenAccess (Payne et al. , ACSAC ‘07) VMScope (Jiang et al. , RAID ‘07) Lares (Payne et al. , Oakland ‘08) … 5 Semantic Gap in Introspection What we want to observe High-level states and events (e.g. system calls, processes) What we can observe Low-level states and events (e.g. raw memory, interrupts) Internal User-mode Applications Monitor … External Monitor Semantic Virtual Machine Gap OS Kernel Hypervisor 6 Addressing the Semantic Gap Guest view casting VMWatcher (Jiang et al. , CCS ‘07) Automatic generation of introspection-based tools Virtuoso (Dolan-Gavitt et al. , Oakland ‘11) Issues Sensitive to internal OS changes or updates Incompatible with existing anti-malware tools 7 Our Goal Support existing in-host process monitors out-of-VM without semantic gap! 8 In-host strace 9 In-host Monitors Process-level monitoring System calls Library calls Instruction execution traces 10 Process Out-Grafting Isolation Monitor protection from malware Compatibility Natural support for fine-grained user-mode process monitoring tools ( strace, ltrace, … ) Efficiency No significant performance overhead due to isolation 11 Rest of This Talk Motivation System Design Implementation & Evaluation Related Work Conclusion 12 Assumptions Trusted hypervisor Untrusted monitored VM Non-goals OS kernel-level monitoring Stealthy monitoring 13 Key Techniques Technique I: On-Demand Grafting Production VM Security VM Stub Suspect Process Monitor User User Kernel Kernel Hypervisor Technique II: Mode-sensitive Split Execution 14 Key Techniques Technique I: On-Demand Grafting Production VM Security VM Monitor User User Kernel Kernel Hypervisor 15 On-Demand Grafting On-Demand Grafting Relocate suspect process to security VM Enable efficient, native inspection Eliminate hypervisor intervention Support existing process monitoring tools Initiate out-grafting as needed Restore process after monitoring 17 When to Out-Graft? Process in user-mode Process in kernel-mode Hypervisor notified when user-mode execution resumes Suspect Process Production VM Physical NX Memory Frames User Kernel Hypervisor 18 What to Out-Graft? Architecture-specific resources No OS kernel-specific resources Main root cause in semantic gap Continued execution of out-grafted process Suspect Process Physical Production Memory VM Frames User EIP Kernel Network Disk File EAX Socket EBX … … Hypervisor 19 How to Out-Graft? Stub protected Production VM by hypervisor Security VM Stub Suspect Process Monitor Physical NXMemory X Frames User User Kernel Kernel Helper Module VCPU Register State Hypervisor Page Mappings 20 Mode-Sensitive Split Execution Mode-Sensitive Split Execution All user-mode execution occurs in security VM All kernel-mode execution occurs in production VM Out-grafted process considers itself running in production VM 22 System Call Redirection Smooth, continued execution of out-grafted process Monitor isolation Production VM No hypervisor Security VM Stub intervention! Suspect Process Monitor - System call number - Arguments User System Call User Kernel Network Kernel Socket Helper Module Disk File Hypervisor (e.g., KVM) 23 Page Fault Forwarding Consistent process memory mapping between VMs No semantic knowledge in security VM (e.g. memory-mapped file) Production VM Security VM Stub Suspect Process Monitor - Faulting address - Read/Write fault Page Page Fault Fault User User Kernel Kernel Page Tables Helper Module Hypervisor (e.g., KVM) 24 Page Fault Forwarding Consistent process memory mapping between VMs No semantic knowledge in security VM (e.g. memory-mapped file) Production VM Security VM Stub Suspect Process Monitor Physical Memory Frame Page Page Fault Fault User User Kernel Kernel Page Tables Helper Module Hypervisor (e.g., KVM) 25 Rest of This Talk Motivation System Design Implementation & Evaluation Related Work Conclusion 26 Implementation Hypervisor: KVM (2.6.36.1) Out-grafting support: +1309 SLOC Extended page tables support (Intel Core i7) Host OS: Ubuntu 10.04 (kernel 2.6.28) Guest OS: Ubuntu 9.04, Fedora 10 27 Security Analysis Monitor isolation and effectiveness System call forwarding Security VM protected from kernel attacks Production VM Security VM Stub Suspect Process Monitor User User Kernel Network Kernel Socket System Call Disk File Hypervisor (e.g., KVM) 28 Security Analysis Stub protection Hypervisor-protected memory and page tables Production VM Security VM Stub Suspect Process Monitor User User Kernel Kernel Hypervisor (e.g., KVM) 29 Security Analysis Out-grafting detection Strong policy for random, arbitrary out-grafting 30 Case Studies thttpd as test process for out-grafting Both disk and network usage Support existing in-host tools, out-of-VM strace, ltrace, gdb OmniUnpack (Martignoni et al. , ACSAC ‘07 ) 31 Case Studies thttpd as test process for out-grafting Both disk and network usage Support existing in-host tools, out-of-VM strace, ltrace, gdb OmniUnpack (Martignoni et al. , ACSAC ‘07 ) 32 strace In-VM strace Out-of-VM strace 33 Code Unpacking Detection OmniUnpack (Martignoni et al. , ACSAC ‘07) Track page writes and executions Detect unpacking when executing a previously-written page Faithful reproduction of algorithm in Linux Kernel module in security VM NX bit support only in Localized overhead in Thorough test of page security VM kernel out-of-VM monitoring fault forwarding 34 Performance Dell T1500, Intel Core i7, 4 cores, 2.6 GHz, 4 GB RAM VM configuration Production VM: 1 VCPU, 2047 MB RAM Security VM: 1 VCPU, 1 GB RAM 35 Performance Inter-VM system call ( getpid ): ~11 µs Process state identification: ~250 µs Slowdown to out-grafted process File-copy time: 35.42% thttpd throughput: 7.38% 36 Performance Production VM Slowdown with a Contending Process Out-grafted 8% 6% 4% 2% 0% -2% -4% -6% 37 Rest of This Talk Motivation System Design Implementation & Evaluation Related Work Conclusion 38 Related Work VM introspection Livewire (Garfinkel et al. , NDSS ‘03) , XenAccess (Payne et al. , ACSAC ‘07) , VMScope (Jiang et al. , RAID ‘07) , Lares (Payne et al. , Oakland ‘08) , VMWatcher (Jiang et al. , CCS ‘07) , Virtuoso (Dolan-Gavitt et al. , Oakland ‘11) Efficient, isolated monitoring SIM (Sharif et al. , CCS ’09) Process migration BLCR (Smith, UCB-TR ‘08) , Zap (Osman et al. , OSDI ‘02) , Migration survey (Nuttall et al. , ACM OS Review ’94) System call forwarding Application protection (Ta-Min et al. , OSDI ‘06) , Monitoring fidelity (Martignoni et al. , ICISS ‘09) Sandboxing, isolation techniques Ostia (Garfinkel et al. , NDSS ‘04) , Janus (Goldberg et al. , Security ‘96) 39 Conclusion Process Out-grafting: An Efficient Out-of-VM Approach for Fine-grained Process Execution Monitoring (CCS ‘11) Technique I: On-Demand Grafting Production VM Security VM Stub Suspect Process Monitor User User Kernel Kernel Hypervisor Technique II: Mode-sensitive Split Execution 40 Thank you! Questions?.