Workshop for Frontiers of Cloud Computing, Dec 1, 2011, IBM T.J. Watson Research Center, NY
Process Out-Grafting: An Efficient “Out-of-VM” Approach for Fine-Grained Process Execution Monitoring
Deepa Srinivasan, Zhi Wang, Xuxian Jiang, Dongyan Xu * North Carolina State University, Purdue University* Malware Infection Trend
New malware samples collected by McAfee Labs, by month*
*Figure source: McAfee Threats Report: Second Quarter 2011, McAfee Labs 2 Anti-Malware Isolation
Traditional anti-malware tools are not well-isolated
Virtual Machine (VM) introspection Isolate tool by placing it outside a VM Analyze states and events externally
User-mode Applications Monitor
Virtual Machine …
OS Kernel
Hypervisor
3 Anti-Malware Isolation
Traditional anti-malware tools are not well-isolated
Virtual Machine (VM) introspection Isolate tool by placing it outside a VM Analyze states and events externally
User-mode Applications Monitor VM Virtual Introspection Machine …
OS Kernel
Hypervisor
4 Out-of-VM Solutions
Livewire (Garfinkel et al. , NDSS ‘03)
XenAccess (Payne et al. , ACSAC ‘07)
VMScope (Jiang et al. , RAID ‘07)
Lares (Payne et al. , Oakland ‘08)
…
5 Semantic Gap in Introspection
What we want to observe
High-level states and events (e.g. system calls, processes)
What we can observe
Low-level states and events (e.g. raw memory, interrupts)
Internal User-mode Applications Monitor
… External Monitor Semantic Virtual Machine Gap OS Kernel
Hypervisor 6 Addressing the Semantic Gap
Guest view casting VMWatcher (Jiang et al. , CCS ‘07)
Automatic generation of introspection-based tools Virtuoso (Dolan-Gavitt et al. , Oakland ‘11)
Issues Sensitive to internal OS changes or updates Incompatible with existing anti-malware tools
7 Our Goal
Support existing in-host process monitors out-of-VM without semantic gap!
8 In-host strace
9 In-host Monitors
Process-level monitoring System calls Library calls Instruction execution traces
10 Process Out-Grafting
Isolation Monitor protection from malware
Compatibility Natural support for fine-grained user-mode process monitoring tools ( strace, ltrace, … )
Efficiency No significant performance overhead due to isolation
11 Rest of This Talk
Motivation
System Design
Implementation & Evaluation
Related Work
Conclusion
12 Assumptions
Trusted hypervisor
Untrusted monitored VM
Non-goals OS kernel-level monitoring Stealthy monitoring
13 Key Techniques
Technique I: On-Demand Grafting Production VM Security VM Stub Suspect Process Monitor
User User Kernel Kernel
Hypervisor Technique II: Mode-sensitive Split Execution
14 Key Techniques
Technique I: On-Demand Grafting Production VM Security VM Monitor
User User Kernel Kernel
Hypervisor
15 On-Demand Grafting On-Demand Grafting
Relocate suspect process to security VM
Enable efficient, native inspection Eliminate hypervisor intervention Support existing process monitoring tools
Initiate out-grafting as needed
Restore process after monitoring
17 When to Out-Graft?
Process in user-mode
Process in kernel-mode Hypervisor notified when user-mode execution resumes
Suspect Process
Production VM Physical NX Memory Frames User Kernel
Hypervisor
18 What to Out-Graft?
Architecture-specific resources
No OS kernel-specific resources
Main root cause in semantic gap
Continued execution of out-grafted process
Suspect Process
Physical Production Memory VM Frames
User EIP Kernel Network Disk File EAX Socket EBX … … Hypervisor
19 How to Out-Graft?
Stub protected Production VM by hypervisor Security VM Stub Suspect Process Monitor
Physical NXMemory X Frames User User Kernel Kernel Helper Module
VCPU Register State Hypervisor Page Mappings
20 Mode-Sensitive Split Execution Mode-Sensitive Split Execution
All user-mode execution occurs in security VM
All kernel-mode execution occurs in production VM
Out-grafted process considers itself running in production VM
22 System Call Redirection
Smooth, continued execution of out-grafted process
Monitor isolation
Production VM No hypervisor Security VM Stub intervention! Suspect Process Monitor
- System call number
- Arguments
User System Call User Kernel Network Kernel Socket Helper Module Disk File
Hypervisor (e.g., KVM)
23 Page Fault Forwarding
Consistent process memory mapping between VMs
No semantic knowledge in security VM (e.g. memory-mapped file)
Production VM Security VM Stub Suspect Process Monitor
- Faulting address
- Read/Write fault Page Page Fault Fault User User Kernel Kernel Page Tables Helper Module
Hypervisor (e.g., KVM)
24 Page Fault Forwarding
Consistent process memory mapping between VMs
No semantic knowledge in security VM (e.g. memory-mapped file)
Production VM Security VM Stub Suspect Process Monitor
Physical Memory Frame Page Page Fault Fault User User Kernel Kernel Page Tables Helper Module
Hypervisor (e.g., KVM)
25 Rest of This Talk
Motivation
System Design
Implementation & Evaluation
Related Work
Conclusion
26 Implementation
Hypervisor: KVM (2.6.36.1) Out-grafting support: +1309 SLOC Extended page tables support (Intel Core i7)
Host OS: Ubuntu 10.04 (kernel 2.6.28)
Guest OS: Ubuntu 9.04, Fedora 10
27 Security Analysis
Monitor isolation and effectiveness
System call forwarding Security VM protected from kernel attacks Production VM Security VM Stub Suspect Process Monitor
User User Kernel Network Kernel Socket System Call Disk File
Hypervisor (e.g., KVM) 28 Security Analysis
Stub protection Hypervisor-protected memory and page tables
Production VM Security VM Stub Suspect Process Monitor
User User Kernel Kernel
Hypervisor (e.g., KVM) 29 Security Analysis
Out-grafting detection Strong policy for random, arbitrary out-grafting
30 Case Studies
thttpd as test process for out-grafting Both disk and network usage
Support existing in-host tools, out-of-VM strace, ltrace, gdb
OmniUnpack (Martignoni et al. , ACSAC ‘07 )
31 Case Studies
thttpd as test process for out-grafting Both disk and network usage
Support existing in-host tools, out-of-VM strace, ltrace, gdb
OmniUnpack (Martignoni et al. , ACSAC ‘07 )
32 strace
In-VM strace Out-of-VM strace
33 Code Unpacking Detection
OmniUnpack (Martignoni et al. , ACSAC ‘07)
Track page writes and executions Detect unpacking when executing a previously-written page
Faithful reproduction of algorithm in Linux
Kernel module in security VM
NX bit support only in Localized overhead in Thorough test of page security VM kernel out-of-VM monitoring fault forwarding
34 Performance
Dell T1500, Intel Core i7, 4 cores, 2.6 GHz, 4 GB RAM
VM configuration Production VM: 1 VCPU, 2047 MB RAM Security VM: 1 VCPU, 1 GB RAM
35 Performance
Inter-VM system call ( getpid ): ~11 µs
Process state identification: ~250 µs
Slowdown to out-grafted process File-copy time: 35.42% thttpd throughput: 7.38%
36 Performance
Production VM Slowdown with a Contending Process Out-grafted 8% 6% 4% 2% 0% -2% -4% -6%
37 Rest of This Talk
Motivation
System Design
Implementation & Evaluation
Related Work
Conclusion
38 Related Work
VM introspection Livewire (Garfinkel et al. , NDSS ‘03) , XenAccess (Payne et al. , ACSAC ‘07) , VMScope (Jiang et al. , RAID ‘07) , Lares (Payne et al. , Oakland ‘08) , VMWatcher (Jiang et al. , CCS ‘07) , Virtuoso (Dolan-Gavitt et al. , Oakland ‘11) Efficient, isolated monitoring SIM (Sharif et al. , CCS ’09)
Process migration BLCR (Smith, UCB-TR ‘08) , Zap (Osman et al. , OSDI ‘02) , Migration survey (Nuttall et al. , ACM OS Review ’94) System call forwarding Application protection (Ta-Min et al. , OSDI ‘06) , Monitoring fidelity (Martignoni et al. , ICISS ‘09) Sandboxing, isolation techniques Ostia (Garfinkel et al. , NDSS ‘04) , Janus (Goldberg et al. , Security ‘96)
39 Conclusion
Process Out-grafting: An Efficient Out-of-VM Approach for Fine-grained Process Execution Monitoring (CCS ‘11)
Technique I: On-Demand Grafting Production VM Security VM Stub Suspect Process Monitor
User User Kernel Kernel
Hypervisor
Technique II: Mode-sensitive Split Execution 40 Thank you!
Questions?