Forenvisor: a Tool for Acquiring and Preserving Reliable Data in Cloud Live Forensics

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TCC.2016.2535295, IEEE Transactions on Cloud Computing

1 ForenVisor: A Tool for Acquiring and Preserving Reliable Data in Cloud Live Forensics

Zhengwei Qi, Chengcheng Xiang, Ruhui Ma, Jian Li, Haibing Guan and David S. L. Wei

Abstract—Live forensics is an important technique in cloud security but is facing the challenge of reliability. Most of the live forensic tools in cloud computing run either in the target Operating System (OS), or as an extra hypervisor. The tools in the target OS are not reliable, since they might be deceived by the compromised OS. Furthermore, traditional general purpose hypervisors are vulnerable due to their huge code size. However, some modules of a general purpose hypervisor, such as device drivers, are indeed unnecessary for forensics. In this paper, we propose a special purpose hypervisor, called ForenVisor, which is dedicated to reliable live forensics. The reliability is improved in three ways: reducing Trusted Computing Base (TCB) size by leveraging a lightweight architecture, collecting evidence directly from the hardware, and protecting the evidence and other sensitive files with Filesafe module. We have implemented a proof-of-concept prototype on the Windows platform, which can acquire the process data, raw memory, and I/O data, such as keystrokes and network traffic. Furthermore, we evaluate ForenVisor in terms of code size, functionality, and performance. The experiment results show that ForenVisor has a relatively small TCB size of about 13 KLOC, and only causes less than 10% performance reduction to the target system. In particular, our experiments verify that ForenVisor can guarantee that the protected files remain untampered, even when the guest OS is compromised by viruses, such as ‘ILOVEYOU’ and Worm.WhBoy. Also, our system can be loaded as a hypervisor with no need of pausing the target OS, and thus our system can also gather live evidence of a target OS running on hardware directly. Finally, we also post the source code of ForenVisor on Github.

Index Terms—Live forensics; virtualization; reliability; ﬁle protection; io monitor. !

1 INTRODUCTION the crime scene [7]. In addition, some target systems, e.g., cloud platforms, cannot tolerate shutdown even for a short Owadays, cloud computing is widely adopted to pro- period of time, since they are dedicated for 24/7 services. vide computing services as a utility [1]. Since a few N Live analysis can remedy some flaws of static analysis. complex techniques, such as virtualization techniques [2], More volatile information, including the process list, the are employed in cloud environments, more vulnerabilities kernel objects, and network traffic, is available for live anal- are introduced to cloud systems [3], which reinforces the ysis [8], [9]. Nevertheless, live analysis introduces reliability importance of digital forensics for cloud computing [4]. problems – the analysis tool itself might be compromised by Digital forensic tools are widely used to collect, analyze, the malware, and the visible volatile data might be faked by and present the evidence of the cybercrime. Though live a rootkit [10]. To address these issues, there have been some analysis and static analysis are differentiated by whether works focused on system integrity monitoring [11], kernel to keep the target system running at analysis time or not, data protection [12] and anti-forensic attack detection [13]. both of them are currently two main approaches adopted However, none of them can guarantee their self-reliability in digital forensics scope. However, current tools of both due to the vulnerability of the Operating System (OS). static and live analysis are facing the challenge of either Virtualization technology is employed to address above comprehensiveness or reliability. problems [8], [14]. The key idea is placing the forensic Static analysis tools [5], [6] obtain digital evidence by process in the Virtual Machine Monitor (VMM) layer, while imaging the disk and analyzing the non-volatile data after the target OS is wrapped as a Virtual Machine (VM). Any the target system has been shut down. The main drawback privileged instruction executed by the VM will cause a of this approach is that the helpful volatile data, such control transfer to the VMM, and therefore the compromised as memory and I/O data, are not available for forensics. VM becomes unable to obstruct the forensic process in the However, I/O data, including keyboard input and network VMM. In addition, the VMM context can supply some low packets, are important information for the reconstruction of level information, such as hardware status, for analysis. However, the semantic gap [15] between the VMM and • Z. Qi, C. Xiang and J. Li are with the School of Software, and the Shanghai VM, i.e. some objects in the VM, such as processes and Key Laboratory of Scalable Computing and Systems, Shanghai Jiao Tong University, Shanghai, 200240 China, e-mail: {qizhwei, xiangchengcheng, files, are not visible to the VMM, brings extra difficulties li-jian}@sjtu.edu.cn. to the analysis. For instance, the VMM could only observe • R. Ma and H. Guan are with the Shanghai Key Laboratory of Scal- the disk blocks accessed by the VM, and needs to parse the able Computing and Systems, and the Department of Computer Sci- file system’s meta data to locate the accessed file. ence, Shanghai Jiao Tong University, Shanghai, 200240 China, e-mail: {ruhuima, hbguan}@sjtu.edu.cn. Current forensic tools based on virtualization technology • D. Wei is with Department of Computer and Information Sciences at are also confronted with some challenges. Firstly, most hy- Fordham University, 441 E. Fordham Road, Bronx, NY 10458, USA, e- pervisors, such as Xen [16] and VMware [17], contain a great mail: [email protected]. amount of code, leading to a high level of vulnerability. The

2168-7161 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TCC.2016.2535295, IEEE Transactions on Cloud Computing

2 second challenge is that for most hypervisors, setting up • We adopt the Filesafe module to protect the evidence the virtualization environment needs to reboot and even data stored locally and sensitive system files instead to reinstall the target system, which inevitably ruins the of storing evidence data remotely via network. Our volatile evidence and thus contradicts the purpose of live approach can avoid having large code size of NIC analysis. driver and avoid network attack. To address the security challenges, TrustVisor [18] re- • We implement the prototype ForenVisor on the Win- duces its code base by leveraging the features of modern dows platform, and perform a comprehensive e- processors. Additionally, BitVisor [19] presents another ap- valuation of code size, efficacy, and performance proach of eliminating device drivers by adopting parapass- overhead. Furthermore, we post the source code of through architecture. Both of them are special purpose hy- ForenVisor on Github1. pervisors for security enhancement, and we take the similar The rest of the paper is organized as follows. Section 2 idea to construct a lightweight hypervisor for reliable foren- introduces some background of virtualization. Section 3 sics. To avoid destructing the volatile data, it is necessary presents the design of ForenVisor. Section 4 illustrates the for the forensic hypervisor to be able to install itself on the implementation of how ForenVisor acquires data directly fly [20]. HyperSleuth [21] is a live forensic framework that from hardware. Section 5 describes the implementation adopts the technique of virtualization on-the-fly. However, issues of the Filesafe module. Section 6 illustrates the ex- HyperSleuth gives the guest OS unrestricted access to I/O perimental results obtained by measuring the code size, devices. It thus loses the important I/O data for forensics, functionality, and performance of our hypervisor. Section 7 and thus cannot protect the forensic data stored on the disk. discusses the limitations of our current work. Section 8 In this paper, we propose ForenVisor, a lightweight discusses the related work in this research area. Finally, hypervisor dedicated for reliable live forensics. ForenVisor Section 9 concludes the paper. is focused on the reliability of live data acquisition and storage, and improves the reliability from three aspects. Firstly, this lightweight hypervisor architecture is adopted to 2 BACKGROUND decrease the code size and protect the forensic process from The classical VMM model is trap-and-emulate, in which a being tampered. Secondly, ForenVisor collects full system trap to the VMM happens whenever the guest OS executes state, including process data, raw memory, and I/O data, a privileged instruction, and then the VMM emulates hard- directly from the hardware, with no need of depending ware to execute the instruction for the guest OS. The x86 on device drivers. Lastly, ForenVisor stores volatile data architecture provides some opportunity to implement this on disk and protects them with Filesafe module [22]. The model. Traditional x86 protected mode has four protection Filesafe module can also supply protection for sensitive files rings, with the OS kernel running on the most privileged in the guest OS, such as logs, which are required for forensic mode (ring0) and user applications running on the least analysis. privileged mode (ring3). Typically ring1 and ring2 are not ForenVisor reduces its Trusted Computing Base (TCB) used, so an intuitive way to protect the VMM is to put the size to improve its reliability. Traditional hypervisors are VMM on ring0, and move the guest OS kernel to ring1 with huge code size as they are designed for general- or ring2. When the guest OS executes some privileged purpose usage. They, in general, consist of device drivers, instruction, we expect a trap to ring0 to transfer control to virtual devices and resource managers. For instance, the Xen the VMM. However, the trap does not always happen, for hypervisor has more than 230K lines of code and its TCB example, the privileged instruction POPF fails silently only also includes a lot of other code of device drivers in Domain when executed above ring0. To solve this problem, different 0, the privileged VM in which the control software runs [23]. virtualization techniques are developed on x86 platform. On the contrary, like other special purpose hypervisors, ForenVisor only needs some special device drivers, which makes it small, namely less than 14K lines of code. 2.1 Software-Based Virtualization In case of a destruction of volatile data, Late- Software-based virtualization implements the VMM in a Virtualization mechanism [24] is leveraged to enable Foren- complete software way in contrast to the new hardware- Visor to be loaded as a driver dynamically, with no need assisted virtualization. There are two types of software- of rebooting the target OS. To hide from the guest OS, based virtualization: full virtualization and paravirtualiza- ForenVisor is designed to be mini-intrusive, which means tion [25]. that the execution of ForenVisor only causes little turbulence Full virtualization means the VMM offers a whole hard- to the memory layout and low performance overhead to the ware interface to the guest OS, so that the guest OS con- target OS. ceives that itself runs directly on the hardware. Several The contributions of our work are summarized as fol- techniques are adopted to achieve that. Firstly, the guest lows: OS’s code is rewritten by dynamic binary translation, assur- ing the VMM can obtain the control whenever a privileged • We devise a lightweight hypervisor, named ForenVi- instruction is executed. For memory segregation, a shadow sor, to attain full system data with enhanced reliabil- page table is kept for each guest OS, and write protection ity. Unlike those existing forensic frameworks [21], is set so that the VMM can get involved in the updating ForenVisor can obtain comprehensive system data, of each page table. The VMM also provides exactly the including I/O data, in addition to process data and memory. 1. https://github.com/supervail/ForenVisor

3 same emulation of devices for the guest OS, and thus system. Different from the paravirtualization approach [25], the guest OS’s driver can run directly on the VMM. In ForenVisor is loaded as a normal driver with no need of spite of the advantage that no modification is done to the modification to the target system. This way, the target sys- guest OS, software-based full virtualization brings a large tem is not aware of the presence of ForenVisor. ForenVisor performance degradation to the guest OS. also prevents its memory space from being detected by Paravirtualization [25] means the VMM does not pretend the target system with the private page table. In addition, itself as hardware, but let the guest OS know its existence ForenVisor needs not suspend the target system, thereby and cooperate with it. The main goal of paravirtualization preventing ForenVisor from being detected, and only incurs is to boom the performance of full virtualization with the less than 10% performance overhead. tradeoff of being required to modify the guest OS. Par- Since ForenVisor is loaded as a system driver, there are avirtualization is implemented in a quite different way. two possible approaches to acquire live data: invoking the Instead of using dynamic binary translation, the guest OS OS interfaces, and acquiring data directly from the hard- is altered statically so that it transfers control to the VMM ware. The former one is easier to implement and has no actively when necessary. For device emulation, the VMM semantic gap with the target OS. However, the data collect- only provides simple device abstractions, and the guest OS ed can be forged or tampered by malicious software, as the is endowed with new device drivers that collaborate with target OS might be compromised by rootkits. Accordingly, the VMM on device management. ForenVisor adopts the latter approach to collect evidence. Although ForenVisor does not bridge the semantic gaps of 2.2 Hardware-Assisted Virtualization these data, prior works on Virtual Machine Introspection (V- MI) [27], [28] can be adopted in ForenVisor for the purpose Hardware-assisted virtualization on x86 platform became of bridging the semantic gaps. possible after Intel and AMD published their processor The reliability of the evidence preservation is guaranteed extension for virtualization, namely VT-x [26] and AMD-V. by file protection mechanisms. ForenVisor stores evidence VT-x and AMD-V are similar techniques, and in this paper files in local disk and protects them with the access control we only present a brief introduction on VT-x, as ForenVisor policy. The policy is on the block level, and any operation is implemented on VT-x. VT-x provides several hardware to the disk will be controlled by it. The key challenge is supports for full virtualization. Firstly, a new protection the semantic gap between the block level and the file level. mechanism is introduced that the VMM will run on the ForenVisor bridges the semantic gap by reconstructing the VMX root-mode, and the guest OS will run on the VMX non- file information based on the raw data residing on the disk. root mode with the original four rings. Every execution of a privileged instruction causes a trap to the VMX root-mode, 3.2 Assumptions so either dynamic binary translation or static modification to the guest OS can be eliminated. Additionally, VT-x also We assume that our adversary can obtain the whole control supplies a way for memory virtualization, namely Extended of the target OS, including kernel-level privileges. However, Page Table (EPT). With EPT, the virtual memory address is hardware-based attacks, such as CPU cache poisoning [29] translated by two page tables: the guest page table that maps and SMM attacks [30], are not taken into account. We do Guest Virtual Address (GVA) to Guest Physical Address not consider hypervisor-based malicious software, such as (GPA), and the host page table that maps GPA to Host SubVirt [31] and Blue Pill [20], either. These kinds of mal- Physical Address (HPA). These hardware supports signifi- ware exploit similar techniques to ForenVisor, and we are cantly decrease the complexity as well as the performance inspired by them to some extent. overhead of a VMM. We assume the load-time integrity of ForenVisor. This could be achieved by leveraging the recent Intel TXT technology [32] which provides a reliable way, called measured 3 DESIGN late launch, to load a clean hypervisor. We also assume a 3.1 Overview trustworthy hypervisor with no vulnerabilities. We have made great efforts on minimizing the TCB size of our ForenVisor is devised with the goal to provide a reliable hypervisor, which will be discussed in detail in the follow- environment for the acquisition, analysis, and storage of ing section. The code size of ForenVisor is much smaller the digital evidence. Since the reliability issue is of our compared with that of a legacy OS or prior hypervisors. focus, ForenVisor does not perform specific live analyses, but rather functions as a framework. Three aspects of reliability problems are taken into account in the design of 3.3 Lightweight Hypervisor Architecture ForenVisor: the safety of the hypervisor, the reliability of Figure 1 shows the lightweight hypervisor architecture for evidence acquisition, and the security of evidence files. reliable forensics. In this architecture, the forensic system ForenVisor guarantees its own safety in two ways. Foren- runs as a lightweight hypervisor in the layer between the Visor is different from other hypervisors due to its design of hardware and the OS, which has a higher privilege level small TCB size in order to reduce its vulnerabilities. This than the OS kernel. Because of different privilege modes, is achieved by exploiting the hardware-assisted virtualiza- applications and the OS are not aware of the existence of the tion technology and by eliminating those forensic-useless forensic hypervisor. As Figure 1 shows, the forensic system modules, such as some device drivers, though needed in consists of five modules: VMCS (Virtual Machine Control the traditional general-purpose hypervisor. Another reliable Structure) Recorder, Memory Dumper, Keylogger, NIC (Net- feature is that ForenVisor is mini-intrusive to the target work Interface Card) Interceptor, and Filesafe. The former

Application Application Application

Process Virtual Keyboard NIC ATA Driver Schedulor Memory Driver Driver

VMCS GPA Inb(0x60) File Guest State EPT Bitmap MMIO DMA Policy System 0 0 0 0 0 1 1 1 0 0 0 0 List Parser HPA

API VMCS Memory Recorder Dumper Keylogger NIC Interceptor FileSafe ForenVisor

CPU Memory Keyboard NIC Disk

Hardware

Fig. 1: The Hypervisor-based Architecture.

four modules are designed for recording the CPU states, the malicious adversaries and thus keeps our forensics process memory contents, the keystrokes, and the network packets confidential. respectively. Instead of collecting data, the Filesafe module plays key role in protecting the data gathered. The Late-Virtualization technique needs the supports of It is of great importance to keep the code size of the hardware-assisted virtualization technology. Our system is hypervisor small to decrease both of system vulnerabili- implemented on the Intel platform VT-x. To assist virtualiza- ty and TCB size. Our architecture achieves this goal in tion, in addition to the traditional 4-layer protection rings, three ways. Firstly, our hypervisor is devised based on another privilege level is introduced to CPU mode. The new hardware-assisted virtualization technology, which elimi- level is named VMX root-mode, which is established for nates a lot of code for hardware simulation. Similar solu- the hypervisor. The guest OS runs in VMX non-root mode tions of hardware-assisted virtualization are supplied by that consists of the original four rings. In the new protection Intel VT-x and AMD-V. Specifically, our implementation is model, a new structure, Virtual-Machine Control Structure based on Intel VT-x technique, and can be ported to the (VMCS), is offered to save the CPU states of both the Host AMD platform easily. Secondly, we apply the parapass- and guest OSes and control the conditions when a mode through approach [33], i.e. only interested operations are switch should happen. intercepted, in our architecture to minimize the code size The forensic hypervisor is loaded as a normal system of the I/O interceptor. Lastly, ForenVisor supports only one driver, and thus has minor influence on volatile data. The virtual machine at a time, which is reasonable since the launching process consists of several steps. The first step is hypervisor is dynamically set up to collect data from a enabling the VMX root-mode of the CPU. Secondly, VMCS native OS running on hardware directly. structure is initialized to register the sensitive events that the hypervisor concerns and the corresponding events’ handlers in the hypervisor. After the configuration of VMCS, the 3.4 Late-Virtualization Approach target OS has been turned into a guest OS wrapped in the Before our system can collect live data, it is necessary hypervisor. When the guest OS executes some privileged to build up a virtual machine environment in advance. instructions, a trap to the hypervisor is incurred and then Converting a native OS to a virtual machine might need the forensic procedure can be performed. After the hyper- to reboot or even reinstall it, and thus will lose live system visor has recorded good enough information, a VMENTRY data inevitably. To avoid the destruction of the live evidence, instruction is executed to continue the guest OS’s running. based on our former work [24], we use Late-Virtualization A potential vulnerability is that the launching procedure technique to set up the forensic hypervisor dynamically. may be compromised by a kernel-mode rootkit, since it is Late-Virtualization technique means inserting a hypervisor executed in ring0 mode rather than the VMX root-mode. To layer between the native OS and hardware dynamically, avoid this, the hypervisor’s image should be placed in read- without disturbing the execution of the running target OS. only media with integrity checked by the Trusted Platform This way, it avoids bothering normal users or alerting Module (TPM).

Guest Page Table Extension Page Table Host Physical Memory 1) AllocProtect allocates a protected file. The file will not be detected by the guest OS. Corresponding control policies Kernel of the blocks of the new file will be added to the policy Address Space Forensics Driver Real list immediately. The file will be set to be read-controlled Forensics Driver and write-controlled. Spare 2) SetPolicy sets the control policy for a file. The parameter Page indicates the access policy to be read-controlled, write- Kernel Page Page controlled, or both. Address Space 3) DelProtect deletes a protected file and relative block-

Page Page level policies will also be deleted at the same time.

Page 4) Read reads the content of a protected file. Any read operation on the file other than this interface will be User Address denied if the file is read-controlled. Space With EPT Translation 5) Write writes some content to a protected file. Any write operation on a write-controlled file will be denied if it Without EPT Translation is not performed through this interface. When a new block is appended to the file, it is added to the policy Fig. 2: The design model of Memory-Hiding Mechanism. list simultaneously.

3.5 Memory Protection Mechanism 4 EVIDENCE COLLECTION Since the forensic hypervisor is loaded as a system driver, its 4.1 Process Data address space is mapped in the page table of the target sys- Each VMCS structure in memory is associated with a logical tem, illustrated by the shadow part in Figure 2. Therefore, processor in the Intel VT-x platform. VMCS has a data the hypervisor needs some memory protection mechanisms area for guest status, which keeps the CPU states before a to protect its private memory from illegal access. EPT and VMEXIT event happens. These states include control regis- memory hiding are the two mechanisms employed for the ters (CR0, CR3, and CR4), debug register, RSP, RIP, RFLAGS, protection purpose. segment registers, and others. To record these states for EPT is an address space protection technique supplied each process, the hypervisor intercepts the instruction MOV by Intel VT-x. As shown by the dotted lines in Figure 2, with the target of CR3, which is executed to change address traditional Memory Management Unit (MMU) translates the space when a process switch happens. The drawback of this GVA directly to the HPA by looking up the guest page means is the performance overhead since traps happen as table. However, while EPT is enabled, GVA will firstly be frequently as process switches. However, since the recorder translated into the GPA through the Guest Page Table. Then, procedure is quite simple, the overhead is moderate, as what GPA is translated to HPA by the extension page table. The we will show later. extension page table, adopted in the Memory Dump phase, These states are practical for analyzing process states, can prevent the guest OS from illegally accessing protected of which one useful application is the detection of hidden physical memory. processes. The stealth rootkit hides processes in three ways: The memory hiding mechanism is designed based on modifying program binaries, hooking into the library call the EPT. When loaded for the first time, the hypervisor path, and manipulating kernel data structures directly. The builds up an extension page table. The extension page table forensic hypervisor can detect all hidden processes through maps most GPA to identical HPA, and therefore GVA can these ways for the reason that they all operate in the kernel be translated correctly with no need of modifying the guest mode, not the root mode. Some work has been done for page table. Nevertheless, the address space of the forensic the hidden process detection based on VMM and cross- driver is mapped into some unused physical memory space, view validation technique [34]. Our system focuses on the not the real one. Accordingly, the guest OS can read and reliability of data collection and storage, and therefore these write the driver’s memory, but makes no influence to the kinds of data analyses were not implemented. hypervisor. When the hypervisor runs in the root mode, EPT is disabled, and then GVA can be translated to the real HPA through the original guest page table. 4.2 Memory Dump As Figure 2 illustrates, the forensic hypervisor adopts EP- T to implement memory virtualization. Unlike traditional 3.6 Filesafe API Shadow Page Table (SPT) technique, which maintains each The Filesafe module is developed based on our prior work process in guest OSes a corresponding shadow page table, [22]. As Figure 1 shows, the VMCS recorder, the memory EPT just needs one page table for each guest OS. In ad- dumper, the keylogger, and the NIC interceptor obtain their dition, hardware supply permission management for EPT needed data respectively. However, they all depend on the pages eliminates many code lines required in SPT and thus Filesafe module to write and protect the evidence data. improves the reliability of the forensic system. Therefore, we abstract the function of the Filesafe module With the help of EPT mechanism, it is not hard to to provide a set of APIs for other modules. The followings implement a memory dumper. Firstly, hypervisor sets all are the descriptions of the main APIs. page entries in EPT to be non-writable to begin the memory

Buffer Descriptors Shadow Circular dump process. After that, when the guest OS writes any Circular Queue Queue memory page, an EPT violation trap happens, and then the Base hypervisor acquires chance to dump the memory page. The mrq rss checksum hypervisor will dump some continuous pages and set them to be writable according to the spatial locality principle to buffer address length

gain better performance. The guest OS continues to write Old Head Copy buffer according to the the page after the trap handler returns. The memory pages address and length New Head which have not yet been written are dumped at the point when the user decides to end the process. Owned by Hardware

4.3 I/O Interception Old Tail Copy new descriptor I/O operations, such as keyboard input and network con- New Tail nection, are crucial for rebuilding the crime scene. There- fore, it is necessary for a forensic system to monitor the I/O activities. ForenVisor supports recording all the three Base+LEN types of I/O operations: Programmed I/O (PIO), Memory Mapped I/O (MMIO), and Direct Memory Access (DMA). Fig. 3: The Buffer Descriptor Circular Queue. From the functional perspective, ForenVisor contains the keyboard logger module and the Network Interface Card (NIC) interception module, which are designed based on The transmitting and receiving process of network pack- our previous work [35]. This part will present the details ets is as follows. As Figure 3 illustrates, two buffer de- combining the perspectives of I/O operation and function. scriptors circular queues are allocated in the memory to transfer the DMA information between CPU and devices. 4.3.1 I/O Port (Keyboard Logger) NIC has some registers to store the queue information: the Logging keyboard input enables us to analyze the whole queue base address is stored in register Transmit Descriptor system behavior. The main I/O operation executed in key- Base Address (TDBA)/Receive Descriptor Base Address board drive is PIO, involving two I/O ports: a data port (RDBA), the queue length is stored in Transmit Descriptor (0x60) and a status port (0x64). When users press or release Length (TDLEN)/Receive Descriptor Length (RDLEN), and a key, CPU will be interrupted to execute the keyboard the head and tail pointers are stored in Transmit Descriptor driver. The driver reads the status port through instruction Head (TDH)/Receive Descriptor Head (RDH). The data, “inb 0x64”, and then read data from the data port by “inb consisting of buffer address, length and other fields, stored 0x60” in case the status is ready. Data from the data port are from head to tail in the queue are handled by hardware. scancodes generated by the keyboard according to the key Data, which comprises checksum and some other fields, stroked. stored in shade part of the queue should be handled by There are two ways to intercept the keyboard input. The the NIC driver. first one is intercepting the interrupt. This is a complicated When sending a packet, the NIC driver allocates a buffer, one since there are numerous different kinds of interrupt. fills it with the packet data, and writes the buffer address Hence, we adopt the second one, i.e. intercepting PIO, which and length to the tail of the transmitting circular queue. can be implemented by configuring the I/O bitmap in After that, NIC device is informed to transmit the packet VMCS. PIO means read and write the I/O port through referred by the buffer address in the head of the queue. the instruction “inb” and “outb” directly. The bitmap tells When the transmission finishes, NIC will write checksum hardware which port is concerned by the hypervisor, and and other data back to the head. It should be noted that the traps happen when associated I/O port instructions are driver may move the tail backwards by more than one step executed. In order to record keyboard input, the system and the device may also handle several buffer descriptors merely set 0x60 bit in the bitmap to be 1, and then the each time. The receiving process is in reverse way. The hypervisor reads from the data port in the trap handler. driver first gives the address of the empty buffer, then the device fills data there, and the driver reads packets received 4.3.2 MMIO and DMA (NIC Interception) from the buffer allocated before. NIC is a hardware dedicated for transmitting and receiv- The sending process is easier to intercept compared ing network packets. Currently, most NIC communicate with the receiving process. When the data for transmission with CPU through the Peripheral Component Interconnect is ready, the driver will move the queue tail by writing (PCI) express bus instead of the old PCI bus. The NIC to the Transmit Descriptor Tail (TDT) register. Writing to read/write data from/to the device by both of MMIO and NIC registers is performed through MMIO, which can be DMA. MMIO means mapping device registers to physical intercepted by setting the corresponding EPT page entries memory address so that the device register can be accessed to non-writable. Then the hypervisor can record the packet by the memory access instructions such as “MOV”. DMA data referred by the buffer address between the old tail and enables data to be transferred directly between memory the new tail. and device’s interior data buffer without CPU intervention. A shadow circular queue is adopted to assist recording DMA frees CPU from numerous device interrupts and thus the receiving packets. In the receiving procedure, the NIC also improves the system performance. will write back to the buffer descriptor after filling up the

buffer. The buffer will be found by the OS driver by another Guest OS copy of the circular queue, which is not kept in the driver context and cannot be obtained by the hypervisor. Therefore, ATA the hypervisor keeps its private copy of the buffer descriptor Driver queue. The interception is also performed through prevent- Hypervisor ing writing to the tail register. When trap happens, the Policy Ctrl hypervisor firstly copies new allocated descriptors stored File System Block-Level between the old tail and the new tail. Then, the hypervisor Parser Policy List copies buffer data according to the address and length between the old head and the new head, which are also maintained by the hypervisor. Hardware Storage Device 5 EVIDENCE PROTECTION Once the volatile data has been acquired, the hypervisor stores them on the disk. After being allocated, the evidence files will be protected by the Filesafe module from being Fig. 4: The Design of Filesafe. Dotted lines represent the accessed by malware or the compromised OS. Besides that, flow of how to set policies. Solid lines depict the flow of users can also set control policies for other sensitive files how policy control works. manually in the hypervisor, which will also be protected by the Filesafe module. the semantic gap problem. With the help of File System 5.1 Bridging the Semantic Gap Parser, users can browse files on the disk and specify which files should be protected when they are setting file security One of the key challenges of Filesafe is to bridge the seman- policy in the hypervisor. Usually, the most convenient way tic gap between the view of the guest OS from the outside of doing that is utilizing the file system interfaces in OS, and the view from the inside. Filesafe has a File System as what [36] does. But in that case, system vulnerabilities Parser and a Policy Controller to address this problem. may be introduced due to these untrusted interfaces. Thus, File System Parser is the bridge mapping files to blocks. It we construct a bridge between the file system and users parses file system information from disk following specific in the underlying layer. As for how to implement such a file system standard. It cooperates with users to set up bridge is file system specific, we only implement a prototype the control policy in the hypervisor. The Policy Controller supporting for FAT32 file system to prove our design is contains a Block-Level Policy List in which access ways of feasible. sensitive files are described. File Allocation Table 32 (FAT32) [37] is a simple and As shown in Figure 4, the flow of how to set policies is prevailing file system. However, it is being replaced by represented by dotted lines. After the hypervisor is installed, a more popular file system, New Technology File System the user could log in to set his/her access policy of sensitive (NTFS) [38]. NTFS is more complicated since it has property files. A console is offered to let the user interact with our of fault tolerance, security, file compression and so on that File System Parser. The user could use command line to are not possessed by FAT32. However, we choose FAT32 browse the file system. Then the user could set read/write as our experiment object because our focus is on how to restriction to sensitive files so that both the block range in protect sensitive files in a file system rather than on how to which these sensitive files are stored and access permission effectively manage a file system. Indeed, our approach can bits will be recorded in the block-level Policy List. also be implemented using NTFS. Solid lines in Figure 4 depict the flow of how policy control works. When the guest OS is running, every read FAT32 consists of three important data structures: the or write request to disk blocks will be intercepted to check boot sector, the file allocation table (FAT) and the directory whether it violates the policies in the Policy List, and if so, it table. The boot sector is the first sector in a FAT32 partition. will be silenced and the guest OS will receive a notification It contains BIOS parameter block (BPB), which portrays the indicating that the operation is failed. The guest OS consid- key information of this partition. FAT is a table with 32-bit ers the read/write error being caused by hardware problem entry in FAT32 file system. Each entry represents the status instead of by the hypervisor protecting the files. Therefore, of a cluster which is the minimum unit of file allocation. the overall process is transparent to the guest OS. Each of the files and directories is stored in one or more The Policy List is stored on the disk as a policy file. Also, clusters. The files and directories are not necessarily stored to protect itself, there is also an access policy for the policy in continuous clusters. Linked list is the data structure used file, which denies any operation from the guest OS. and FAT is such a table that describes this structure. The value in each entry may indicate a free cluster, the next cluster number of a file, the last cluster of a file, or a bad or 5.2 File System Parser Implementation reserved cluster. Directory table is a special type of file that File System Parser analyzes file system and maps file to represents a directory. Each file or directory stored within disk blocks where the real data of file reside. It helps users it is represented by a 32-byte entry in the table. Each entry browse files outside of the guest OS. File System Parser is records information of each file, such as name, attributes, an important module of the Filesafe in hypervisor to solve size and especially the cluster number of the first cluster of

8 TABLE 1: Abbreviation TABLE 2: The code size of ForenVisor.

Abbr. Module Name Source Lines of Code BPB BIOS Parameter Block hypervisor base 4773 RSC Reserved Sector Count memory dumper 664 NF Number of FATs SPF Sectors Per FAT I/O Interceptor 1189 SPC Sectors Per Cluster ATA Driver 4410 BPS Bytes Per Sector File system parser 2300 SN Sector Number SO Sector Offset Total 13336 CN Cluster Number LSN Logic Sector Number

Hypervisor Linux the file data. 4 Qemu ATA Driver After analyzing the FAT32 file system, we can obtain file 3 information reconstructed from the raw data residing on the disk. For instance, to get the blocks that the file named A 2 occupies, File System Parser first locates the root directory TCB Size (100KL) entry of the FAT32 partition. If A is in the root directory, 1 by matching the file name of each entry in this directory, 0 the parser could get the entry representing A as well as ForenVisorTrustVisor BitVisor Xen KVM the cluster number of the first cluster of A’s data. Next, it should query FAT to get the next cluster in a linked list using Formula 1 and Formula 2. When all the cluster numbers Fig. 5: Comparison of the TCB size of virtual environ- of A’s data have been collected, cluster number should be ments. converted to sector number using Formula 3 and Formula 4. In Formula 4, Logic Sector Number (LSN) is the sector index in this partition. On the other hand, in FAT32, there is another term representing a location, i.e. Logical Block Address (LBA) which is the sector index of the entire disk. F AT EntrySN = BP B.RSC + (CN ∗ 32/8)/BP B.BP S (1) When the parser obtains the file content, LSN should be F AT EntrySO = (CN ∗ 32/8)%BP B.BP S (2) converted to LBA by adding an offset stored in the partition F irstDataSN = BP B.RSC + BP B.NF ∗ BP B.SP F (3) table of boot sector because LBA is the absolute disk address LSN = F irstDataSN + (CN − 2) ∗ BP B.SP C(4) of the file while LSN is only a relative address. If A is in the subdirectory, as directories are treated as files that contain the directory information, the subdirectory should be accessed recursively the same way as the procedure 6 EVALUATION described above. The abbreviations used in formulas are listed in Table 1. In this section, we first show the TCB size of ForenVisor. Next, we describe the functional examination and show Once we have the mapping from file to disk blocks, the results. Finally, we evaluate the performance of our we could set the security policy, such as read only or not implementation on an Intel machine with Core i5-650 3.20 accessible, for all the blocks of sensitive files. When the guest GHz, 3GB DDR3 RAM, an Intel 82578DM Gigabit Ethernet OS is running, Policy Control will enforce these policies to NIC and 250 GB Seagate SATA2 disk. A Windows XP with protect sensitive files. The guest OS usually accesses the disk sp3 runs on this machine, and it will become a guest OS by specifying three parameters: LBA, number of sectors, and after our hypervisor is mounted on. sector data. In Filesafe, the operation is intercepted and the three parameters are sent to the Policy Control. In the view of the guest OS, any attempt to read a read-protect file will 6.1 TCB Size get an empty file and to write to a write-protect file will ForenVisor eliminates some general purpose components result in a failure operation. But deleting a write-protect file that are unnecessary for forensics. The total code size is will always succeed, because in FAT32 file system, deleting about 13336 Source Lines of Code (SLOC), with 4773 SLOC a file will only set the first byte in the directory entry to as the hypervisor base. The memory dumper and I/O 0xE5 rather than erase all the data of that file. Thus it could interceptor have fewer code lines, 664 SLOC and 1189 SLOC be easily recovered with erasing the first byte. Formatting respectively. The only driver that ForenVisor has is the ATA a partition will also work well since formatting will only driver that consists of 4.4K SLOC. In addition, ForenVisor erase the data in FAT. As all the sector numbers of sensitive has another 2.3K SLOC for the file system parser. The code files are listed in Policy List, FAT could be reconstructed for size of each component of ForenVisor is listed in Table 2. those sensitive files. Filesafe does these recovery works after In Figure 5, we compare the TCB size of some popular the guest OS is shutdown. contemporary virtual environments. We can see that the

9 TABLE 3: Function comparison.

Functions ForenVisor HyperSleuth Xen KVM Process data ! ! ! ! Physical memory ! ! ! ! Network trafﬁc ! - ! ! Fig. 6: The packet data captured by windump File protection ! - - - Dynamic loaded ! ! - -

TCB size of ForenVisor is far smaller than that of general purpose hypervisors, such as Xen and KVM. Xen and KVM support any kind of OSes as well as hardware. Thus, they have to include a few device drivers or code of software simulation. On the other hand, ForenVisor is a special purpose hypervisor for live forensics. By utilizing cutting-edge Intel- VT, ForenVisor just intercepts critical events, rather than traps and emulates. Compared with Bitvisor [19], another special purpose hypervisor, ForenVisor still has a much Fig. 7: The packet data captured by ForenVisor smaller TCB size. However, ForenVisor’s TCB size is a bit larger than that of TrustVisor [39]. The reason is that TrustVisor is just dedicated to code integrity as well as data selected viruses, the guest OS suffers a severe damage, how- integrity, and thus has no component for data gathering. We ever, the protected files remain unchanged. The experiment do not display the TCB size of HyperSleuth, another special shows that Filesafe can guarantee strong protection even purpose hypervisor with similar functions to ForenVisor, when the OS is compromised. because it is not open-source accessed. 6.2.3 Comparison of Functionality 6.2 Function Evaluation Table 3 displays a functionality comparison between Foren- 6.2.1 Data Collection Visor and other hypervisors, including special purpose hypervisors, e.g., TrustVisor and HyperSleuth, and gener- We load ForenVisor as a system driver after the test sys- al purpose hypervisors, e.g., Xen and KVM. The function tem has booted up. We configure ForenVisor to record the features compared comprise which data the hypervisors process data, dump the raw memory, and intercept the I/O can gather and whether they can be dynamically loaded, operations simultaneously. Four files, named ‘proc’, ‘mem’, which is critical for preserving dynamic evidence. As Table 3 ‘key’, and ‘net’, are generated to store these data, separately. shows, HyperSleuth is a similar special purpose hypervisor The process data, including many CPU registers, are divided loaded dynamically for live forensics. However, it only by different processes. Some preliminary information, such gathers limited data without capturing the crucial network as the number of processes and the page table of a process, traffic, and it provides no protection for files, which can also can be extracted from these data easily. The ‘mem’ file is be critical evidence, in the guest OS. The general purpose in the raw structure and has the same size as the physical hypervisors, namely Xen and KVM, can gather as much memory. The ‘key’ file stores the scancode of the keys stroked, information as ForenVisor, but they are not able to protect and the ‘net’ file stores the link layer packets. files in the guest OS either. Furthermore, there exists a fateful 6.2.2 Filesafe Module drawback with those general purpose hypervisors in which they cannot be dynamically loaded, thereby limiting their ForenVisor can protect files on the target system from be- usability. ing ruined by all kinds of worms, for example, ILOVEY- OU and Worm.WhBoy. ILOVEYOU [40] is a script written in Visual Basic. It is widely spread in the world vi- 6.3 Case Study a an email titled ‘ILOVEYOU’ and attached ‘Love-Letter- We conduct a case study of obtaining real data from a For-You.TXT.vbs’. Once infected, the virus insanely covers compromised system to show the validity of our approach. contents of files such as image and audio files, especially The attack scenario is as follows. The target OS is com- important system files. Also, it adds ‘.vbs’ to the end of these promised by a rootkit that gains the OS kernel privilege. The files’ name. Careless users will be disguised to open these rootkit communicates with its controller through network, files and cause more loss. Worm.WhBoy is written in Delphi. and it tries to hide its network communication by falsifying This virus deletes files with extension file name, ‘.gho’, such the packet captured by forensic tools. that user cannot use ghost to recover their operating system. We implement such a rootkit as a windows driver. To In addition, the virus adds virus code to binary files and show the effect, the rootkit simply sends ‘this is malign’ adds problematic URLs to html files. Both the two viruses to another computer. The rootkit also tampers with the cause great damage to users’ sensitive files and operating ‘wpcap.dll’ file, a library file of WinPcap, which is an systems. industry-stand windows packet capture library adopted by In the experiment, the test files, a.jpg, b.mp3, a1.gho, b1.exe most packet capture tools [41]. The altered WinPcap library and c1.html, are protected by Filesafe. When infected by the tampers with the captured packets, so we can only obtain

50 180% Without Forensic Read Response Time With Forensic 160% Write Response Time 40 Read Throughput 140% Write Throughput 30 120% 100% 20 80% Base Ratio 60%

10 Normalized to 40%

0 20% perlbenchbzip2 gcc mcf gobmkhmmersjeng libquantumh264refomnetppastar xalancbmk 0% native response time/throughput 4k 16k 32k 64k Packet size

Fig. 8: ForenVisor performance impact - SPEC CINT2006 Fig. 9: ForenVisor performance impact - Network access

140% Random Read Random Write the fake data when using packet capture tools relying on 120% Sequential Read Sequential Write WinPcap, such as WinDump and Wireshark [41]. 100% Figure 6 shows a packet captured by WinDump. The first 80% line shows the packet is from computer ‘TEST2’ to ‘TEST1’, and the underneath lines are divided into two parts: the left 60% is the hex data, and the right shows the data in char. The 40% payload of the packet is “this is benign”, which has been 20% tampered with. Figure 7 shows the packet data captured by Normalized to native throughput 0% ForenVisor. The first line shows the memory address of the 4KB 16KB 32KB 64KB packet buffer and the packet length. Then the second part is Block size the hex data, after which is the data in char. The payload of Fig. 10: ForenVisor performance impact - Disk access the packet is “this is malign”, which is real data sent by the rootkit. that there is a gap between the read and write response time. 6.4 Performance Overhead As we have described in section 4, the receiving interception We execute both compute-bound and I/O-bound appli- needs to keep an extra shadow descriptor queue, which cations with ForenVisor. For compute-bound applications, accounts for the phenomenon. we use the SPEC CINT2006 suite [42]. The SPECint suite Results of Iometer benchmark for disk I/O are pre- consists of a series of long-running computationally inten- sented in Figure 10. We evaluate disk I/O performance of sive applications so as to measure the CPU and memory ForenVisor with various block sizes and various operation management performance. For I/O-bound applications, we types. When the block size increases to 64KB or higher, the utilize Iometer [43] to evaluate the network transfer and disk performance overhead becomes negligible. For the sake of I/O performance. brevity, results of block size larger than 64KB are not shown in the figure. 6.4.1 SPEC CINT2006 This figure indicates that the I/O operation overhead of Results of SPECint benchmarks are presented in Figure 8. ForenVisor is a constant value, rather than in proportion to In the experiment, we compare performance of the bench- block size. According to the bars whose block size is 4KB, we marks running on the native computer and on ForenVisor. see that random operation is much slower than sequential On average, the overhead of our hypervisor is less than 8%, operation. Sequential operation overhead of ForenVisor is but the mcf benchmark shows more than 20% overhead. notable because sequential operations take shorter time, We attribute this to EPT performance overhead and hope while random operation overhead is not notable because the hardware will be improved in the near future. Besides, random operations take much longer time. When we com- we also observe high overhead (up to 80%) measured by pare the bars whose block size is 4kB with others, we see SPECint in other work [39], [44] implemented with EPT that when the block size increases and the I/O operations support. take more time, the overhead incurred by ForenVisor becomes negligible. 6.4.2 I/O We also evaluate the performance overhead of network data 6.4.3 Application transfer and disk I/O with Iometer benchmark. We test the performance of an Apache http server to reveal Results of Iometer benchmark for NIC I/O are shown the overhead that ForenVisor incurs on real-world applica- in Figure 9. Both of the response time and throughput of tions. In a LAN environment with the speed of 100Mbps, read and write are normalized to the native system. With http load is used to generate workloads of different con- increasing block size, the response time declines and the currency levels, ranging from 10 to 250. Figure 11 shows throughput raises slightly. The reason is that the whole the results that ForenVisor causes no obvious overhead on response time decreases and throughput increases, and thus the throughput and nor on the latency. The reason is that the overhead cuts down relatively. It should be pointed out the overhead of ForenVisor is so small that is hidden by

13 700 tolerance, security, and file compression. We choose FAT32 Throughput (no FV) Throughput (FV) 600 12 Latency (no FV) as our experiment object because our focus is on how to Latency (FV) 11 500 protect sensitive files in a file system rather than on how to 400 effectively manage a file system. 10 300 In the cloud environment, the adoption of ForenVisor 9 Throughput may be limited in the Infrastructure as a Service (IaaS)

(MBytes/sec) 200 Latency (msecs) 8 100 platform, where there is already a hypervisor which may also depends on hardware-assisted virtualization. Howev- 7 0 0 50 100 150 200 250 er, ForenVisor can be easily deployed in the Platform-as- Concurrency number a-Service (PaaS) architecture that adopts the Docker en- gine [47] instead of hypervisors. Fig. 11: ForenVisor performance impact - Apache Server

the variation of the network speed. Hence, ForenVisor is 8 RELATED WORK practical in real applications without decreasing the user experience. 8.1 Cloud Forensics The cloud environment poses new challenges to the tradi- 7 LIMITATIONS tional digital forensic models [48]. Some theoretical work has been done to address the issue. Martini et al. proposed Our proof-of-concept implementation of ForenVisor is sub- an integrated conceptual forensic framework based on M- ject to some limitations. We discuss the limitations in this cKemmish’s work [49] and NIST [4], the two widely used section and provide some possible solutions on which we traditional forensic frameworks. To bridge the gap between are working. theoretical models and practical application, research works The current version of ForenVisor does not cover all have focused on developing tools for specific real cloud forensic procedures. In general, it is believed that a complete systems, such as XtreemFS [50], OpenStack [51] and Google digital forensic model should include six phases, namely Drive [52]. However, the most complicated part, evidence identification, preservation, collection, examination, analy- preservation, still remains unsupported in most cloud en- sis [45]. Our current implementation focuses on acquiring vironments [53]. Our work effectively provide a reliable and protecting evidence, which is the work of preservation approach for grabbing and preserving data in the cloud phase. However, ForenVisor in general supplies a reliable environment. environment for every stage of live forensics, and it is not hard to transplant existing tools of other phases to ForenVisor. 8.2 Virtualization-based Live Analysis Our current implementation of ForenVisor does not include an NIC driver, which means that the evidence needs to There has been some work on analyzing the guest OS in hy- be saved locally. Saving evidence locally has several draw- pervisor or VMM. The idea was firstly proposed as VMI [54] backs. Firstly, it may overwrite the deleted files due to block by Garfinkel et al. in 2003. Subsequently, many researchers reuse, resulting in the destruction of the original evidence. elaborate the idea by leveraging VMI for some specific In addition, local saved evidence restricts the scalability of purposes, such as honeypot [55] and malware analysis [56]. live forensics, especially in cloud environments, where the There are also some researches about digital forensics based forensics may involve hundreds of physical devices. Though on VMI. Nick L. Petroni Jr. proposes FATkit, an extensible out of the focus of this paper, a potential collection phase can framework for extracting and analyzing volatile system effectively help address the aforementioned problems. memory [57]. As Xen [16] gains popularity, Brian Hay and In the collection phase, ForenVisor sends its evidence to Kara Nance present the VIX tools suite, which is developed a master server so that the evidence can be gathered in one for virtual introspection on Xen [58]. However, the instal- place. To achieve this purpose, while keeping the small TCB, lation of the aforementioned hypervisors or VMMs need a minimalistic NIC driver needs to be added to ForenVisor, to reboot the target system, which may cause intolerable like what is done in HyperSleuth [21]. However, even if evidence loss. the evidence is sent through network, local disk cache of Lorenzo Martignoni et al. propose HyperSleuth [21] to the dynamic evidence is necessary as network throughput address this problem. HyperSleuth can be installed and is limited for dumping a large amount of dynamic data, removed on-the-fly on commodity production systems. Al- such as the physical memory. Filesafe module is useful though ForenVisor has similar launching process to Hy- for protecting the local disk cache as well as user files perSleuth, ForenVisor possesses some other good features. that are potential evidence. To prevent the disk cache from First, ForenVisor does not need another trusted host to save overwriting the original deleted files, a dedicated partition evidence data, but HyperSleuth needs it. Secondly, instead for evidence caching is necessary. of relying on pure software-based approaches, ForenVisor is Also, our choosing FAT32 for prototype implementation more prone to leveraging hardware-assisted virtualization might seem to be another limitation, since FAT32 is being technique, thus can effectively reduce TCB size. Lastly, replaced by NTFS [46]. However, both FAT32 and NTFS ForenVisor can acquire the whole system data, including the are optional for ForenVisor. Compared with FAT32, NTFS process data, raw memory, and I/O data, which are helpful is more complicated since it has properties, such as fault for the reconstruction of the original crime scene.

12 8.3 Virtualization-based File Security TCB size and protecting the protected files from tampered. HyperSpector [59] is a virtual distributed monitoring envi- Those integrity verifications will be of our future work to ronment that employs multiple Intrusion Detection Systems further improve hypervisor security of our system. (IDS) to protect a distributed system. A specific IDS VM mounts the file system of the target OS as shadow file 9 CONCLUSIONAND FUTURE WORK system and enforces the access control on that file system. SVFS [60] adopts a similar approach that uses a Data Virtual In this paper, we presented a lightweight hypervisor de- Machine stores and protects sensitive files for other applica- voted to acquiring and preserving data for reliable live tion VMs. Though this approach may be a good solution forensics. The reliability is improved in three ways: the for distributed systems, it is not suitable for the design lightweight architecture, the data acquisition mechanism, aiming at reducing TCB size because introducing another and the evidence protection mechanism. We leverage the VM inevitably increases the TCB size and performance hardware-assisted virtualization technique and eliminate overhead. Our approach does not need any additional VMs unused device drivers to reduce the TCB size, thereby but only a thin hypervisor. decreasing the vulnerability of our hypervisor. Our proof- BitVisor [19] is a thin hypervisor for enforcing I/O device of-concept prototype system is implemented and evaluated security. It proposes the parapass-through architecture, which on the Windows platform. The evaluation results show that makes most of the I/O requests from the guest OS go di- our system has a small code size, anticipated functionality, rectly to devices instead of being handled by the hypervisor. and moderate overhead. Our experiments also show that Their later work [36] utilizes BitVisor to prevent persistent ForenVisor can guarantee strong protection to those protect- rootkits by protecting system-critical files from writing op- ed files, even when the guest OS is compromised by viruses, erations. Besides, the later work introduces a mode named such as ‘ILOVEYOU’ and Worm.WhBoy. Administrator Mode in which they make security policy by using user-level tool in the guest OS, while in this mode ACKNOWLEDGEMENTS guest OS is assumed to be secure. Using the interfaces of the file system in the guest OS to get disk blocks of sensitive This work was supported by 863 Program of China (No. files is a simple and fast way to solve the semantic gap. 2012AA010905) and NSFC (Grant No.61272100, 61202374). However, this results in hypervisor relying on the guest OS Ruhui Ma is the Corresponding Author. to provide file security, which makes hypervisor not isolated to the guest OS and incurs system vulnerability. Filesafe, REFERENCES an important module of ForenVisor, makes no assumption about the guest OS. We leverage the methodology of VMI [1] R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, and I. Brandic, instead of depending on OS interfaces. By reconstructing “Cloud computing and emerging it platforms: Vision, hype, and reality for delivering computing as the 5th utility,” Future Genera- a file system from raw data on disk, our hypervisor is tion computer systems, vol. 25, no. 6, pp. 599–616, 2009. truly independent of the guest OS and thus offers a strong [2] S. U. Malik, S. U. Khan, and S. K. Srinivasan, “Modeling and anal- protection mechanism for sensitive files. ysis of state-of-the-art vm-based cloud management platforms,” Cloud Computing, IEEE Transactions on, vol. 1, no. 1, pp. 1–1, 2013. [3] C. Modi, D. Patel, B. Borisaniya, H. Patel, A. Patel, and M. Rajara- 8.4 Enhancing Hypervisor Security jan, “A survey of intrusion detection techniques in cloud,” Journal of Network and Computer Applications, vol. 36, no. 1, pp. 42–57, 2013. The security of the hypervisor layer gains importance due to [4] B. Martini and K.-K. R. Choo, “An integrated conceptual digital the wide adoption of the virtualization technology in cloud forensic framework for cloud computing,” Digital Investigation, platform. An intuitive approach to improve security is to vol. 9, no. 2, pp. 71–80, 2012. [5] Guidance Software, Inc., “EnCase,” http://www. decrease the TCB size of hypervisor, aiming at which many guidancesoftware.com/, 2001. thin hypervisors have been proposed. NOVA [61] propos- [6] AccessData Group, “FTK,” http://www.accessdata.com/, 2003. es a microhypervisor-based architecture, which takes the [7] B. Carrier and E. H. Spafford, “An event-based digital forensic investigation framework,” in Digital forensic research workshop, similar idea of Microkernel system [62], decomposes fine- 2004, pp. 11–13. grained functions into a microhypervisor, multiple VMMs [8] B. Hay and K. Nance, “Forensics examination of volatile system and some other modules, thereby effectively reducing the data using virtual introspection,” SIGOPS Oper. Syst. Rev., vol. 42, TCB size. Some researchers take the idea to build special- pp. 74–82, April 2008. [9] B. Dolan-Gavitt, A. Srivastava, P. Traynor, and J. T. Giffin, “Robust purpose hypervisors with small TCB. TrustVisor [39] is signatures for kernel data structures,” in ACM Conference on Com- a thin hypervisor that provides code integrity and data puter and Communications Security, E. Al-Shaer, S. Jha, and A. D. integrity for applications. Another similar hypervisor is Keromytis, Eds. ACM, 2009, pp. 566–577. BitVisor [19], which specializes in enforcing I/O device [10] G. Hoglund and J. Butler, Rootkits: subverting the Windows kernel. Addison-Wesley Professional, 2006. security. Both of TrustVisor and BitVisor take advantages [11] Y. Kinebuchi, S. Butt, V. Ganapathy, L. Iftode, and T. Nakajima, of the hardware-assisted virtualization technology. “Monitoring integrity using limited local memory,” Information Besides reducing the TCB size, some researches focus on Forensics and Security, IEEE Transactions on, vol. 8, no. 7, pp. 1230– 1242, 2013. checking the integrity of hypervisor, including the control- [12] J. Li, Z. Wang, T. Bletsch, D. Srinivasan, M. Grace, and X. Jiang, flow integrity [63], runtime integrity [23], and memory in- “Comprehensive and efficient protection of kernel control data,” tegrity [64]. XMHF [64] is an extensible and modular hyper- Information Forensics and Security, IEEE Transactions on, vol. 6, no. 4, visor framework, which provides an automated verification pp. 1404–1417, 2011. [13] S. Rekhis and N. Boudriga, “A system for formal digital forensic of memory integrity, based on the DRIVE methodology. Our investigation aware of anti-forensic attacks,” Information Forensics scheme enhances security mainly by reducing hypervisor and Security, IEEE Transactions on, vol. 7, no. 2, pp. 635–650, 2012.

[14] S. Krishnan, K. Z. Snow, and F. Monrose, “Trail of bytes: efficient ’08. New York, NY, USA: ACM, 2008, pp. 91–100. [Online]. support for forensic analysis,” in ACM Conference on Computer Available: http://doi.acm.org/10.1145/1346256.1346269 and Communications Security, E. Al-Shaer, A. D. Keromytis, and [35] X. Zhong, C. Xiang, M. Yu, Z. Qi, and H. Guan, “A virtualization V. Shmatikov, Eds. ACM, 2010, pp. 50–60. based monitoring system for mini-intrusive live forensics,” Inter- [15] P. M. Chen and B. D. Noble, “When virtual is better than real,” in national Journal of Parallel Programming, pp. 1–17, 2013. Workshop on Workstation Operating Systems (now HotOS)/Workshop [36] Y. Chubachi, T. Shinagawa, and K. Kato, “Hypervisor-based pre- on Hot Topics in Operating Systems, 2001, pp. 133–138. vention of persistent rootkits,” in SAC, 2010, pp. 214–220. [16] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. L. Harris, A. Ho, [37] M. Corporation, “Microsoft extensible firmware initiative fat32 file R. Neugebauer, I. Pratt, and A. Warfield, “Xen and the art of system specification,” no. 12, 2000. virtualization,” in SOSP, M. L. Scott and L. L. Peterson, Eds. [38] Wikipedia, “NTFS,” http://en.wikipedia.org/wiki/NTFS, 1993. ACM, 2003, pp. 164–177. [39] J. M. McCune, Y. Li, N. Qu, Z. Zhou, A. Datta, V. D. Gligor, and [17] C. A. Waldspurger, “Memory resource management in vmware A. Perrig, “Trustvisor: Efficient tcb reduction and attestation,” in esx server,” in OSDI, 2002. IEEE Symposium on Security and Privacy, 2010, pp. 143–158. [18] J. M. McCune, Y. Li, N. Qu, Z. Zhou, A. Datta, V. Gligor, and [40] Wikipedia, “ILOVEYOU,” http://en.wikipedia.org/wiki/ A. Perrig, “Trustvisor: Efficient tcb reduction and attestation,” in ILOVEYOU, 2000. Security and Privacy (SP), 2010 IEEE Symposium on. IEEE, 2010, [41] R. Technology, “WinPcap,” http://www.winpcap.org/. pp. 143–158. [42] Standard Performance Evaluation Corporation, “SPEC CPU2000,” [19] T. Shinagawa, H. Eiraku, K. Tanimoto, K. Omote, S. Hasegawa, http://www.spec.org/cpu2000/, 2000. T. Horie, M. Hirano, K. Kourai, Y. Oyama, E. Kawai, K. Kono, [43] Intel Corporation, “Iometer,” http://www.iometer.org/, 1998. S. Chiba, Y. Shinjo, and K. Kato, “Bitvisor: a thin hypervisor for [44] M. Xia, M. Yu, Q. Lin, Z. Qi, and H. Guan, “Enhanced privilege enforcing i/o device security,” in VEE, 2009, pp. 121–130. separation for commodity software on virtualized platform,” in [20] J. Rutkowska, “Subverting vistatm kernel for fun and profit,” Black ICPADS, 2010, pp. 275–282. Hat Briefings, 2006. [45] J. Slay, Y.-C. Lin, B. Turnbull, J. Beckett, and P. Lin, Towards a [21] L. Martignoni, A. Fattori, R. Paleari, and L. Cavallaro, “Live and formalization of digital forensics. Springer, 2009. trustworthy forensic analysis of commodity production systems,” [46] “File Allocation Table,” Dec. 2015, page Version ID: in Recent Advances in Intrusion Detection. Springer, 2010, pp. 297– 695816671. [Online]. Available: https://en.wikipedia.org/w/ 316. index.php?title=File Allocation Table&oldid=695816671 [22] J. Wang, M. Yu, B. Li, Z. Qi, and H. Guan, “Hypervisor-based pro- [47] R. Rosen, “Linux containers and the future cloud,” Linux J, vol. tection of sensitive files in a compromised system,” in Proceedings 240, 2014. of the 27th Annual ACM Symposium on Applied Computing. ACM, [48] B. Martini and K.-K. Choo, “Cloud forensic technical challenges 2012, pp. 1765–1770. and solutions: A snapshot,” Cloud Computing, IEEE, vol. 1, no. 4, [23] A. M. Azab, P. Ning, Z. Wang, X. Jiang, X. Zhang, and N. C. pp. 20–25, Nov 2014. Skalsky, “Hypersentry: enabling stealthy in-context measurement [49] R. McKemmish, What is forensic computing? Australian Institute of hypervisor integrity,” in Proceedings of the 17th ACM conference of Criminology Canberra, 1999. on Computer and communications security. ACM, 2010, pp. 38–49. [50] B. Martini and K.-K. R. Choo, “Distributed filesystem forensics: [24] M. Yu, Q. Lin, B. Li, Z. Qi, and H. Guan, “Vis: virtualization Xtreemfs as a case study,” Digital Investigation, vol. 11, no. 4, pp. enhanced live acquisition for native system,” in Proceedings of the 295 – 313, 2014. [Online]. Available: http://www.sciencedirect. Second Asia-Pacific Workshop on Systems. ACM, 2011, p. 13. com/science/article/pii/S1742287614000942 [51] J. Dykstra and A. T. Sherman, “Design and implementation of [25] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. L. Harris, A. Ho, frost: Digital forensic tools for the openstack cloud computing R. Neugebauer, I. Pratt, and A. Warfield, “Xen and the art of platform,” Digital Investigation, vol. 10, pp. S87–S95, 2013. virtualization,” in SOSP, M. L. Scott and L. L. Peterson, Eds. [52] D. Quick and K.-K. R. Choo, “Google drive: Forensic analysis ACM, 2003, pp. 164–177. of data remnants,” Journal of Network and Computer Applications, [26] G. Neiger, A. Santoni, F. Leung, D. Rodgers, and R. Uhlig, “Intel vol. 40, pp. 179 – 193, 2014. [Online]. Available: http://www. virtualization technology: Hardware support for efficient proces- sciencedirect.com/science/article/pii/S1084804513002051 sor virtualization.” Intel Technology Journal, vol. 10, no. 3, 2006. [53] D. Quick, B. Martini, and R. Choo, Cloud storage forensics. Syn- [27] A. Joshi, S. T. King, G. W. Dunlap, and P. M. Chen, gress, 2013. “Detecting past and present intrusions through vulnerability- [54] T. Garfinkel, M. Rosenblum et al., “A virtual machine introspection specific predicates,” in Proceedings of the twentieth ACM based architecture for intrusion detection.” in NDSS, 2003. symposium on Operating systems principles , ser. SOSP ’05. New [55] X. Jiang and X. Wang, “out-of-the-box monitoring of vm-based York, NY, USA: ACM, 2005, pp. 91–104. [Online]. Available: high-interaction honeypots,” in Recent Advances in Intrusion Detec- http://doi.acm.org/10.1145/1095810.1095820 tion. Springer, 2007, pp. 198–218. [28] B. Dolan-Gavitt, T. Leek, M. Zhivich, J. Giffin, and W. Lee, “Vir- [56] A. Dinaburg, P. Royal, M. Sharif, and W. Lee, “Ether: malware tuoso: Narrowing the semantic gap in virtual machine introspec- analysis via hardware virtualization extensions,” in Proceedings of tion,” in Security and Privacy (SP), 2011 IEEE Symposium on. IEEE, the 15th ACM conference on Computer and communications security. 2011, pp. 297–312. ACM, 2008, pp. 51–62. [29] R. Wojtczuk and J. Rutkowska, “Attacking smm memory via intel [57] N. L. Petroni Jr, A. Walters, T. Fraser, and W. A. Arbaugh, “Fatkit: cpu cache poisoning,” Invisible Things Lab, 2009. A framework for the extraction and analysis of digital forensic [30] ——, “Attacking intel trusted execution technology,” Black Hat DC, data from volatile system memory,” Digital Investigation, vol. 3, 2009. no. 4, pp. 197–210, 2006. [31] S. T. King and P. M. Chen, “Subvirt: Implementing malware with [58] B. Hay and K. Nance, “Forensics examination of volatile system virtual machines,” in Security and Privacy, 2006 IEEE Symposium data using virtual introspection,” ACM SIGOPS Operating Systems on. IEEE, 2006, pp. 14–pp. Review, vol. 42, no. 3, pp. 74–82, 2008. [32] Intel Trusted Execution Technology Architecture Overview. Denver, [59] K. Kourai and S. Chiba, “Hyperspector: virtual distributed moni- CO: Intel Corporation, 2006. toring environments for secure intrusion detection,” in Internation- [33] T. Shinagawa, H. Eiraku, K. Tanimoto, K. Omote, S. Hasegawa, al Conference on Virtual Execution Environments, 2005, pp. 197–207. T. Horie, M. Hirano, K. Kourai, Y. Oyama, E. Kawai, [60] X. Zhao, K. Borders, and A. Prakash, “Towards protecting sensi- K. Kono, S. Chiba, Y. Shinjo, and K. Kato, “Bitvisor: a thin tive files in a compromised system,” in IEEE Security in Storage hypervisor for enforcing i/o device security,” in Proceedings Workshop, 2005, pp. 21–28. of the 2009 ACM SIGPLAN/SIGOPS international conference [61] U. Steinberg and B. Kauer, “Nova: a microhypervisor-based se- on Virtual execution environments, ser. VEE ’09. New York, cure virtualization architecture,” in Proceedings of the 5th European NY, USA: ACM, 2009, pp. 121–130. [Online]. Available: conference on Computer systems. ACM, 2010, pp. 209–222. http://doi.acm.org/10.1145/1508293.1508311 [62] D. L. Black, D. B. Golub, D. P. Julin, R. F. Rashid, R. P. Draves, R. W. [34] S. T. Jones, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau, Dean, A. Forin, J. Barrera, H. Tokuda, G. Malan et al., “Microkernel “Vmm-based hidden process detection and identification using operating system architecture and mach,” in Proceedings of the lycosid,” in Proceedings of the fourth ACM SIGPLAN/SIGOPS USENIX Workshop on Micro-Kernels and Other Kernel Architectures, international conference on Virtual execution environments, ser. VEE 1992, pp. 11–31.

[63] Z. Wang and X. Jiang, “Hypersafe: A lightweight approach to Haibing Guan received Ph.D. degree from provide lifetime hypervisor control-ﬂow integrity,” in Security and Tongji University in 1999. He is a Professor of Privacy (SP), 2010 IEEE Symposium on. IEEE, 2010, pp. 380–395. School of Electronic, Information and Electronic [64] A. Vasudevan, S. Chaki, L. Jia, J. McCune, J. Newsome, and A. Dat- Engineering, Shanghai Jiao Tong University, and ta, “Design, implementation and veriﬁcation of an extensible and the director of the Shanghai Key Laboratory of modular hypervisor framework,” in Security and Privacy (SP), 2013 Scalable Computing and Systems. His research IEEE Symposium on. IEEE, 2013, pp. 430–444. interests include distributed computing, network [65] M. L. Scott and L. L. Peterson, Eds., Proceedings of the 19th ACM security, network storage, green IT and cloud Symposium on Operating Systems Principles 2003, SOSP 2003, Bolton computing. Landing, NY, USA, October 19-22, 2003. ACM, 2003.

David S. L. Wei received his Ph.D. degree in Computer and Information Science from the Uni- Zhengwei Qi received his B.Eng. and M.Eng versity of Pennsylvania in 1991. He is currently a degrees from Northwestern Polytechnical Uni- Professor of Computer and Information Science versity, in 1999 and 2002, and Ph.D. degree from Department at Fordham University. He is an As- Shanghai Jiao Tong University in 2005. He is a sociate Editor of IEEE Transactions on Cloud professor at the School of Software, Shanghai Computing, and an Associate Editor of Journal Jiao Tong University. His research interests in- of Circuits, Systems and Computers. Dr. Wei clude program analysis, model checking, virtual focuses his research efforts on cloud computing, machines, and distributed systems. wireless sensor networks and cognitive radio networks.

Chengcheng Xiang received his B.E degree from Shanghai Jiao Tong University in 2013. Currently, he is a graduate student at Shanghai Key Laboratory of Scalable Computing and Sys- tems, School of Software, Shanghai Jiao Tong University. His research interests mainly include program analysis, operating system, and virtualization.

Ruhui Ma received his Ph.D. degree in computer science from Shanghai Jiao Tong University, China, in 2011. He has worked as a postdoc at Shanghai Jiao Tong University (2012 and 2013), at McGill University, Canada (2014), respectively. He is an Assistant Professor in the Depart- ment of Computer Science and Engineering at SJTU. His main research interests are in virtual machines, computer architecture and network virtualization.

Jian Li received his M.S. and B.S. degree from the University of Henri Poincare (France, 2003) and TianJin University (2001). He obtained his Ph.D. in Computer Science from the Institute Na- tional Polytechnique de Lorraine-Nancy, France in 2007.He has worked as a postdoc at Univer- sity of Toronto and at McGill University in 2007 and 2008. He is an Associate Professor in the School of Software at SJTU. His research interests include real-time scheduling theory, network protocol design and embedded system.