Virtual Machine Extrospection: a Reverse Information Retrieval in Clouds 1P.Vijay Bhaskar Reddy2n.Venkateswarlu

International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

Virtual Machine Extrospection: A Reverse Information Retrieval in Clouds 1P.Vijay Bhaskar Reddy2N.Venkateswarlu

2 Scholar, Dept. of Master of Computer Applications, Narayana Engineering College, Gudur.

1 Assistant Professor, Dept. of Master of Computer Applications, Narayana Engineering College, Gudur.

Abstract—In a virtualized environment, it is not Asthehypervisor runsatamoreprivilegedlevelthan difficult to retrieve guest OS information from its its guest OSes, at this level, one can control hypervisor. However, it is very challenging to physical resources, monitor their access, and be retrieve information in the reverse direction, i.e., isolated from tampering against attackers from the retrieve the hypervisor information from within a guest OS. Monitoring offline-grained information guest OS, which remains an open problem and of the guest OSes from the underlying hypervisoris has not yet been comprehensively studied before. called virtual machine introspection (VMI)[1]. In this paper, we take the initiative and study this However, at the guest OS level retrieving reverse information retrieval problem. In information about the underlying hypervisor particular, we investigate how to determine the becomes very challenging, if not impossible. In host OS kernel version from within a guest OS. thispaper,welabelthereverseinformationretrievalwit We observe that modern commodity hypervisors h the coined term virtual machine introduce new features and bug fixes in almost extrospection(VME).While VMI has been widely every new release. Thus, by carefully analyzing used for securitypurposesduring the past decade, the seven-year evolution of Linux KVM the reverse directionVME—theprocedurethat development (including 3485 patches), we can retrieves the hypervisor information from identify 19 features and 20 bugs in the hypervisor theguestOS level—is a new topic and has not detectable from within a guest OS. Building on beencomprehensively studied before. our detection of these features and bugs, we VME can be critically important for both malicious attackers and regular users. On one present a novel framework called Hyperprobe hand, from the attackers’ perspective, when an that for the first time enables users in a guest OS attacker is in control of a virtual machine (VM), to automatically detect the underlying host OS either as a legal resident or after a successful kernel version in a few minutes. We implement a compromise of the victim’s VM, the underlying prototype of Hyperprobe and evaluate its hypervisor becomes its attacking target. This threat effectiveness in six real world clouds, including has been demonstrated in [8], [9], where an Google Compute Engine (a.k.a. Google Cloud), attacker is able to mount a privilege escalation HP Helion Public Cloud, ElasticHosts, Joyent attack from within a VMware virtual machine and Cloud, CloudSigma, and VULTR, as well as in a a KVM-based virtual machine, respectively, and controlled testbed environment, all yielding then gains some control of the host machine. promising results. Although these works demonstrate the possibility of such a threat, successful escape attacks from the Index Terms—Virtualization, Hypervisor, guest to the host are rare. The primary reason is Extrospection, Linux, KVM. that most hypervisors are, by design, invisible to the VMs. Therefore, even if an attacker 1.INTRODUCTION gainsfull control of a VM, a successful attempt to prevalent,a variety of security methodologies break out of theVM and break into the hypervisor havebeende- veloped at the hypervisor level, requires an in-depth knowledge of the underlying includingintrusionand hypervisor, e.g., type and version of the malwaredetection[1],[2],honeypots[3],[4],kernelro hypervisor. However, there is no straightforward otkitdefense [5], [6], and detection of way for attackers to obtain suchknowledge. covertlyexecutingbinaries[7]. These security On the other hand, benign cloud users services depend on the key may also need to know the factorthatthehypervisorisisolatedfromitsguestOSes. underlyinghypervisor information. It is com- http://indusedu.org Page 239 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

monly known that hardware and software Hyperprobe, we perform experiments in a systems both have various bugs and controlled testbedenviron- ment. For 11 of the 35 vulnerabilities, and different hard- kernel versions we studied, Hyperprobe can ware/software may exhibit different correctly infer the exact version vulnerabilities. Cloud customers, when number;fortherest,Hyperprobecannarrowitdown to making decisions on the choice of a cloud within 2 to 5versions. provider, may want to know more The remainder of the paper is organized as information about the underlying hardware or follows. Section 2 describes the background of our software. This will help customers determine work. Section 3 presents the design of Hyperprobe. whether the underlying hardware/software Section 4 detailsthe implementation of Hyperprobe can be trusted, and thus help them decide with several case studies. Section 5 presents whether or not to use this cloud service. experimental results on virtual machines in the However, for security reasons, cloud cloud and our controlled testbed. Section 6 providers usually do not release such discusses some potential extensions to the sensitive information to the public framework. Section 7 sur- orcustomers. veysrelatedwork,andfinally,Section8concludesthep Whereas research efforts have been made to aper. detect the existence of a hypervisor [10], [11], [12], [13], from a guest OS, to the best of our 2 BACKGROUND knowledge, there is no literature describing how Hypervisor, also named as virtual machine to retrieve more detailed information about the monitor, is a piece of software that creates and hypervisor, e.g., the kernel version of the host manages VMs. Tradition- ally, hypervisors such as OS, the distribution of the host OS (Fedora, VMware and Virtual PC use the technique of SuSE, or Ubuntu?), the CPU type, the memory binary translation to implement virtualization. type, or any hardware information. In this Later on, x86 processor vendors including Intel paper, we make an attempt to investigate this and AMD released their architecture extensions to problem. More specifically, as a first step support virtualiza- towards VME, we study the problem of tion.Thosehypervisorsthatusebinarytranslationarec detecting/inferring the host OS kernel version alled software-only hypervisors, and recent from within a guest OS, and we expect our hypervisors that take advantage of these processor work will inspire more attention on mining the extensions are calledhardware assisted hypervisors information of a hypervisor. The major research [20]. In this paper, we focus on a popular hardware contributions of our work are summarized assisted commodity hypervisor, Linux KVM. We asfollows: develop our framework and performexperiments Wearethefirsttostudytheproblemofdetec- on a physical machine with Linux OS as the host, ing/inferring the host OS kernel version from which runs a KVM hypervisor, and a VM is within a VM. Exploring the evolution of Linux running on top of the KVM hypervisors, we analyze various features hypervisor.OurstudycoversLinuxkernelversionsfro and bugs introduced in the KVM hypervisor; and m then we explain how these features and bugs can 2.6.20 to 3.14. While 2.6.20, released in February be used to detect/infer the hypervisor kernel 2007, is the first kernel version that includes version. KVM, 3.14, released in March 2014, is the latest stable kernel at the time of this study. More .We design and implement a novel, practical, auto- specifically, we study the evolution of KVM over matic, and extensible framework, called the past seven years and make three major Hyperprobe, for conducting the reverse observations. In this section, we briefly describe information retrieval. Hy- perprobe can help users Linux KVM and report ourobservations. in a VM to automatically detect/infer the 2.1 LinuxKVM underlying host OS kernel version in less than five minutes with highaccuracy. KVM refers to kernel-based virtual machine. Since .We perform our experiments in six real world Linux kernel version 2.6.20, KVM is merged into clouds, including Google Compute Engine [14], the Linux mainline kernel as a couple of kernel HP Helion Public Cloud [15], ElasticHosts [16], modules: an architecture JoyentCloud [17], CloudSigma [18], and VULTR [19], and our experimental results are very promising. To further validate the accuracy of http://indusedu.org Page 240 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

Qemu Qemu emulating a CPU is hard. Since a modern CPU App App Normal User Normal User defines hundreds of registers, emulating the Process behaviors of various registers correctly is Guest Guest Process OS OS challenging. Failing to do so usually causes various unexpected results. In fact, register

Linux Kernel KVM related bugs have been reported on a regularbasis. During our study, we discover that these Fig. 1: KVM Overview features and bugs can help us determine the independent module called kvm.ko, and an underlying hypervisor kernel version. A more architecture de- pendent module called either detailed description of our design approach is kvm-intel.ko or kvm-amd.ko. As a hardware presented in Section 3. assisted virtualization technology, KVM relies heavily on the support of the underlying CPUs 1.KVM has recently started supporting non-x86 and requires different implementations for platform, such as ARM and PPC; however, in this study, we only consider patches for x86 different CPU vendors, such as Intel VT-x and platforms, i.e., the number 3485 does not AMD SVM. Figure 1 illustrates the basic include the patches for the non-x86platforms. architecture of KVM. KVM works inside a host kernel and turns the host kernel into a 2.2 Intel VT-x Extension hypervisor. On top of the hypervisor, there can be multiple VMs. Usually KVM requires a user-level tool called Qemu to emulate various devices, and they communicate using predefined ioctlcommands. Over the years, KVM has changed significantly. The original version in 2.6.20 consists of less than 20,000 lines of code (LOC); but in the latest 3.14 version, KVM modules consist of about 50,000 LOC. The

reason of such growth is that 3485 KVM related patches have been released by Linux As a hardware assisted hypervisor, KVM relies on mainline kernel1. By carefully analyzing these the virtualization extensions of the underlying patches, we make a few important observations processors. In 2006, both Intel (VT-x) and AMD about the evolution of the KVM (AMD-SVM) introduced hardware virtualization developmentprocess. First, while ideally hypervisors should The key concept of Intel VT-x is that the CPU is betransparent to guest OSes, this is not realistic. split into the root mode and the non-root mode. In particular, during its development process, Generally, the hypervisor runs in the root mode and on the one hand, KVM exposes more and more its guests run in the non-root mode. Transitions processor features to a guest OS; on the other from the root mode to the non- root mode are hand, KVM has been provided with many called VM entries, and transitions from the non- paravirtualiza- tion features. These changes root mode to the root mode are called VM exits. improve performance but at the cost of The hypervisor can specify which instructions and lesstransparency. events cause VM exits. These VM exits actually Second, for the sake of better resource allow the hypervisor to retain control of the utilization, KVM has also included several underlying physical resources. An example of a virtualization-specific features, e.g., nested VM exit is, when a guest OS attempts to access virtualization [21] and kernel same page merg- sensitive registers, such as control registers or ing (KSM) [22], many of which can be detected debug registers, it would cause a VM exit. A from within the guest OS. handler defined by the hypervisor will then be Third, similar to all other large projects, KVM invoked, and the hypervisor will try to emulate the have bugs. Among the 3485 patches, about behavior of the registers. As men- tioned above, 30% of them arebug fixes. In particular, we given the large number of registers, register notice that a common type of bugs in KVM is emulation is hard anderror-prone. related to registers. This reflects the fact that The first generation of Intel VT-x processors http://indusedu.org Page 241 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

mainly simplifies the design of hypervisors. But space, we are not able to explain each of the since then, more and more features have been features/bugs, but we will choose some of the included in their later processor models. To name more interesting ones and explain them in the a few, Extended Page Table (EPT), which aims to next section as case studies. In this section, reduce the overhead of address translation,is we elaborate on how we use these features introduced by Intel since Nehalem processors, and and bugs to infer the underlying hypervisor VMCS shadow, which aims to accelerate nested kernel version. virtualization, is introduced since Haswell. Once KVM Features these new hardware features are released, modern KVM releases new features regularly. One hypervisors such as KVM and Xen, provide their may infer the underlying hypervisor kernel support for these new features on the softwareside. version using the following logic: if feature A is introduced in 2.6.30 and feature B is 3.DESIGN introduced in 2.6.35, then if one can detect Hyperprobe framework has the following goals: feature A but not B, one may infer that the •.Practical: The framework should detect the underlying host kernel version is between underlying hypervisor kernel version within a 2.6.30 and 2.6.34. However, this may lead to reasonable amount of time with high accuracy and inaccuracies. Since even if feature B is precision. As more test cases are added to provide introduced into the Linux mainline kernel on more vantage points of different kernel versions, a particular release, the feature could be its accuracy and precision should also beimproved. disabled by system administrators. Therefore, even if feature B is not detected, it does not • Automatic: The framework should run test cases, mean the underlying hypervisor kernel collect and analyze results automatically without manual inter- vention. To this end, the test cases version is older than 2.6.35. Such 2 customizations could impact precision. should not crash the guest or hostOS. To avoid such inaccuracies, Hyperprobe • Extensible: The framework should be uses the following strategy to handle the easilyextended to detect/infer future Linux kernel existence or non-existence of a kernel versions and to add more vantage points to feature: if we detect a feature exists, we previously released kernel versions. To this end, assert that the underlying hypervisor kernel the whole framework should be modular, and version is no older than the version in which adding modules to the framework should beeasy. this feature was first introduced. By designing test cases that detect these features, 2.Kernel bugs that cause guest or host OS to crash we report a minimum version number. This are verycommon, but we purposely avoided using them in our test cases. One could utilize these bugs number can be viewed as the lower bound of to gain more vantage points, but they should be the underlying hypervisor kernel version. used with greatcaution. KVM Bugs and Bug Fixes Technical Challenges KVM has bugs and bug fixes like any other To meet these design goals, we faced several software. If bugs can be detected from within challenges: even though the hypervisor the guest OS, then one may infer the introduces new features fre- quently, how underlying hypervisor kernel version using many of them are detectable from within the following logic: assuming bug A is fixed the guest OS? Similarly, how many in kernel version 2.6.30, and bug B is fixed in hypervisor bugs are detectable from within kernel version 2.6.35. If one detects that bug the guest OS? A does not exist but bug B does, one may After manually analyzing the infer that the underlying hypervisor kernel is aforementioned 3485 patches, we found a between 2.6.30 sufficient number of features and bugs that meet our requirements. Tables 1 and 2 illustrate the features and bugs we have selected for our framework. To exploit each, it would require an in-depth knowledge of the kernel and also a good understanding of the particular feature/bug. Due to limited

http://indusedu.org Page 242 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

and 2.6.34. Similarly, this may lead to the upper bound of the underlying hypervisor inaccuracies, as a bug could be manually fixed in kernel version. Along with the test cases that an older kernel without updating the entire kernel. detect kernel features, which can report a lower Therefore, the non-existence of a bug does not bound, we can then narrow down the hypervisor necessarily mean the kernel is newer than a kernel to a range of versions. Figure 2 illustrates particular version. an example: upon the detection of feature A and To avoid such inaccuracies, Hyperprobe uses bug B, we report that the hypervisor has kernel the following strategy to handle the existence version 2.6.30 as the lower bound and 2.6.34 as or non-existence of a kernel bug: if a bug is the upper bound. detected, we assert that the underlying kernel version is older than the kernel version where 4 IMPLEMENTATION this bug is fixed. By creating test cases that Our framework implementation consists of detect kernel bugs, we report a maximum 3530 lines of C code (including comments). version number. This number can be viewed as To meet the extensible goal, we http://indusedu.org Page 243 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

http://indusedu.org Page 244 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

although the time cost may vary across different Case Studies: Kernel Features machines, it remains nearly the same when we Kernel Same page Merging switch from EPT to shadow page table, or from shadow page table to EPT. • Memory over commitment, a mechanism to save When running a benchmark that requires memory, was originally implemented in VMware [23]. frequent memory mapping changes, EPT offers Later on, this mechanism is introduced into Linux significant performance improvements over kernel since version 2.6.32, and it is called kernel shadow page table. Partic- ularly, we choose the same page merging (KSM) [22]. This is a crucial classic forkwait microbenchmark, which has been widely employed [20], [25], [26],to evaluate feature in a virtualized environment, where there could virtualization performance. The main part of this be a large number of similar VMs running on top of benchmark repeats the operation of process one hypervisor. Consequently, if we can detect KSM creation and destruction very aggressively. is enabled, we can ascertain that the underlying Similar to [26], we have tested the forkwait hypervisor kernel is newer than or equal to version microbenchmark across 30 VMs (with different 2.6.32. hardware and software configurations), and have consistently observed that EPT offers approxi- Extended Page Table (EPT) mately 600% performance gains over shadow page table. Traditionally, commercial hypervisors including KVM, Xen, and VMware, all use the shadow page table technique to manage VM memory. The Therefore, our algorithm can be elaborated as shadow page table is maintained by the hypervisor follows. First we allocate a memory page, and stores the mapping between guest. compute the average time to access one byte of virtual address and machine address. This mechanism the memory page, and use this average time as requires a serious synchronization effort to make the a baseline. Next, we run the forkwait shadow page table consistent with the guest page table. microbenchmark, compute the average time to In particular, when a workload in the guest OS requires fork-wait one process, and record the ratio frequent updates to the guest page tables, this between these two average times (average time to synchronization overhead can cause very poor fork-wait one process divided by average time to performance. To address this problem, recent access one byte of memory page). On all VMs we architecture evolution in x86 processors presents the have tested, this ratio is larger than 100,000 when extended/nested page table technology (Intel EPT and the hypervisor is using shadow page table, and it AMD NPT). With this new technology, hypervisors do is usually between 10,000 to 20,000 when the not need to maintain shadow page tables for the VMs, hypervisor is using EPT. Therefore, we can and hence avoid the synchronization costs of the choose a threshold, and if the ratio is less than shadow page table scenario. The difference between that threshold, we assume the underlying shadow page table and extended page table is hypervisor is using EPT; otherwise, we assume it illustrated in Figure 4. is using shadow page table. Our current implementation uses 30,000 as the threshold. Before kernel 2.6.26, KVM uses shadow page table to virtualize memory. Since kernel 2.6.26, KVM starts to support Intel EPT and enable it by default. Therefore, if we can detect the existence of EPT from within the

guest OS, we can assume the underlying hypervisor kernel is newer than or equal to version 2.6.26. Algorithm 3 describes the EPT detection mechanism, and we derive this algorithm from the following observations:

• On a specific VM, no matter whether the underlying hypervisor is using shadow page table

or EPT, the average time to access one byte in memory is very stable. We have measured this across 30 virtual machines (with different hardware and software configurations). Note that

http://indusedu.org Page 245 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

Emulating Hyper-V and Support Microsoft Enlighten- ment Microsoft Enlightenment is an optimization made by Mi- crosoft to Windows systems when running in a virtualized environment. The key idea is to let the guest OS be aware of the virtualized environment, and therefore tune its behavior for performance improvement. Windows systems released after 2008, such as Windows Server 2008, and Windows

Vista, are fully enlightened [27], [28], which means they take full advantage of the possible enlightenments. Microsoft Enlightenment was originally designed for Hyper-V, but Microsoft provides APIs for other hypervisors to utilize this optimization. Since kernel 2.6.34, KVM has started utilizing these APIs and supporting Mi- crosoft Enlightenment. According to the Hyper- V specification [29], [30], several synthetic registers are defined, including HV X64 GUEST OS ID, HV X64 HYPERCALL, HV X64 VP INDEX, as well as the EOI/TPR/ICR APIC registers. Details of these registers are shown in Table 3. Before kernel 2.6.34, accessing these registers would gener- ate a general protection fault, but since kernel 2.6.34, they should be accessible whether accessing from a Windows or Linux guest OS. Thus, we attempt to access these registers. If they are accessible, we assume the kernel version is newer than or equal to version 2.6.34; otherwise, the feature may not be present, but we do not make any assertion.

http://indusedu.org Page 246 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

5 EVALUATION ranging from 1 to 16. Hyperprobe shows that Google is using a kernel version between 3.2 To demonstrate how Hyperprobe performs in the and 3.3 in its hypervisors. According to a recent wild, we ran its test suite on VMs provisioned work [35] and some online communications from different public cloud providers to detect written by Google engineers [36], [37], Debian 7 their hypervisor kernel versions. In most cases, is most likely used in its hypervisors as this we were able to narrow the suspected hyper- Linux distribution is widely used in its visor kernel versions down to a few; in one case, production environments. The default kernel of we even had an exact match. However, as public Debian 7 is 3.2.0-4, agreeing with our findings. cloud providers do not disclose detailed information about the hypervisors they are using (for obvious security reasons), we had to find other means to confirm these results, such as user forums and white papers. Our results do coincide with what are being reported via these side channels. To more rigorously verify the accuracy of Hyperprobe, we also evaluated it in a controlled testbed environment across 35 different kernel versions with very encouraging results.

a. Results in Real World Clouds The public cloud providers we selected in this study include Google Compute Engine, HP Helion Public Cloud, ElasticHosts, Joyent Cloud, and CloudSigma. (all KVM- based) In our experiments, we intentionally created VMs with different configurations to test the detection robustness and accuracy of our framework. The results are shown in Tables 4, 5, 6, 7, and 8. Running the test suite and analyzing the collected results take less than 5 minutes to complete, which is fairly reasonable from a practical point of view. In fact, we observe that the running time is mainly dominated by those test cases that require sleeping or running some microbenchmarks. In what follows, we detail our findings for each cloud provider.

i. Google Compute Engine Google Compute Engine is hosted in data centers located in Asia, Europe, and America. One can choose the number of VCPUs per VM

http://indusedu.org Page 247 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

http://indusedu.org Page 248 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

features and bugs are common in both versions. For VULTR Cloud, we performed some architectures, others may not be. And these manual efforts to figure out its upper bound after architecture-specific features and bugs can further launching a CentOS 7 x64 system, with 1 virtual improve the accuracy of Hyperprobe’s reported CPU, 768 MB memory, and a 15GB SSD disk. The results. The result for CloudSigma was mainly host machine is located at New Jersey. We found based on the common features and bugs, and thus, that the bit 63 of the CR3 register is not writable. Hyperprobe was not able to narrow down the This fact matches with a KVM bug description kernel versions as much as it could for the Intel- [43], and the bug is fixed in the kernel version based cloud providers. 3.19. This suggests that the underlying kernel version in VULTR Cloud is between 3.14 to 3.18. Summarizing Findings in Public Clouds Results in a Controlled Testbed We found several interesting facts about these To better observe if what Hyperprobe detects is clouds: really what is deployed, we ran the same test suite Even if a cloud provider has multiple data centers in a controlled testbed environment across all the spread across various geographic locations, it is 35 major Linux kernel releases (2.6.20 to 3.14) very likely that they are using the same kernel since KVM was first introduced. The testbed is a version and distribution. This confirms the Dell Desktop (with Intel Xeon 2.93GHz Quad- conventional wisdom that standardization and Core CPU and 2GB memory) running OpenSuSE automation are critical to the maintainability of an 11.4. We used OpenSuSE 11.4 as the guest OS IT environment as it grows more complex. Modern running a 3.14 Linux kernel. We manually cloud providers’ data centers are as complicated as compiled each of the 35 kernels and deployed it as they can get. the kernel used in our hypervisor. After each set of Cloud providers usually do not use the latest experiments, we shut down the guest OS and kernel. At the time of our study, the latest stable rebooted the host OS. Linux kernel is version 3.14, which was released in The results are listed in Table 9. To sum up, from March 2014, and our experiments were performed Table 9, among the 35 host OS kernel versions, we in June 2014. can find an exact match for 11 of them; for 15 of Cloud Evolution them, we can narrow down to 2 versions; for 4 of them, we can narrow down to 3 versions; for 4 of After 18 Months, i.e., in January 2016, we them, we can narrow down to 4 versions; and for 1 performed a new set of measurements in three of them, we can narrow down to 5 versions. cloud environments: Google Cloud, Elastic

Hosts Cloud, and VULTR Cloud [19]. The pur- pose of this additional measurements is to 6 DISCUSSION verify whether cloud vendors do upgrade their In this section, we discuss some potential hypervisor kernels. enhancements. For Elastic Hosts, we launched one virtual Other Hypervisors machine located in their San Jose Data Center, Our current framework is developed for KVM, but the virtual machine is configured with a the approach we propose should certainly work 2000MHZ virtual CPU, 1024MB memory, and for other popular hypervisors such as Xen. In fact, is installed with a Debian 8.2 Linux operating we notice that KVM and Xen share many of the system. Surprisingly, we found that Elastic same features and bugs. For instance, They both Hosts are still using the kernel version between support the Microsoft enlightenment feature, and 3.6 to 3.8, which is the same as what we also notice that some MSR register bugs exist Hyperprobe reported 18 months ago. One in both KVM and Xen. Therefore, we plan to possible reason is that we were using free trial include the support for Xen hypervisors in our virtual machines in both our measurements in framework. 2014 and 2016, and these virtual machines are Meanwhile, we are also trying to enhance our located at some specific servers that the cloud framework for closed-source hypervisors, such as providers do not perform regular maintenance. VMware and Hyper-V. Even though their source codes are not available, the vendors provide a For Google Cloud and VULTR Cloud, Hyperprobe release note for each major release, which clearly re- ports that their kernel versions are equal to or state their new features. And the bugs of these newer than 3.14, which is the upper limit of hypervisors are also publicly available. Hyperprobe. Thus, with the current implementation of Hyperprobe, we are just able to http://indusedu.org Page 249 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

More Features are time consuming and incur significant overhead. We describe two more interesting kernel features To address this problem, the latest Intel processors, for which we had planned to create test cases, but such as the Intel Xeon E52600 v2 product family, have not yet done so due to the unavailability of provide a new feature called APIC virtualization or certain processors in our lab. APICv. With APIC virtualization, most interrupt related activities do not need to trap into the Pause Loop Exiting hypervisor. Thus, overhead is reduced and I/O Lock holder preemption is a problem that throughput is increased. commonly exists in a virtualized environment Since Linux kernel 3.9, APIC virtualization is [44], [45]. This problem happens when a supported in KVM. In its implementation, virtual CPU holding a lock in a virtual machine accessing most APIC registers does not trap, but is scheduled out by the hypervisor scheduler. there are a few exceptions. For example, the local This can happen because the hypervisor has no APIC ID register, the Task Priority register, and the semantic knowledge of guest locks. The Current Counter for Timer register will still trap. consequence is that when the other virtual Therefore, accessing these APIC registers will CPUs are scheduled in, they have to wait until incur much more time than accessing the other the lock holder is scheduled in and releases the APIC registers. Based on this timing difference, we lock. This is a waste of time. can detect the APIC virtualization feature. And a To address this problem, recent generations of successful detection of this feature suggests that x86 processors present a new feature (Intel the underlying hypervisor kernel should be newer Pause Loop Exiting and AMD Pause Filter) than or equal to 3.9. aimed at detecting virtual CPU busy waiting. When this virtual CPU is detected, a trap to the Open Source host occurs (this is called Pause Loop Exiting, We have implemented Hyperprobe as a because spinlock are usually implemented with framework, which includes different test cases, but a loop of the pause instructions), and the each test case is totally separated from all the other hypervisor can then make a better scheduling test cases. In other words, each test case can be decision. Since Linux kernel 2.6.33, Pause developed separately. Such a key property allows it Loop Exiting (PLE) is supported in KVM. to meet one of our design goals: extensibility. In When PLE happens, the hypervisor would let a fact, we have made it open source on github3, and waiting vCPU sleep for 100 microseconds we hope we can rely on a community of users to hoping the lock holding vCPU is scheduled in use it and contribute additional test cases. The to release the lock. This, to some extent, more vantage points (i.e., test cases) we have, the resolves the lock holder preemption problem. higher precision our detection can achieve. And To detect PLE, we can simply run a lot of pause this will certainly accelerate our development instructions, and see if it traps. Because of the process and our support for the other hypervisors. 100 microseconds sleeping, the execution time In addition, developing multiple test cases for one of those pause instructions should be far longer kernel version would also reduce the possibility of when PLE is enabled than when it is not. false positives. And a successful detection of PLE suggests that the underlying hypervisor kernel should be 7 RELATED WORK newer than or equal to 2.6.33. We survey related work in three categories: This PLE feature is only available in recent Intel detection of a specific hypervisor, attacks against processors: Intel Nehalem-ex and Intel Westmere, hypervisors, and operating system fingerprinting. but not in the processors we are using, thus, we

were not able to develop a test case for this Detection of Hypervisors feature. Since virtualization has been widely used for APIC virtualization deploying defensive solutions, it is critical for attackers to be able to detect virtualization, i.e., APIC virtualization is yet another feature that detect the existence of a hypervisor. To this end, exists in recent generations of x86 processors. several approaches have been proposed Traditionally, guest in- terrupts are virtualized by

the hypervisors, but this requires a lot of exits from the guest to the hypervisor and then re-entries from 1. https://github.com/jidongxiao/hyperprobe the hypervisor to the guest. These context switches http://indusedu.org Page 250 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

vulnerabilities. Ormandy [53] classified the for detecting the underlying hypervisors and are security threats against hypervisors into three briefly described as follows. categories: total compromise, partial RedPill [46] and Scooby Doo [47] are two compromise, and abnormal termination. A total techniques proposed to detect VMware, and compromise means a privilege escalation they both work because VMware relocates attack from a guest OS to the hypervisor/host. some sensitive data structures such as Interrupt A partial compromise refers to information Descriptor Table (IDT), Global Descriptor leakage. An abnormal termination denotes the Table (GDT), and Local Descriptor Table shut down of a hypervisor caused by attackers. (LDT). Therefore, one can examine the value of According to the definition above, gaining the IDT base, if it exceeds a certain value or hypervisor information by Hyperprobe belongs equals a specific hard-coded value, then one to a partial compromise. assumes that VMware is being used. However, these two techniques are both limited to Operating System Fingerprinting VMware detection and are not reliable on machines with multi-cores [48]. By contrast, Operating system fingerprinting is crucial for the detection technique proposed in [48] is both attackers and defenders. Prior research in more reliable but only works on Windows guest this area can be divided into three The first OSes. Their key observation is that because category is network based fingerprinting, a LDT is not used by Windows, the LDT base popular technique mainly used by attackers. In would be zero in a conventional Windows particular, tools like Nmap [54], [55] and system but non- zero in a virtual machine Xprobe [56] have been widely used and environment. Therefore, one can simply check extensively studied. These tools work by for a non-zero LDT base on Windows and examin- ing the TCP/IP traffic patterns and determine if it is running in VMware matching them against a database of known environment. results. The second category is virtualization A variety of detection techniques based on based fingerprinting [57], [58]. The key idea of timing anal- ysis have also been proposed in OS- Sommelier [57] is, in a cloud environment, [10], [49].The basic idea is that some when the guest OS memory is present, system instructions (e.g., RDMSR) are intercepted by administrators can compute a hash for the hypervisors and hence their execution time is kernel code of each guest OS; as different guest longer than that on a real machine. One can OSes should produce a different hash value, detect the existence of a hypervisor by system administrators can differentiate each measuring the time taken to execute these guest OS, achieving the goal of guest OS instructions. Note that all these previous works fingerprinting. In [58], the authors observed can only detect the presence of a hypervisor that, in a virtualized environment where and/or its type, but none are able to retrieve memory deduplication works at the hypervisor more detailed information about the underlying level, the memory deduplication mechanism hypervisor, such as its kernel version. usually causes accumulated access delay for the deduplicated memory pages. Therefore, one Attacks against Hypervisors can load different OS images into its own memory; if there is another virtual machine Modern hypervisors often have a large code running the same OS co- resident with the base, and thus, are also prone to bugs and attacker’s virtual machine, the identical pages vulnerabilities. Considering a hypervisor’s will be deduplicated, and by measuring the critical role in virtualized environments, it has access delay, one can detect whether or not that been a particularly attractive target for specific operating system is running in co- attackers. Vulnera- bilities in hypervisors have resident virtual machines. been exploited by attackers, as demonstrated in The third category is USB based [59], [60]. In a prior work [8], [9]. Perez-Botero et al. [50] represen- tative work of this type, Bates et al. [59] characterized various hypervisor vulnerabilities proposed to use USB devices to identify different by analyzing vulnerability databases, host systems. The main premise of this work is that including SecurityFocus [51] and NIST’s there is a timing variation between different Vulnerability Database [52]. Their observation operating systems when communicating with a is that specific USB device. Leveraging this time almost every part of a hypervisor could have variation and some machine learning techniques, http://indusedu.org Page 251 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

system administrators can determine the identity of the 14th ACM conference on Computer each host system. and Communications Security (CCS), Compared with all these OS fingerprinting 2007, pp. 103–115. techniques, Hyperprobe differs in two aspects. [5] L. Litty, H. A. Lagar-Cavilla, and D. Lie, First, it has a different threat model. Hyperprobe ―Hypervisor support for identifying works inside a virtual machine, and attempts to covertly executing binaries.‖ in USENIX Security Symposium, 2008, pp. 243–258. retrieve the information of the underlying [6] K. Kortchinsky, ―Cloudburst: A vmware hypervisor, specifically its kernel version. Second, guest to host escape story,‖ Black Hat it em- ploys a very different approach. In USA, 2009. particular, our implementation mainly relies on [7] N. Elhage, ―Virtunoid: Breaking out of the knowledge of the evolution of KVM. As far as kvm,‖ Black Hat USA, 2011. we know, we are the first to systematically examine [8] T. Garfinkel, K. Adams, A. Warfield, and the KVM patches over the past seven years and J. Franklin, ―Compati- bility is not study the evolution of KVM development. transparency: Vmm detection myths and realities.‖ in Proceedings of the 9th USENIX workshop on Hot topics in 8 CONCLUSION operating systems (HotOS), 2007. In this paper, we investigated the reverse [9] P. Ferrie, ―Attacks on more virtual information retrieval problem in a virtualized machine emulators,‖ Symantec environment. More specifically, we coined the Technology Exchange, 2007. [10] J. Franklin, M. Luk, J. M. McCune, A. term virtual machine extrospection (VME) to Seshadri, A. Perrig, and describe the procedure of retrieving the hyper- L. Van Doorn, ―Remote detection of visor information from within a guest OS. As a virtual machine monitors with fuzzy first step towards VME, we presented the design benchmarking,‖ ACM SIGOPS and development of the Hyperprobe framework. Operating Systems Review, vol. 42, no. 3, After analyzing the seven-year evolution of Linux pp. 83–92, 2008. KVM development, including 35 kernel versions [11] J. Xiao, Z. Xu, H. Huang, and H. Wang, and approximately 3485 KVM related patches, we ―Security implications of memory implemented test cases based on 19 hypervisor deduplication in a virtualized features and 20 bugs. Hyperprobe is able to detect environment,‖ in Pro- ceedings of the the underlying hypervisor kernel version in less 43rd Annual IEEE/IFIP International than five minutes with a high accuracy. To the best Conference on Dependable Systems and of our knowledge, we are the first to study the Networks (DSN), 2013. problem of detecting host OS kernel version from [12] ―Google compute engine,‖ within a VM. Our framework generates promising https://cloud.google.com/products/ results in six real clouds, as well as in our own compute-engine/. [13] ―Hp helion public cloud,‖ testbed. http://www.hpcloud.com/. REFERENCES [14] ―Elastichosts,‖ http://www.elastichosts.com/. [1] M. Vrable, J. Ma, J. Chen, D. Moore, E. [15] ―Joyent,‖ http://www.joyent.com/. Vandekieft, A. C. Snoeren, [16] ―Cloudsigma,‖ G. M. Voelker, and S. Savage, https://www.cloudsigma.com/. ―Scalability, fidelity, and containment in [17] ―Vultr cloud,‖ https://www.vultr.com/. the potemkin virtual honeyfarm,‖ ACM [18] K. Adams and O. Agesen, ―A SIGOPS Operating Systems Review, vol. comparison of software and hardware 39, no. 5, pp. 148–162, 2005. techniques for x86 virtualization,‖ Proceedings of the 11th international [2] X. Jiang and D. Xu, ―Collapsar: A vm- conference on Architectural Support for based architecture for network attack Programming Languages and Operating detention center.‖ in USENIX Security Systems (ASPLOS), vol. 41, no. 11, pp. Symposium, 2004, pp. 15–28. 2–13, 2006. [3] R. Riley, X. Jiang, and D. Xu, ―Guest- [19] M. Ben-Yehuda, M. D. Day, Z. Dubitzky, transparent prevention of kernel rootkits M. Factor, N. Har’El, with vmm-based memory shadowing,‖ in A. Gordon, A. Liguori, O. Wasserman, Recent Advances in Intrusion Detection and B.-A. Yassour, ―The tur- tles project: (RAID). Springer, 2008, pp. 1–20. Design and implementation of nested [4] N. L. Petroni Jr and M. Hicks, virtualization.‖ in Proceedings of the 9th ―Automated detection of persistent kernel USENIX conference on Operating control-flow attacks,‖ in Proceedings of http://indusedu.org Page 252 This work is licensed under a Creative Commons Attribution 4.0 International License International Journal of Research in Engineering, IT and Social Science, ISSN 2250-0588, Impact Factor: 6.565, Volume 09, Special Issue 4, May 2019, Page 239-253

Systems Design and Implementation ment, software testing and anomaly detection, (OSDI), vol. 10, 2010, pp. 423–436. and energy and power management.

[20] A. Arcangeli, I. Eidus, and C. Wright, ―Increasing memory density by using ksm,‖ in Proceedings of the Linux Haining Wang received his Ph.D. in Computer Symposium, 2009, pp. 19–28. Science and Engineering from the University of [21] C. Waldspurger, ―Memory resource Michigan at Ann Arbor in 2003. He is a Pro- management in vmware esx server,‖ fessor of Electrical and Computer Engineering Proceedings of the 5th symposium on at the University of Delaware, Newark, DE. His Operating Systems Design and research interests lie in the areas of security, Implementation (OSDI), vol. 36, no. SI, networking system, and cloud computing. pp. 181–194, 2002. [22] K. Suzaki, K. Iijima, T. Yagi, and C. Artho, ―Software side channel attack on memory deduplication,‖ Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP 11’ POSTER), 2011. [23] J. Ahn, S. Jin, and J. Huh, ―Revisiting hardware-assisted page walks for virtualized systems,‖ in Proceedings of the 39th Interna- tional Symposium on Computer Architecture (ISCA). IEEE Computer Society, 2012, pp. 476–487. [24] N. Bhatia, ―Performance evaluation of intel ept hardware assist,‖ VMware, Inc, 2009.

[25] N. Smyth, Hyper-V 2008 R2 Essentials. eBookFrenzy, 2010. [26] M. Russinovich, ―Inside windows server 2008 kernel changes,‖ Microsoft TechNet Magazine, 2008.

[27] ―Hypervisor top-level functional specification 2.0a: Windows server 2008 r2,‖ http://www.microsoft.com/en- us/download/ details.aspx?id=18673.

Jidong Xiao received his PhD degree in Com- puter Science from the College of William and Mary, Williamsburg, in 2016. He is an assistant professor at Boise State University. His research focuses on cybersecurity, with a particular em- phasis on operating system security and cloud security. Lei Lu received his PhD degree in computer science from the College of William and Mary, Vir- ginia, in 2014. He is currently a senior member of technical staff at VMWare Inc., Palo Alto. Hai Huang received his PhD degree in Com- puter Science and Engineering from University of Michigan, Ann Arbor in 2006. He is a Re- search Staff at IBM T. J. Watson Research Cen- ter. His research interests include cloud computing, file and storage systems, systems manage-

http://indusedu.org Page 253 This work is licensed under a Creative Commons Attribution 4.0 International License