Security in Live Virtual Machine Migration

SECURITY IN LIVE VIRTUAL MACHINE MIGRATION

A Thesis by

Shah Payal Hemchand

B.E., Government College of Engineering, North Maharashtra University, Jalgaon, 2008

Submitted to Department of Electrical and Computer Engineering and faculty of the Graduate School of Wichita State University in partial fulfillment of the requirements for the degree of Master of Science

December 2011

SECURITY IN LIVE VIRTUAL MACHINE MIGRATION

The following faculty members have examined the final copy of this thesis for form and content, and recommend that it be accepted in partial fulfillment of the requirement for the degree of Master of Science with a major in Computer Networking.

______Ravi Pendse, Committee Chair

______Abu Asaduzzaman, Committee Member

______Linda Kliment, Committee Member

iii

DEDICATION

To the Almighty, my family, for their continuing support and patience; to my WSU friends for their significant advice and time throughout the completion of my thesis.

ACKNOWLEDGEMENTS

I sincerely thank my thesis advisor, Dr. Ravindra Pendse for his devoted motivation and supervision throughout my career at Wichita State University. His guidance helped me complete my thesis successfully. By working as a Graduate Research Assistant for him I gained knowledge, and professional work ethics.

I take this opportunity to thank Amarnath Jasti for his constant support and guidance throughout my thesis. His suggestion and advice helped me understand the technology and gain more knowledge. His opinion towards my academic and career were valuable. I would like to thank members of the committee for their effort and time.

I would like to extend my gratitude towards to Yonatan Assefa and all those who directly or indirectly helped motivate me with my research.

ABSTRACT

Virtualization has become an essential technology for organizations. With on-demand services from vendors, a substantial rise in the use of virtualization has been noticed. Today there are various kinds of virtualization techniques offering different advantages. One of the important features of virtualization is live virtual machine (VM) migration. In live VM migration, the controls of a VM are migrated from one physical host to another. Workload balancing, and server maintenance becomes easy by migrating the VM. The ability to reboot or shut down the physical server without affecting running applications is greatly beneficial to an organization.

With this indispensable feature of live VM migration, the security factor is still unanswered. Very little research has been done in exploring the security concerns inherent while data moves between the two physical machines. This thesis looks at this poorly explored area and attempts to propose a proper solution, and thereby maintain security. Man–in–the–middle attack could be created by sniffing data between the hypervisors and confidentiality is lost. Data in transit could be read and then tampered with or misused, and can create havoc in the network and bring it down completely.

The research shows how a malicious attacker can sniff the data while performing live

VM migration over Xen hypervisor and exploit information. Using this experiment the author proposes strategies that can be used to have a secure live migration process.

TABLE OF CONTENTS

Chapter Page

1. INTRODUCTION 1

1.1 Overview 1 1.2 Storage Area Network 2 1.3 Security 4 1.4 Problem Identification 6 1.5 Organization of Thesis 6

2. VIRTUALIZATION 7

2.1 Definition 7 2.2 Origin of Virtualization 7 2.3 Need of Virtualization 8 2.4 Types of Virtualization 9

3. LITERATURE SURVEY 13

3.1 Security in Virtualization 13 3.2 Related Work 15

4. XEN OVERVIEW 19

4.1 Xen Hypervisor 19

4.1.1 Why Xen? 20 4.1.2 Other Features of Xen 20 4.1.3 Working of Xen 21

4.2 Requirements for Live VM Migration 22

vii

TABLE OF CONTENTS (cont…)

Chapter Page 5. TEST BED AND RESULTS 24

5.1 Hardware 24 5.2 Software 5.2.1openSUSE 24 5.3 Test Bed 24 5.4 Results 26 5.5 Proposed Solution 29 5.5.1 IPSec Tunnel 29 5.5.2 Hypervisor Encryption 31

6. FUTURE WORK AND CONCLUSION 32

6.1 Future Work 32 6.2 Conclusion 32

REFERENCES 34

viii

LIST OF FIGURES

Figures Page

1 Basic SAN Environment 3

2 Key elements of Data Center 4

3 OS Virtualization 10

4 Paravirtualization 11

5 Full Virtualization 12

6 Test Bed 25

7 Three-way handshake between host A and host B 26

8 Three-way handshake capture 27

9 Wireshark capture showing ascii values 28

10 Live VM migration through IPSec tunnel 30

11 Encrypted data after VM migration through IPSec tunnel 30

LIST OF ABBREVIATIONS

AoE ATA over Ethernet

DoS Denial – of – Service

IDS Intrusion Detection System

IPS Intrusion Prevention System iSCSI Internet Small Computer System Interface

NFS Network File System

OS Operating System

SAN Storage Area Network

VM Virtual Machine

VMM Virtual Machine Monitor

Chapter 1 Introduction 1.1 Overview In an IT environment, aggregation of computing technology and storage resources enables an enterprise to reduce the operational cost, and improve efficiency and flexibility in the hardware utilization. Organizations have huge servers, costly data centers, consume a lot of power, require and space and generates large amount of heat. In earlier times, it was cumbersome to maintain the larger mainframe computers. However all the work was done by the mainframe and workstations were the dumb terminals [1]. Organizations had to invest a considerable amount of time and money into the mainframe technology to keep up with tasks. Mainframe

Computers were quite expensive, occupied more space for hardware, and required more human attention and more resource consumption. Organizations required more hardware for different projects, and to manage more data and software; huge data centers were required. To meet business requirements, exponentially growing data, organizations sought effective and maximal storage capabilities. All this led to the rise of virtualization and Storage Area Network (SAN) [2] to gain higher network efficiency. Virtualization came into play to consolidate the individual servers and today more and more servers are opting for SAN storage, thereby making the local hard drives insignificant.

With the advent of virtualization [3], hardware is consolidated in data centers to a single modern server that runs several virtual servers, thereby maintaining high performance and efficiency. Less hardware utilization leads to less consumption of electricity, less space used and reduced number of electrical components; leading to Green Computing [4]. Virtualization has gained extensive attention in the growing world of technology. Hardware miniaturization and the capability of installing more software require an organization to use multi-tenant services. Due to 1 rapid growth and popularity, virtualization is said to be, “The Fundamental block for today‟s technological world”. Virtualization is further explained in detail in Chapter 2.

1.2 Storage Area Network (SAN)

Every small organization has a strategy to build a central storage so servers, new software or larger applications which need more memory are constantly developed. Storing data on a local hard disk is becoming archaic. So the idea of having a central storage that constitutes a SAN could be as simple as just plugging in the terminal in the mainframe world.

SAN is a collection of hubs, fabric, and software. It is a high-speed network of storage elements, such as shared storage arrays, and clusters servers, where one or more interfaces create complete connectivity. A basic SAN environment is as shown in Figure 1. In a typical SAN environment, uninterrupted availability of data is very important for the survival of an organization. It is essential to have a reliable infrastructure to ensure that data is available at all times. The key characteristics of a typical data center are capacity, security; availability, performance, data integrity and scalability [5]. If all are met appropriately then an organization can have unremitting services, as shown in Figure 2. Following is the brief description of key characteristics of a data center:

Figure 1 Basic SAN Environment a) Capacity: As per the growing needs and user demands, capacity requirements should

be increased to provide uninterrupted services. Data center operations must have

adequate resources to store a large amount of data. If more resources are required,

then the data should be reallocated rather than adding new resources. b) Security: Proper integration and authentication should be maintained in the network

to ensure enough security. Clients should have complete access to allocated resources

on the specific storage arrays. c) Availability: In an organization, data should be available all the time without any

inconvenience.

Figure 2 Key elements of Data Center

d) Performance: The system should have the ability to provide optimal performance by

offering services at high speed.

e) Data Integrity: Data integrity should be maintained by mechanisms such as error

detection and correction.

f) Scalability: The storage should be able to grow with user demands. The system

should be able to allocate additional processing capabilities for storage on demand

without interrupting the other services.

1.3 Security

With the upcoming needs of organizations, more attention is required to provide security in a network [6]. Storage and virtualization are the evolving technologies, and so are the threats

4 and vulnerabilities. Features in such technologies can lead to unwanted and unexpected security threats. Often technology is implemented without being aware of the security consequences.

Eavesdropping, data theft, and hacking are the basic security concerns within an organization and over the Internet. Security could be broken accidentally or intentionally. If the critical infrastructure is compromised, the consequences would be severe and lead to data loss. SAN is considered wealthy since it has valuable information and sensitive data. If SAN fabric is configured using conventional methods without any security controls, then compromising the network not be difficult.

Organizations should think carefully about their data storage needs for a secure storage and an integrated solution for threats and vulnerabilities. Security should be implemented layer- by-layer, so that if one layer is compromised, the assets of the other layers are under protection.

Proper security measures, tools, and encryption techniques should be implemented to have a secure organization. The technology should be understood to the core and eventually security should be maintained by taking preventive measures across the network. A virtual environment is more susceptible to security threats. Possible ways to break into virtualization are to compromise a Virtual Machine (VM), hack the hypervisor, insert a rootkit, and sniffing the traffic. All these possible ways need attention in an organization to have fail-proof security.

With virtualization, live VM migration [7] has gained more popularity. Live VM migration attracted the users because of its transparent nature, and clean separation between hardware and software. Administrators can consolidate the system load, improve power efficiency, and improve infrastructure maintenance and flexible relocation of the sources [8].

Security is a bigger concern in Live VM migration as it poses threats to VM as well as network.

Data is in flight as well as at rest, which could be sniffed and security can collapse.

1.4 Problem Identification

An indispensable feature in the world of virtualization is live migration of a Virtual

Machine (VM). Live VM migration means migration of an entire OS and its associated applications from one physical machine to another. An administrator can ensure complete utilization of the available resources on any one machine or another. But with some added advantage comes a threat. Security is the biggest concern while doing migration and has not received its due attention. The VMs are migrated “live”, without disrupting the applications and the network with the least minimal downtime. During the migration, an attacker in the middle can gain control over the VM or capture the traffic and try to recover the data. The traffic between the hypervisors can be sniffed and is more vulnerable to man-in-the-middle attacks or

Denial-of-Service (DoS). Confidentiality, data integrity, and essential credentials are lost easily.

Similarly, this could cause killing of the available resources for that particular VM. After losing vital data, the host machines might have to shut the VM‟s to protect themselvess from an attack.

1.5 Organization of Thesis

This thesis brings a new vision and direction to less focused areas of securing the live

VM migration. Chapter 1 provides a brief overview of security in SAN and virtualization.

Chapter 2 explains virtualization, needs and types of virtualization in depth. Chapter 3 provides a literature survey of how VM is migrated with minimal downtime and securing live VM migration. Chapter 4 gives a detailed explanation how live VM migration could cause a big security threat and about Xen hypervisor. Chapter 5 consists of the experiment and the results.

Chapter 6 concludes the thesis and recommendations are made for future work for securing live

VM migration.

Chapter 2

Virtualization

2.1 Definition Virtualization is the emulation of hardware in the software platform. Virtualization in simple words is defined as creating a Virtual Machine (VM) or a virtual copy of a device or a resource, such as a storage device, network or an Operating System (OS). The VM fetches resources from the physical resources (such as servers, and OS) and makes them look like a multiple logical resource and vice-versa. Each application needs one server, which increases cost. Virtualization offers the ability to run multiple applications and OS on the same machines.

Thus a small number of servers in the data center can perform better and accomplish more work than before

2.2 Origin of Virtualization

Christopher Strachey, the first Professor of Computation at Oxford University and leader of the programming research group, brought the term “Virtualization” to life, in his paper „Time

Sharing in Large Fast Computers‟. The term Virtual Machine Monitor (VMM) came into being in 1960 [9] as a software abstraction layer partitioning a network platform into one or more

VMs. Bob Muglia, Senior Vice-President for server and tools business at Microsoft Cooperation says “Virtualization is an approach to deploying computer resource that isolates different layers

– hardware, software, data, network and storage from each other” [10]. IBM wanted to logically partition the mainframe computers into VM [18, 19 and 20]. IBM pioneered virtualization in

1970 [9, 11 and 12], but it was eventually perfected by others. Hardware level VMs disappeared during the 1980‟s and 1990‟s [13, 14]. The dilemma of utilizing efficient hardware was faced by the IT departments where eventually virtualization was applied to x86 architecture in order to

7 improve the issues such as high maintenance, infrastructure costs, management costs and disaster protection. In 1990, Java VM [15] was developed apart from IBM‟s mainframe computers.

VMware Workstations were first utilized in 1999 [16] which allowed running virtual machines.

VMM became popular in 2005 and 2006 in academia and industry.

In 2005-2006, Xen researchers introduced the term hypervisor [17]. Since then Xen became the de facto virtualization architecture as open source virtualization. Intel, AMD, Sun

Microsystems and IBM are building different virtualization strategies which would give the technical world a new plethora.

2.3 Need for Virtualization

The demand for virtualization is growing because of the following advantages [21]:

 Underutilized hardware: Many data centers have machines that utilize only 10-15% of

total processing capacity and the remaining 85-90% is unused.

 Smaller footprint: Over the last two decades, technology has made a big jump and has

reached a platform where everything is a click away. The rise of internet accelerated this

transformation. Real-time communication and video-conferences are possible because of

the growing internet. This has led to the need for huge servers causing a real estate

problem for organizations. Space is required for such data centers. This is one of the

important reasons to adapt virtualization. Virtualization helps in reducing the cost of

building more data center space.

 More system administration: Due to more hardware, every machine needs constant

monitoring, regular updates, OS, application and software installations. This increases the

amount of labor. With the ease of virtualization, the system administration‟s job is

reduced.

 Ease of testing and development: Testing becomes easy with the deployment of

different OS environments and isolating the applications per VM.

 Mobility: Virtual Machines can be migrated from one physical host to another, giving

better system mobility. This also improves the resource utilization.

2.4 Types of Virtualization:

There are different types of virtualization techniques as per the application or services.

Following are the types of virtualization [22]:

2.4.1 Server Virtualization

In Sever Virtualization, the resources of servers are hidden from its users. The users don‟t need to understand the management of resources. Server Virtualization is done to increase the utilization and sharing of resources. There are different ways to do Server Virtualization as:

2.4.1.a Operating System (OS) Virtualization

OS virtualization means running an OS on top of an existing one. The main reason for

OS virtualization is to provide the set of libraries that the application might interact with. The application in hosted OS can‟t see any other applications in another virtual OS. OS virtualization is offered by Sun and SW Soft [23] with the commercial product Virtuozzo [24]. OpenVZ [25] is another open source OS virtualization. A typical OS Virtualization is shown in Figure 3.

Guest Operating Guest Operating Guest Operating System System System

Host Operating System

Physical Hardware

Figure 3 OS Virtualization

2.4.1.b Hardware Emulation

A hardware emulated environment (usually hypervisor) is presented to the guest OS in the hosted environment. This is also referred to as the Virtual Machine Monitor (VMM). The guest OS and the VMM form a complete consistent package and that can be completely migrated from one physical host to another. A hypervisor can virtualize the physical host resources such as

CPU and memory, creating a virtual environment. The VMs running in a virtual environment have an illusion of being in non-virtualized ones. A hypervisor is responsible for allocating resources to each VM on demand.

2.4.1.c Paravirtualization

The guest OS is modified to run in paravirtualization. It is a subset of Server

Virtualization; it has a thin software interface between the hardware and the VM, not identical to that of the underlying hardware. This is shown in Figure 4. The time spent in performing the operations is reduced because of the modified interface. The guest VMs are aware that they are

10 running in a virtualized environment. The virtual devices rely on physical device drivers of the underlying host. The device interaction is similar to that of full virtualization [26].

User Application User Application User Application

Guest Operating Guest Operating Guest Operating System System 1 System 2 Dom 0 Dom U Dom U

Dom 0 Interface Hypervisor Hypervisor

Physical Hardware

Figure 4 Paravirtualization

The abstraction formed with paravirtualization means that OS performs much better than full virtualization. But the flexibility and security is lost. Since the OS needs to be modified, flexibility is compromised. The readily available OS can‟t be used for paravirtualization. The guest OS is close to the hardware and has more control. This creates risk of impacting the lower hardware level and compromising the entire network.

2.4.2 Storage Virtualization

The amount of data being sent and received is growing enormously. With so many VMs, the space required to store data is decreasing daily. Hence, there arises Storage Virtualization. If data is stored on one machine, a bottleneck will be created if the single machine stops operating.

Organizations use specialized hardware and software along with array controllers, and disk drivers to provide reliable storage for data processing. The storage is virtually centralized and could be delivered over a Fiber channel, iSCSI, NFS or other storage protocols.

2.4.3 Full Virtualization

The guest OS remains unmodified in Full Virtualization. It provides a virtual environment, namely a complete simulation of the underlying hardware as depicted in Figure 5.

Every feature of the hardware, such as interrupts, I/O operations, memory access, and full instruction set which runs on the bare machine, should be reflected on the VM. The guest OS is not aware that it is running in virtual world.

User Application User Application User Application

Guest Operating Guest Operating Guest Operating System System System

Hypervisor

Physical Hardware

Figure 5 Full Virtualization

Full Virtualization helps isolate the users, and emulating the hardware to achieve security and reliability, improve the efficiency, simplifies migration and portability. The hypervisor [27] is used to allow the I/O devices to go to the guest OS machines by imitating the physical devices in the VMM. It does not need any actual hardware to support virtualization which adds the overhead performance [28]. Full virtualization is slower than any other virtualization techniques.

Chapter 3

Literature Survey

Senior Director of Product Management for VMware, Patrick Lin says, “Virtualization plays a role as an opportunity and as well as a threat” [29]. Security is the most important aspect every organization would strive for. A detailed discussion on different security threats and measures are discussed here.

3.1 Security in Virtualization

Organizations have taken a risk with the technology and getting accustomed to compete in the world considering the aspects of security. It is difficult to maintain a secure environment, where frequent access to critical business applications is necessary. Going global requires many data centers, and innovative business practices. Security experts [30] know that detecting threats and information leaks is an order of magnitude more difficult than stopping malware. To protect against this kind of threat very few measures have been taken to secure the crucial and sensitive data.

As per Citrix whitepaper [31], workforce is mobile, and distributed for people working remotely from home, offices or mobile devices having internet access. Security becomes necessary and even more elusive because it brings the associated vulnerability. Protecting the cooperate resources and assets is not easy. A single security breach can cost millions of dollars, thereby compromising the organizations sensitive information. With iron clad security and safeguards in place, a failed system should not bring an entire organization to its knees. A proper strategy should be followed. Data should reside on servers and within data centers and not on the

13 client devices [32]. If only limited access is available over the network, data is secure even if a laptop or a computer is compromised.

Virtualization is causing a transformation in the traditional way of resource utilizations, speeding IT response to a new, well-structured business environment [33]. Any computer is prone to attacks and threats. However, the virtualized computer is more likely to be exposed to attacks since security is quite weak due to more holes to patch, and too many interconnections.

Common defensive steps to limit user access are in place, such as the Intrusion Detection System

(IDS)/ Intrusion Prevention System (IPS), and firewalls. The attacks possible in virtualization are compromising the hypervisor or the kernel, a host module attack, VM escape, VM hopping, VM monitoring from host, Hyperjacking, and VM sprawl [34], etc. Attackers and hackers [35, 36] have an eye on one weak entry and can insert malware to detect a virtual environment and change itself accordingly. One of the major attacks is the man-in-the-middle attack during live

VM migration. During live VM migration, data is in clear text, allowing man-in-the-middle attack on a VM‟s hypervisor [37]. Virtualization and live migration enable new features benefitting the end user. This combination gives rise to novel security threats. The hypervisor implementing live migration feature can possibly expose both the OS and the guest VM to attacks, resulting in integrity loss.

A VMM encapsulates the vivid and volatile states of a VM. When a VM is suspended, its state is mapped to files in the local system of the host. Transition of a VM from physical host to another is called live migration. Live migration provides workload balancing, and resource sharing. Following are the advantages of live VM migration [38, 39]:

- Maintenance and Upgrade: When a physical host needs to be restarted, all the VMs

running on that machine are migrated over to another identical physical host without

causing interruptions in the services.

- Load Balancing: Depending on availability of resources VMs can be migrated

depending on the workload.

- Power Efficiency: If a physical host has only a couple of VMs and the workload is

low, they could be migrated over to other devices. This would reduce the power and

resources used.

3.2 Related Work

To perform live VM migration, state of the art uses a pre-copy approach [40]. This technique transfers the memory pages first and then copies pages modified later Pre-copy is most common for live VM migration as well as for wide area migration [41]. VM migrations have been discovered as a tool providing mobility to users working at different times on different machines. Thus they set an example by transferring an OS instance from one machine to another.

This collective project [42] aims to optimize the links, and reduce service downtime. Other projects [43 and 44] have concentrated over long time span by pausing, stopping and then migrating the memory pages during live VM migration.

Process migration moves a process from one physical host to another. Zap [45] uses partial OS virtualization for migration of process domains (pods) with the help of a Linux kernel.

VMware added VMotion as an advantage for OS migration to their VirtualCenter management software [46]. Process migration had been another area of interest for some time and more research was done in the 1980‟s [47, 48, 49, 50, 51, 52, 53, 54, and 55]. However, process

15 migration didn‟t gain popularity because of the residual dependency limitation and mobility. A better explanation and survey on process migration technique has been done in [56].

In contrast to process migration, OS migration handles all the limitations of the process migration and still does the VM migration efficiently. The difference between OS migration and process migration is that OS migration overcomes the problem of residual dependencies with the help of a narrow interface between a virtualized OS and the hypervisor. The administrator need not be concerned with what is running within the VM; instead they can migrate the OS and its associated processes as one unit.

VM migration really means to migrate the control and memory of a VM from one physical host to another, without causing any service disruption. Extensive research has been done on how the memory pages are migrated over during live VM migration. In general, memory migration [57] can be described as:

a) Push phase: During migration, the source VM is not suspended but is running while a

few pages are pushed to the destination machine. To make sure there is consistency

between the memory pages, the pages that are modified during the migration must be

re-sent.

b) Stop-and-copy phase: The VM to be migrated is stopped. The VM is only started

after the memory pages are transferred across the destination host.

c) Pull phase: If the destination host happens to access a page which is not yet

transferred from the source host, those pages are faulted in (“pulled”) across the

network from the source VM.

The best example of memory migration is stop-and-copy which involves starting a new

VM only after stopping the original VM and transferring all the pages. The downtime [58] and total migration time are proportional to the amount of physical allocated memory of the VM, and this gets affected since the migration of memory pages takes quite a long time. To have the least amount of downtime, pure demand migration was adopted which uses a short stop-and-copy phase. The essential kernel data structures are copied to the destination host. The new VM is started after the complete migration of memory pages and the other pages are fetched across the network at first use. This leads to less downtime but increased total migration time. Performance degradation also occurs due to a considerable set of pages being faulted across.

The best option for VM migration is the pre-copy migration [59]. The pre-copy approach provides a balanced way of migration by combining the bounded iterative push phase along with a short stop – and – copy phase. The pre-copying of memory pages occurs in rounds hence the term “iterative”. The pages which are modified during the first round are copied during the next round. There will be a small set of pages for each VM which are updated often. Thess turn out to be poor pages for pre-copy migration. The Writable Working Set was designed to calculate the number of iterations for typical workloads.

Wide research has been carried out on the topic of making live VM migration efficient and with minimal downtime. The live VM migration feature came into existence quite some time ago. It was performed from one server to another server located in the same room or from one rack to another rack which has physical security, and has minimal chances for loss of data. As the demand grew for virtualization, live VM migration was performed in a LAN, from one data center to another data center located in different places. Now the question arises: “How secure is the live VM migration”? Due to wide spread locations, physical security is not possible, invites

17 data sniffing and poses a security threat. It is observed that during live VM migration, the data can be sniffed and compromised. Any third party can gain access to the network, sniff the ongoing traffic and visualize the data. Three different threats were explored during live VM migration [60]. The traffic going over the data plane is sensitive and is not secured. The transit path is open to any outside attacker to read the data during the transfer causing a man-in-the- middle attack. This area of security requires more research since it is open for any attack. Using the tool Xensploit, several ways to attack the live VM migration were evaluated. This includes control plane, data plane and migration module. This thesis looks at how security can be provided during live VM migration.

Chapter 4

Xen Overview

Xen provides an exceptional feature called live VM migration. To perform live VM migration certain requirements must be fulfilled. During the migration, an attacker in the middle can gain control over the VM or capture the traffic since it is easy to decode the sniffed traffic and recover the data. The traffic between the hypervisors could be intercepted and is more vulnerable to man-in-the-middle attack or Denial-of-Service (DoS). Confidentiality, data integrity, and essential credentials are lost easily. Similarly, this could cause killing of the available resources for that particular VM. After losing vital data, the host machines might have to shut the VMs preventing them from being compromised. Not much research has been done in securing the live VM migration. As mentioned in chapter 3, threats during live VM migration are discussed [60] but no research has been done on how the traffic goes between the hypervisors.

This thesis concentrates on how the traffic flows between the hypervisors. A small experiment was conducted to show that during live VM migration data goes in clear text and is vulnerable to attack. The following section gives an overview of how Xen works and supports live VM migration.

4.1 Xen Hypervisor Xen is open source industry software for virtualization. The virtual world has been gaining popularity with the ease of the Xen hypervisor. Xen offers an efficient, strong and secure feature set for virtualization. The OS supported by Xen are Windows, Linux, Solaris, so it is neutral. Being independent, Xen allows Domain 0 to be the unique VM and it has control over all the other VMs.

4.1.1 Why Xen? Xen is popular because it has features which dominate in the virtualization world and hence is in demand. Xen doesn‟t contain any device drivers [62]. The VMs installed are isolated from each other. They are not aware of any other VM running on the same physical OS.

4.1.2 Other Features of Xen

Xen is chosen to be implemented because of its following salient features:

- Privileged Access: Domain 0, also called Dom0 has access to communicate with the

hardware and other Guest VMs.

- Small Base Code: The Xen hypervisor layer is thin, and has a tiny code, restricting the

areas for attacks.

- OS Separation: The Xen hypervisor is separated from the physical hardware. There is no

way to attack the actual OS from the hypervisor.

Xen allows paravirtualization. Paravirtualization allows the guest OS to communicate with the hypervisor in order to improve the memory, CPU, I/O and other resources. Since a VM is aware of run in a virtual world, varieties of tasks are achieved in accordance with the hypervisor. The overall maximum performance is achieved since all the VMs are isolated from each other. The task given to each VM is processed during the normal operation without any OS overhead.

Xen version 4.0 was used for live VM migration. Xen 4.0 is the fastest and most secure virtualization software today [63]. The delivery of Xen 4.0 with state of the art of features and hardware support highlight the strength and commitment of the open source Xen.org community

[63].

4.1.3 Working of Xen

The Xen hypervisor is an abstraction layer of software located on the hardware below any

OS. The hypervisor plays an important role in the execution of a VM. CPU Scheduling, memory allocation, and disk sharing are all controlled by the hypervisor. Xen consists of two domains,

Domain 0 and Domain U [64]. Domain 0, known as Dom0, is modified Linux kernel. Dom0 is the only VM having access to the physical I/O resources. Dom0 also communicates with the other VM (Domain U, known as Dom U) running on the physical host. Every machine running

Xen should have Dom0 running before any other VM starts running. Dom0 has two drivers; the

Network Backend Driver and the Block Backend Driver. The Network Backend Driver processes all the requests coming from the Dom U. The Block Backend Driver checks with the local storage disk to process the read or write request of the other VMs.

Domain U paravirtualized guests do not have direct access to the OS and are often referred as having unprivileged access. All paravirtualized guests are known as DomU PV Guest.

The Dom U PV Guests are aware of having no access to the physical hardware and are running in a virtual world. Fully virtualized VMs running on Xen are known as Dom U HVM Guests. A

Domain U PV Guest consists of two drivers; PV Network Driver and PV Block Driver. PV

Network Driver is not located within a Domain U HVM Guest and instead requires another special daemon Qemu-dm which is initialized for each VM in Dom0. This supports access to the disk. Any request made by the Dom U PV Guests must communicate with the Xen hypervisor via Dom 0. There is a direct link between Dom0 and Dom U PV Guests known as an event channel running through the Xen hypervisor.

4.2 Requirements for Live VM Migration

To perform live VM migration, there are certain requirements for live VM migration. The following precautions should be taken for a successful VM migration:

- The two hosts should be configured with the same version of Xen

- Both hosts should be in the same layer 2 network and IP subnet.

- The disk image and the configuration file should be stored on the shared storage. Both

hosts should have access to it.

- The processor on which Xen is installed should be the same. It can cause problems for

migration if the hardware is different. Migration works for the Intel Xeon processor with

four cores to an Intel Core2Duo processor with two cores. If the hardware is entirely

different, migration is not supported.

- The Xen configuration for the Xen daemon is stored in the file “xend-conFiguresxp”. It

can be found at /etc.xen/xend-conFiguresxp. This file needs to be modified on both

hosts.

The xend-config file consists of the following lines:

(xend-relocation-server no) (xend-relocation-port 8002) (xend-relocation-address “ ”) (xend-relocation-hosts-allow “ ”)

The above lines need to be modified to

(xend-relocation-server yes) (xend-relocation-port 8002) (xend-relocation-address “ ”) (xend-relocation-hosts-allow “ ”)

The xend-relocation-server is set to“yes” so that the receiving host can accept the relocated guests. The second line specifies that xend should listen on the default port 8002. The third line specifies which IP address the Xen daemon should listen on for migration requests. If this field is left blank, then it allows listening on all addresses and interfaces. The last line limits the number of hosts to contact the server for migration. Once the above requirements are done, live VM migration should be done easily. The different shared storages available are iSCSI,

NFS, ATA over Ethernet (AoE) [65]. We have used NFS (Network File System) as the shared storage.

Chapter 5

Test Bed and Results

Goals and Limitations:

The research focuses on the following main goals:

 Tracing the session establishment details

 Running any application that leads to data flow in clear text

 Loss and misuse of essential credentials

5.1 Hardware

Two Dell Optiplex 780 (x86_64) servers were used for installing openSUSE OS. The processor used was Intel Core 2 quad cpu [email protected] consisting of 4GB Memory. A Cisco Layer 3

Switch 3550 was used for routing between the two host machines.

5.2 Software

5.2.1 openSUSE

To perform live VM migration, a Xen hypervisor was chosen. Xen was installed on openSUSE 11.3. Novell sponsors Linux based OS openSUSE which includes a command line interface and a graphical user interface (GUI). It offers various different GUI as such as KDE,

SC, and GNOME. It has the updated Linux kernel 2.6.34. Installation of openSUSE is simple and it is user friendly [64]. The setup for Xen is simple and creating VMs is fairly easy. [61]

5.3 Test Bed

The physical hosts A and B run Xen hypervisor. A virtual machine VM 1 was created on host A. The hosts A and B were connected to layer 3 switch in the same vlan 1 as shown in

Figure 6. A Dynamic Host Configuration Protocol (DHCP) was configured on the switch to get

24 the ip addresses automatically. Connectivity between the hosts and the VM was checked with a ping test. Traffic was sniffed using Wireshark which is a tool used to capture and analyze the traffic going through the network. Usually Ethernet network is supported on wireshark. To see the migration data, a Switch Port Analyzer (SPAN) session was configured on the switch. After the VM was createda a small script was written executing a small known pattern continuously

15,000 times. The pattern was written to a text file, so the script was doing read and write at the same time. A continuous ping from the host A to VM 1 was started.

Figure 6 Test bed

To start with the migration, host A sent a SYN (synchronize) packet request to initiate a connection with host B. Host B replied with an ACK (acknowledge) to host A‟s SYN packet agreeing, upon building a connection and sends its own SYN packet (both messages are combined to form a SYN + ACK). Host A then sends an ACK to host B‟s SYN thereby agreeing to establish the connection. Figure 7 shows the pictorial representation of the three way

25 handshake between host A and host B. To perform the capture, wireshark was turned on using a laptop connected to the switch in the same vlan. The script was executed and at the same time

VM1 was migrated with the command “xm migrate – l 172.16.0.3”. The migration was completed. The VM1 was now residing on host B and the script was still being executed on the host B. The script continued on the host B from the point it stopped on the host A. Even though the VM migrated, connectivity to the VM was not lost and the service was not disrupted. This is shown by the continuation of the script and no ping packets being lost.

Figure 7 Three-way handshake between host A and host B

5.4 Results

For testing purposes, only the script was running and no other application was running.

This proves that, during the live VM migration, the service will not be lost or it will be lost for a minimal time. In the wirsehark capture, the ascii value of the string being printed in the script was seen. It was easy for an attacker to decode the ascii value and see the data being transferred, hence data is not secure during live VM migration. If any other applications are running or any important text files containing confidential data are open, then that data is not secure. The below

Figure 8 show the wireshark capture where the three way handshake is established between the source and destination hosts.

Figure 8 Three-way handshake capture

All three packets (SYN, SYN+ACK, ACK) are clearly visible indicating that the connection has been established between the hosts A and B. Figure 9 shows capture of the ascii value for the respective string being printed in the script. The string being printed was a known data pattern of xyxyxyxyxyyxxxxxxyyyyyyyzzzzzzzzz. In the wireshark captures, the ascii value for the above pattern was seen. Hence it‟s quite easy for anybody to decrypt these values and recover the data being transmitted. The ascii value for x is 78, y is 79 and z is 7a which is clearly visible in the wireshark capture. The ascii value is highlighted in red in the wireshark capture.

Along with the data, we can also see the source port, and destination port being used for migration. The destination could also be compromised if too many requests are being sent on the same destination port from some other host and the migration can never complete.

Figure 9 Wireshark Capture showing the ascii values

The above test shows that live VM migration is not secure and is the biggest security threat in virtualization. The continuous execution of the string during live VM migration shows

28 that the string was not interrupted when the VM migrated. The wireshark capture shows the exact ascii value for the string being tested which makes it easy to sniff the data and create man- in-the-middle attack. This simple experiment proves that data is in clear text and is open for attacks.

5.5 Proposed Solution

Security measures to protect the data during live VM migration should be designed in such a way that data integrity, and confidentiality is maintained. There is always a cost to be paid in order to make a gain some extra benefit. Following are a few ways for a secure live VM which migration could be implemented without losing data.

5.5.1 IPSec tunnel

One way of securing secure live VM migration is through building an IPSec tunnel. If the live VM migration is done across the Internet Protocol Security (IPSec) tunnel, then the cost to be paid would be the downtime of the VM. Since the migration would be done through the tunnel, extra overhead and processing would be done and the downtime of the VM might increase. This would cause disruption in the service for a longer duration, but the data would be secure since it will be encrypted. IPSec is a protocol for securing the Internet Protocol (IP) traffic. Authentication and encryption of each IP packet in the communication session is done when passing through the tunnel. In [66], the author explains operation, authentication and encryption techniques in detail. IPSec tunnel could be used in protecting the data flow at server- to-server levels or from edge router-to-edge router. When live VM migration is performed through the IPSec tunnel, data would be encrypted and difficult to trace. Figure 10 shows that an

IPSec tunnel has been created. Live VM migration is done through the tunnel to ensure data is encrypted.

Figure 10 Live VM migration through IPSec tunnel

Figure 11 shows how data would be encrypted and consists of some random characters which are not easy to decrypt. The red highlighted portion shows that data is encrypted.

Figure 11 Encrypted data after VM migration through IPSec tunnel

While building the tunnel, various parameters have to be considered like Network

Address Translation (NAT), ip route, and application level inspection. All these network

30 parameters add more headers and cause more delay in processing the packet at each stage. In a network, there could be many devices which process the packet, add headers, and check the route before forwarding the packet. During this checking, there could be a possibility of losing the essential packets. If the core network has a low bandwidth, then there can be the following risks:

 Loss of ACK packets: If the ACK is lost, the connection could be dropped and VM

migration is not initiated.

 Loss of regular packets: There could be intermittent delay or loss of other essential

packets and VM migration can take more time to complete. Also there could be a

chance of the VM crashing and migration failinh. IPSec is an end-to-end protection

scheme at the Internet Layer which can give security by encrypting the packets but

can cause other problems which are not affordable in an IT organization.

5.5.2 Hypervisor Encryption

Another approach is to do encryption at the hypervisor level. If the Xen hypervisor does the encryption and sends the data during live VM migration, chances of hacking decrease considerably. Strong encryption keys should be used. Decryption should be done by the Xen hypervisor itself to maintain the data integrity and confidentiality. If encryption is done at the hypervisor level, there would be less overhead, less downtime, less migration time. There would not be a requirement for application inspection through the tunnel. If a host builds a SSH connection or any secure connection, encryption would be done by the hypervisor. The hypervisor should maintain the encryption and decryption keys to ensure that the data is secure.

Chapter 6

Conclusion and Future Work

6.1 Conclusion

This research addresses the security concerns in live VM migration and proposes mitigation techniques. With the advent of virtualization and its new features, there are many security holes. This research conducts a simple experiment to show how data is visible during live VM migration. If there is an application running on a VM and if that VM is live migrated to another physical host, then the data being exchanged in that specific application is in clear text and could be compromised using a tool wireshark. Using wireshark, a man-in-the-middle attack can be created by sniffing the traffic flowing between the two hypervisors. To have a secure live

VM migration, migration should be done through an IPSec tunnel or using proper encryption techniques with strong private and public keys.

6.2 Future Work

This thesis brings to attention the less secure feature of virtualization. Performing live

VM migration is beneficial in many ways if done securely. Live VM migration has a limitation of migrating in the same subnet. To overcome this limitation, a virtual switch should be configured to note VM‟s new location and routes the packets as per the new default gateway. To overcome the security issue, IPSec keeps data secure but has the disadvantage of delaying the migration. This is due to more downtime of the application on the VM, and extra processing of the packets. One should concentrate on encryption at the hypervisor level. Embedded hypervisor

[67] is a type 1 hypervisor that supports multiple VMs. A very important feature on the

32 embedded hypervisor is secure encapsulation for any subsystem. Embedded hypervisor gives less exposure to malicious hackers and protects the complete network from being attacked.

REFERNCES

REFERENCES

[1] Walter Tichy, Technology Review of Mainframe Computer Systems and their Alternatives [2] SAN Basics, A Technical White Paper from MetaStor Storage Solutions, 1999 [3] Kara Nance, Brian Hay, Matt Bishop, Virtual Machine Introspection Observation or Interference, IEEE Computer and Society, 2008 [4] Server Virtualization: A Step Toward Cost Efficiency and Business Agility, Avanade Perspective, 2009. [5] G. Somasundaram, Alok Shrivastava, Information Storage and Management, EMC Education Services, 2009.

[6] SAN Security: A Best Practices Guide. Incorporating SAN security into the enterprise with the Brocade Secure Fabric OS, Brocade Communications Systems, Inc. [7] Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hanseny , Eric July, Limpach C., Pratt I., Wareld A., Live Migration of Virtual Machines [8] Ma F., Liu F., Liu Z., “Live Virtual Machine Migration based on Improved Pre-copy Approach,” Network Management Research Centre, Beijing Jiaotong University Engineering Research Center of Network Management Technology for High Speed Railway, Ministry of Education, 2010 [9] Rosenblum M., Garfinkel T., "Virtual Machine Monitor: Current Technology and Future Trends", Published by the IEEE Computer Society, 2005 [10] Al-Rabayah O., Virtualization Concept and History http://www.remoteitservices.com/content/virtualization-concept-and-history, 2010

[11] Wahlig E., hardware Based Virtualization Technologies http://www.redhat.com/f/summitfiles/presentation/June2/ECO/Red%20Hat%20Summit% 20Breakout%20A%20Elsie.pdf, 2006

[12] Fleury E., "Virtualization (Wake up Neo, The Matrix got you)", http://www.labri.fr/perso/fleury/courses/SS07/download/lectures/01-Virtualization.pdf, 2007.

[13] Rosenblum M., "The Reincarnation of Virtual Machines", http://www.datatrend.com/trendseletter/Issue_09_Articles/ReincarnationoftheVirtualMac hine.pdf, 2004

[14] Mijat R., Nightingale A., "Virtualization is Coming to a Platform Near You", White Paper ARM, 2010

[15] Veners Bill. 2000, Inside the Java Virtual Machine http://www.artima.com/insidejvm/ed2/jvm.html

[16] George K. Thiruvathakula, Konstantin Laufer, Konrad Hinsen, and Joe Kaylor, "Virtualization for Computational Scientists". Computing in Science & Engineering, Copublished by the IEEE CS and the AIP, 2010

[17] What is Xen hypervisor? , Xen, http://www.xen.org/files/Marketing/WhatisXen.pdf , 2004

[18] IBM Virtualization, Virtualization on System z, 2009

[19] IBM Virtualization. Virtualization for Linux, 2009

[20] Adair, R.J., R.U. Bayles, L.W. Comeau, and R.J. Creasy,A Virtual Machine System for the 360/40, Report 320-2007,May 1966, IBM Cambridge Scientific Center: Cambridge,MA.

[21] Thomas Burger, The advantages of using Virtualization Technology in the Enterprise, http://software.intel.com/en-us/articles/the-advantages-of-using- virtualization-technology-in-the-enterprise/, Aug 2010.

[22] Bernard Golden, Clark Scheffy, Virtualization for dummies, Sun and AMD special Edition, 2009

[23] Virtuozzo,http://www.swsoft.co.uk/, 2001

[24] OS Virtualization, http://www.parallels.com/products/pvc46/info/virtualization/, 2001

[25] Kirill Kolyshkin , Virtualization in Linux, 2006

[26] A. Mann. The pros and cons of virtualization. BTQ, 2007. http://www.btquarterly.com/?mc=pros-cons-virtualization\&page=virt-view%research

[27] J. Kirch. Virtual machine security guidelines. The center for Internet Security, September 2007. http://www.cisecurity.org/tools2/vm/CIS_VM_Benchmark_v1.0.pdf.

[28] Oliver Garraux, Information Technology Fundamentals, How Virtualization Works and its Effects on IT, Oct 2007.

[29] K. J. Higgins. VM‟s create potential risks. Technical report, darkreading, 2007. http://www.darkreading.com/document.asp?doc_id=117908.

[30] Roberts Paul, Secure your enterprise data. InfoWorld IT Strategy Guides at http://www.infoworld.com/ad/sponsored_resources.html, 2007.

[31] Citrix , The End of Application Deployment: Virtualized Applications Streamline, Secure and Manage Your Business. www.citrix.com. 2008

[32] Gruman, Galen ABC: An Introduction to Mobile Security – CIO, March 08, 2007 From: http://www.cio.com

[33] Dell Virtualize at the Speed of Your Business, 2009.

[34] Doug Hyde, A Survey On the Security of Virtual Machines, April 2009.

[35] Porter, M. E., Competitive strategy: Techniques for analyzing industries and competitors New York : The Free Press, 1980

[36] Soror, A.A. Aboulnaga, A. Salem, K., Database Virtualization: A New Frontier for Database Tuning and Physical Design. Data Engineering Workshop, 2007 IEEE 23rd International Conference (April, 2007)

[37] Jim Carr, Two vulnerabilities found in VMware virtualization products, http://www.scmagazineus.com, February 2008.

[38] A. Ganguly, A. Agrawal. P.O. boykin, and R. Figueiredo, “WOW: Self-Organizaing Wide Area Overlay Networks of Virtual Work-stations;” Proceedings of the 15th IEEE International Symposium on High Perfomance Distributed Computing (HPDC), pages 30-41, 2006.

[39] P.Ruth, J.Rhee, D. Xu, R. Kennell and S. Goasguen, “Autonomic Live Adaption of Virtual COmputational Environment in a Multi-Domain Infrastructure.” IEEE International COnference on Autonomic Computing (ICAC'06), 2006.

[40] NELSON, M., LIM, B.-H., AND HUTCHINS, G., “Fast transparent migration for virtual machines.” In Usenix, Anaheim, CA (2005), pp. 25–25.

[41] Bradford, R., Kotsovinos, E., Feldmann, A., And Schi Oberg, H, “Live wide-area migration of virtual machines including local persistent state,” Proceedings of the International Conference on Virtual Execution Environments (2007), pp. 169–179.

[42] C. P. Sapuntzakis, R. Chandra, B. Pfaff, J. Chow, M. S. Lam, and M.Rosenblum. “Optimizing the migration of virtual computers;” Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI-02), December 2002.

[43] M. Kozuch and M. Satyanarayanan. Internet suspend/ resume. Proceedings of the IEEE Workshop on Mobile Computing Systems and Applications, 2002.

[44] Whitaker, A., Richard S. Cox, Marianne Shaw, and Steven D. Gribble. “ Constructing services with interposable virtual hardware,” Proceedings of the First Symposium on Networked Systems Design and Implementation (NSDI '04), 2004.

[45] S. Osman, D. Subhraveti, G. Su, and J. Nieh. “The design and implementation of zap: A system for migrating computing environments,” Proc. 5th USENIX Symposium on Operating Systems Design and Implementation (OSDI-02), pages 361-376, December 2002.

[46] VMWare, Inc. VMWare VirtualCenter Version 1.2 User's Manual. 2004.

[47] Michael L. Powell and Barton P. Miller., “Process migration in DEMOS/MP,” Proceedings of the ninth ACM Symposium on Operating System Principles, pages 110- 119. ACM Press, 1983.

[48] Marvin M. Theimer, Keith A. Lantz, and David R. Cheriton, “Preemptable remote execution facilities for the V-system.” Proceedings of the tenth ACM Symposium on Operating System Principles, pages 2 - 12. ACM Press, 1985.

[49] Eric Jul, Henry Levy, Norman Hutchinson, and Andrew Black, “Fine-grained mobility in the emerald system. ACM Trans,” Comput. Syst., 6(1):109-133, 1988.

[50] Douglis, F., and John K. Ousterhout, “Transparent process migration: Design alternatives and the Sprite implementation,” Software - Practice and Experience, 21(8):757-785, 1991.

[51] A. Barak and O. La'adan. “The MOSIX multicomputer operating system for high performance cluster computing,” Journal of Future Generation Computer Systems, 13(4- 5):361-372, March 1998.

[52] D. Milojicic, F. Douglis, Y. Paindaveine, R. Wheeler, and S. Zhou. Process migration. ACM Computing Surveys, 32(3):241-299, 2000.

[53] Douglis, F., “Transparent process migration in the Sprite operating system,‟ Tech. rep., University of California at Berkeley, Berkeley, CA, USA, 1990.

[54] KERRIGHED. http://www.kerrighed.org, 1998.

[55] Thain, D., Tannenbaum, T., And Livny, M., “Distributed computing in practice: the condor experience,” Concurr. Comput. : Pract. Exper. 17 (2005), 323–356.

[56] Milojicic, D., Douglis, F., Paindaveine, Y., Wheeler, R., And Zhou, S., “Process migration survey,” ACM Computing Surveys 32(3) (Sep. 2000), 241–299.

[57] Venkatesha S., Sadhu S., Kintali S.Department of Computer Science, University of California, Santa Barbara, “Survey of Virtual Machine Migration Techniques,” 2009.

[58] Nelson M., Beng-Hong L., and Hutchins G., “Fast Transparent Migration for Virtual Machines,” VMware, Inc. In 2005 USENIX Annual Technical Conference, 2005.

[59] Marvin M. Theimer, Keith A. Lantz, and David R. Cheriton, “Preemptable remote execution facilities for the V-system,” Proceedings of the tenth ACM Symposium on Operating System Principles, pages 2 - 12. ACM Press, 1985.

[60] openSUSE 11.1 Start-up, http://www.novell.com/documentation/opensuse111/pdfdoc/opensuse111_startup/opensu se111_startup.pdf, March 2009

[61] SUSE Linux Enterprise Server 11SP1, Virtualization with Xen,http://www.novell.com/documentation/sles11/pdfdoc/book_xen/book_xen.pdf , Aug 2010

[62] Spector ,S., and Xen.org Community, Why Xen?, 2004.

[63] Xen 4.0 Data Sheet, http://www.xen.org/files/Xen_4_0_Datasheet.pdf, 2004.

[64] XEN, How does XEN Work? , http://www.XEN.org, December 2004.

[65] Brantley C., Coraid Inc, ATA Over Ethernet, 2005.

[66] IPSec Tunnel Creation, SANS InfoSec Reading Room, 2003

[67] Robert D., LynuxWorks San Jose, CA Virtualization and hypervisors aid embedded design http://www2.electronicproducts.com/Virtualization_and_hypervisors_aid_embedded_des ign-article-FAJH05_LynuxWorks-Apr2008-html.aspx, 2008