Automated File Extraction in a Cloud Environment for Forensic Analysis

DEGREE PROJECT FOR MASTER OF SCIENCE IN ENGINEERING COMPUTER SECURITY

Automated ﬁle extraction in a cloud environment for forensic analysis

Kevin Gustafsson | Emil Sundstedt

Blekinge Institute of Technology, Karlskrona, Sweden, 2017

Supervisor: Kurt Tutschku, Department of Communication Systems, BTH

Abstract

The possibility to use the snapshot functionality of OpenStack as a method of securing evidence has been examined in this paper. In addition, the possibility of extracting evidence automatically using an existing operation tool has been investigated. The usability of snapshots in a forensic investigation was examined by conducting a series of tests on both snapshots and physical disk images. The results of the tests were then compared to evaluate the usefulness of the snapshot. Automatic extraction of evidence was investigated by implementing a solution using Ansible and evaluating the algorithm based on the existing standard ISO 27037. It was concluded that the snapshots created by OpenStack behaves similar enough to disks to be useful in a forensic investigation. The algorithm proposed to extract evidence automatically seems to not breach the standard.

Keywords: Forensic, Qcow, OpenStack, Snapshot

Sammanfattning

Möjligheten att använda OpenStacks ögonblicks funktion som metod för att säkra bevis har granskats i detta papper. Dessutom har möjligheten att extrahera bevis automatiskt med ett beﬁntligt automatiseringsverktyg undersökts. Användbarheten av ögonblicksbilder i en rättslig utredning undersöktes genom att genomföra en serie tester påbåde ögonblicksbilder och fysiska disk avbilder. Resultaten av testerna jämfördes sedan för att utvärdera användbarheten av ögonblicksbilden. Automatisk utvinning av bevis undersöktes genom att implementera en lösning med Ansible och utvärdera algoritmen baserat påden beﬁntliga standarden ISO 27037. Det drogs slutsatsen att de ögonblicksbilder som skapats av OpenStack beter sig tillräckligt lika en fysisk disk för att avbilderna ska vara användbara vid en råttslig utredning. Den algoritm som föreslås att extrahera bevis automatiskt tycks inte bryta mot standarden.

Nyckelord: Forensik, Qcow, OpenStack, Ögonblicksbild

iii

Preface

This thesis is the last part of ﬁve year education on Blekinge Institute of Technology. The education will provide a degree of master of science in engineering computer security. We would like to thank City Network AB for the time they spent on helping us and also for the workspace in their oﬃce. A special thanks to our advisor Anders Carlsson who have helped us with contact information and provided us with ideas on how to solve the task. Thanks to Vida Ahmadi for helping us get in contact with City Network. Thanks to Kurt Tutschku who have been our supervisor throughout the thesis. Thanks Jonas Virdegård and Jim Keyzer who have helped us as external resources.

"I am the wisest man alive, for I know one thing, and that is that I know nothing." - Socrates

Nomenclature

Notations Acronyms API Application Programming Interface CSP Cloud Service Provider DEFR Digital Evidence First Responder GDPR General Data Protection Regulation IaaS Infrastructure-as-a-Service IDS Intrusion Detection System IoT Internet of Things IP Internet Protocol ISO International Organization for Standardization kB Kilobyte LVM Logical Volume Management NTP Network Time Protocol PaaS Platform-as-a-Service PID Process identiﬁer Qcow QEMU copy on write RAM Random Access Memory RB Rättegångsbalken SaaS Software-as-a-Service SSH Secure Shell UUID Universally Unique Identiﬁer VM Virtual Machine YAML YAML Ain’t Markup Language

vii

Table of Contents

Abstract i Sammanfattning (Swedish) iii Preface v Nomenclature vii Notations ...... vii Acronyms ...... vii Table of Contents ix 1 Introduction 1 1.1 Introduction ...... 1 1.2 Background ...... 1 1.3 Objectives ...... 1 1.4 Delimitations ...... 2 1.5 Thesis question ...... 3 2 Theoretical Framework 5 2.1 What is cloud computing ...... 5 2.2 Forensic science ...... 9 2.3 Automated tools ...... 15 2.4 Technical standards ...... 16 2.5 Laws and preliminary investigation ...... 18 2.6 Similar work ...... 19 3 Method 21 3.1 Tests of Qcow disk image ...... 21 3.2 Algorithm for automated extraction ...... 24 3.3 Analysis of Qcow using forensic tools ...... 26 3.4 Prove non-repudiation of snapshots ...... 27 4 Results 29 4.1 Test results ...... 29 4.2 Proposed algorithm for automated extraction ...... 30 4.3 Findings in Qcow snapshot ...... 31 4.4 Proving non-repudiation of snapshots ...... 31 5 Discussion 35 5.1 Proposed method ...... 35 5.2 ISO 27037 ...... 36 5.3 Why we chose Ansible ...... 36 5.4 OpenSSH or Paramiko ...... 37 5.5 Test results ...... 37 5.6 Identifying a virtual machine ...... 37 5.7 Using backing and overlay ﬁles ...... 38 5.8 CLI history ﬁle ...... 39 5.9 Virtual Introspection ...... 39 5.10 Ethics ...... 39 5.11 Sustainable Development ...... 40 6 Conclusions 41

ix 6.1 Is it possible to use a snapshot as evidence? ...... 41 6.2 Prove non-repudiation of a snapshot ...... 41 7 Recommendations and Future Work 43 7.1 Implementation in OpenStack ...... 43 7.2 Implementation on a hypervisor level ...... 43 7.3 Check additional hypervisors ...... 43 7.4 Additional ﬁle systems ...... 44 7.5 Automatic learning and extraction ...... 44 7.6 Test in court ...... 44 7.7 Checkpoint snapshot ...... 44 7.8 Snapshotting in containers ...... 44 References 45

x 1 INTRODUCTION

1.1 Introduction Crimes has been an inconvenient truth since the beginning of humanity and have unfortunately become a fact of the society of today. As the capabilities of technology have evolved throughout the past decade, so has the methods of committing crimes. Currently all types of crimes involve electronic equipment in some way, if not to commit the action itself but to communicate or ease the task. This is unlikely to change as computer systems play a more important role in our everyday lives, cell phones are more likely than not to be present and additional equipment is being introduced daily, Internet of Things (IoT) for example.

As illegal actions, such as selling and buying drugs, obtaining and distributing child pornography, money laundering, piracy, etc. moved into the Internet, so did the investigators of crimes. The standard procedure when conducting a forensic investigation includes a collection of electronics that may contain evidence for the case. This means that the physical hardware found is collected and brought to a forensically sound lab for analysis.

Cloud technology allows users to move their activity away from in-house hardware and into the cloud instead. This means that any illegal activity such as distributing illegal material which was previously conducted via in-house solutions or otherwise outsourced to rental servers, can now be moved into the cloud. Due to the nature of the cloud, data can now be moved seamlessly between servers, data centres and even countries. This seamless structure adds an extra layer of obstacles during a forensic investigation as data can be spread across multiple servers and countries, rendering a physical collection impossible. Also there exist laws which prevent authorities from shutting down an entire cloud just to investigate a single user’s actions.

1.2 Background City Network Hosting AB is a Swedish company currently located in Karlskrona but with infrastructure around the world. Their initial business was to oﬀer a web hosting service to individual users as well as companies. Their focus has lately shifted from web hosting to cloud computing. Because they are oﬀering infrastructure to their customer, they might be hosting systems included in potential crimes. A solution which could be used to secure digital evidence located in a cloud would enhance the perception of the business as well as their credibility.

Also in May 2018 a new General Data Protection Regulation (GDPR) will be applied. This new EU regulation will replace the current regulations in Sweden. The regulation shifts all the power of information related to individuals back to the individual. In case the regulation is not followed as expected companies can face severe penalties of up to 4% of their turnover. Due to this, there is an interest for City Network Hosting AB to be able to secure evidence of systems running in their cloud to be able to dismiss any accusations as well as aid a pending investigation.

1.3 Objectives Our objectives with this project are to investigate available snapshot functionality found in the cloud environment OpenStack, which could potentially be used to secure evidence. The outcome of the snapshot function will be evaluated against a traditional disk copy, and a proposed algorithm for securing a snapshot and extracting evidence found within will be proposed. A

1 simple proof of concept solution will be implemented as far as possible to check out how practical our proposed solution is. We will most likely come up with alternatives to our solutions that we will not be able to implement due to our time constraint.

We will focus on preserving non-repudiation when securing and extracting evidence. Non- repudiation within the ﬁeld of digital security is composed of two parts:

• Integrity of data The integrity of potential evidence should be secured. This means that it should be possible to prove that any data extracted has not been altered. • Authenticity It should be possible to prove that extracted data has been extracted from the alleged system.

1.4 Delimitations Due to City Network Hosting ABs generosity and agreeing to work with us, we have chosen to only focus our work on the same type of setup as they are currently running. This means that we are only looking into a solution that would primarily work with OpenStack which is running KVM/QEMU as the virtualization layer. Any other type of setup, cloud orchestrator or virtualization technique is out side of this scope.

1.4.1 Forensic process A standard complete digital forensic process consist of the following steps: 1. Preparation This is the phase during which the examiner plans the case, expected search warrant needed, special equipment, researching required knowledge, expected systems, etc. 2. Survey Surveying a crime scene is when potential sources of evidence should be identified and noted, this includes both hardware and digital evidence. 3. Documentation When potential sources of evidence has been identified they has to be properly documented. Documentation is a crucial part of all stages during an investigation, and any handling of evidence and actions made should be carefully documented. 4. Preservation Sources of potential evidence should be preserved in such a way that the digital evidence can be authenticated at a later time. Methods of preservation differ between cases and sources, changes made to to potential evidence should always be kept to a minimum. 5. Examination and Analysis Collected potential evidence is then moved to a special facility designed for the purpose of digital evidence analysis. Everything collected is then inspected, and found evidence are preserved and documented. 6. Reconstruction When the examiner believes all evidence collected has been identified and preserved, then the events are reconstructed, trying to acquire a complete picture of events.

2 7. Reporting The examination is ﬁnalised by writing a reporting of all the ﬁndings. This is one of the most important stages as it is the report that is usually presented in court.

While all of the stages in a digital forensic process are important, we are only focusing on potential methods of preserving digital evidence found in a virtual machine running on OpenStack. We assume appropriate preparations have been taken and potential sources have been identiﬁed.

1.4.2 Acquisition constraints We do not consider special cases where there might exist legal constraints which prevent an analyst from collecting some evidence. These types of constraints would prevent the use of snapshotting, thus prevent any proposed algorithm in this paper. In those cases where the disk is allowed to be copied we assume the analyst obtained the proper authorization to read the data.

1.4.3 Encrypted systems It is possible to encrypt virtual machines running within a cloud, much like it is possible to encrypt a physical hard drive of a running system. We do not consider this possibility, as an encrypted virtual machine would render our solution useless. An encrypted virtual machine, in this case, will most likely aﬀect an investigation in the same ways as an encrypted physical system would, thus falling out of our scope.

1.4.3.1 Attached volumes In a cloud it is possible to attach volumes to the virtual machines. Our solution will not take that into account. The volumes in OpenStack are using Logical Volume Management (LVM) to create a logical disk, and we are only looking at the snapshot of a virtual machine which generates a QEMU copy on write (Qcow) disk image.

1.5 Thesis question Is it possible to use the snapshot functionality in OpenStack as an alternative to a forensic disk clone? If the OpenStack snapshot functionality is an alternative to a forensic disk clone, is it possible to automatically extract data from the snapshot in a forensically sound manner? Is it possible to prove the non-repudiation of the snapshot?

2 THEORETICAL FRAMEWORK

2.1 What is cloud computing The modern concept of Cloud Computing was introduced by Amazon back in 2006 when they introduced Elastic Compute Cloud, which was a service where excessive resources were oﬀered to the public for a modest cost. This concept has since then evolved into a hot topic in the digital world. The National Institute of Standards and Technology deﬁnes Cloud Computing as [3]:

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of conﬁgurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management eﬀort or service provider interaction.

This type of service allows its users to pay only for the resources required at the moment, a so-called pay-per-use model. The solution is commonly referred to as an elastic solution as it also provides the capabilities to add and remove resources seamlessly as the demand grows and shrinks. As customers only purchase what they need from a Cloud Service Provider (CSP), there is no need for the customer to purchase and maintain their own infrastructure. Instead, this is now up to the CSP [4].

A report written at UC Berkeley Reliable Adaptive Distributed Systems Laboratory explains [5] that Cloud Computing introduces three new aspects from the hardware’s point of view:

1. Cloud Computing creates the illusion of inﬁnite hardware being available on-demand. 2. Resources can be added as demand increases. This means that anyone can start out small and if there ever will be a need for additional resources, those can be added seamlessly. 3. Cloud consumers do not have to make a long-term commitment. This means that it is possible to pay for resources by the hour.

As Dale says, the three deﬁnitions above only describe the physical nature of Cloud Computing. This model will also blur the lines of ownership of data. The key concept for cloud consumers is that data that was previously stored in-house is now being moved to outsourced locations [6].

Different models of Cloud Computing solutions can be offered to customers. Three models are discussed alongside Cloud Computing. Software-as-a-Service (SaaS), this is a model where complete applications running on the cloud infrastructure are offered to the customers. Examples of SaaS applications are the ones hosted by Google (Google Drive, Google Mail, Google Calendar, etc.) or the music streaming service Spotify. Platform-as-a-Service (PaaS) is the second model which allows the Cloud consumers to host their own applications on the CSPs infrastructure. Solutions such as AWS Elastic BeanStalk, Microsoft Azure, and Google Apps Engine runs on PaaS. The last model is Infrastructure-as-a-Service (IaaS), this is a model where the CSPs infrastructure is offered to the customers. The infrastructure can be used to launch and host Virtual Machines (VMs). See fig 2.1 for a visual presentation.

2.1.1 OpenStack overview OpenStack is an open source project initiated by NASA and Rackspace back in 2010 [7]. A non-proﬁt corporate was established in 2012 whose purpose is to promote OpenStack. Over 500

5 Figure 2.1: Management distribution between cloud models. companies around the globe come together to develop and maintain the project. At the core, OpenStack is a cloud orchestrator framework for setting up an IaaS-cloud, which potentially could be transformed to PaaS and SaaS.

The architecture of OpenStack has been designed to support deployments in a modular fashion. This was achieved by developing components into independent services. The general purpose of a service is to abstract and manage a set of resources. The modular approach allows for multiple diﬀerent deployments of OpenStack and if a particular service is not part of the desired deployment, then it can be safely left out. All services available in OpenStack belongs to one of three groups:

• Control is the group which contains all services that expose an Application Programming Interface (API), such as interfaces, databases, and the message bus.

• Network contains services that run and manages the network of the deployment. • Compute groups all services that manage the VMs.

In a smaller deployment, all the desired services can be deployed on the same node. It is generally recommended to distribute the deployment of services onto diﬀerent nodes as to support scaling. OpenStack was designed to scale horizontally (adding additional resources), thus it is recommended to at least deploy the compute service on a separate node from the control

6 and network services as additional nodes could then be added [10].

OpenStack was developed with the intention to abstract the underlying infrastructure and its communication. This is realised by the use of vendor-provided drivers, which prevents technology lock-ins [8]. OpenStack is also capable of running on top of multiple diﬀerent hypervisors (described below) such as KVM, QEMU, VMWare, Xen, and Hyper-V.

Because OpenStack uses a modular architecture, services can be added and removed as needed. Currently three services are required to run a core version of OpenStack [9]: • Nova is the controller node of OpenStack. It is responsible for the life cycles of instances (VMs) in the cloud (spawning, scheduling, terminating, etc.). • Glance service stores and manages images used to launch instances in the cloud. • Keystone service is responsible for authentication and authorization of actions made against available services in OpenStack.

The services mentioned above presents the bare core, additional services are deployed to add functionality. Examples of such services are Neutron (networking), Swift (object storage), Cinder (block storage) and Horizon (dashboard).

Figure 2.2: OpenStack architecture.

2.1.2 Virtualization techniques A hypervisor, also called Virtual Machine Monitor, is a hardware, software, or ﬁrmware [27] that can create and run VMs. A computer system that runs a hypervisor which runs one or more VMs is called a host machine and each VM is called a guest machine. In this paper we will use the term VM instead of guest machine. The hypervisor is presenting the VMs with a virtual operating platform, a virtual hardware, and manages execution of the VMs. There is no limit to which operating systems that shares the same hardware which means Linux, Windows and OS X can all run on the same physical hardware.

There are two types of hypervisors, type 1 and type 2:

7 • Type 1 - The type 1 hypervisor is running directly on the hardware and does not rely on a host operating system. The ﬁrst hypervisors IBM created was of this type. Modern versions are Xen, Oracle VM Server, Hyper-V, and VMware ESX/ESXi. • Type 2 - These runs on top of the host operating system, just as a normal program does. All virtual machines launched on a type 2 hypervisor will run as a separate process on the host system. The modern versions are VirtualBox, VMware Player, VMware Workstation, and QEMU.

In this paper we will look at QEMU and KVM [17]. KVM is a hypervisor, or to be precise an accelerator, that runs VMs with the help of QEMU. KVM does not perform any emulation by itself, thus needing QEMU to emulate the operating system. Since KVM is built into the Linux kernel, it runs code directly on the physical hardware, the kernel switches mode for the process to a guest mode. This means we are running a thread with the VM directly on the hardware, but the VMs operating system is running on QEMU within the host. For the KVM all VMs are just threads running on the hardware, all memory checks, scheduling, etc. is done by QEMU. The advantage of this is that we can use the QEMU commands on the host to alter the VM, which is more ﬂexible if we later need to switch host, but still have the speed as the VM runs directly on the physical hardware.

To be able to manage all these layers OpenStack uses the libvirt library. Libvirt is an API that can be used both locally and remotely to manage the lifecycle of VMs running in KVM. Libvirt does not only support KVM, but that is the only hypervisor we will look at. A command is sent from the virsh, the command line tool for libvirt, to the libvirt library. Libvirt then sends the command to QEMU which sends it to the KVM. At the KVM layer, it is run on the hardware. If the hypervisor instead was a Xen, we would use the Xen tools to send the command down directly to the hypervisor. It is possible to run libvirt on a Xen hypervisor.

Figure 2.3: Command ﬂow in libvirt.

As stated before we are only considering QEMU and KVM, but OpenStack have the possibility

8 to run diﬀerent hypervisors. The algorithm we are proposing should work on all hypervisors as long as the virtual disk is handled the same way.

2.2 Forensic science Forensics or forensic science, is "the application of scientific knowledge and methodology to legal problems and criminal investigations", according to an English dictionary. The early days lacked standardised forensic practices; this led to criminals avoiding punishment. Criminal investigators relied heavily on confessions and witnesses. One of the first real breakthroughs in forensic is the use of fingerprints. When it became possible to connect a crime scene and a person through fingerprints, it became easier to prove that a specific person committed the crime, or at least was at the crime scene. Today the forensic science has come a long way and contains well-defined standards on how evidence should be collected and what is allowed to do in a specific scenario.

Along with new technologies, the forensic must evolve. Standards and laws need to be changed and updated. During the early 80s, the personal computers became more accessible which meant that more people started to use it. At the same time, a new type of crime was recognised, hacking. To ﬁnd and convict criminals in this new medium a new forensic method had to be formed, thus leading to the term digital forensic.

2.2.1 Digital forensic Digital forensic is the name for recovering and investigating material found on a digital device. A digital device is a device that is capable of storing digital data, for example, computers, mobile phones, memory cards, etc. Even networks can be included in this term, even if the networks do not store any digital data. With the rapid advancement in technology, the term digital forensic had to be split into sub-areas. Today forensic investigators are specialised in a speciﬁc digital medium or system. For example there are investigators that are considered Android forensics specialists, database forensics specialist, network forensics specialist, etc. This paper will investigate cloud forensic and snapshot forensic.

A digital investigation have many applications. The most common task is to support or disclaim a hypothesis in court, but this is not the only application scope. Corporations can use forensics to investigate internal corporate information leak or data breach, something that the company does not necessarily want the police to be involved in but the company wants to know when and where an event happened.

A digital investigation usually consists of four steps, collection, acquisition, analyse and report [1].

• Collection - Collection, according to Swedish police standards, is the search, collection and securing of information stored on a digital medium intended to be used as evidence. During this step everything is documented with photographs and video, when possible. The purpose with this is mainly to be able to reconstruct the scene in a lab during the analysis phase, but also to ensure nothing is lost during transport.

During the ﬁrst years of collecting digital evidence the police always shutdown systems that were running and brought them back to a lab for analysis. This is not always the best way of doing it, and in some cases it is even illegal, according to the principle of proportionality.

9 The principle of proportionality states that the damage caused by an investigation must be proportional to the crime being investigated. It is for example not allowed to collect all disks from a cloud provider if a single virtual machine or user is the subject of a minor crime.

When a system is running with encrypted disks is encountered the best option is to do as much of the forensic process on the live system as possible. When the system is powered oﬀ the data is encrypted and if the key is unknown, lost. The action of performing an analysis of a running system is called live forensic.

• Acquisition - Sometimes during an investigation, collection of the digital system of interests is not feasible, meaning that the systems cannot be removed from its original location. This could happen because of physical or legal constraints. When encountering this situation the data of interest is acquired logically. This means that the data is copied logically and brought back to a lab. What is acquired and how much is determined from case to case.

If all data is to be acquired a forensic copy of all disks are made, this is to ensure the original data is not modiﬁed. If data, which is not involved in the crime or event is found, it can be deleted from the copy. Conﬁdential data concerning a third party is also to be treated accordingly, not to cause unnecessary suspicion, expense, or inconvenience.

Sometimes data that requires a speciﬁc authorization is present on a system. In these cases it might be preferred to acquired only speciﬁc data instead of a complete copy. An example of data that is not to be acquired during an analysis is email correspondence between the suspect and a lawyer.

• Analyse - There is no clear line between the analysis phase and the collection/acquisition and they are usually done in parallel. In this phase, the data that was collected/acquired in the previous phase is analysis with the purpose to secure data that will be used as evidence. The collection/acquisition and analysis phase is iterative, meaning that it is repeated multiple times as new potential evidence is found. The analysis should be done with consultation with the inquiry leader.

The analyst attempts to recreate the occurring events once it has been understood. This is done to conﬁrm the hypothesis of what happened.

• Report - The documentation from all the methods and actions taken and any evidence discovered are compiled into one report. This also includes the actions done during the collection/acquisition. The report of the investigation is the most essential document. Usually, a lawyer or prosecutor will only read this report and base the case upon it.

Keep in mind that the steps above are very generic and simpliﬁed. They are also based on the method the Swedish police are using and does not necessarily follow other countries standards, even if the general steps are similar.

10 2.2.2 Cloud forensics Today it is possible to run and maintain a whole company using cloud services. This requires the investigators to find new ways to handle data since the old standards are not suitable for a cloud environment [11]. The first problem occurs when data is no longer bound to a physical location or specific disk. When stored in the cloud a file could be fractured into smaller pieces and spread across multiple disks. This creates a problem when an investigator shall collect evidence for a case. As described above a usual way is to collect evidence is to take the physical disk. This is unrealistic to do in most cases when working in a cloud since it will damage the cloud provider and all users running virtual machines on the same server. In extreme cases, this is done, for example during the investigation of the torrent site The Pirate Bay.

Since it is not feasible to collect the disks in a cloud environment, new methods are required. The method researched in this paper will look into the possibility to use a snapshot, a copy of the virtual disk at a speciﬁc time, as a forensic disk copy.

2.2.3 Physical versus logical copy of a disk When referring to a physical disk copy, we mean the speciﬁc method of requiring a disk. The physical disk copy this is when an analyst takes the physical disk, puts it in a dedicated write blocker and then copies the disk bit by bit to another hard drive. This is to ensure that nothing will change on the original disk. The original drive is then stored in a safe place.

A logical copy is when we are not using dedicated hardware to make the copy. A simple "dd"-command in Linux would be suﬃcient to make this kind of copy. Since the disk we are copying is connected to a computer, we cannot know that the disk is in a read-only state, even if we mount it that way. The computer we attached the disk to could be broken or infected. This is why an analyst should use a dedicated write blocker from a known provider as often as possible. In the case of analysis in the cloud, an analyst needs to take a logical copy since the physical disk is not accessible in most cases.

2.2.3.1 Non-volatile Data Non-volatile memory is a storage that has the capability to hold saved data even when the power is cycled.Examples of non-volatile memory are ﬂash memory, SD-cards, optical disks, and magnetic devices. Magnetic memory refers to hard disk drives, ﬂoppy disks, and magnetic tape. This type of memory is usually used as a secondary storage or consistent long-term storage in a system.

The most common PC will store its boot partition and file system on a secondary storage, usually a hard disk drive or a solid-state disk. In a forensic analysis this is the device where most of the data is found. This data is easy to collect since it does not require any special equipment to create a logical copy. During an investigation a disk copy is preferred to the acquisition of only certain files. If only found files of interest are copied, important data that could have been hidden on the disk is lost. Such data could be found in slack space of files or in unallocated memory. Slack space is the bytes in a block that is not used by the file allocating the block. For example if a file with a size of three Kilobyte (kB) is occupying a block with a size of four kB the block has one kB of unused memory which could be used to hide data. To ensure all data is collected a full disk image could be created of the disk.

11 A bit by bit copy also ensures that deleted files that have not yet been overwritten are accessible in the disk image. Files on a disk are stored differently depending on the file system. On a Unix system all files are associated with an inode, which is defined by an integer called inode number. Inodes stores metadata about the file and a reference to where the data is stored on the physical disk. What happens with the inode when a file is deleted depends on the file system in use, for example, deletion of a file on ext3 will result in a complete wipe of information in the inode which is then marked as unallocated. But on ext4 the status of the inode will be changed to unallocated, which means that any metadata is left untouched until the inode is allocated again. The file is not deleted or overwritten on the disk and can still be accessed if the location is known. This means that all deleted files on a system can be recovered until they are overwritten on the physical disk. The method for extracting these files are called file carving. The same method of extracting files can be used on a Windows operating system, but instead of inodes, Windows uses a Master File Table to keep track of files.

2.2.3.2 Volatile data Data found in digital systems which are stored on a volatile storage medium, such as Random Access Memory (RAM), are usually referred to as volatile data. That is because this type of data is usually lost forever if the system is powered down. Volatile data increases the diﬃculty of collecting relevant evidence of a digital system because it must be collected on a live system. When collecting data on a live system, the examiner will alter the system in various ways, which might compromise additional evidence located on the system. Because of the risks involved when collecting volatile data the examiner should weigh the pros and cons of doing so (usually at the scene, however criteria are commonly deﬁned beforehand). If the examiner decides to collect volatile data then it is good practice to use a script written on an alternative system which performs the actual collection. This method is preferred because all actions will alter the system in various ways and a script eliminates the risk of typos and unnecessary commands being executed on the system.

There are different types of volatile data that can be collected from a digital system, data interesting for the case at hand will differ from case to case and from system to system. Volatile data that might be of interest are contents of memory, network configuration, network connection, running processes, open files, login sessions and operating system time. Note that this list is not exhaustive. Volatile data of interest will change from case to case depending on the reason of the investigation. For an example network connections, login sessions, network configuration might be of particular interest if the system has been involved in malicious activity taking place on the network. Whereas running processes and open files might be of more interest if the investigation concerns piracy or child pornography. If there is uncertainty concerning which volatile data to collect, then the examiner should always collect as much data as possible as the possibility to collect the data will be lost once the system is powered down. Since some information is more time sensitive than other, volatile data should be collected according to the time sensitivity. An example of differences is that network connections, and login sessions might time out at any time, while it is unlikely that the network configuration or operating system time will change. Recommended order of collection:

1. Network connections - Active network connections can reveal outgoing connections which could be additional systems of interest for the case. Incoming connections could potentially show active backdoors on the system. Netstat is a software commonly present on system which can be used to print active network connections.

12 2. Login sessions - Active login sessions could show malicious sessions active on the system, which could have been used to launch attacks. 3. Contents of memory - The memory of a system could potentially store a lot of interesting information only available while the system is powered on. Examples of interesting data could be malicious software which only resides in memory, keys used to encrypt the entire or a part of the system and data which has not yet been written to persistent storage. 4. Running processes - Processes on a system could be started which will wipe any traces of itself from the persistent storage, thus only running in RAM. This means that any traces of the process/program are lost upon system shutdown. 5. Open files - Files that are open on a system could point an examiner in the direction of interesting data. 6. Network configuration - It is important to document how the network of the system is configured, if there are multiple network interfaces, what Internet Protocol (IP) addresses are present on the system, if the routing table has been modified. 7. Operating system time - The time of the operating system must be documented as it is the key to being able to determine the sequence of events. This time might also differ between systems as they might be located in different time zones or might just have been configured incorrectly.

If possible, the examiner should also check the system for potential rootkits that has been installed on the system. The reason is that they might have been installed with the purpose to feed false data when trying to collect the volatile data.

2.2.3.3 Snapshotting in QEMU A snapshot is the exact copy of a disk at a specific time [12]. Since DevStack is our test environment which is running QEMU, each instance will be stored as a Qcow2 or Qcow3 disk image depending on version. There is no significant difference between the two, except some added functionality in Qcow3. Therefore we are treating the two versions the same way. We are not analysing the first version of Qcow files since they are not used in the newer OpenStack deployments.

Qcow images are so-called copy-on-write images. The technique is to use a disk image file called Qcow image, as a backing file and use overlay files which stores changes. When an instance is launched all necessary data is read from the backing image. When the instance then writes anything to disk a new Qcow image is created to store these changes on, this is called an overlay. This way of launching and storing data will create smaller disks for each instance launched from the same backing file, since only changes are written to the overlay. Each instance launched from the same backing image will have its own overlay with changes of the backing image. The backing image will be opened in read-only mode when an instance is running, preventing changes. A change of this image will affect all instances launched from that specific backing file.

When a snapshot is taken of a Qcow image, the current overlay is saved and a new one is created. The old overlay will still use the original backing image as its base, but the new overlay will instead use the old as its backing image. The snapshots are in this way linked with each other and all snapshot overlays are required to be able to run the system without data loss. It is

13 important that every backing ﬁle is stored and loaded in read-only mode [13]. See ﬁgure 2.4 for a visual presentation.

Figure 2.4: Qcow Snapshot.

If a user wants to remove the backing image, it is possible to do by doing a merge of all the snapshots into one overlay that includes the data needed from the backing image. This is what OpenStack does. The command used is the Block Pull or Block Stream in some part of the source code. This commands creates a new Qcow image to act as a new overlay. The libvirt then streams, or pulls, the data from all sources into the new file. Every block that is occupied on the overlays is copied over to the new overlay. Because the copy only checks if the block is occupied and not if it is allocated inside the Qcow image it should be possible to recover deleted files from the snapshot. When the merging is complete, a Qcow image containing all data from every snapshot and the backing file has been created. This new file will not have any backing file and new instances could be launched from this image. See figure 2.5 for a visual presentation.

Figure 2.5: Snapshot Merge.

Note that the current versions of OpenStack do not support snapshotting with multiple snapshots merging into one. The current solution only uses one overlay for each instance and each time this overlay is snapshoted a new launchable Qcow image is created. This is, in a forensic standpoint, much better since the risk of overwriting data becomes smaller than when merging several overlays into one.

14 When the snapshot command is issued in OpenStack, the Nova service will handle the request [14]. Nova then looks on which compute node the instance to be snapshotted is running on and opens a connection to the hypervisor running on that node. Nova then sanitises all parameters received by the request and passes them into the hypervisor driver library, in our case this will be libvirt since DevStack is using QEMU in our solution. Libvirt then creates a new Qcow image, denies the running machine CPU time (causes all execution on the VM to halt) and starts the stream/pull command to mirror the disk snapshotted [15]. When the stream is complete a new standalone Qcow image has been created and the VM is once again permitted CPU time.

2.2.4 Importance of timestamps When analysing a computer system during a forensic investigation, the examiner must be carefully consider available timestamps. This is because the interesting information found in a system has to be pieced back in order to be able to determine the sequence of events. In the best of worlds, the system clock found on the systems would have been synchronized with a hierarchical network of systems designed to synchronise clocks. Unfortunately this is not always the case. System clocks might not have been synchronized , the internal clock of the system might drift by ticking either too fast or too slow, the clock might have been incorrectly configured by a system administrator, etc. Events like these increases the difficulty of piecing events back in the correct sequence. In addition it is not improbable that systems clocks have been configured to different time zones.

Due to these facts it falls on the investigator to take note of all the diﬀerent time conﬁgurations that might be present on the systems. This is so that timestamps later found on the systems can be recalculated to a standard base, from which they can be pieced into order of event.

Network Time Protocol (NTP) is a time protocol designed by David L. Mills with the purpose to synchronise time between computer systems. Computer systems conﬁgured to use NTP communicates with an accurate time server or a set of servers to retrieve the time. Due to the network latency when communicating with other systems, a modiﬁed version of Marzullo’s algorithm called "intersection algorithm" is being utilized to calculate and correct time inaccuracy introduced by the network latency. The current version in use is version 4 which has been standardized and can be found in RFC 5905 [16].

2.3 Automated tools Server configuration and management was traditionally performed manually by the personnel managing the systems. As the amount and complexity of systems grew so did the risk of mistakes and the time required to perform those tasks. The introduction and growth of the Cloud has also affected configuration needs. Typically large system has to be configured in a very specific way and even a small mistake could have devastating consequences. CSPs offers the ability to quickly launch VMs to be used by the consumer, these machines has to be configured quickly for production.

Software configuration management tools exists to automate the process of configuring systems. The automated process inherently speeds up, uniform the configuration, and minimizes potential human mistakes. There exist various tools where the majority of them are open-source. Four of the most popular automation tools are Ansible, Puppet, Chef and Salt. A brief explanation

15 of the four can be found below:

• Ansible - is a clientless solution which uses Secure Shell (SSH) to connect to nodes. It is designed with focus on security and reliability. Tasks in Ansible are deﬁned as "Playbooks" which are written using YAML Ain’t Markup Language (YAML) syntax. Various modules developed and maintained by Ansible is available for use and user contributed modules can be accessed for additional functionality. Modules are traditionally written in Python as it is the language Ansible is written in, but any language is supported.

• Puppet - is the most mature tools out of the four. It uses a client-server model to communicate with the nodes, which means that agents (client software) is installed on all nodes. Puppet uses a custom language to describe how systems should be conﬁgured. Ruby is the language under the hood of Puppet and thus it is the language to be used when writing modules.

• Chef - uses a client-server model for communication with the nodes, but also supports a standalone mode. Ruby and Erlang is the languages used to write Chef (Ruby/Erlang on the server, Ruby on the client). Self-written modules have to be written in Ruby.

• Salt - is a Python-based configuration management tool which uses a client-server model when communicating with the nodes. Tasks are defined as scripts which are called Salts, these can be added or removed to define configuration scenarios. User-written modules should be written in Python to comply with the Salt module design.

Configuration and management tools are a relatively new solution and new options are becoming available, thus the list above is not exhaustive. These tools offer slightly different possibilities and options for configuration and management and therefore suits different needs.

2.4 Technical standards Standards are norms established to standardise technical systems and methods. Standards can be issues by companies, regulatory bodies, standard organizations, etc. Who issues a standard and why depends on the need of a closed group and/or the public. Companies issues private standards as to standardise methods and functionality within the company while standard organizations issues standards to publicly unify something. One example of a standard organization is International Organization for Standardization (ISO). Their headquarter is located in Geneva, Switzerland but they are issuing standards aﬀecting 162 countries as of March 2017.

Standards designed to describe digital forensic guidelines and best practises are sparse. Generally legal authorities in each respective country have to establish guidelines for how digital forensics investigation should be carried out. This includes planning, evidence collection, analysis and reporting. These guidelines aims to unify the forensic process and ensure the highest probability of evidence authenticity within a specific country. Guidelines may or may not impose difficulties when investigating between countries as regulations may differ.

ISO published in October, 2016 a standard (27037) which proposes guidelines for identiﬁcation, collection, acquisition and preservation of evidence found in digital systems [25]. This standards is described more in detail below.

16 2.4.1 ISO 27037 This standard describes activities when identifying, collecting, acquire and preserve digital evidence. Guidelines described concerns digital storage, devices and equipment that might contain digital evidence. Four general requirements are discussed.

• Auditability - All actions made during an investigation should be documented. This is because the process should be available for evaluation and all decisions should be argued for. • Repeatability - An independent Digital Evidence First Responder (DEFR) should be able to follow the documentation of the acquisition process and end up with the same result. There might be circumstances when this is not possible, the DEFR should be able to argue for this. • Reproducibility - An independent DEFR should be able to reproduce the acquisition process under different conditions and when using different tools and still end up with the same result. Reasons for deviations should be argued for. • Justifiability - All actions and decisions made during the acquisition should be justifiable. This means that the DEFR should be able to show and argue that the decisions made was the best choice.

General requirements established when handling processes are: any handling of evidence or potential evidence should be kept to a minimal. Any changes made to collected evidence should be carefully documented, the process should comply with any rules established locally and actions should never be taken beyond the competence of the investigator. The initial four key processes of the standard is described as follows:

• Identification - of digital storage and devices which potentially contains evidence. All identified sources should be ordered according to volatility and order of collection/acquisition should be determined. Sources containing potential evidence might not be easily identified as they might be hidden, stored on a remote site, located in the cloud, etc. • Collection - is a process where sources of potential digital evidence are collected to be transported to a controlled laboratory. Sources might be encountered either powered on or powered off and will likely require different approaches, locally established guidelines should be followed. • Acquisition - is a process to gather data where the sources of potential digital evidence can not be removed from its original location. In this situation an acquisition is desirable. The process should produce a digital copy of the source. The DEFR should verify that the copy is an exact copy of the source. If verification is impossible (e.g. on live systems or faulty disk) the DEFR should document the situation and prepare to defend the decision. • Preservation - of evidence either collected or acquired is of uttermost importance to prevent potential evidence from tamper or destruction. The DEFR should be able to demonstrate the integrity of the evidence (prove that it is the same evidence as initially collected).

The standard also describes the importance of document any handling of evidence as to preserve the trustworthiness (also known as preserving the chain of custody). There are

17 also general recommendations regarding surrounding requirements such as personnel, roles, responsibilities, competence etc.

2.5 Laws and preliminary investigation 2.5.1 Rättegångsbalken Rättegångsbalken (RB) is a legal document in Sweden which consists of procedural laws [26]. This document is mainly a collection of laws concerning trails (who can be prosecuted, when, why, etc.). There are also laws which describes how preliminary investigations should be carried out and evidence handling. A few paragraphs found in RB are especially of interest when discussing legal actions in a cloud environment.

Paragraph Swedish English Translation

23:4 Förundersökningen ska bedrivas så A preliminary investigation shall be att inte någon onödigt utsätts för mis- conducted so as to not cause some- stanke eller orsakas kostnad eller olä- one unnecessary suspicion, expense genhet. or inconvenience.

27:1 Föremål som skäligen kan antas ha Objects that can reasonably be betydelse för utredning om brott assumed to be of importance for eller vara avhänt någon genom the investigation of crimes or taken brott eller förverkat på grund av away from someone by crime or brott får tas i beslag. Detsamma forfeited because of the offense may gäller föremål som skäligen kan be confiscated. The same applies antas ha betydelse för utredning om to objects that can reasonably be förverkande av utbyte av brottslig assumed to be of importance of verksamhet enligt 36 kap. 1 b the investigation of forfeiture in §brottsbalken. exchange for operations of crime under chapter 36. 1b §brottsbalken. Tvångsmedel enligt detta kapi- tel får beslutas endast om skälen för Coercive measures under this åtgärden uppväger det intrång eller chapter may be adopted only if the men i övrigt som åtgärden innebär reasons for the measures outweigh för den misstänkte eller för något the intrusion or but for the rest of annat motstående intresse. the measures for the suspect or for any other conflicting interest.

18 27:10 Beslagtaget föremål skall tas i för- Conﬁscated objects shall be in the var av den som verkställt beslaget. custody of the person who executed Om det kan ske utan fara och eljest the seizure. If it can be done without är lämpligt, får dock föremålet läm- danger and otherwise appropriate, nas kvar i innehavarens besittning. may the object be left in the holder’s Ett föremål som lämnas kvar i in- possession. An object left in the nehavarens besittning skall förseglas holder’s possession shall be sealed eller märkas som beslagtaget, såvida or marked as seized, unless it appears det ej framstår som obehövligt. to be unnecessary.

Ett föremål som ej tas i förvar eller An object not seized or sealed may be förseglas får nyttjas av innehavaren, used by the holder, unless otherwise om ej annat beslutas. decided.

27:15 För säkerställande av utredning om To ensure the investigation of crimes brott må byggnad eller rum tillstän- may building or room be closed, ac- gas, tillträde till visst område förbju- cess to the speciﬁc area be banned, a das, förbud meddelas mot ﬂyttande prohibition of removal of certain ob- av visst föremål eller annan dylik jects or other similar measures taken. åtgärd vidtagas.

The laws mentioned above are just examples of Swedish laws that aﬀects legal investigations of cloud environments.

2.6 Similar work Work similar to the work in this thesis has been done by Sameera Almulla, Youssef Iraqi & Andrew Jones and can be read in their paper "Digital Forensic of a Cloud Based Snapshot" [29]. Their research goals were to investigate the snapshot functionality of the Xen hypervisor, possibilities to use the snapshot during a forensic investigation and if it can be done without reconstructing the originating environment. Their proposed solution was to create a snapshot of a VM using the snapshot function in the Xen hypervisor. A copy-on-write image was created by the operation, which was analysed using the open-source forensic tool DFF [28]. They found that during an analysis of the snapshot, ﬁles deleted using the graphical interface, command line and shredding tools could be successfully retrieved. They concluded that the copy-on-write snapshot generated by the Xen hypervisor could possibly be used as a method of extracting and preserving evidence in a forensically sound manner. They also concluded that the original environment was not required.

3 METHOD

3.1 Tests of Qcow disk image We will attempt to answer our ﬁrst research question: "Is it possible to use the snapshot functionality in OpenStack as an alternative to a forensic disk clone?" by using an experimental method. A series of experiments will be conducted on both a snapshot generated by OpenStack of a VM and on a forensic disk clone of a physical server. The two systems will be installed to mimic each other as much as possible. The outcome of the experiments conducted on the two systems will be compared.

The experimental method was chosen because we are foremost interested in how a snapshot generated by OpenStack resembles a forensic disk clone. This method allows us to create a hypothesis based on information learned during pre-study. Then a series of experiments can be conducted which simulates real-world scenarios and compare the results. The comparison should clearly show if a snapshot could be used as an alternative to a forensic disk clone.

A literature study could have been conducted to be able to conclude an answer to our research question. However, we felt that the available documentation and research is inadequate to draw a conclusion of the usefulness of the Qcow2 format. Also, the nature of conducting experiments on the actual disk clones resembles actual work that could be made during a forensic investigation. We would have preferred to some kind of a simulation technique to compare the usefulness of the Qcow2 format. A simulator would have allowed us to test various scenarios with just minor diﬀerences and execute those multiple times to verify the result. However, we were unable to ﬁnd a simulator which could get the job done.

Another way to extract ﬁles from a cloud would be to use introspection. That is a technique where it is possible to extract data directly from the VM by injection into RAM and run the program from there. This technique is not fully developed for KVM, and it would also fall outside of the scope for this paper.

All test are performed on DevStack running Cinder v. 2.0.1, Glance v. 2.6.0, Neutron v. 6.1.0, Nova v. 7.1.0 and Keystone v. 3.8. Each test will create its own instance so as to not affect the result. Before the tests, the OS_ variables will be set to allow the playbook to connect to DevStack and create an instance. export OS_USERNAME=admin export OS_PASSWORD=z2o3xjE4eN6PSLux963e export OS_AUTH_URL= h t t p : / / 1 0 . 1 . 0 . 3 7 : 5 0 0 0 / export OS_PROJECT_NAME=demo All tests will be repeated on a physical machine running Ubuntu Server 16.04 (ext4 as the file system). On this device, a forensic extraction will be made using the dd command in Linux. Between each test, the disk will be overwritten with a file created by dd before any test were performed.

3.1.1 Scenario 1: Find a file on a snapshot This test aims to find an existing file on a Qcow disk. Hypothesis: The file will be found on the system.

21 1. Createtestenvironment -Inthecloud:CreateanUbuntuServer16.04instanceand injectanSSH-keyforeasyaccess. Onthephysicalmachine:CreateacleaninstallationofUbuntuServer16.04onthedisk. 2. ConnecttotheVM(Cloudonly) -UseSSHtoconnecttotheinstanceusingSSH-key. 3. Createaﬁleandwritetodisk -Createaﬁlewithsomeuniquesearchablecontent. echo "Thisis myfile"> myFile sync

Thesynccommandisusedtoensurethatthefileiswrittentodisk.Withoutthis,thereisa riskthefilemightresideinRAMandnotonthedisk.Thesnapshotfunctionshoulddo thisbuttobesurethisisdonemanually. 4. Createanimageoftheinstance -Inthecloud:CreateasnapshotoftheVMusing Horizon. Onthephysicalmachine:Createanimageofthediskusing dd . 5. Findthefileonthediskimage -Catthefileandusegreptofindthefilecontent. cat | grep a "Thisis myfile"

The"-a"ﬂagtellsgreptotreattheoutputasASCII.

3.1.2 Scenario2:Findadeletedfileonasnapshot ThistestaimstoshowthatafilethathasbeendeletedcanstillbefoundonaQcowdisk. Hypothesis: Thefilecontentwillbefoundevenifthefileisdeleted.

1. Repeatstep1-3fromTest1,subsection3.1.1 2. Deletetheﬁle -Removetheﬁlefromthedisk. rm myFile sync

3. Createanimageoftheinstance -Inthecloud:CreateasnapshotoftheVMusing Horizon. Onthephysicalmachine:Createanimageofthediskusing dd . 4. Findthefileonthediskimage -Catthefileandusegreptofindthefilecontent. cat | grep a "Thisis myfile"

3.1.3 Scenario3:Findfileswiththesamename Thistestisdonetoseeifwecanfindtheoldcontentofafileorifitwillbeoverwrittenwithnew content. Hypothesis: Webelievewewillbeabletofindbothcontentsofthefile.

1. Repeatstep1-3fromTest1,subsection3.1.1 2. Deletetheﬁle -

22 rm myFile sync

3. Createanewﬁlewiththesamename -Usethesamenamesincewewanttocheckitwe stillgetseetheoldcontentonthedisk. echo "Newcontentsamefilename"> myFile sync

4. Createanimageoftheinstance -Inthecloud:CreateasnapshotoftheVMusing Horizon. Onthephysicalmachine:Createanimageofthediskusing dd . 5. Findthefileonthediskimage -Catthefileandusegreptofindthefilecontent. cat | grep a "Newcontentsame filename"

6. Findthecontentoftheoldfile -Catthefileandusegreptofindthefileoldcontent. cat | grep a "Thisis myfile"

3.1.4 Scenario4:Findoriginalfile-contentfromachangedfile Thistestaimstocheckwhathappenswithafileifthecontentisupdated.Iftheoldcontentwill stillbeaccessible. Hypothesis: Bothcontentswillbefoundonthedisk.

1. Repeatstep1-2fromTest1,subsection3.1.1 2. Createaﬁleandwritetodisk -Createaﬁleinsidethehomefolder.Writesomecontent thatiseasytosearchfor. echo "Thisistheoriginalcontent"> myFile sync

3. Changethecontentofthefile -Sincewealreadyhaveafileopenthefileandchangethe content.Weareusingatexteditorthatdoesnotcreateanewfilewhenwritingtodisk.We usednanoforthis,vimcreatesanewfilewhichchangestheinode. nano myFile

Changethecontentofthefileto"Thisisthemodifiedcontent" 4. Createanimageoftheinstance -Inthecloud:CreateasnapshotoftheVMusing Horizon. Onthephysicalmachine:Createanimageofthediskusing dd . 5. Findthefileonthediskimage -Catthefileandusegreptofindthefilecontent. cat | grep a "Thisisthe modified content"

23 6. Findtheoldcontentonthesnapshotdiskimage -Cattheﬁleandusegreptoﬁndthe oldcontent. cat | grep a "Thisistheoriginal content"

3.1.5 Scenario5:Findadeletedfileonafilleddisk Thistestwillcheckwhathappensifthewholediskisused.Thusthedeletedfilehasbeen overwritten. Hypothesis: Thecontentwillnotbefound.

1. Repeatstep1-3fromTest1,subsection3.1.1 2. Deletetheﬁle - rm myFile sync

3. Fillthediskwithrandomdata -Createafilethatisbigenoughtofillthecontentofthe disk.Thisshouldoverwriteallblocksthatarenotinuse,thuspreventingusfromfinding thedeletedfile. dd if =/dev/urandom of=./file bs=4kiflag=fullblock, count_bytescount=15G

3.2 Algorithmforautomatedextraction Wewilluseacomparativemethodtodeﬁneanalgorithmwhichcouldbeusedtoextractpotential evidenceautomatically.Thealgorithmwillbedeﬁnedbystudyingexistingrecognisedmethods andtryingtoadaptthosetothecloud. Ourproposedalgorithmwillbecomparedanddiscussedagainsttheexistingmethods.

Thealgorithmshouldpreferablybeusedinanactualcaseandtestedincourtasthatisthe ﬁnalevaluation.Unfortunately,thisisnotpossible,asithastobeestablishedandrevisedbefore beingappliedtoarealcase.Inaddition,acasecouldspanoverseveralyears,whichisnot feasibleinthispaper.

Analternativemethodofestablishingthealgorithmcouldbetoconductinterviewswith peopleinvolvedinactualcases(prosecutors,laymen,forensicworkers,etc.).Questionslike "Whatinvalidatesevidences?"and"Whatshouldbeconsideredwhencollectingevidence?" shouldbeasked.Weoptednottotakethisapproach,asitwouldrequireagreatcommitment ofthepeoplebeinginterviewedandalargeamountofparticipants,whichwouldbediﬃcultto

24 get hold of. We have however discussed the algorithm with forensic analysts that works for the Swedish police and the Swedish tax Agency. Both parties agree that the algorithm could be tested in a court case.

Our algorithm will be run and tested on DevStack. DevStack is a test environment that is easy to setup and runs OpenStack with QEMU. This machine has been running in City Cloud, City Networks public cloud. City Cloud is running on KVM. This should not have any eﬀect on our solution. The test environment is running Ubuntu Server 16.04, 16 GB ram, 4 cores and 100 GB of storage.

DevStack is installing all necessary OpenStack components to be able to run a cloud. The components are Nova, Glance, Cinder, Neutron and horizon. Versions of these components have changed during the project since we are using the master branch of the DevStack git to make the installation. Several installations have been done because DevStack has a tendency to crash.

The algorithm proposed will be tested against a scenario where an arbitrary virtual machine has been infected or is acting strangely. This strange behaviour would be recognised by an Intrusion Detection System (IDS) which executes an Ansible playbook, containing our algorithm. This scenario will be tested by creating a virtual machine which will be considered a regular VM running in the cloud. A ﬁle which could be of interest is created (which is the ﬁle to be retrieved automatically).

3.2.1 Choice of automation tools We are using Ansible as our automation tool. This is because Ansible is a clientless solution which only requires SSH to access the clients. Most of the modules are written in Python and Python is required on the machine the modules are running on. Ansible transfers modules to the targeted clients and executes them in the speciﬁed order. This should not impose a problem since Unix distributions generally have Python installed.

3.2.1.1 Raw commands Ansible oﬀers the option to use raw commands directly via SSH to a client. When a module is used Ansible uploads the module into the /var/tmp directory from which it runs the module. A majority of the modules require Python to run. The raw options allow Ansible to install Python, or any other software as if the command was written in the command line interface. In a forensic analysis, it is important to keep the alteration of the machine under investigation to a minimum. If a normal module is used, Ansible will write data to the disk which most certainly will alter the system and potentially destroy evidence. Ansibles raw module solves this for us. Since we are not uploading any ﬁles to the machine, we will keep the system alteration to a minimum. The raw option allows us to run Unix commands to monitor volatile data of the system. Commands we opted to run are:

1. netstat -anp - Netstat can be used to monitor the networking subsystem, this means active connections, routing table, interfaces, etc. The "-a" option tells netstat to output both listening and non-listening sockets. "-n" prevents netstat from looking up the symbolic host but prints the IP addresses instead. The "-p"-flag tells netstat to included the Process identifier (PID) of the program owning each socket. 2. netstat -rn - The "-r"-flag is used to print the kernel routing table.

25 3. w -i - The w-program shows all logged on users on the system and what they are currently doing. "-i"-flag is used to print IP addresses of each user instead of hostnames. 4. ps -aux - The ps-command outputs information about running processes on the system. "-ax" generally prints all processes. "-u" prints processes of users on the system. 5. lsof -n - lsof stands for "list open files" and does just that. "-n" tells lsof to print IP addresses instead of hostnames. 6. ifconfig - This command outputs information about network interfaces on the system. 7. date -R - Prints the date of the system. The "-R"-flag forces the output to be formatted according to RFC 2822. An example of RFC 2822 format is: Tue, 18 Apr 2017 10:01:37 +0200

These commands follows the standard on which order the information should be gathered according to the recommendations stated in section 2.2.3.2. The list could easily be expanded to include additional programs and procedures.

3.2.1.2 Gather_facts module This module is automatically run on the machine before any other modules are run. Since it is a forensic analysis, all additional modules could alter the system. For the same reason as for the raw command, this is not desirable and thus disabled.

3.2.1.3 Ignore_errors option If Ansible goes into an error state, the whole playbook will stop its execution. In some of the plays, a module failure is not considered a complete failure of the execution and the playbook should continue with the next task. An example is the raw command "which lsof", which will return the path of the program lsof if the program is installed. A failure in this task will only notify the play that the program is not installed, not that something went wrong with the play, and the playbook should continue.

3.2.2 Snapshot solution The proposed algorithm will be using the snapshot function in OpenStack to create a stand-alone Qcow image of an instance. The overlay disk of the instance will be merged together with the backing ﬁle. This is the best solution when using OpenStack. Another solution would be to use Ansible to connect to the control node hosting the instance of interest and create a direct copy of the overlay disk. The problem with this alternative is that we do not know which node the instance is running on, also copying a disk of a running system might result in a corrupt copy. To be able to connect to the hosting node the IDS system needs to have access to each and every node available in OpenStack. For simplicity, it is easier to use functionality exposed via OpenStack. Functionality in OpenStack also ensures that the state of the targeted instance is unchanged to ensure the created the snapshot is consistent.

3.3 Analysis of Qcow using forensic tools The forensic tools used is The Sleuth Kit. A collection of command line tools together with a C library. These tools allow a user to analyse a disk image. The Sleuth Kit is the backbone in Autopsy and other open source forensic tools. The web interface for Autopsy was used during the investigation.

26 Start the web interface and create a new case. Open the snapshot image and browse the file system. The deleted file names can be found in the folder together with the inode for the file. The Autopsy web interface were not able to the find the content of the file. Since we know the content of the file, we used the Linux command grep to find it directly in the snapshot. We were able the find the content which shows that another forensic tool can find the content of a deleted file in snapshot when Qcow disks are used.

3.4 Prove non-repudiation of snapshots The integrity of the created snapshots will be proved by using hashing (one-way encryption). Hashing should be sufficient since any changes made to the snapshot will result in a different hash value. We will study the workflow of the snapshot functionality and compare different stages where the hashing could be performed.

We will attempt to ﬁnd methods of proving the origin of the snapshot (authenticity) by studying unique data found in VMs. Available unique data will be located by analysing the ﬂow at VM launch.

4 RESULTS

4.1 Testresults WhenexecutingthetestsonaphysicalharddriveandaQcowdisktheresultisthesame.Ifthe copywasdonewithadedicatedhardware,aso-calledphysicalcopy,itmighthavebeenpossible toextractﬁlesthathasbeenoverwritten.Duetoeconomicconstraints,thiswasnotpossibleto do. Theresultsofthetestscanbeseenbelow:

4.1.1 Scenario1 Test1showsthataﬁlecreatedonaninstancecanbefoundonthesnapshotdiskandonalogical copyfromaphysicaldisk.Thisisexactlyasitshouldbe.Ifthisdidnotwork,thesnapshot functionwouldbeuseless.Thistestalsoveriﬁesthegeneralmethodweareusingtoperform thesetests.Thehypothesiswascorrect.

ubuntu@server:/tmp/tests$ cat Test_VM_1_Snap| grep a "Thisis myfile" Thisis myfile echo "Thisis myfile"> myFile

4.1.2 Scenario2 Thistestshowsthatadeletedfilemightberecoveredfromsnapshotsandlogicalcopies.Deleted filescanbefoundinsnapshotsbecauseduringthemergeofthesnapshottheoverlayfileis writingeveryblockthatcontainsdataintothenewQcow.WhenthenewQcowiswritten,the hypervisordoesnotwastetimebysortingwhichblocksarereallyinuse.Therefore,allblocks arecopied.AsfaraswecanseethebehaviourremindsonhowanSSDdiskisused.Itwill continuetouseallblocksbeforeitstartstoreusedeletedblocks.Theoverlaycantherefore be20GBonthehost,buttheVMisonlyallocating5GBofdata.Therestisolddatathat hasnotyetbeenoverwritten.TherearecommandsinQEMUthatcompromisesthesedisks, butthosearenotruninthesnapshotfunction.Thefilecouldbefoundonthelogicalcopy becauseitremainsonthediskuntilithasbeenoverwritten.Thehypothesisforthistestwascorrect.

ubuntu@server:/tmp/tests$ cat Test_VM_2_Snap| grep a "Thisfile hasbeendeleted" Thisfile hasbeendeleted

4.1.3 Scenario3 ThistestdidnotresultinaﬁndingoftheﬁleonaQcowimageoronalogicaldiskcopy.We haveretestedthismultipletimesandendedupwiththesameresult.Webelievethelocationis reusedonthephysicaldiskscenario,butweareunabletoexplainthebehaviourinQcowdisks.

ubuntu@server:/tmp/tests$ cat Test_VM_3_Snap| grep a "Thisistheoriginalfile" echo "Thisistheoriginalfile"> myFile

29 4.1.4 Scenario4 Whenafileismodifiedwewereabletofindboththeoldandthenewcontent.Thisindicates thatchangesmadetoafileisstoredinanewblock.Whenexecutingthetest,wemodifiedthe existingcontent.Thehypothesiswascorrect.

ubuntu@server:/tmp/tests$ cat Test_VM_4_Snap| grep a "Thisistheoriginalcontent" Thisistheoriginalcontent ubuntu@server:/tmp/tests$ cat Test_VM_4_Snap| grep a "Thisisthe modifiedcontent" Thisisthe modifiedcontent

4.1.5 Scenario5 Theonlythingthatindicatesthatthisfileeverexistedisthatthecommandusedtocreatethe fileissavedinthe~/.bash_historyfile.Asweexpecteditisnotpossibletofindanytraceofthe contentofthefilesincethewholediskhasbeenfilledwithrandomdata.Ifwewereabletotake thephysicaldiskitmightbepossibleextractthedatabuttotestthisaspecialhardwareisrequired. Aproblemmightalsobethatwewanttoextractdatathatispartofafile,notawholefile.We arenottryingtherecoverthefilefromthediskitself,wealreadyhavetheQcowimage.Itwould takeaprogramtorecogniseafilesysteminsideafilesystem,andcarveinthesecondfilesystem. Insidethesecondfilesystemwehaveafilethatisoverwrittenthatneedstobecarved.Thesame resultoccurredwhenusingalogicaldiskcopyoftheharddrive.Hypothesisforthetestwascorrect.

ubuntu@server:/tmp/tests$sudo cat Test_VM_5_Snap| grep a "Thisissomeoriginalcontent" echo "Thisissomeoriginalcontent"> myFile

4.2 Proposedalgorithmforautomatedextraction Ourinvestigationsofexistingrecognisedmethodsresultedinthealgorithmdescribedbelow.It hasbeenimplementedwithAnsibleandtestedagainstourtestenvironmentasdescribed.The outcomeoftheplaybookisalogﬁle,whichdescribesallactionstakenonthesuspicioussystem, thetargetedﬁlewashashedanddownloadedtoacontrolledmachineandacopyofthesnapshot wasdownloadedforsafe-keeping.

1. Checktime -Thefirststepistocheckthetimeonthelocalmachine,thiswillbethetime ontheIDSsystem.Thistimeisimportantsincewecannottrustthetimeonthemachine wearesnapshotting. 2. Createaforensicmachine -Thisstepcouldbeskippediftheanalysisisdoneondifferent computer.Thismachineispre-installedwithalltoolsneededfortheanalysis. 3. Configureforensicmachine -Checkifalltheprogramsareinstalled,ifnotinstallthem. 4. Collectvolatiledata -IfaccessisgrantedtothesuspiciousmachineviaaSSHkeythen connecttothemachineandrunLinuxcommandstogetinformationaboutthesystem.All thesecommandsarerunwiththerawmoduletominimizethealterationoftheaccessed system.

30 5. Snapshot - When all volatile data has been collected the machine is snapshotted. This will create an exact copy of the disk at this exact moment. A hash of the snapshot will be generated by OpenStack upon creation. 6. Download the snapshot - Since we can not rely on that the snapshot disk image will be untouched in the cloud it is downloaded for safe keeping. 7. Create a volume of the snapshot - This is done to be able to read the disk from the forensic machine. Since OpenStack does not support direct attachment of images it is necessary to create a volume of it. 8. Attach and mount the volume in read-only mode - To ensure that we do not alter any data on the volume we mount it in read-only mode. In a forensic examination in a physical environment the disk would be connected to a hardware writeblocker, in our case that can not be done. 9. Extract interesting files - If any files we know contain important information then the playbook extracts them. These files have to be specified before the playbook is executed. The files are hashed on the targeted machine, copied to a controlled machine using SCP and hashed again afterwards to verify its correctness.

4.2.1 Ansible logfile output When the playbook is run all actions will be logged to a logfile. This logfile contains timestamps, actions, user used during SSH, IP used for SSH and more. This file is necessary to prove what actions have been taken and when. If the playbook is started manually it is important to log who started it, what time, why it started and so on. This is outside our scope but has to be considered.

4.3 Findings in Qcow snapshot When analysing the snapshot file we mainly used two programs. The first one was The Sleuth Kit. In this program we were able to see deleted files when browsing the file system. The file name is probably stored in the "." file that exists inside each folder in Unix systems. The files are flagged as deleted but the file name can still be read. In some cases this might be enough evidence to convict someone. The Sleuth Kit was not able to reconstruct the file content but that might be because we were using the free version. A forensic tool with more features will most likely find the content of the file and map it to the filename. The Sleuth Kit was able to find the inode number of the deleted file, which indicates that if we followed it we might find the file content.

The second tool we used was the grep command. Grep searches for a pattern in a file and prints matches. We used this to search for the known content of the deleted file. The tests showed that the content of a deleted files exists on the snapshot. This indicates that the Qcow disk image could be treated much like a physical disk. Another forensic program would be needed to be able to analyse Qcow disk image fully. A recommendation would be to use EnCase Forensic [20] or Forensic Toolkit (FTK) [21]. Both of those are widely used programs and are used by the Swedish authorities. Both of them require a licence key to use, which we lack and can therefore not do the analysis by ourselves.

4.4 Proving non-repudiation of snapshots Inside each instance launched in an OpenStack environment is a directory, /var/lib/cloud/, that contains the Universally Unique Identiﬁer (UUID) of the instance and all start-up scripts

31 Figure 4.1: The logﬁle generated by the ansible playbook execution. used by OpenStack. This ID is unique for all instances launched in the same cloud, and most likely between other OpenStack environments as well [19]. It is generated during instance creation by OpenStack. This ID is what OpenStack uses to separate instances from each other. This ID can not be changed by the native OpenStack commands, it would be possible to change this with admin rights on Nova and full access to the database, something a normal user will not have. Therefore it is impossible for an end user in OpenStack to change this.

Figure 4.2: The folder /var/lib/cloud holds a lot of information that could be of use during an investigation.

The UUID is the ﬁrst thing that would help to tie a snapshot to an instance. Since OpenStack

32 have all the IDs stored in a database it is possible to map the UUID found in a snapshot to an instance-ID. If there is a match between those IDs then the snapshot is probably of that instance. One problem is that the user has full access to files and directories located on their instance, which means that a user can change or delete them. As shown below it is possible to see deleted files in a snapshot and even old content of a file as long as that part of the disk is not overwritten. A user can, if he knows about these files, change all the IDs inside this folder and fill up the disk with random data to overwrite the old content. If this is done before the snapshot is taken then the connection between the UUID in the snapshot and the ID stored in OpenStack is lost.

When a snapshot is taken of an instance then the ID of that instance is stored in a database, to keep track of which instance the snapshot originated from. This data can be changed by the end user but the OpenStack project Cinder will log which instance was snapshotted and when. The snapshot will get its own UUID when created. This means that with access to the Cinder log it is possible to map the snapshot to the instance even if the end user changes the data of the snapshot.

This is the result we found during our studies of the snapshot functionality and VMs in an OpenStack environment. Alternative methods are discussed later as we found them to not be applicable in the environment as is, but could potentially improve the possibility to prove non-repudiation.

5 DISCUSSION

5.1 Proposed method The proposed method used when extracting data from the cloud is the following: 1. Check time - The first step is to check the time on the local machine, this will be the time on the IDS system. This time is important since we can not trust the time on the machine we are snapshotting. It is done using the date command in Linux. Since the playbook will be started automatically by an IDS we assume the time on the local machine is correct. If this time is not correct the whole IDS will have problem, since all times would be wrong. 2. Authenticate - Before any commands to OpenStack can be used an auth-token needs to be acquired. Doing a request to Keystone with the variables username, password and project_name does this. This request is done by using the program shade and is done by the local machine, the IDS. This task will return two variables, service_catalog and auth_token. Sevice_catalog contains all endpoints we have access to with the auth_token. The auth_token is like any other token, instead of using username and password from now on we will use the token. 3. Create a forensic machine - This step could be skipped if the analysis is done on different computer. This machine is pre-installed with all tools needed for the analysis. This is done by sending a request to OpenStack using the token. The task is using shade to make the request. It also creates and injects an SSH-key into the instance. 4. Configure forensic machine - Check if all the programs are installed, if not install them. The SSH-key is required for Ansible to connect the instance. 5. Collect volatile data - If access is granted to the suspicious instance via a SSH-key then connect to it and run Linux commands to get information about the system. All these commands are run with the raw module to ensure Ansible does not download anything to the instance, which could alter the disk. As discussed before anything that writes data to disk could erase content of a deleted file. As long as we only do the raw command and do not gather facts about the instance we should keep the impact to a minimum. 6. Snapshot - When all volatile data is collected the instance is snapshotted. This will create an exact copy of the disk at this exact moment. We are using the token to send a request to Nova in OpenStack, which takes the snapshot for us. The name of the snapshot will consist of the time gathered at step one and the ID of the instance we are snapshotting. The task will return the ID of the snapshot, among other data of the snapshot. 7. Download the snapshot - Since we can not rely on that the snapshot disk image will be untouched in the cloud it is downloaded for safe keeping. This is done by a request to Glance. OpenStack is treating the snapshots as normal images and we can download it as any other image. The image is downloaded to the IDS at the moment. If deployed the snapshot should be moved directly to a separate storage for safe keeping. 8. Create a volume of the snapshot - This is done to be able to read the disk from the forensic instance. Since OpenStack does not support direct attachment of images it is necessary to create a volume of it. The volume will contain all data from the image. Doing this step will create an easy way to browse the file system for files. Finding deleted files in is this volume is not recommended. For finding deleted files use a copy of the original image and mount it using a forensic tool. The browsing option is good for finding evidence that are not hidden, deleted or encrypted. To create a volume we are doing a request to Cinder, using the image as a base. This will create a LVM volume and not a Qcow.

35 9. Attach and mount the volume in read-only mode - To ensure that we do not alter any data on the volume we mount it in read-only mode. In a forensic examination in a physical environment the disk would be connected to a hardware writeblocker, in our case that can not be done. We are making two requests for this. Both the requests are sent to Nova. The ﬁrst one is to attach the volume to the instance and the second one is to mount the volume. As for now the mount point will be at /mnt/. Since we just created the forensic instance no volume is currently attach and we can mount on that point.

10. Extract interesting files - If any files we know contain important information then the program extracts them. These files have to be specified before the program starts.

5.2 ISO 27037 The proposed automated algorithm in this thesis is believed to fulﬁl the four general requirements for digital evidence handling described in the ISO 27037 standard.

• Auditability - is fulﬁlled because the algorithm is automatically executed using Ansible. The logs generated by Ansible contains detailed logging of each activity performed during the acquisition of the snapshot and ﬁles. This means that any independent party should be able to evaluate each action.

• Repeatability - is partially fulﬁlled. The measurement procedure, method and instrument can easily be used due to the automation of the algorithm. However the algorithm cannot be reapplied to the same system at any given time and still produce the same result because it is applied to a running system.

• Reproducibility - might be fulﬁlled depending on the instrument. The proposed solution relies heavily on the Qcow format and the snapshot creation performed by QEMU. This means that the snapshot functionality will determine the output. On a more general level, the algorithm should be applicable to alternative environments that has implemented a snapshot functionality. Like the repeatability requirement, the algorithm is applied to a running system making it diﬃcult to reproduce the test at a later time.

• Justiﬁability - of the proposed algorithm should be possible because a minimal amount of actions is made when taking and managing the snapshot.

5.3 Why we chose Ansible We conducted a simple study of Puppet, Ansible, Chef and Salt where we looked at the obvious differences between the four. We also checked the relation between the options and OpenStack. What we found was that all of them can by used to communicate with OpenStack (either via existing modules or via the OpenStack API). Ansible is written in Python and so is OpenStack which is a good fit. One key benefit of Ansible for our proposed solution is that Ansible uses a clientless solution when communicating with the nodes. This is preferable to us because this allows our solution to provide the option to connect directly to VMs of interest while minimising the modification of the system (compared to the requirement of a client software). Another upside of choosing Ansible is that it is the automated operation tool of choice for City Network Hosting AB.

36 5.4 OpenSSH or Paramiko Ansible have to option to use SSH or Paramiko. Paramiko is a Python implementation of SSHv2 protocol, can be used as both client and server. Our playbook is only using OpenSSH. The reason is that we can not assume all hosts will have python installed on them and using OpenSSH will then be the best choice. When connecting to a suspicious host we also want to use the raw command option and this option cause least amount of problem when using OpenSSH.

5.5 Test results The first two test results were exactly as we thought. A file should be on the snapshot if not deleted and the deleted content should be found because of the Qcow properties. The third test was a failure for us. The hypothesis for this test was that we should be able to find the old content of the file. This because Qcow saves changes of files and not necessarily overwrites the file on disk, as a normal hard drive would do. This might be caused by how Unix system treats the inode. When a file is created it is designated an inode. When the file is deleted the inode is flagged as free in the inode table. If the next file created have the same file name the same inode will be used again. If another file name is chosen this file will have the next inode. This might suggest that the file system is reusing the same disk space thus overwriting the old content. This seems not the be the same in a Qcow. Even when another file is created between the removing of the file and creation of a file with the same name the original content seems to be overwritten.

Since the fourth test was successful and we were able to find both content of the file all theories on how Qcow handles deleted files with same names does not make sense. If the part of the disk is overwritten when a modification is made we should not be able to find the old content on the disk in the fourth test. If the next file created overwrites parts of the file we should not be able to find content of deleted files. In test five we created a file that almost took the whole disk and were still able to find the deleted file, indicating that this large file occupies the next block after the deleted file. When continue to write to disk the deleted file were overwritten.

When comparing the result to the to the ones S. Almulla and Y. Jones got in their investigation [29] we see that the Qcow will act almost the same way as a cow disk. Both tests show that a deleted file could be recovered. The main difference is that we were unable to recover a shredded file. This could be due to lack of good forensic tools. In this paper we also tested if old file content could be found after the file was changed. A similar test was not performed by the others and could be interesting.

When comparing the results from the Qcow and the logical disk copy of the hard drive we got the same result on all tests. This shows that the Qcow could be used as an alternative to a logical disk copy. If possible it is always better to take the whole disk because it is then possible to retrieve data that has been overwritten.

5.6 Identifying a virtual machine In OpenStack every instance, user, snapshot, tenant, etc are assigned its own unique ID, a UUID. This ID is used by OpenStack to identify everything and most commands require the user to send the ID with the command for i to execute, in some cases the name can be used. Names is not unique in OpenStack, a user can have several instances with the same name, and can therefore cause some commands to fail due to several matches when names are used

37 as identiﬁer. As said before the UUID is unique and is therefore used in OpenStack as the identiﬁer.

Every virtual machine gets a UUID generated and is stored in the OpenStack database. In the database it is mapped to a name and a number that identiﬁes the instance in the hypervisor. The database also stores the user that created the instance and which tenant, also called project, it belongs to. With this it is possible for OpenStack to identify an instance when required. This is usually enough to identify an instance inside of OpenStack. The problem with a snapshot is that we can remove if from the OpenStack environment. For example we are downloading the image, and doing the analysis outside of OpenStack.

The logfile generated by Ansible could be used to strengthen the claim that a snapshot is from a specific instance. Together with the ID inside the snapshot this could be enough to prove that the snapshot originates from a specific instance. If the instance ID inside the machine is changed it is possible to use the native OpenStack logs to cross-reference the timestamps. Since our playbook is started from an IDS the time on the IDS and the OpenStack system should be the same. Therefore a cross-reference is possible.

5.6.1 Inject a unique file As described earlier all instances in OpenStack gets its own UUID, which can be used to uniquely identify each machine and a possible connection to the owner. However, this ID may or may not be possible to use to reliably tie the machine to an owner (due to potential modifications). An option to this ID would be to inject a unique file into each new instance at creation. This file could be the UUID of the launched machine, but encrypted. Any attempts to modify this file would result in a corrupted encryption file (which could be an indication of malicious activities). This solution would force the CSP to store and manage the encryption keys for each machine running within their cloud.

This ﬁle should be injected into the instance in a folder that would not be changed by mistake. Injecting the ﬁle into the home directory would be a bad idea since it is visible to the user and could be deleted by mistake. A better path would be in /bin/ or equivalent.

5.7 Using backing and overlay files An alternative way of extracting data would be to take the backing file and the overlay file separately. This requires modifications inside the overlay, path to backing needs to be changed, or that the backing file is located with the exact same path as it had before. If the overlay file is read and the backing file is not found it would result in an error, when trying to mount it, or we will only get the modifications of files created of files that were on the backing file. All files created layer will be on the overlay.

For convenient reasons it is better to use the snapshot function. The function will merge the two files into one. The merge file is technically not the same file as the disk, since the snapshot created it, it is impossible to hash it and say that the files are the same. It would be possible mount both the overlay and the snapshot and hash every file on the system and that way proves they contain the same information. This will not work when the instance is running. The running instance will constantly change the overlay. In a shutdown state it would be possible but if the instance is shutdown it cannot trigger the IDS to start the extraction.

38 If the instance is running during the snapshot the snapshot function will force the instance to write all data, that is currently in memory but should be written to disk. If only the overlay is taken it is a risk that some data is not yet written to disk and therefore missed. For this reason it is more important to get all data than keep the exact integrity with two files, as long as the files inside the overlay and backing file is the same as on the snapshot. The integrity on the files inside the virtual machine must be kept.

5.8 CLI history ﬁle Unix-like systems stores commands entered in a terminal or terminal emulator by default in a ".bash_history"-ﬁle. This will also be the case of this kind of systems running in a cloud environment. This means that it is also possible during a forensic investigation to analyse any commands executed on a system (much like traditional digital forensics). Two settings concerning the bash history management is HISTSIZE and HISTFILESIZE. The HISTSIZE determines how many commands entered should be stored in memory. The HISTFILESIZE determines how many commands that should be saved to disk (e.g. to "~/.bash_history"). One interesting part of this and VMs running in an OpenStack environment is that the machines uses Qcow as the back-end storage. E.g. all changed will be written to disk until the maximum capacity of the VM has been reached. This means that it can be possible to extract more of the ".bash_history" than can be found in the actual machine.

5.9 Virtual Introspection Our proposed solution struggles when it comes to collecting volatile data of a VM. One new area of cloud analysis is taking shape which is called "Virtual introspection". Virtual introspection is a technique of inspecting the content of a VM outside the VMs own environment. This technique could be used during a digital forensic investigation of a VM running in the cloud. Main advantage of this technique in conjunction with digital forensics is the possibility to read and analyse the memory of a running VM, either from a Virtual Machine Monitor or from a privileged VM. An introspection analysisuhm virtu could then theoretically be performed on a live target system without the risk of system modiﬁcations or detection [24].

Virtual introspection is still a new, evolving concept which needs additional work and testing before being applicable in a digital forensics case. The technique seems promising and should oﬀer some advantages of live forensics in a cloud environment compared to an investigation of a non-virtualized system.

5.10 Ethics The algorithm proposed in this paper has been designed with the CSP in mind. This means that it could be applied by the CSP at anytime. This can most certainly be discussed if it is ethically correct or not, because the algorithm can be used to create a point in time snapshot of any VM currently running in the cloud and stored elsewhere for analysis. It could be argued that this solution would be a breach of privacy, and it most certainly could be. As the proposed algorithm could be executed based on suspicious activity of a VM there is a risk that legit VMs could be snapshotted and investigated, violating the privacy. However methods of investigating VM running in the cloud is a necessity as criminals are exploring various cloud solutions. Privacy concerns could be addressed through service level agreements between the CSP and the VM owner.

39 5.11 Sustainable Development As stated before the methods used by the police today are out-dated when it comes to the cloud. A new method is required to extract the data. As the cloud usage is increasing it will most likely be part of every investigation in the future. If the data should be used in court the methods needs to be tested and evaluated, our algorithm is just the one proposal on how the extraction could be done.

The algorithm presented is developed to a cloud environment. Since most of the basic functionalists will be the same through all clouds the basic concept will be the same, take a snapshot and analyse it. When using a cloud a user will share the physical hardware with other users with will use less power than if every user had their own server. This will save the environment since less electricity will be needed.

Our solution will also allow a forensic team to extract data remotely. Today the police will go to the physical server and collect them to analyse. This require that at least one car is brought to collect the server. By doing the collection remote it negates the need for a car. It will also negate the need for a physical writeblocker since the extraction is not done locally.

40 6 CONCLUSIONS

6.1 Is it possible to use a snapshot as evidence? In Swedish court, there is a thing called free sifting of evidence. This means that everything can be used as evidence and therefore the snapshot can be used. The relevant question is if the snapshot contains enough information to be useful. According to the tests we made, we believe that a snapshot handled by OpenStack contains enough information to be useful. We were able to recover deleted ﬁles, both the content and the ﬁlename, and as far as we can see the end user cannot change the snapshot in any way. It is also possible to take the snapshot without altering the instance.

The snapshot will have the same problem as a physical disk regarding overwritten ﬁles, if not worse performance. On a physical disk it is possible to dig out data that has been overwritten up to eight times, sometimes it is possible to extract data from up to twelve writes. When doing this, the analyst is looking at the physical track on the disk. Every time the disk is written to, the track is either a bit higher or lower than before, this indicates the pattern the rack had before. Those traces, can be followed and old ﬁles can be carved. This method requires the physical disk and is thus impossible to do on a snapshot. This means that a physical disk has to be overwritten multiple times to not have any trace but a logical copy will only need to overwritten one time, as shown in our tests. Traces in slackspace will be the same between the two disks.

Is it possible to use the snapshot functionality in OpenStack as an alternative to a forensic disk clone? Yes, it is possible unless the logical disk have been shredded, then the physical disk is required.

If the OpenStack snapshot functionality is an alternative to a forensic disk clone, is it possible to automatically extract data from the snapshot in a forensically sound manner? Yes, we believe it is. An IDS could start in an automated tool could extract ﬁles that contains important information. Syslog would be a ﬁle that an analyst would always look at. To have a program extract them would be convenient.

6.2 Prove non-repudiation of a snapshot If only data from the snapshot is taken into account, this is not possible. The snapshot will contain the ID of the instance snapshotted, but this ID can be changed by the user. A user can then change this value into another instance to trick the analyst. The snapshot by itself has no way to show non-repudiation.

If the snapshot and the log from Ansible is available, it is easier to show that the snapshot is from a speciﬁc instance. The logs will contain the ID of both the snapshot and the instance. Since Ansible is using native API calls to OpenStack, it enforces the use of the instance ID. As stated before this ID is unique and can not be changed by the end user. With this in mind, it is possible to say the snapshot is from the instance with the ID within the playbook.

OpenStack have its own log on when a snapshot is taken. The timestamp from the Ansible log can be used as a reference on when the snapshot was taken. If access to the OpenStack log is accessible, together with the Ansible log and the snapshot, it is possible to prove beyond reasonable doubt that the snapshot is from a speciﬁc instance, unless the OpenStack environment

41 is compromised.

As discussed before the option to inject a file could be used to identify a VM. This require that the cloud provider saves the data in a database that the analyst have access to. It also requires that the file is injected during creating of the VM. This way of identifying a VM would be very effective. The user can change the content of the file, but that will prevent the provider from decrypt the file, indicating someone does not want the instance to be identified. The encryption prevents the user from changing the values to another instance ID. The instance value should be stored inside the encrypted file. It should also be possible to extract the old content of the file since QCOW is only storing changes. We proved that the old content could be extracted if the disk has not been overwritten.

Is it possible to prove the non-repudiation of the snapshot? With only the snapshot no. With logs from Ansible, it is debatable. With logs from Ansible and OpenStack then yes. With an injected ﬁle it is possible.

42 7 RECOMMENDATIONS AND FUTURE WORK

7.1 Implementation in OpenStack As for now, our solution is using OpenStack API to make calls to the hypervisor. To make a better soultion OpenStack should have the option to make a forensic copy. Our solution is using the admin user to make a snapshot. The snapshot will then be available for the user since it will belong to same project as the instance. OpenStack could have an option, for an admin user, to make a forensic copy. This copy will then not be available for the normal user. This will make the whole process more secure since the end user will not see or know that the snapshot have happened.

It is possible to make these changes by ourselves but time constraints required us to use the existing API calls. The call should make a snapshot of an instance, move the snapshot into another project only accessible by the admin and if called on remotely, download the created image.

7.2 Implementation on a hypervisor level The snapshot should be taken as closely as possible to the hypervisor. Making a direct connection would be the best option. As discussed earlier this would require an admin to know which node the instance is running on. It could be possible to connect control node and query OpenStack for that information and then with a third party program connect directly to the hypervisor and make a snapshot. Doing this a snapshot disk image will be created on the node, forcing us to have access to the storage in some way to be able to download the image.

In a forensic standpoint, this would be the best option since we can use as few programs as possible, keeping the risk of altering data to a minimum. The main problem would be that this is more complex than just do the API calls to OpenStack and let OpenStack handle the call.

Another advantage of accessing the hypervisor directly is that it is possible to access RAM of the virtual machine. This means that an admin user does not need a SSH-keys to extract volatile data [22]. By injecting code into the RAM of the virtual machine, it is possible to run all commands without the machine knowing the code is run [23]. This would be the best solution in a forensic analysis. This means that we could run code inside the machine, by injecting RAM into the machine and run the programs there. This would not alter the current ram since we are injecting our own and the machine will not know about the programs run in this new ram. This prevents malware or rootkits to detect it and altering the output from the commands.

7.3 Check additional hypervisors This paper has only looked into QEMU and KVM. When using KVM, all virtual machines will get a Qcow disk. Other hypervisors will use different type of disks and those needs to be examined individually. The process of extracting data would be similar to this paper. If static sized disk is created, it would most likely have the same characteristics as a physical disk. When checking other hypervisors, it would be interesting to investigate how a modified file is treated. Could the old content of a modified file be recovered when using a cow disk?

43 7.4 Additional file systems All test were made on a Unix system running ext4, the VM were also using ext4. The Qcow should work the same way in theory on a different file system, but tests need to be done to confirm this.

7.5 Automatic learning and extraction The algorithm presented in this thesis only touches on the subject of extracting a set of predefined files of a snapshotted system. Functionality regarding extraction of files could be further explored, perhaps including a system, which recognises files, which are traditionally of interest.

7.6 Test in court The focus of this thesis was to investigate the usefulness of Qcow disks compared to physical disks. As well as investigating the possibility to extract evidence from Qcow disks automatically. Unfortunately it is impossible to conclude if the method proposed in this thesis will hold up in court. The only option to verify this is to use the method proposed and test it in a real case.

7.7 Checkpoint snapshot When using KVM/QEMU it is possible to use a checkpoint instead of a snapshot. The diﬀerence is that the current state of the VM are snapshotted, this then includes the RAM. When doing this, it should be possible to extract the content from RAM directly from the snapshot ﬁle. This means that there are no need to connect to the VM and destroy potential evidence when collecting volatile data.

7.8 Snapshotting in containers This thesis only touched on the subject of snapshotting in the cloud orchestrator OpenStack. It could be useful to extend this investigation to snapshotting of container environments (such as Docker). This is because containers could be used to carry out malicious activities in the same way as VMs. We reckon that one of the obstacles of container environments are the rate of which containers are created and destroyed. Making the time window of securing the evidence small.

44 REFERENCES

[1] Kronqvist, S.Brott och digitala bevis, Tredje upplagan, (2013) [2] Casey, E. Digital Evidence and Computer Crime, Third Edition, (2011) [3] Mell, P. Grance, T. The NIST Definition of Cloud Computing, Special Publication, pp.800-145. September (2011) [4] Carretero, J. Blas, J.G, Introduction to cloud computing: platforms and solutions, February (2014) [5] Armbrust, M. Fox, A. Griffith, R. Joseph, A.D. Katz, R. Konwinski, A. Lee, G. Patterson, D. Rabkin, A. Stoica, I. Zaharia, M. Above the Clouds: A Berkeley View of Cloud Computing, February (2009) [6] Prince, J.D. Introduction to Cloud Computing, Journal of Electronic Resources in Medical Libraries, 8(4):449-458, (2011) [7] Oracle White Paper. OpenStack Cloud Infrastructure, June (2016) [8] Abdelrazik, A. Bunce, G. Cacciatore, K. Hui, K. Mahankali, S. Van Rooyen, F. Adding Speed and Agility to Virtualized Infrastructure with OpenStack, (2015) [9] Beernaert, L. Matos, M. VilaÃ§a, R. Oliveira, R. Automatic Elasticity in OpenStack, Proceedings of the Workshop on Secure and Dependable Middleware for Cloud Monitoring and Management 2012(2), December (2012) [10] Radez, D. OpenStack Essentials - Second Edition, (2016) [11] Jahankhani H. Hosseinian-Far, A. Challenges of Cloud Forensics (2017) [12] Features/LiveBlockMigration, (2016), http://wiki.qemu-project.org/Features/LiveBlockMigration [Last Accessed: 10 April 2017] [13] QCOW2 backing files & overlays, (2012), https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html [Last Accessed: 10 April 2017] [14] OpenStack Compute (Nova), (2017), https://github.com/openstack/Nova [Last Accessed: 10 April 2017] [15] Libvirt git, (2017) https://github.com/libvirt/libvirt [Last Accessed: 10 April 2017] [16] NTP, RFC 5905, https://tools.ietf.org/html/rfc5905 [Last Accessed: 10 April 2017] [17] Kernel Virtual Machine, https://www.linux-kvm.org/page/Main_Page [Last Accessed: 11 April 2017] [18] Libvirt Virtualization API, (2017), https://libvirt.org/ [Last Accessed: 11 April 2017] [19] A Universally Unique IDentifier (UUID) URN Namespace, 2005, http://www.ietf.org/rfc/rfc4122.txt [Last Accessed: 18 April 2017] [20] EnCase Forensic, https://www.guidancesoftware.com/encase-forensic [Last Accessed: 18 April 2017]

45 [21] Forensic Toolkit (FTK), http://accessdata.com/solutions/digital-forensics/forensic-toolkit-ftk [Last Accessed: 18 April 2017]

[22] Tobin, P. Kechadi, M-T. Virtual Machine Forensics by means of Introspection and Kernel Code Injection, (2014) [23] Conover, M. Chiueh, T. Code Injection From the Hypervisor: Removing the need for in-guest agents, https://www.blackhat.com/presentations/bh-usa-09/CONOVER/BHUSA09-Conover- SADEintoVM-SLIDES.pdf [Last Accessed: 20 April 2017]

[24] Hay, B. Nance, K. Forensics Examination of Volatile System Data Using Virtual Introspec- tion, April (2008) [25] ISO/IEC 27037:2012 Information technology - Security techniques - Guidelines for identiﬁ- cation, collection, acquisition and preservation of digital evidence, October (2016) [26] Justitiedepartementet DOM, L5 och Å1942:740 Rättegångsbalken [Last Accessed: 25 April 2017]

[27] Barrett, D. Kipper, G. Virtualization and Forensics: A Digital Forensic Investigator’s Guide to Virtual Environments, (2010) [28] DFF, http://digitalforensicsframework.blogspot.se/ [Last Accessed: 27 April 2017]

[29] Almulla, S. Iraqi, Y., Jones, A. Digital Forensic of a Cloud Based Snapshot, (2016)

46 Blekinge Institute of Technology, Campus Gräsvik, 371 79 Karlskrona, Sweden