Justifying Integrity Using a Virtual Machine Verifier

Justifying Integrity Using a Virtual Machine Veriﬁer

Joshua Schiffman, Thomas Moyer, Christopher Shal, Trent Jaeger, and Patrick McDaniel Systems and Internet Infrastructure Security Laboratory Computer Science and Engineering Department, Pennsylvania State University University Park, PA 16802 {jschiffm,tmmoyer,cshal,tjaeger,mcdaniel}@cse.psu.edu

Abstract—Emerging distributed computing architectures, as Xen and Linux, these integrity problems may occur in such as grid and cloud computing, depend on the high integrity either the computing architecture or the customer’s use of execution of each system in the computation. While integrity that architecture. While architecture providers assert their measurement enables systems to generate proofs of their integrity to remote parties, we find that current integrity mea- diligence in providing a protected environment, such as surement approaches are insufficient to prove runtime integrity military grade physical protection of the EC2 cloud data for systems in these architectures. Integrity measurement centers [4], we aim for a principled approach for proving approaches that are flexible enough have an incomplete view of high integrity distributed computations. runtime integrity, possibly leading to false integrity claims, and While our ultimate goal is a single approach for gener- approaches that provide comprehensive integrity do so only for computing environments that are too restrictive. In this paper, ating proofs of integrity for a complete distributed compu- we propose an architecture for building comprehensive runtime tation, we find that current approaches for proving integrity integrity proofs for general purpose systems in distributed for a single system are insufficient for these architectures. computing architectures. In this architecture, we strive for Integrity measurement mechanisms build proofs of system classical integrity, using an approximation of the Clark-Wilson integrity, but to date such proofs either enable verification of integrity model as our target. Key to building such integrity proofs is a carefully crafted host system whose long-term partial integrity for general purpose systems [5]–[10] or en- integrity can be justified easily using current techniques and a able verification of comprehensive integrity for limited (e.g., new component, called a VM verifier, which comprehensively single application) systems [11]–[13]. Integrity measurement enforces our integrity target on VMs. We have built a prototype approaches that only justify incomplete integrity can give based on the Xen virtual machine system for SELinux VMs, a false sense of integrity when used on general purpose and find that distributed compilation can be implemented, providing accurate proofs of our integrity target with less than systems, unless they are used very carefully, so we aim 4% overhead. for comprehensive integrity. Previous approaches to prove VM system integrity [14] used incomplete integrity [5], so Keywords-cloud computing, integrity measurement, virtual machines the verifier had to make assumptions about the integrity of dynamic data on the base system and guest VM data (e.g., system-specific configurations and application data) I.INTRODUCTION that may lead to vulnerabilities. Integrity measurement ap- With the emergence of a variety of distributed comput- proaches that require custom systems are not practical for ing architectures, such as cloud computing [1] and stream these emerging, distributed computing architectures, as their processing [2], many applications are moving from central- limitations (e.g., a single application and complete system ized to distributed architectures, even architectures involving restarts) are not compatible with these architectures. multiple administrative interests. Cloud computing, in partic- In this paper, we design and implement a comprehen- ular, has gained significant mindshare in the past year. Cloud sive integrity measurement approach for general purpose architectures provide a simple interface to configure and systems. The design of our approach is based on three execute complex, large-scale applications for a modest fee insights. First, we leverage emerging work on practical without committing capital to resources or administration. integrity models [15]–[17] that define practical integrity In such distributed architectures, it is difficult for either the model for general purpose systems. Our approach generates computing architecture (e.g., cloud) provider or the customer proofs that a remote party can use to determine whether to determine whether the entire distributed computation was a virtual machine’s execution adheres to an approximation executed in a high integrity manner on all systems. Concerns of Clark-Wilson integrity [15]. Second, comprehensive in- may arise because one of the systems may use outdated, tegrity measurement [11], [12] requires secure initialization vulnerable code, have misconfigured policies or services, or (i.e., secure boot [18]), as well as runtime integrity. We may have been previously attacked with a rootkit, triggered leverage a secure initialization approach that binds boot- by malicious code or data [3]. As both the base system time integrity to the installation of the system [19]. Third, and guest VMs use conventional systems support, such we find that measuring integrity of general purpose systems II.PROBLEM DEFINITION 1) Upload Source Package Package Archive A. Scenario distcc distcc distcc distcc Node Node Node Node 3) Get 5) Upload As an example of a distributed service, consider Canon- Launchpad Source Binaries Web Portal Xen Xen ical’s Personal Package Archives (PPA), which compiles Host Host developer-supplied source code into packages for all sup- distcc ported platforms . This enables developers to concentrate Controller 2) Request to 4) Distribute Build Packages Compilation Tasks on the development of source code while the PPA does the work of building and hosting binary packages for the Ubuntu user community. Figure 1. A Personal Package Archives provides compilation services for developers to build binary packages for a variety of platforms. We envision The specific architecture of the PPA build system is such a service as a distributed compilation cluster (e.g. distcc, whereby not public, but we know that they use Xen-based virtual a web portal uploads packages and processes compilation requests. machine systems to host compiler nodes. Thus, we envision results in a large number of integrity-relevant events that the PPA system architecture as shown in Figure 1. Here, must be captured and justified to the remote party. Instead the PPA Launchpad frontend receives source packages from of recording all events and requiring the remote party to developers. As PPA may receive many packages from many deduce their impact, we leverage enforcement of integrity developers and needs to support several platforms for such to remove the need to convey these events. By enforcing an compilation, we envision a set of distributed compilations integrity property, we only need to prove to the remote party implemented by one or more distcc distributed compilers. that the VM was initialized securely and that this integrity The PPA frontend initiates a distcc compilation through property is enforced thoroughly at runtime. a distcc controller, which retrieves the package to be compiled from a package archive where it was uploaded pre- In this work, we make the following contributions: viously. The controller distributes the job to multiple nodes distcc • We define an integrity measurement approach that that perform the actual compilation. Note that starts from well-defined origins (e.g., installation) and can also be configured to enable nodes to offload work to enforces a comprehensive view of system integrity subnodes. The distcc controller will ultimately return the to generate practical integrity proofs for VM-based compiled code back to the package archive, where PPA can distributed computing nodes. then post the binary packages for users to download. • We demonstrate a two-level integrity measurement ap- Clearly, Canonical, as well as the developers and Ubuntu proach whereby an easily verified fixed base system en- user community depend on the integrity of the build process, forces a well-defined integrity property on application but this build process presents several challenges. First, the VMs, removing the need to verify the application VM portal and the archive are permitted to interact with un- details in favor of verifying the base system’s ability to trusted parties (users and developers), but the build process enforce a particular integrity property. depends on them preserving their integrity. Second, compiler • We deploy our integrity measurement approach on ap- nodes may delegate their work to other systems without the plication VM systems that perform distributed compila- knowledge of the portal. We need to ensure that all systems tions (distcc) and web serving (Apache) to evaluate that are depended upon for high integrity operations are our approach. For Apache, we show how to comprehen- verified as high integrity. Third, we do not know whether sively enforce CW-Lite integrity [15] on an application the components are configured correctly. Such configuration VM, and demonstrate that only a 4% increase in goes beyond code verification to access control policies, compilation time results from using our architecutre in firewalls, and configurations. a distcc cluster. A prior proposal for the Shamon system [14] enabled the construction of integrity-verifiable distributed computations. In Section II, we describe an example scenario and review However, Shamon imposes two restrictions that prevent the our goals for its integrity. In Section III, we examine current challenges above from being addressed. First, a centralized integrity measurement techniques and identify the need for node verifies the integrity of all compute nodes. In our comprehensive statements of system initialization and en- architecture, no single node may see all the systems. Second, forcement to justify its integrity. In Section IV, we describe the VMs in Shamon are isolated from all untrusted systems. the architecture of our integrity measurement approach. In That is not practical in an environment where developers Section V, we detail its implementation on Xen systems provide the input (source packages), and users download running Linux application VMs. We perform an evaluation the output (binary packages). Third, the initial integrity of the security and performance of the implementation on of a VM is determined by a centralized component, so it distcc and Apache application VMs in Section VI. In cannot support VMs whose integrity is defined elsewhere. Section VII, we conclude. In general, the main flaw in the Shamon approach is that it does not trust application VMs to make integrity decisions. unnecessary. We will leverage one such model, called CW- Instead of providing a high integrity platform for running Lite [15], in providing practical runtime integrity, where VMs whose runtime integrity can be evaluated, Shamon CW-Lite only permits untrusted inputs to high integrity pro- dictates tightly defined VM integrity and isolates those VMs grams through filtering interfaces identified by the program. from untrusted components. This is too restrictive for an open system, such as PPA, so we propose a more general III.RELATED WORKIN INTEGRITY MEASUREMENT approach that assesses VM integrity in an open environment. We identify three key tasks in building a runtime integrity proof: (1) initialization; (2) enforcement; and (3) mea- B. Integrity surement. While current integrity measurement approaches We claim demonstration of a distributed computation’s support these tasks in some form, they are either incomplete correctness requires each participating system (VM and (i.e., do not satisfy runtime integrity) or demand limitations base) to satisfy the runtime integrity requirements for that on the system deployment. computation. Historically, runtime integrity has been as- We start with the measurement task. Several integrity sessed using integrity models. The first comprehensive in- measurement approaches [5]–[7] only measure system’s tegrity models were proposed by Biba [20]. In Biba integrity, operations that impact integrity. The idea is to measure every subjects and objects are assigned integrity labels based on event that impacts the integrity of the system (i.e. integrity- their initial integrity state, and these labels are arranged in an relevant events) and send a proof of these events to a remote integrity lattice where information can only flow from higher party who determines whether these events result in a high integrity entities to lower integrity entities. For example, integrity system. These measurement-only approaches are a subject can only read objects higher (or equal) in the insufficient because they do not place requirements on data integrity lattice and write objects lower (or equal) in the or external sources. As a result, a distcc compiler code integrity lattice. The initial integrity state of the system must may be correct, but it may use incorrect configurations or be ensured, which the Clark-Wilson integrity model [21] depend on untrusted nodes for inputs. Despite not measuring makes explicit by defining an integrity verification procedure every integrity-relevant event, such approaches generate a (IVP), a process to verify the integrity of the system at large number of low-level measurements that must be parsed initialization time to guarantee a high integrity starting point. for verification. Even worse is the case verifiying heteroge- Runtime integrity is necessary to ensure all inputs the neously constructed distcc nodes whose measurements compilation nodes depend upon, such as the code distcc may be different but valid. runs, its configurations, the source code it compiles, and Next, some integrity measurement approaches perform compilation results from others, are sufficiently protected a degree of integrity enforcement in addition to mea- from untrusted entities. From a distcc node’s perspective, surement [8]–[10]. For example, the Satem [9] system it is important that the host system and distcc application only allows the system to execute approved code objects. be derived from high integrity origins. In addition, the inputs BIND [10] only uses a high integrity input if it was gen- to the node must also be from high integrity nodes. Finally, erated from operations verified to be high integrity. There nodes must be able to protect itself from low integrity are two advantages to having enforcement in addition to inputs. If each distcc node protects its runtime integrity measurement. First, it reduces the ad hoc proofs generated effectively, including verification that it only obtains inputs by measurements alone, so the remote party only needs to from other nodes that protect their runtime integrity, then verify that the system follows some integrity guidelines. In it may be possible to build a proof for the PPA server that Satem proofs, a remote party only needs to verify that the such compilations are done correctly. In this paper, we will Satem system enforces code execution completely and that examine what is required of each node to prove runtime it uses input from an authority that the party trusts. Second, integrity for a commercial application like distcc. enforcement can reduce the number of integrity-relevant A major challenge in proving runtime integrity is handling events that need to be included in a proof. PRIMA [8] untrusted inputs. The Clark-Wilson integrity model [21] enforces a mandatory access control (MAC) policy that previously identified this problem, and stated that high aims to protect processes with high integrity labels. As a integrity programs (called transformation procedures) must result, we don’t need to measure code whose integrity is be able to immediately discard or update untrusted inputs. not guaranteed, we just have to show that such code cannot Additionally, formal assurance of the correctness of these impact the high integrity processes. programs is necessary to justify such behavior. However, it is Unfortunately, these enforcement approaches are limited now widely believed that formal assurance of applications is in two ways: (1) enforcement is incomplete and (2) en- impractical. Thus, new interpretations of integrity have been forcement comes too late. First, each of these systems, proposed [15]–[17]. These require high integrity programs Satem, BIND, and PRIMA, demonstrate a useful integrity to only accept untrusted inputs at interfaces where integrity enforcement, but we need all three of these enforcement decisions can be enforced thus making full formal assurance mechanisms and more if we are to cover all integrity-relevant events. The remote party will need to verify that the code Application VM executed is acceptable (Satem), that the resulting process is Local Attestation Application protected from low integrity processes (PRIMA), that high Enforcement Service integrity inputs are truly from high integrity sources (BIND), and that low integrity inputs are really handled according Host System to Clark-Wilson requirements (none). Second, all of these VM Proof Enforcer approaches depend on a high integrity initialization (secure IVP Generator boot) of the system, which provides the function of a Clark- VM Verifier Wilson IVP. As a system could be modified in prior boot cycles, ensuring that the initialization of a system satisfies Trusted Platform Module integrity requirements is challenging. A few integrity measurement approaches build a proof of Figure 2. The overview of the VMV architecture. The host system’s integrity using secure boot. Outbound Authentication checks integrity is proven using an IVP satisfying installation. The VMV contains each layer of the system before loading the subsequent services to establish, enforce, and report the integrity of an application VM. The local enforcement mediates integrity-relevant events and the attestation layer [11]. Flicker uses specialized hardware support to load service handles VM attestation requests. a program from a secure state [12]. Both techniques build proofs using integrity measurement hardware justifying the a comprehensive integrity property, in this case CW-Lite. integrity of the secure boot process. However, both run only one application in a tightly prescribed environment. For IV. ARCHITECTURE example, a Flicker computation takes a formatted executable, input parameters, and predefined stack region, and runs In this section, we present our architecture for enforcing that executable on those inputs without support from any CW-Lite integrity on Application VMs and reporting their OS or services outside the Flicker environment. In both integrity to remote verifiers. Figure 2 shows an overview of cases, the number of integrity-relevant events that can occur our architecture in which an integrity verifiable host system while a computation is running are minimized. While both (e.g. using a ROTI approach [19]) enforces an integrity approaches allow execution of individual applications [13], property over the execution of application VMs. Such en- [22], neither is intended to support integrity measurement forcement is implemented and measured by a new compo- of a distcc node, which runs several processes, uses nent called the VM Verifier (VMV). The VMV consists of conventional OS services, and depends on the integrity of three parts: (1) a VM IVP, which performs VM initialization remote nodes. The Bear approach aims for secure booting of by verifying that VM’s integrity prior to loading; (2) an general purpose systems [23], although the basis for initial enforcer, which provides integrity enforcement by ensuring integrity is ad hoc and enforcement is incomplete. that VMs run under the control of an integrity property; and Our Goal: Our goal is to define an integrity measure- (3) a proof generator, which uses measurements to produce ment approach that provides: (1) effective initialization to attestations of the VM’s integrity for remote verifiers. In this define a high integrity system foundation; (2) comprehen- architecture, an attestation consists of a traditional integrity sive enforcement to make integrity guarantees predictable measurement of the host system, measurement of the en- to remote parties and reduce the number of integrity- forced integrity property, and any additional measurements relevant events that must measured; and (3) measurement necessary for the enforcer to show the mechanisms by which of integrity-relevant events and enforcement properties nec- it enforces that property. essary to justify the integrity of distributed computing, such We extend application VMs with two services: (1) local as the distcc system. We are motivated to tackle this enforcement that may optionally be uploaded into the VM by challenge using prior work on developing an initialization the enforcer to supplement the enforcement of integrity poli- mechanism, called a root-of-trust-installation [19] (ROTI). cies and (2) an attestation service that assists applications The ROTI binds the integrity of a privileged VM to its in obtaining proofs from the proof generator. We note that installer. The ROTI enables a secure boot of the privileged local enforcement may be trusted by the VMV enforcer to VM to yield a proof that a particular installer generated this achieve the integrity property. In this case, the enforcer must system, and that the system protects its integrity at runtime. justify the integrity of local enforcement and its ability to If a remote party trusts the installer, then it can trust the protect the integrity of the Application VM. The attestation execution of the privileged VM. We use the ROTI approach service need not be trusted, as all integrity proof statements to justify the integrity of the base system, but we cannot are signed by the proof generator and TPM. use it for application (distcc) VMs, as it requires that the system’s persistent state is largely static. We aim for a A. Initializing the Host System and Application VMs way to take advantage of the a ROTI-justified base system Initialization defines mechanisms for securely booting the to enforce integrity on application VMs, enabling a proof of host and VM necessary to ensure the initial integrity of trusted components. Figure 3(a) demonstrates the initializa- to a remote system is established, local enforcement traps tion process of the host, which is derived from the root- the request and verifies the remote system’s integrity is of-trust-installation (ROTI) approach [19]. First, a trusted justified by an attestation (Figure 3(b)). Optionally, a mutual installer (e.g., on a CD-ROM) installs and configures a high attestation of the local VM may be requested by the remote integrity host system, including our VMV. The resultant node. Finally, the IPsec connection is established allowing system is bound to the installer via a ROTI Proof. This proof inputs to the system. contains the hash of the installed system and the identity of the installer, binding the system origin to the installer. C. Measuring Integrity of Application Execution The proof is cryptographically bound to the machine’s TPM In order to prove an Application VM’s integrity to a using a key certified by the TPM’s private endorsement key. − remote verifier, the VM must produce a proof that demon- We state the IVP proof as I − , where AIK is the AIK strates (a) the host system running the VM is able to private Attestation Identity Key produced by the TPM. When enforce an application policy on the VM and (b) such a the host system is booted, a host IVP process measures the VM is actually running on that host system. (a) requires installed system into the TPM, so it can be verified by a the host system must be verifiably booted into a known remote party in the same manner as a code load. good state that protects its integrity. Additionally, the host Once the host has been initialized, it must verify the VM must demonstrate it enforces the integrity property on the is also initialized in a similar high integrity manner. The pro- VM. Proving (b) requires associating the running VM with cess of starting a VM occurs when an application authority the attesting host system’s physical identity (e.g. the TPM’s sends a signed request to the VMV. In Figure 3(a), the VMV signing key) via an identity key assigned to the VM. downloads a VM’s disk image from a VM store server to the 1) Generating the Attestation: Figure 3(c) shows the VM IVP. The VM IVP using the verification requirements steps in generating the VM’s integrity proof. A remote in the signed request to validate the integrity of the VM system first initiates the attestation process by sending a downloaded. Such requirements could include attestations nonce N to the Application VM’s attestation service. This of the VM’s last run. This is analogous BIND [10] where nonce is then forwarded to the host system’s VMV. Next, each use of an input requires an integrity verification, but the proof generator builds an integrity proof for the host, the VMV approach is stronger, as no VM is used without Φ (N). We represent the proof as: validation of requirements from a trusted authority. Host

B. Enforcing Integrity over Application VMs ΦHost(N) = QUOTE(P,N)AIK− + M + IAIK− (1)

During the lifetime of the VM, all integrity-relevant Here, QUOTE(P,N)AIK− is the result of the TPM’s quote operations must be mediated by the VM reference monitor. operation, which provides a signature of the measured state This requires mediation of both local and remote events. of the host system using the private key of a TPM AIK−. We assume each application VM has an accompanying M contains a list of integrity-relevant operation measure- application policy PApp, defining the integrity policy to be ments (e.g. all code loaded on the host), used to compute enforced on the VM. Our CW-Lite integrity policy specifies the hash chain value P . The result of the host’s IVP proof the trusted subject labels that define the application VM’s IAIK− is included to be compared to the filesystem hash TCB and the code that can run in those subject labels. recorded by the IVP. Next, the generator creates the proof After a VM’s integrity has been initialized (Section IV-A), for the Application VM, ΦVM (N) as follows: the enforcer obtains the VM’s policy from its application au- + thority (Figure 3(b)), installs the attestation service and local ΦVM (N) = KVM + PApp + ΦHost(N) enforcement into the VM, and configures them according to + + SIG(KVM ||PApp||ΦHost(N))AIK− (2) the policy. In our implementation (Section V), we employ an integrity enforcing kernel that mediates code loading using When the VM is first configured by the enforcer, a fresh an integrity policy in the VM. Certain operations are difficult IPsec key pair KVM is generated for the VM, which is to enforce externally to the VM, such as access control, so used to uniquely identify the VM’s identity. This key pair we allow the VMV enforcer to configure such local enforce- is placed in the VM’s local key store. This identity is tied ment in the VM. Of course, the integrity of such enforcement to the VM proof by a signature of ΦHost(N) concatenated must be carefully configured and justified to remote parties. with the VM’s public key and an identifier for the policy Our architecture also supports the use of externel integrity PApp that the VMV is enforcing on the VM. This proof, monitoring approaches such as VM introspection [24] from ΦVM (N) ties the integrity of the host system and the the host and kernel integrity monitors [25]. enforced integrity policy of the VM to the VM’s public While the application VM is running, all network inter- key using the host system’s authoritative private key. Next, faces are mediated high integrity services only recieve input ΦVM (N) is passed back to the VM’s attestation service, + from high integrity sources over IPsec. Before a connection which sends ΦVM (N) + KVM to the remote verifier. Remote System Remote System

Filter Low Request Integrity Sources Application VM Attestation Application VM Install Host Verify VM System Disk Image Local Attestation Application Local Attestation Enforcement Service Application VM Enforcement Service Store Conﬁgure Host System Host System Guest Host System

VM Proof Application VM Proof VM Proof Enforcer Enforcer Enforcer IVP Generator Authority IVP Generator IVP Generator VM Verifier Obtain VM Verifier VM Verifier Policy Trusted Platform Module Trusted Platform Module Trusted Platform Module

(a) Initialization (b) Enforcement (c) Measurement

Figure 3. (a) demonstrates the installation of the host system veriﬁcation of VMs before execution. (b) The application policy is obtained and used by the enforcer to conﬁgure the VM. (c) Attestation requests are forwarded by the attestation service to the proof generator, which builds an attestation and returns it to the requester.

2) Verifying the VM’s Integrity: When the remote verifier remote systems. Finally we show how a remote verifier receives the VM’s attestation, it first validates the signatures obtains and validates the integrity of the reporting system. in the proofs for correctness. It must then determine the host A. Overview system’s integrity. To do this, it inspects ΦHost(N) by first checking that the quote QUOTE is correct by reconstructing Figure 4 shows the implementation of our prototype the PCRs P with M and verifying both the signature and N design. The host system is implemented as a Xen VM are correct. This check ensures the reported measurement list system with the host VM domain (Dom0) containing the came from the machine associated with the AIK. The verifier VMV. The VMV is comprised of several services including then assesses IAIK− to confirm that the host system’s IVP the VM manager, enforcer, and attestation service. The VM process measured the same installed system as in the IVP Manager handles application VM disk images and verifying proof and then compares the entries in M against a set of their integrity. The Enforcer installs the programs and policy known-trusted measurements. We envision that an authority files in the VM disk image to enforce CW-Lite integrity. It over the VM will provide the information necessary for also supports the mediating VM integrity from outside the obtaining these measurements. This check confirms (a) the VM using integrity monitoring techniques [24], [25]. Finally, host booted into a trusted state and that (b) only code from the VMV’s proof generator produces attestation of the host that installation was loaded. Thus, a remote verifier is able system using the TPM’s quote and signing operations and to ascertain the integrity of the host system using traditional vouches for the integrity of the application VMs running on integrity measurement techniques. the host. Within the application VM, the enforcer installs The verifier then compares PApp against the policy it the port authority, which verifies the integrity of remote expects the Application VM to use. Given such a policy, systems before establishing IPsec connections. It also sets the host configures the VM to support PVM . This resultant up the VM’s attestation service to forward requests from policy represents the combined changes the enforcer makes remote parties to the host system’s attestation service. We to the VM to support PApp. Since the installer is trusted now discuss how these components are implemented and to generate a host that enforces an application policy on how they meet the goals of our architecture. the VM, a successful validation of Φ (N) implies P Host VM B. Initializing the Host System and Application VMs satisfies PApp. Moreover, the host system has claimed key + KVM speaks for a VM with the integrity specified by One of the goals when building the host system was ΦVM (N). Using this key, the verifier can establish an IPsec simple integrity verification and the ability to justify the tunnel with the application VM that cryptographically ties integrity of its VMs. To that end, we used an install based both the VM’s identity and integrity to the tunnel. on the ROTI approach [19], which configures the host filesystem and genrates a proof of the system’s state and V. IMPLEMENTATION filesystem at the time of the installation. A remote verifier We now describe our implementation of our architecture. can compare this proof to the runtime measurements of the We first detail the host system and its VMV components host to detect any deviations between boots. The resulting to enforce the application specific integrity policy on VMs. host system consists of a Xen Hypervisor 3.2 with MAC Next we explain how the VMV modifies the VM to mediate enforcement of VMs and resources by the built-in Flask / both CW-Lite integrity events in the VM and inputs from Xen Security Module (XSM) [26] built-in. Dom0 runs a stripped down Ubuntu distribution and a 2.6.24 Linux kernel enforcer supplies an initrd that loads two policy files from modifed to enable SELinux and IMA to run concurrently. the policy package via the sysfs files \selinux\ts_load A dedicated integrity measurement framework is currently and \sys\kernel\security\enforce. The first pol- being added to Linux, which will obviate the need for this icy file defines the SELinux subject types that represent the modification [27]. Further details on the host’s configuration TCB. The second is a list of acceptable code hashes. These is describe in the ROTI work. hashes are defined by the distribution and vendors that ap- Managing Application VMs: The VMV consists of plication VM’s uses. Whenever a kernel module is loaded, a the VM manager, enforcer, and proof generator services. process memory maps a file (such as a library) as executable, The VM manager receives requests from an application or an execve call triggers the bprm_set_security authority to start and stop VMs and validates the request LSM hook1, the validation procedure is called. using certificates obtained at install time. Once a request is Our validation procedure proceeds as follows. The kernel received, the VM image is downloaded from its VM store, first examines whether the current process’ label is one of which we implement as a simple web service. The VM the policy specified trusted subjects. If it is not, no further manager then checks the VM disk integrity. Included with examination is performed since the process is not part of the the image is an integrity proof generated by the previous host TCB. Otherwise, the measurement list is checked to see if system to run the VM. The VMV checks this proof using the file had been previously measured. If it was not, the code the verification technique for remote attestations described is hashed and compared to the policy database of trusted later in Section V-D. By ensuring the previous host was code hashes. A successful match will cause the hash of the able to enforce the application policy’s CW-Lite properties, file to be placed in the measurement list. Otherwise, the the host is able to build transitive trust of all previous runs operation that triggered the validation check is denied. If of the VM. This concept is similar to BIND [10], but we the file had been previously measured, it must have been in additionally verify that the host system was initialized to a the acceptable hash list. Hence, the triggering operation is high integrity state before each run. permitted to continue as normal. Applications frequently depend on inputs from remote C. Enforcing Integrity over Application VMs sources. In order to protect those applications and other Our implementation enforces a CW-Lite runtime property trusted network facing services from low integrity sources, on VMs as specified by an application policy. This policy our architecture defines that an Input Verifier mediate all defines a TCB using SELinux subject types and a set of network communication. We implement this verifier as the trusted code. We now look at how the VMV’s enforcer uses port authority, a service the VMV installs in every VM. This this policy to achieve CW-Lite integrity in the VM. service configures the VM’s iptables to default deny all The VMV’s enforcer is invoked by the VM Manager non-IPsec traffic to high integrity services except for those to configure the VM and create its domain for execution. that can filter low integrity inputs (via application policy). First, the application policy, a signed Debian package, is When a connection to or from a remote party is attempted, retrieved from the application authority web service. The the port authority requests an attestation of the remote party. VM image is mounted locally and the package installs the This attestation is verified using the application policy’s policy files, attestation service, and port authority com- verification criteria (see Section V-D). If the remote party ponents. We describe the contents of the policy package is deemed high integrity, the other party may optionally further in Section VI-A. Next, the enforcer configures the request a mutual attestation. Along with the attestation is VM domain’s RAM, vCPU, networking configurations, and a certificate of the attesting system’s IPsec key. Once both XSM security label. Since VMs other than Dom0 do not sides have agreed to permit the connection, an IPsec security have access to physical platforms, it is critical to link VM’s association is created using this key. identity to a physical platform like the EK stored in the The IPsec tunnel permits the connection to pass the TPM [28]. To solve this issue, the enforcer generates an iptables firewall and identifies the inputs as coming IPsec key pair and places a copy of it in the VM and VMV’s from a high integrity source. Additionally, the Port Authority key store. The VMV vouches for this key by generating a periodically requests fresh attestations from the remote party. fresh certificate for each attestation of the VM. Finally, the If the integrity proof fails to pass the verification criteria, hypervisor is invoked to create and start the VM domain. the IPsec tunnel is torn down, and the security association The enforcer runs the VM with a Linux kernel using a is removed. Additionally, aggressive dead peer detection modified SELinux LSM that performs a validation check on is used to discover if a peer has lost connection. The all code loaded into trusted process. This allows the enforce- connection is also torn down when this occurs to prevent the ment of an application policy that specifies both the permis- possibility of the remote party rebooting into a low integrity sible code and process that must be protected. This is similar to PRIMA [8], but additionally enforces high integrity code 1This is the first point during an execve when a process begins to loading instead of just measuring it. At boot time, the transition to a new subject type by the verifier . Further discussion of how this is defined is Integrity Application IPsec Port Enforcing Tunnel Authority VM in Section VI-A. A successful validation assures the verifier Kernel Exchange Attestaton Key Application that the host system is enforcing an acceptable integrity Attestations Service Store Policy policy on the VM associated with the attested IPsec key. This process also allows the host to revoke a VM’s key by Build Proof Configure VM Host System no longer providing a valid certificate when requested. Remote Dom0 Application Verifier Proof Get Enforcer VI.EVALUATION Generator Policy Authority

ROTI VM Download & A. Security Evaluation VM Store Proof Manager Verify We now evaluate how our design achieves the goals of TPM Xen Hypervisor enforcing and reporting VM runtime integrity. First we examine our proof of concept system’s ability to resist Figure 4. The VM manager obtains and validates VM disk images. The compromise. We then study how the VMV enforces an enforcer manages the VM’s runtime integrity according to an application integrity policy on an Apache webserver VM and reports policy by mediating integrity relevant events in the VM with an integrity this to remote parties. Proper enforcement of VM runtime enforcing kernel and remotely from network inputs using the port authority. The attestation service works with the proof generator to build an attestation integrity requires (1) a high integrity host system and (2) the of the host system and vouch for the VM’s integrity as well as its ownership host system’s enforcing components to be protected from of an IPsec key stored in the VM’s key store. compromise. The security of the ROTI configured Xen- based host has been studied in previous work [19], but system and using the negotiated IPsec session keys. it is important to note that a multitude of vulnerabilities have been found recently [30]. Approaches such as privilege D. Measuring Integrity of Application Execution separating Xen domains [31] and the use of SMM based Figure 4 illustrates the process of a remote party request- integrity scanners [25] offer ways of mitigating these threats. ing an attestation from the application VM. First a nonce is Constructing an Application Policy Package: How the sent to the application VM, initiating the procedure. When system protects the application VM’s integrity is defined by the VM’s attestation service receives the request, the nonce its application policy. In our system, we aim for CW-Lite in- is forwarded over an internal network socket to the VMV’s tegrity [15], so an application policy defines the application- proof generator, which uses it to obtain a TPM quote. The specific inputs necessary to justify CW-Lite integrity: (1) the proof generator then creates a fresh certificate for the VM’s set of high integrity code for the Application VM; (2) the public IPsec key using a hash of the key, the nonce, and the set of trusted subject labels in the VM’s mandatory access application policy package. This certificate is signed using control (MAC) policy; and (3) the criteria to assess remote the private portion of the TPM’s AIK. Finally, TPM quote, systems. At present, distcc does not have an SELinux the certificate, and the ROTI proof are forwarded to the policy module necessary for building a complete application remote party via the VM’s attestation service. Alternatively, policy package. So we created a proof of concept application the VM manager may also request an attestation of the VM policy package for an Apache webserver VM. Apache is when it is shutdown. In this scenario, the manager uses a a significantly more complex system than distcc, so we hash of the VM disk image as a nonce to the proof generator. expect that the same approach will apply once an SELinux The resulting attestation binds the current state of the VM policy module is available. (after shutdown) to the host’s integrity. First, we obtained the list of acceptable code hashes for To validate an attestation, the requester first obtains the the Application VM by parsing all packages in the Ubuntu public AIK of the host’s TPM. There are various methods 8.10 repository for executable files, libraries and kernel for implementing this including the use of a trusted third modules. The resulting set contained 34,000 hashes resulting party or direct anonymous attestation [29], but it is outside in about 648KB. The policy package also includes the strict the scope of this design. Next, the attestation is checked for SELinux policy and Apache policy module. Second, we syntactical correctness including signatures (quote, certifi- obtain the trusted subjects that identifies the MAC policy cate, ROTI proof), the value of the nonce, and that the PCRs subject labels that must load only trusted code. SELinux match the measurement list. Next, the ROTI proof’s installer uses type enforcement to label process so we identified a hash is check against the set of known good installers. set of subject types that represent the target system TCB. If this passes, the file system hash the quote must match We used PALMS [32] to find this set. Further details of this the one in the ROTI proof. Success means the host booted process can be found in our previous work [33]. Third, our into a trusted state. Next, the measurement list is checked verification criteria for remote systems contains the hashes of for illegal measurements. This ensures only files from the the trusted installer image and application policy packages. installer’s distribution were loaded in the host. This good In addition, the acceptable code and data measurements measurement set is defined by a verification criteria obtained that a trusted host system would have are included. This Operation Time (seconds) Attestation Exchange 1.087 ± 0.010 cache at boot slowed the hashing process. One possible im- Generate IPsec Configuration File 0.00027 ± 0.000022 provement would be to incrementally measure the filesystem Generate IPsec Connection Specification 0.417 ± 0.020 as the files are requested instead of measuring everything IPsec SA Setup 0.588 ± 0.026 at once. This approach would complicate the verification Table I as the measurements would be added nondeterministicly PORT AUTHORITY MICRO-BENCHMARKSSHOWINGTIMEREQUIREDFOR thus requiring inspection of a full measurement list of files BUILDING AN INTEGRITY VERIFIED IPSECTUNNEL. instead of a single aggreagate. Starting a VM involves verifying the disk image, the comprises all load time code and policy measurements that policy package, configuring the VM and finally creating the should appear in Dom0. Since a trusted installer places the Xen domain. For our VMV verification policy, verifying same files in every system it installs, these measurements VM and policy package integrity requires checking for would be included with the installer image. a valid signature on both. The combined time to verify VM Integrity: Within the application VM, both local our compressed 137MB distcc VM image and 1.5MB and remote dependencies must be mediated to assure high policy package took about 1 second to perform. A more integrity computation. In our desgin, the host installs a sophisticated verification such as validating the VM integrity reference monitor in the VM comprised of the integrity proof will increase the time accordingly. The average time enforcing kernel and the Port Authority. The kernel uses to mount the VM disk, install programs and configuring the SELinux to enforce the application policy installed by the enforcement policies added approximately 1.35 seconds to VMV’s enforcer. The trusted processes that form the TCB the Xen domain creation procedure. are all mediated so that only trusted code is loaded into Enforcement: The VM’s PRIMA enforcing kernel in- types consdidered high integrity. High integrity types may troduces an initial delay to the program execution due to only receive flows from high integrity sources or from low hashing of the code and looking up the result in the policy integrity flows via a filtering interfaces, which immediately hash table. Since the average size of an executable was discards or upgrades inputs to high integrity. This is sim- 39KB (with the largest being 6MB) and in-kernel SHA1 ilar to DLM’s notion of endorsers [34]. Remote inputs, is 135MB/s on our setup, code hashing added a negligible are similarly mediated via the port authority. As a result, time. The PRIMA module adds on average 12.85 ms with only verifiably high integrity network sources can connect negligible variance to the execution time after the program to trusted services unless the service exposes a filtering is hashed. This means the initial delay to hash a program is interface that can handle unverified sources. relatively short. Measurement: Before a connection is established with B. Performance Evaluation the VM, the port authority must create an IPsec tunnel. Ta- To evaluate the practicality of deploying a VMV system, ble I shows the individual steps in creating that connection. we constructed a distcc cluster. We developed a ROTI Unsurprisingly, the majority of the 1.53 seconds required to installer that configures each host system, a VMV service, setup the IPsec connection come from the attestation, which and a distcc VM image with an application policy. We then is due to the slow speed of the TPM. evaluated the overhead introduced by our implementation Application Performance: We used our setup to com- on compilation time. Specifically, we focused on overhead pile two different code-bases: the Linux 2.6.28 kernel and from initializing the host, starting a VM, local enforcement, openssh. We gathered macro benchmarks of the compila- and finally from mediating network connections. We use tion time with and without the port authority to determine a testbed of 3 Dell PowerEdge M605 blades with 8-core the effect of verifying remote host integrity, setting up 2.3GHz Dual Quad-core AMD Opteron processors, 16.0GB IPsec connections, and polling remote hosts on the com- RAM, and 2x73GB SAS Drives in RAID1. These systems pilation time. During the run with port authority enabled, were connected over an isolated Gigabit Ethernet switch on a all connections required mutual attestation, and a repeated quiescent network. Each blades formed a host system using attestation every thirty seconds. As shown in Table II, the Xen 3.2 and Ubuntu 8.10 as Dom0. Each distcc node overhead introduced by our system was minimal with less VM consisted of a 362MB Ubuntu 8.10 installation, 1GB than 4% increase in compilation time. After profiling the RAM, one vCPU and a Xen 2.6.24 Linux kernel modified build process we found the majority of the overhead was due with our PRIMA enforcing LSM module. to IPsec on network throughput. We further experimented Initialization: At boot-time, the IVP script runs SHA1 with different encryption suites and null encryption, but on the 630 MB filesystem and logs it in PCR 13 in the found similar times. Our results indicated the largest per- TPM before root is rotated to the primary partition. SHA1 in formance impact came from processing IPsec traffic. When userspace (tested with openssl) is normally 270MB/s on we experimented with running miltiple compliation tasks our machines, but the total IVP time takes approximately one concurrently, we found TPM response time increased only minute. The added delay was due to the kernel’s empty page when additional hosts were connected to the profiled system. Time (mins.) Task % diff. Lines of Code Unattested Attested [14] J. M. McCune, T. Jaeger, S. Berger, R. Caceres, and R. Sailer, Linux Kernel 7:01.84 7:17.24 3.65% 6,450,761 “Shamon: A System for Distributed Mandatory Access Con- Proceedings of the IEEE 22nd ACSAC OpenSSH 0:22.67 0:23.74 3.98% 68,626 trol,” in , Washington, DC, USA, 2006. Table II [15] U. Shankar, T. Jaeger, and R. Sailer, “Toward automated 6 NODE DISTCC CLUSTERCOMPILATIONTIMEWITHANDWITHOUT information-flow integrity verification for security-critical ap- REATTESTATIONS EVERY 30 SECONDS. plications,” in Proceedings of the 2006 ISOC Networked and Distributed Systems Security Symposium, February 2006. This is due to the slow clock speed of he TPM, causing [16] N. Li, Z. Mao, and H. Chen, “Usable Mandatory Integrity Protection For Operating Systems,” in Proceedings of the a significant queuing effect. However, this has no direct 2007 IEEE Symposium on Security and Privacy, May 2007. affect on the performacne of the compilation itself, except if [17] W. Sun, R. Sekar, G. Poothia, and T. Karandikar, “Practi- the attesation delay becomes large enough to invalidate the cal Proactive Integrity Preservation: A Basis for Malware link. On possible method of addressing this issue is to use Defense,” in Proceedings of the 2008 IEEE Symposium on Security and Privacy. aysnchronous attestation techniques [35]. [18] W. A. Arbaugh, D. J. Farber, and J. M. Smith, “A Secure and Reliable Bootstrap Architecture,” in In Proceedings 1997 VII.CONCLUSION IEEE Symposium on Security and Privacy. In this paper, we proposed an architecture for proving [19] L. St. Clair, J. Schiffman, T. Jaeger, and P. McDaniel, “Establishing and Sustaining System Integrity via Root of comprehensive runtime integrity for application VMs. The Trust Installation,” in Proceedings of the 23rd ACSAC, 2007. architecture was based on a carefully crafted host system [20] K. J. Biba, “Integrity Considerations for Secure Computer whose long-term integrity can be justified easily using cur- Systems,” MITRE, Tech. Rep. MTR-3153, April 1977. rent techniques and a new component, called a VM verifier, [21] D. D. Clark and D. Wilson, “A Comparison of Military and Commercial Security Policies,” in 1987 IEEE Symposium on that can enforce our integrity target on VMs comprehen- Security and Privacy, May 1987. sively. We built a prototype VM Verifier for SELinux VMs, [22] J. G. Dyer, M. Lindemann, R. Perez, R. Sailer, L. van Doorn, and find that integrity-verified distributed compilation can S. W. Smith, and S. Weingart, “Building the IBM 4758 Secure Coprocessor,” Computer, vol. 34, no. 10, pp. 57–66, 2001. be implemented with less than 4% overhead. [23] J. Marchesini, S. Smith, O. Wild, and R. MacDonald, “Ex- REFERENCES perimenting with TCPA/TCG Hardware, Or: How I Learned to Stop Worrying and Love The Bear,” Dartmouth College, [1] “Amazon Elastic Compute Cloud,” http://aws.amazon.com. Tech. Rep. TR2003-476, 2003. [2] B. Gedik, H. Andrade, K.-L. Wu, P. Yu, and M. Doo, [24] L. Litty, H. A. Lagar-Cavilla, and D. Lie, “Hypervisor Support “SPADE: The System S Declarative Stream Processing En- for Identifying Covertly Executing Binaries,” in Proc. 17th gine,” in Proceedings of the 2008 ACM SIGMOD Conference. Usenix Security Symposium, San Jose, CA, Jul. 2008. [3] A. Baliga, P. Kamat, and L. Iftode, “Lurking in the Shadows: [25] P. A. Loscocco, P. W. Wilson, J. A. Pendergrass, and C. D. Identifying Systemic Threats to Kernel Data (Short Paper),” McDonell, “Linux Kernel Integrity Measurement Using Con- in 2007 IEEE Symposium on Security and Privacy, May 2007. textual Inspection,” in Proceedings of the 2007 ACM Work- [4] Amazon, “Amazon Web Services Security,” White paper. shop on Scalable Trusted Computing, New York, NY, USA. http://s3.amazonaws.com. [26] G. Coker, “Xen Security Modules (XSM),” http://www.xen. [5] R. Sailer, X. Zhang, T. Jaeger, and L. van Doorn, “Design org/files/xensummit 4/xsm-summit-041707 Coker.pdf. and Implementation of a TCG-based Integrity Measurement [27] “integrity: Linux Integrity Module.” [Online]. Available: Architecture,” in Proceedings of the 13th USENIX Security http://lwn.net/Articles/287790/ Symposium, San Diego, CA, USA, Aug. 2004. [28] S. Berger, R. Caceres,´ K. A. Goldman, R. Perez, R. Sailer, [6] H. Maruyama, F. Seliger, N. Nagaratnam, T. Ebringer, and L. van Doorn, “vTPM: Virtualizing the Trusted Platform S. Munetoh, S. Yoshihama, and T. Nakamura, “Trusted plat- Module,” in Proceedings of the 15th USENIX Security Sym- form on demand,” in IBM Technical Report RT0564, 2004. posium, Berkeley, CA, USA, 2006. [7] Microsoft Corporation, “Next generation secure computing [29] E. Brickell, J. Camenisch, and L. Chen, “Direct anonymous base,” http://www.microsoft.com/resources/ngscb/. attestation,” in Proceedings of the 11th ACM conference on [8] T. Jaeger, R. Sailer, and U. Shankar, “PRIMA: Policy- Computer and communications security, 2004. Reduced Integrity Measurement Architecture,” in Proceedings [30] R. Wojtczuk, “Subverting the Xen hypervisor.” www. of the 11th ACM SACMAT, 2006. blackhat.com/presentations/bh-usa-08/Wojtczuk. [9] S. Shrivastava, “Satem: Trusted Service Code Execution [31] D. G. Murray, G. Milos, and S. Hand, “Improving xen across Transactions,” in Proceedings of the 25th IEEE Sym- security through disaggregation,” in Proceedings of the 4th posium on Reliable Distributed Systems, 2006. ACM SIGPLAN/SIGOPS international conference on Virtual [10] E. Shi, A. Perrig, and L. van Doorn, “BIND: A Fine- execution environments, 2008, pp. 151–160. Grained Attestation Service for Secure Distributed Systems,” [32] B. Hicks, S. Rueda, L. St.Clair, T. Jaeger, and P. McDaniel, in Proceedings of the 2005 IEEE Symposium on Security and “A Logical Specification and Analysis for SELinux MLS Privacy, 2005. Policy,” in Proceedings of the 12th ACM SACMAT, 2007. [11] S. W. Smith, “Outbound Authentication for Programmable [33] D. Muthukumaran, A. Sawani, J. Schiffman, B. M. Jung, and Secure Coprocessors,” in Proceedings of the European Sym- T. Jaeger, “Measuring integrity on mobile phone systems,” in posium on Research in Computer Security (ESORICS), Oct. Proceedings of the 13th ACM SACMAT, 2008. 2002. [34] A. C. Myers and B. Liskov, “Protecting privacy using the de- [12] J. M. McCune, B. J. Parno, A. Perrig, M. K. Reiter, and centralized label model,” ACM Trans. Softw. Eng. Methodol., H. Isozaki, “Flicker: an execution infrastructure for tcb min- vol. 9, no. 4, pp. 410–442, 2000. imization,” in Proceedings of the 3rd ACM SIGOPS/EuroSys [35] T. Moyer, K. Butler, J. Schiffman, P. McDaniel, and T. Jaeger, 2008. “Scalable Web Content Attestation,” in To appear in Proceed- [13] J. M. McCune, A. Perrig, and M. K. Reiter, “Safe Passage ings of the IEEE 25nd ACSAC, 2009. for Passwords and Other Sensitive Data,” in Proceedings of NDSS, 2009.