The following paper was originally published in the Proceedings of the 8th USENIX Security Symposium Washington, D.C., USA, August 23–26, 1999 V A U L T E D V P N : C O M P A R T M E N T E D V I R T U A L P R I V A T E N E T W O R K S O N T R U S T E D O P E R A T I N G S Y S T E M S Tse-Huong Choo THE ADVANCED COMPUTING SYSTEMS ASSOCIATION © 1999 by The USENIX Association All Rights Reserved For more information about the USENIX Association: Phone: 1 510 528 8649 FAX: 1 510 548 5738 Email: [email protected] WWW: http://www.usenix.org Rights to individual papers remain with the author or the author's employer. Permission is granted for noncommercial reproduction of the work for educational or research purposes. This copyright notice must be included in the reproduced paper. USENIX acknowledges all trademarks herein. Vaulted VPN: Compartmented Virtual Private Networks On Trusted Operating Systems Tse-Huong Choo Hewlett-Packard Laboratories, Bristol, United Kingdom Abstract be compromised if an error occurs in the network protocol stack. Virtual Private Networks for IPSec based on an intermediate packet-redirector in network-protocol This report attempts to describe another approach that stacks are becoming increasingly common for many seeks to avoid some of these issues by standard operating systems and represent a well- compartmentalizing the components of a traditional understood method for retro-fitting such systems with VPN into isolated, independent processes. In IPSec support. This report describes how a different particular, we aim to show how a more secure version design structured around a Trusted Operating System of a conventional VPN can be built to make use of the can offer better security, performance and robustness. security-features present in a trusted operating system We describe in detail an implementation of an IPSec and also satisfy the goals of high-performance and VPN consisting of a series of compartmented, robustness. concurrently executing IPSec stacks. The motivations and security-related benefits behind each design The techniques used in designing a more secure VPN decision are discussed. In addition, we show how a will typically introduce performance degradations that configuration of independent IPSec stacks based on are otherwise not present on a standard VPN-design. this design can be configured to execute in parallel For example, the general principle of reducing a for greater performance on single-threaded kernels, monolithic system to a series of smaller, trusted and how its design allows individual component- components involves added overhead in failures without affecting the system as a whole. synchronization between these components. Therefore, the issues of security and performance have 1 Introduction to be considered as inextricably linked. These two issues, together with robustness, is fundamental to There is an increasing prevalence of IPSec Virtual many of the topics presented here. Private Networks (VPNs) based around an approach of inserting a plug-in into the network protocol stack This report describes the design and operation of the of mainstream operating-systems. This plug-in Vaulted VPN, which is a software-based IPSec VPN typically redirects any incoming or outgoing network built on a slightly modified version of the HP-UX packet in order to perform IPSec-specific processing 10.24 Virtual Vault Operating System. We begin by such as encryption or decryption. This approach is describing briefly the security-features of HP-UX well understood and has been mentioned in many 10.24 relevant to the Vaulted VPN and continue by standard texts, most notably the IETF RFCs for IPSec describing the architecture and operation of the [2][3]. Vaulted VPN, followed by a security analysis of each aspect of VPN’s operation. However, this approach is not without its weaknesses. The important, but often conflicting design goals of We also show how compartmented VPNs such as the security, performance and robustness often leads to Vaulted VPN can take advantage of parallel cases where the development of one goal requires processing and how they can continue to function even trade-offs in the other. For example, placing IPSec- if some portions fail – not an unimaginable occurrence processing in the kernel or operating system level is given the many well-documented attacks using desirable performance-wise because of copy- malformed IP packets [19] [20]. This represents an avoidance of packets between kernel and user-mode unexpected level of robustness considering the spaces. However, the kernel or operating system can monolithic nature of most network protocol stacks. Although this report mentions the use of HP-UX 10.24 2.3 Least Privilege specifically as a target platform, the principles shown here could be applied to full-blown CMW platforms On a traditional UNIX system, the ability to override [9] such as HP-UX 10.26 or Sun Microsystems’ security controls rests entirely within the root-account. Trusted Solaris 2.5. On trusted systems, this all-powerful ability has been broken down into a large number of individual 2 Features of the HP-UX 10.24 Virtual privileges, each of which confers its holder the ability to perform a specific action. Every action that could Vault Operating System affect security is restricted with a named privilege, and only processes with the appropriate privileges are The target platform of the Vaulted VPN is the HP-UX allowed to perform these actions. Following the 10.24 Virtual Vault Operating System. The HP-UX principle of least privilege, this finer level of 10.24 operating system is a UNIX-style Multi-Level granularity allows specific programs to be assigned Secure (MLS) operating system that implements with the minimum-required set of privileges in order additional security-features beyond those normally for them to perform their assigned tasks. available on standard UNIX systems. It is a CMW- derived operating system [6] [9] and features the Mandatory Access Controls (MAC) and principles of 3 IPSec Protocol least-privilege found on full CMW-type systems, but differs in that it lacks the Trusted Windowing and A significant fraction of commercial VPNs are based Trusted Networking features of a full CMW-type on IPSec technology which defines the encoding of system. This section describes the security-features the encrypted IP packets. Normal unencrypted IP used by the Vaulted VPN. packets are made up of an IP-header followed by a variable length payload in clear-text which may itself 2.1 Mandatory Access Control include headers for higher-level protocols, e.g. TCP or UDP. In addition to the standard UNIX-style security features, HP-UX 10.24 also implements a Mandatory IPSec packets appear to be normal IP packets to Access Control policy based on the Bell-LaPadula switching and routing hardware, but feature internal formal model [11]. Mandatory Access Control (MAC) payloads which may be encrypted, authenticated or is a set of rules that govern how data may be accessed both. There are two formats used: the AH format [5] on the trusted system. The policy is called mandatory which allows data to be authenticated using a keyed because users cannot choose which data will be hash-function, and the ESP format [4] in which data is regulated under this policy – the trusted system encrypted and optionally authenticated, depending on enforces MAC consistently. In this model, all which encryption algorithm is selected. resources or objects in the system are given sensitivity labels, which consists of a classification and a number In addition, IPSec allows for two main modes of of compartments. Access to an object is controlled by operation, namely transport and tunnel modes. Tunnel- its sensitivity label. To have read-access, a process mode allows entire IP packets of arbitrary protocols to must have a sensitivity label which ‘dominates’ that of be encapsulated within another IP header to allow for the object being read. For write-access, the labels must authentication of the original IP packet’s header, and is match exactly. typically used in creating an encrypted tunnel between gateways. 2.2 Discretionary Access Control The IPSec protocol may be imagined to be operate at a HP-UX 10.24 has discretionary access controls (DAC) level below or at the IP-layer of network protocol which are similar to those found on standard UNIX stacks. The secret keys used to encrypt the payloads systems, specifically the read-write-execute mode-bits are derived earlier using some higher key-agreement based on user and group identities. It is most effective protocol (such as IKE [1]) or simply manually keyed. when used in conjunction with Mandatory Access Control, for example, when used to restrict access to everyone but a special pseudo-user used by specific trusted processes. 4 Vaulted VPN Architecture Since each process is placed at a given sensitivity level, and access to objects at different sensitivity- This section gives an overview of the architecture of levels is controlled strictly by the operating system the Vaulted VPN by describing the purpose and using MAC controls - one can imagine that a process function of each of its components. An explanation is at a given sensitivity-level to be in a separate given on how packets are routed in and out of the ‘compartment’ from other processes of differing kernel, where IPSec processing takes place and how levels. The term ‘compartment’ is used loosely here to Internet Key Exchange (IKE) [1] negotiations are refer to the segmentation that arises as a result of triggered to form IPSec Security-Associations. applying MAC controls to labeled objects, and does not refer explicitly to the term used in defining 4.1 Design of Conventional Bump-in-the- sensitivity-labels.