Program-mandering: Quantitative Privilege Separation Shen Liu∗ Yongzhe Huang Frank Capobianco Dongrui Zeng∗ [email protected] [email protected] [email protected] The Pennsylvania State University The Pennsylvania State University [email protected] University Park, Pennsylvania University Park, Pennsylvania The Pennsylvania State University University Park, Pennsylvania Stephen McCamant Trent Jaeger Gang Tan [email protected] [email protected] [email protected] University of Minnesota The Pennsylvania State University The Pennsylvania State University Twin Cities, Minnesota University Park, Pennsylvania University Park, Pennsylvania ABSTRACT ACM Reference Format: Privilege separation is an effective technique to improve software Shen Liu, Dongrui Zeng, Yongzhe Huang, Frank Capobianco, Stephen McCa- security. However, past partitioning systems do not allow program- mant, Trent Jaeger, and Gang Tan. 2019. Program-mandering: Quantitative Privilege Separation. In 2019 ACM SIGSAC Conference on Computer and mers to make quantitative tradeoffs between security and perfor- Communications Security (CCS ’19), November 11–15, 2019, London, United mance. In this paper, we describe our toolchain called PM. It can Kingdom. ACM, New York, NY, USA, 18 pages. https://doi.org/10.1145/ automatically find the optimal boundary in program partitioning. 3319535.3354218 This is achieved by solving an integer-programming model that optimizes for a user-chosen metric while satisfying the remaining security and performance constraints on other metrics. We choose 1 INTRODUCTION security metrics to reason about how well computed partitions Privilege separation in software systems refers to the process of enforce information flow control to: (1) protect the program from decomposing a system into multiple modules, each loaded into a sep- low-integrity inputs or (2) prevent leakage of program secrets. As arate protection domain. Privilege separation prevents a software a result, functions in the sensitive module that fall on the optimal system from being compromised completely by a single vulnerabil- partition boundaries automatically identify where declassification ity because any compromised protection domain cannot directly is necessary. We used PM to experiment on a set of real-world access the code or data of the parts of the system running in other programs to protect confidentiality and integrity; results show that, protection domains. Calls to functions in other protection domains with moderate user guidance, PM can find partitions that have are converted into remote procedure calls (RPCs) and data access better balance between security and performance than partitions is restricted to protection domains where necessary. found by a previous tool that requires manual declassification. While there is clear potential to improve software security through the use of privilege separation, programmers face challenges in CCS CONCEPTS leveraging privilege separation to achieve security guarantees, to • Security and privacy → Software and application security; refactor software systems into working modules, and to maintain • Software and its engineering → Automated static analysis; efficient performance in the decomposed system. First, systems Dynamic analysis. that are privilege separated often aim to assign least privilege [22] permissions to each protection domain, but it is unclear whether KEYWORDS such permission assignments achieve security guarantees. For ex- ample, OpenSSH was manually refactored by Provos et al. [20] Automatic program partitioning, privilege separation, integer pro- into one privileged server process and many unprivileged monitor gramming processes, each of which handles a user connection. In this setup, a compromised monitor process should not affect the server process ∗Both authors contributed equally to this research and are co-first authors. or other monitor processes. However, only later was it shown that the privilege separated OpenSSH achieved an approximation of the strong Clark-Wilson integrity model [23]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed Second, manually privilege separating complex software is labor for profit or commercial advantage and that copies bear this notice and the full citation intensive. Automatic privilege separation aims to partition software on the first page. Copyrights for components of this work owned by others than the with little user involvement. For instance, Privtrans [4] takes as author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission input C source code and user annotations about sensitive data and/or a fee. Request permissions from [email protected]. and declassification, and employs static analysis to separate the CCS ’19, November 11–15, 2019, London, United Kingdom application into a master process that handles privileged operations © 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-6747-9/19/11...$15.00 and an unprivileged slave process. Automatic privilege separation https://doi.org/10.1145/3319535.3354218 has great potential for improving software security. Third, while security is the motivating goal in performing privi- • PM is a privilege-separation framework that guides users lege separation, the performance implications of the resulting pro- to make quantitative tradeoffs between security and perfor- gram must also be carefully considered. No matter how protection mance. By converting privilege separation into an integer- domains are isolated (e.g., via the OS process isolation or via some programming problem, it automatically computes the op- hardware mechanism such as Intel’s SGX), there is invariably a timal partition, with respect to user-specified budgets on performance cost when data and control cross protection-domain security and performance. boundaries; as a result, refactoring a monolithic application into • PM is the first system that combines quantitative information multiple modules in different protection domains comes with a flow with privilege separation. This not only provides a performance cost, incurred by changing local data/code accesses security metric that aligns well with security goals common into remote data/code accesses, which cross the partition boundary. in applications, but also reduces users’ burden of performing Importantly, the performance cost depends on how the application manual declassification—the optimal partition computed by is partitioned; that is, how boundaries of modules are drawn in the PM automatically gives where data should be declassified. application and what code is duplicated. If considering only per- • We have implemented PM and evaluated it on a set of real formance, one would just put all code into one protection domain, world programs. Our experience shows that PM helps users reverting back to the original monolithic application; however, se- make quantitative trade-offs among multiple factors. After curity would not be improved. Similarly, considering only security observing initial partitions, users could use PM to improve could result in bad performance. the balance between security and performance by setting Therefore, being able to achieve security goals while still allow- simple constraints, in an iterative process. ing users to make tradeoffs between security and performance is essential for the success of privilege separation. We call security and performance partitioning factors, as they are critical for partition- 2 RELATED WORK ing. Many previous tools [4, 14–16, 21, 29, 30], however, consider Several tools have been proposed to assist programmers in man- only security during privilege separation. Some of these tools (e.g., ually partitioning their applications, including Privman [13] and [4, 15]) require user-specified data declassification, a process that Wedge [3]. However, they require programmers to manually figure allows sensitive data flow from a sensitive domain (i.e., the domain out a good partition boundary. A number of tools [4, 14–16, 21, 29, that processes sensitive data) to an insensitive domain,toprevent 30] have been created for automatically partitioning applications leakage of secret data and/or use of low-integrity data. For exam- using program analysis. These tools’ partitioning algorithms, how- ple, suppose there is an authentication function that uses a secret ever, consider only security but not other factors; as a result, they password and a client function f that invokes the authentication do not allow tradeoffs among multiple factors. Further, these tools function. By declassifying the return value of the authentication partition programs, but lack consideration for helping users achieve function, f can then be put into the insensitive domain. This man- security goals; e.g., they expect users to manually choose where ual declassification process, however, burdens programmers as they to declassify information flows and sometimes do not account for have to decide where to perform declassification. In general, pro- flows to external channels. grammers have to evaluate the performance impact of boundary ProgramCutter [28], Swift [6], and SOAAP [12] are partitioning crossings and the possible security impact of information flows at systems that consider both security and performance, but none