SemperOS: A Distributed Capability System Matthias Hille, Technische Universität Dresden; Nils Asmussen, Technische Universität Dresden; Barkhausen Institut; Pramod Bhatotia, University of Edinburgh; Hermann Härtig, Technische Universität Dresden; Barkhausen Institut https://www.usenix.org/conference/atc19/presentation/hille This paper is included in the Proceedings of the 2019 USENIX Annual Technical Conference. July 10–12, 2019 • Renton, WA, USA ISBN 978-1-939133-03-8 Open access to the Proceedings of the 2019 USENIX Annual Technical Conference is sponsored by USENIX. SemperOS: A Distributed Capability System Matthias Hille† Nils Asmussen† ∗ Pramod Bhatotia‡ Hermann Härtig† ∗ †Technische Universität Dresden ‡The University of Edinburgh ∗ Barkhausen Institut Abstract systems have received renewed attention recently to provide Capabilities provide an efficient and secure mechanism for an efficient and secure mechanism for resource management fine-grained resource management and protection. However, in modern hardware architectures [5, 24, 30, 36, 44, 64, 67]. as the modern hardware architectures continue to evolve Today the main improvements in compute capacity with large numbers of non-coherent and heterogeneous cores, are achieved by either adding more cores or integrating we focus on the following research question: can capability accelerators into the system. However, the increasing core systems scale to modern hardware architectures? counts exacerbate the hardware complexity required for global In this work, we present a scalable capability system to drive cache coherence. While on-chip cache coherence is likely to future systems with many non-coherent heterogeneous cores. remain a feature of future hardware architectures [45], we see More specifically, we have designed a distributed capability characteristics of distributed systems added to the hardware system based on a HW/SW co-designed capability system. by giving up on global cache coherence across a whole We analyzed the pitfalls of distributed capability operations machine [6, 28]. Additionally, various kinds of accelerators running concurrently and built the protocols in accordance are added like the Xeon Phi Processor, the Matrix-2000 accel- with the insights. We have incorporated these distributed erator, GPUs, FPGAs, or ASICs, which are used in numerous capability management protocols in a new microkernel-based application fields [4,21,32,34,43,60]. These components also OS called SEMPEROS. Our OS operates the system by contribute to the number of resources an OS has to manage. means of multiple microkernels, which employ distributed In this work, we focus on capability-based systems and how capabilities to provide an efficient and secure mechanism their ability to implement fine-grained access control combines for fine-grained access to system resources. In the evaluation with large systems. In particular, we consider three types of we investigated the scalability of our algorithms and run capability systems: L4 [30], CHERI [67] (considered for The applications (Nginx, LevelDB, SQLite, PostMark, etc.), which Machine [31]), and M3 [5] (see Section 2.1). For all these are heavily dependent on the OS services of SEMPEROS. The capability types it is not clear whether they will scale to modern results indicate that there is no inherent scalability limitation hardware architectures since the scalability of capability for capability systems. Our evaluation shows that we achieve systems has never been studied before. Also existing capability a parallel efficiency of 70% to 78% when examining a system schemes cannot be turned into distributed schemes easily since with 576 cores executing 512 application instances while they either rely on centralized knowledge, cache-coherent using 11% of the system’s cores for OS services. architectures, or are missing important features like revocation. Independent of the choice which capability system to use, 1 Introduction scaling these systems calls for two basic mechanisms to be fast. First, it implies a way of concurrently updating access Capabilities are unforgeable tokens of authority granting rights to enable fast decentralized resource sharing. This rights to resources in the system. They can be selectively means fast obtaining or delegating of capabilities, which delegated between constrained programs for implementing acquires or hands out access rights to the resources behind the the principle of least authority [48]. Due to their ability of capabilities. The other performance-critical mechanism is the fine-grained resource management and protection, capabilities revocation of capabilities. Revoking the access rights should be appear to be a particularly good fit for future hardware possible within a reasonable amount of time and with minimal architectures, which envision byte granular memory access to overhead. The scalability of this operation is tightly coupled to large memories (NVRAM) from a large numbers of cores (e.g. the enforcement mechanism, e.g. when using L4 capabilities The Machine [31], Enzian [18]). Thereby, capability-based the TLB shootdown can be a scalability bottleneck. USENIX Association 2019 USENIX Annual Technical Conference 709 We base our system on a hardware/software co-designed ca- L4 M3 CHERI pability system (M3). More specifically, we propose a scalable Scope Coherence Dom. Machine Address space distributed capability mechanism by building a multikernel Enforcement MMU / Kernel DTU / Kernel CHERI co-proc. Limitation Coherence Dom. Core count no revoke OS based on the M3 capability system. We present a detailed analysis of possible complications in distributed capability systems caused by concurrent updates. Based on this inves- Table 1: Classification of capability types. tigation we describe the algorithms, which we implemented L4 microkernels [30,36,40,62], (2) instruction set architecture in our prototype OS—the SEMPEROS multikernel. (ISA) capabilities, as implemented by the CAP computer [49] Our OS divides the system into groups, with each of them and recently revived by CHERI [67] and (3) sparse capabilities being managed by an independent kernel. These independently which are deployed in the password-capability system of the managed groups resemble islands with locally managed re- Monash University [2] and in Amoeba [63]. sources and capabilities. Kernels communicate via messages to Capabilities can be shared to exchange access rights. ISA enable interaction across groups. To quickly find objects across capabilities and sparse capabilities can be shared without the whole system—a crucial prerequisite for our capability involving the kernel since their validity is either ensured by scheme—we introduce an efficient addressing scheme. hardware or checked by the kernel using a cryptographic Our system design aims at future hardware, which might one-way function in the moment they are used. In contrast, connect hundreds of processing elements to form a powerful sharing of partitioned capabilities is observed by the kernel. rack-scale system [28]. To be able to experiment with such To analyze capability systems regarding their scalability, we systems, we use the gem5 simulator [14] Our evaluation inspect their enforcement mechanism, their scope, and their focuses on the performance of the kernel, where we showcase limitation in Table 1. The three categories of capability systems the scalability of our algorithms by using microbenchmarks in Table 1 represent a relevant subset of capability systems for as well as OS-intensive real-world applications. We describe the scope of this work: (1) L4 capabilities which are partitioned trade-offs in resource distribution between applications and capabilities employed in L4 µ-kernels [30,36,40,62], (2) M3 the OS to determine a suitable configuration for a specific capabilities which are a special form of partitioned capabilities application. We found that our OS can operate a system which involving a different enforcement mechanism explained in the hosts 512 parallel running application instances with a parallel following, and lastly (3) CHERI capabilities which are ISA efficiency of 70% to 78% while dedicating 11% of the system capabilities implemented by the CHERI processor [64, 67]. to the OS and its services. L4 capabilities utilize the memory management unit To summarize, our contributions are as follows. (MMU) of the processor to restrict a program’s memory access. • We propose a HW/SW co-designed distributed capability Access to other resources like communication channels or system to drive future systems. Our capability system process control are enforced by the kernel. Since L4 is built 3 extends M capabilities to support a large number of for cache coherent machines both the scope of a capability and non-cache-coherent heterogeneous cores. the current limitation is a coherence domain. • We implemented a new microkernel-based OS called In contrast, M3 introduces a hardware component called SEMPEROS that operates the system by employing data transfer unit (DTU) which provides message passing multiple microkernels and incorporates distributed between processing elements and a form of remote direct mem- capability management protocols. ory access. Consequently, memory access and communication • We evaluated the distributed capability management channels are enforced by the DTU and access to other system protocols by implementing the HW/SW co-design for resources by the kernel. The DTU is the only possibility SEMPEROS in the gem5 simulator [14] to run real for