SkyBridge: Fast and Secure Inter-Process Communication for Microkernels Zeyu Mi, Dingji Li, Zihan Yang, Xinran Wang, Haibo Chen Shanghai Key Laboratory for Scalable Computing Systems Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University Abstract ACM Reference Format: Zeyu Mi, Dingji Li, Zihan Yang, Xinran Wang, Haibo Chen. 2019. Microkernels have been extensively studied over decades. SkyBridge: Fast and Secure Inter-Process Communication for Mi- However, IPC (Inter-Process Communication) is still a ma- crokernels. In Fourteenth EuroSys Conference 2019 (EuroSys ’19), jor factor of run-time overhead, where fine-grained isolation March 25–28, 2019, Dresden, Germany. ACM, New York, NY, usually leads to excessive IPCs. The main overhead of IPC USA, 15 pages. hps://doi.org/10.1145/3302424.3303946 comes from the involvement of the kernel, which includes the direct cost of mode switches and address space changes, as well as indirect cost due to the pollution of processor struc- 1 Introduction tures. Microkernels have been extensively studied over the past In this paper, we present SkyBridge, a new communica- four decades [22, 28, 29, 36, 42, 43, 48, 60, 63]. The key tion facility designed and optimized for synchronous IPC design is to deprivilege most kernel functionalities into dif- in microkernels. SkyBridge requires no involvement of ker- ferent servers residing in isolated user processes. The ker- nels during communication and allows a process to directly nel provides basic functionalities, such as process manage- switch to the virtual address space of the target process and ment, capability enforcement and inter-process communica- invoke the target function. SkyBridge retains the traditional tion (IPC). Such a decentralized design makes the OS archi- virtual address space isolation and thus can be easily inte- tecture robust against run-time errors, which means a fault grated into existing microkernels. The key idea of SkyBridge within one server would not affect other servers and the ker- is to leverage a commodity hardware feature for virtualiza- nel. Removing most functionalities from the kernel also re- tion (i.e., VMFUNC) to achieve efficient IPC. To leverage sults in a small Trusted Computing Base (TCB), making it the hardware feature, SkyBridge inserts a tiny virtualization less vulnerable to attacks and possible for comprehensive layer (Rootkernel) beneath the original microkernel (Subker- formal verification [36]. Given such advantages, microker- nel). The Rootkernel is carefully designed to eliminate most nels [29, 34] are widely used in various areas where high re- virtualization overheads. SkyBridge also integrates a series liability matters, such as aerospace, automotive and medical of techniques to guarantee the security properties of IPC. devices. We have implemented SkyBridge on three popular open- In a microkernel, any communication between different source microkernels (seL4, Fiasco.OC, and Google Zircon). user processes is based on IPC, which is an intensively-used The evaluation results show that SkyBridge improves the operation. For example, if a client process writes data into speed of IPC by 1.49x to 19.6x for microbenchmarks. For an external block device, it first communicates with the file real-world applications (e.g., SQLite3 database), SkyBridge system, which in turn notifies the disk device driver to write improves the throughput by 81.9%, 1.44x and 9.59x for the data into the block device. All the communication is done three microkernels on average. via IPC. In fact, IPC is known as a major factor of run- time overhead [20, 27, 40, 52], which determines the per- formance of applications on microkernels. Transferring con- trol across process boundaries is expensive, which requires at least: a trap into the microkernel (SYSCALL instruction), Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not data copying for arguments, one address space switch (even made or distributed for profit or commercial advantage and that copies bear two switches if considering the recent Meltdown attack [45]), this notice and the full citation on the first page. Copyrights for components and an upcall back to the user level. Such operations must be of this work owned by others than ACM must be honored. Abstracting with repeated upon IPC return. Some asynchronous implementa- credit is permitted. To copy otherwise, or republish, to post on servers or to tion of IPC even involves costly scheduling work. redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. A large body of research has been done to optimize the EuroSys ’19, March 25–28, 2019, Dresden, Germany IPC performance. Software-based solutions try to shorten © 2019 Association for Computing Machinery. the IPC path by removing unnecessary operations. seL4 [36] ACM ISBN 978-1-4503-6281-8/19/03. $15.00 uses the IPC fastpath for the case of Call and ReplyWait hps://doi.org/10.1145/3302424.3303946 system calls where the IPC message fits in CPU registers, 1 EuroSys ’19, March 25–28, 2019, Dresden, Germany Zeyu Mi, Dingji Li, Zihan Yang, Xinran Wang, Haibo Chen and no capabilities are transferred. For a fastpath, the mes- a tiny virtualization layer (called Rootkernel) only consist- sage will be sent immediately and the control flow will be ing of the most primitive functionalities for SkyBridge while directly transferred without entering into the costly sched- eliminating VM exits during an IPC. uling logic. Similarly, some software-based solutions like Second, existing ways [41, 46] of leveraging VMFUNC LRPC [8] also eliminate the scheduling overhead and allow require non-trivial modification to the microkernels and thus a process’s thread to execute requested procedures in the re- tremendous engineering effort. SkyBridge proposes a light- ceiver’s address space. However, all such approaches still weight method to efficiently switch virtual address spaces require the involvement of the kernel and thus their perfor- among different processes which can be easily integrated mance (around 1000 cycles for an IPC roundtrip) do not sat- into microkernel architectures. isfy the requirement of IPC-intensive workloads, as shown Third, it is difficult to design a secure IPC facility without in Section 2. Hardware-based solutions propose new hard- the involvement of the kernel, especially when one malicious ware extensions to boost IPC operation. dIPC [55] puts all process can exploit the VMFUNC instruction to corrupt IPC participants into a single address space, and the ker- other processes [46]. SkyBridge guarantees that there is only nel is removed from the IPC path. The process isolation one legal entry point for switching address spaces among is achieved by the newly designed tagged memory. Such processes, which prevents a malicious process from invok- hardware-based solutions usually require non-trivial modifi- ing self-prepared VMFUNC instructions to corrupt other pro- cation to both hardware and software, which have less poten- cesses. SkyBridge also requires a process to register to other tial for practical adoption. processes before communicating with them and introduces a Therefore, we argue that there is a need for an IPC tech- calling-key table mechanism to enforce such a policy. nique that satisfies the following requirements. We have implemented SkyBridge on three different micro- kernels (seL4 [36], Fiasco.OC [1] and Google Zircon [5]) • Efficient: the IPC path does not involve the kernel. and deployed them on a commodity Intel Skylake machine. • Lightweight: the IPC can be readily deployed on com- Our evaluation shows that SkyBridge significantly improves modity hardware and can be easily integrated into ex- the performance of IPC by 1.49x, 5.86x, and 19.6x for seL4 isting microkernel architecture. (fastpath), Fiasco.OC and Zircon respectively. For a real- • Secure: the IPC design does not break the microkernel world application like a multi-tier SQLite3 workload, Sky- isolation abstraction. Bridge improves the performance by 81.9%, 1.44x and 9.59x for such three microkernels on average. In this paper, we present a new IPC design that meets Contributions. The contributions of the paper are summa- such requirements. Our design, called SkyBridge, allows one rized as follows: process (sender) to directly execute the requested procedure in another process’s (receiver) address space without trap- • A detailed analysis of the performance overheads of ping into the kernel. SkyBridge has two main technical ad- IPC in state-of-the-art microkernels. vantages. First, SkyBridge still places each process in its • A new design which can significantly improve the per- own virtual address space which fits well with the design formance of the microkernel IPC without any modifi- and implementation of existing microkernels. Second, Sky- cation to the hardware. Bridge leverages one Intel hardware feature for virtualiza- • An implementation of SkyBridge and an evaluation us- tion, named EPT (extended page table) switching (the VM- ing real-world benchmarks on three different microker- FUNC instruction), to change the virtual address space at nels. the user level. By configuring the receiver’s EPT, SkyBridge maps the page table of the sender to that of the receiver. 2 Motivation and Background Therefore, after switching the EPT by VMFUNC, the hard- 2.1 Deconstructing Synchronous IPC ware uses the receiver’s page table to translate all subsequent virtual addresses. SkyBridge also provides a separated stack In this section, we evaluate the performance costs associ- for each receiver’s thread in its virtual address space. To sup- ated with the traditional synchronous inter-process call (IPC) port long IPC, SkyBridge provides shared buffers for the in microkernels. We use seL4 [36] (v10.0.0) on an Intel Sky- IPC participants when large messages are transferred. Each lake processor to conduct all the experiments. seL4 is known buffer is bound to one receiver’s thread for concurrency.