mmapx: Uniform memory protection in a heterogeneous world Reto Achermann* David Cock* Roni Haecki* University of British Columbia ETH Zurich ETH Zurich Vancouver, Canada Zurich, Switzerland Zurich, Switzerland Nora Hossle* Lukas Humbel* Timothy Roscoe* ETH Zurich ETH Zurich ETH Zurich Zurich, Switzerland Zurich, Switzerland Zurich, Switzerland Daniel Schwyn* ETH Zurich Zurich, Switzerland ABSTRACT CCS CONCEPTS Modern Systems-on-Chip (SoCs) are networks of hetero- • Software and its engineering ! Operating systems; Mem- geneous cores, intelligent devices, and memory, connected ory management; Virtual memory. through multiple configurable address translation and protec- tion units like IOMMUs and System MMUs. KEYWORDS Modern OS kernels like Linux are based on traditional operating systems, memory management, address spaces MMUs and have no clear abstractions to represent this com- plexity, mostly leaving IOMMU configuration to device dri- ACM Reference Format: vers. This has led to a recent spate of serious bugs, and increas- Reto Achermann, David Cock, Roni Haecki, Nora Hossle, Lukas ing concern over “cross-SoC” attacks on memory security. Humbel, Timothy Roscoe, and Daniel Schwyn. 2021. mmapx: Uni- Workshop To address this, we propose a new kernel primitive, mmapx, form memory protection in a heterogeneous world. In on Hot Topics in Operating Systems (HotOS ’21), May 31–June 2, based on a decoding net a rich and detailed representation of 2021, Ann Arbor, MI, USA. ACM, New York, NY, USA, 8 pages. the memory addressing semantics of a complex SoC from the https://doi.org/10.1145/3458336.3465273 recent formal methods literature. mmapx provides a uniform facility for securely configuring all the address translation facilities in a system. 1 INTRODUCTION mmapx leverages existing Unix facilities wherever possi- Memory management in a modern operating system provides ble: the file system for naming, discovery, and coarse-grained protection via a simple model: client software (either a user- access control, and file descriptors for fine-grained autho- space process or an in-kernel driver) requests access to a rization. We show how mmapx can eliminate bugs caused by physical resource (either memory or device registers) by iden- device drivers programming IOMMUs directly, but also the tifying its location in the system-wide physical address space. detail captured by the underlying model has further benefits The OS checks authorization, and then maps the appropriate while incurring minimal overhead. physical region into the client’s virtual address space. Un- fortunately, these days this appealingly simple model just *Authors are in alphabetical order doesn’t work. Modern SoCs have complex interconnects and peripheral Permission to make digital or hard copies of part or all of this work for devices, and much of what Linux models as devices are really personal or classroom use is granted without fee provided that copies are not complete processors with their own firmware or independent made or distributed for profit or commercial advantage and that copies bear operating systems, often incorporating their own memory this notice and the full citation on the first page. Copyrights for third-party translation units. For protection to be useful, these devices components of this work must be honored. For all other uses, contact the must be sandboxed behind a correctly-configured IOMMU owner/author(s). (or System MMU). HotOS ’21, May 31–June 2, 2021, Ann Arbor, MI, USA © 2021 Copyright held by the owner/author(s). However, programming IOMMUs is complex and error- ACM ISBN 978-1-4503-8438-4/21/05. prone, and is often delegated to individual device drivers, https://doi.org/10.1145/3458336.3465273 which have ambient authority in the kernel. The task is made 159 HotOS ’21, May 31–June 2, 2021, Ann Arbor, MI, USA Achermann, et al. worse by the need to enforce a changing partial correspon- QuRT Linux dence between the virtual address space seen by the device, and that seen by a process, since the OS needs to share datas- A57 DSP DSP tructures with devices as much as protect itself from them. Registers MMU The result is that buggy, compromised, or just plain malicious devices or drivers can do an end-run around the OS protection MMU SMMU model by exploiting holes in the IOMMU-based protection domain [14, 22]. DRAM System Address Space Surprisingly, modern OSes provide no good abstractions for uniformly handling this problem, leaving low-level con- figuration of protection up to individual device drivers. This Figure 1: Relevant actors (A57 and DSP), translation is in contrast to, e.g. access control in file systems or au- units (MMU, SMMU) and memory regions (gray) of the thority over process address space, where well-established Qualcomm SoC. Note that the DSP MMU is not con- subsystems enforce OS policy. trolled by the host Linux, but the SMMU is. We propose a new primitive, mmapx, for clients to request general memory mappings. Unlike existing interfaces, mmapx is explicit about which address space it is mapping a region between the model of hardware behavior baked into the OS from, and which address space it is mapping a region into. kernel, and the reality of modern SoC platforms. This allows clients to be precise in specifying what memory QualPwn affects mobile SoCs running Android and starts is exposed to devices or coprocessors via IOMMUs. Like with a bug in the WLAN process running not on the CPU but mmap(), mmapx refers to memory regions using file descrip- a DSP core on the chip, which runs the proprietary QuRT OS. tors, which provide capability-like protection. However, re- A series of exploits allows compromise of another process on gions for mmapx are acquired using a file system (/dev/as), the DSP, which itself communicates with its corresponding which captures the memory topology of the machine in detail, device driver in the Linux kernel on the application cores and allows basic authorization to leverage the full Unix file using DMA. Since the driver trusts the device, it can be tricked system protection model via two different rights on an address into granting the device full access to application memory by space: map and grant. reprogramming the system MMU. We describe mmapx here from the point of view of user- Our focus in this paper is preventing incorrect granting of space Linux programs but the basic model works in a micro- memory access rights to devices and drivers. A simplified kernel architecture, or within a monolithic kernel. In the latter view of the hardware is shown in Figure 1: two processors, case, protection within the kernel is only advisory, unless running a different OS, with different MMUs, but sharing a mechanism like Nooks [28] is available. Crucially, how- the same memory. The Linux kernel driver is tasked with ever, even in this case mmapx provides a policy framework configuring the SMMU to only allow legitimate access to for protecting the kernel itself from malicious or buggy dri- buffers shared between the two cores, and it fails to do this. vers, device firmware, or other cores not running the kernel Linux offers little functionality to help with this task. In- itself by ensuring that IOMMUs and other translation units stead, it relies on a naive model where a set of process vir- are correctly programmed. tual address spaces are mapped to a single physical address Moreover, while mmapx resembles a high-level primitive space, and protection against DMA-capable devices using like mmap(), this belies its true power. mmapx builds on our an IOMMU or SMMU is delegated to drivers. Indeed, the existing work on formalizing address translation and decod- SMMU is often programmed to give a device the same view ing, and its concept of an “address space” region is flexible of memory as that of the corresponding software process, and powerful enough to capture the functionality of the full whether in user space or the kernel. range of TLBs, and even individual levels of a multi-level For example, the recent Linux Heterogeneous Memory page table. This allows mmapx to express “delegation” of Manager [29] attempts to unify device memory management MMU page table structures to virtual machines, for example, using a specialized page structure to replicate translation as in Arrakis [24] and Ix [6]. across device and CPU address spaces, in order to simplify programming with GPU and FPGA accelerators. We argue this is inappropriate for devices: access by the device should be restricted as much as possible, rather than 2 MOTIVATION giving the device free rein over application memory [22]. The “QualPwn” exploit [14] is emblematic of the problem However, Linux provides no help in maintaining partially we address in this paper. It makes brutally clear the mismatch replicated mappings between heterogeneous devices or cores: 160 mmapx: Uniform memory protection in a heterogeneous world HotOS ’21, May 31–June 2, 2021, Ann Arbor, MI, USA there are simply no abstractions for explicitly changing SMMU base asid right z }| { z}|{ z }| { mappings. /dev/as /0 /{grant,map,info} We are not first to point this out[13], and so-called “cross- SoC” attacks [4, 14, 26] and other DMA-related vulnerabili- ties [18, 21, 23, 25, 32] which exploit the fact that the OS is Figure 2: The path structure of /dev/as not in control of software running on other cores on the SoC is becoming a serious security concern. Neither does moving the device driver out of the kernel solve the problem, whether decoding net representation of the hardware using standard into Linux user space or using a microkernel-oriented design. device discovery and Devicetree representations [20]. Our solution to this is to provide a single primitive for con- For example, the Linux kernel running on the A57 in Fig- figuring any translation hardware in the system, with a clear ure 1 would detect the presence of the modem coprocessor and secure authorization framework that leverages existing from a Devicetree entry, and would create the corresponding mechanisms in the kernel.