Keystone: An Open Framework for Architecting Trusted Execution Environments Dayeol Lee David Kohlbrenner Shweta Shinde [email protected] [email protected] [email protected] UC Berkeley UC Berkeley UC Berkeley Krste Asanović Dawn Song [email protected] [email protected] UC Berkeley UC Berkeley Abstract 1 Introduction Trusted execution environments (TEEs) see rising use in The last decade has seen the proliferation of trusted exe- devices from embedded sensors to cloud servers and en- cution environments (TEEs) to protect sensitive code and compass a range of cost, power constraints, and security data. All major CPU vendors have rolled out their TEEs threat model choices. On the other hand, each of the current (e.g., ARM TrustZone, Intel SGX, and AMD SEV) to provide vendor-specific TEEs makes a fixed set of trade-offs with a secure execution environment, commonly referred to as little room for customization. We present Keystone—the first an enclave [1, 51, 64]. TEEs have use-cases in diverse de- open-source framework for building customized TEEs. Key- ployment environments ranging from cloud servers, mobile stone uses simple abstractions provided by the hardware phones, ISPs, IoT devices, sensors, and hardware tokens. such as memory isolation and a programmable layer under- Unfortunately, each vendor TEE enables only a small por- neath untrusted components (e.g., OS). We build reusable tion of the possible design space across threat models, hard- TEE core primitives from these abstractions while allowing ware requirements, resource management, porting effort, platform-specific modifications and flexible feature choices. and feature compatibility. When a cloud provider or soft- We showcase how Keystone-based TEEs run on unmodified ware developer chooses a target hardware platform they are RISC-V hardware and demonstrate the strengths of our de- locked into the respective TEE design limitations regardless sign in terms of security, TCB size, execution of a range of of their actual application needs. Constraints breed creativity, benchmarks, applications, kernels, and deployment models. giving rise to significant research effort in working around these limits. For example, Intel SGXv1 [64] requires statically CCS Concepts: • Security and privacy ! Trusted com- sized enclaves, lacks secure I/O and syscall support, and is puting; Hardware security implementation; Software and ap- vulnerable to significant side-channels35 [ ]. Thus, to exe- plication security. cute arbitrary applications, the systems built on SGXv1 have Keywords: Trusted Execution Environment, Hardware En- inflated the Trusted Computing Base (TCB) and are forced clave, Secure Enclave, RISC-V, Memory Isolation, Side-Channel to implement complex workarounds [18, 22, 31]. As only Attack, Hardware Root of Trust, Open Source Intel can make changes to the inherent design trade-offs in SGX, users had to wait for changes like dynamic resizing of ACM Reference Format: enclave virtual memory in SGXv2 [63]. Unsurprisingly, these Dayeol Lee, David Kohlbrenner, Shweta Shinde, Krste Asanović, and other similar restriction have led to a proliferation of new and Dawn Song. 2020. Keystone: An Open Framework for Archi- TEEs on other ISAs (e.g., OpenSPARC [30], RISC-V [36, 80]). tecting Trusted Execution Environments. In Fifteenth European However, each such redesign requires considerable effort Conference on Computer Systems (EuroSys ’20), April 27–30, 2020, and only provides another fixed design point. Heraklion, Greece. ACM, New York, NY, USA, 16 pages. https://doi. We advocate that the hardware should provide security org/10.1145/3342195.3387532 primitives instead of point-wise solutions and in this paper leverage RISC-V’s primitives to construct highly customiz- Permission to make digital or hard copies of part or all of this work for able TEEs. We can draw an analogy with the move from tradi- personal or classroom use is granted without fee provided that copies are tional networking solutions to Software Defined Networking not made or distributed for profit or commercial advantage and that copies (SDN), where exposing the packet forwarding primitives to bear this notice and the full citation on the first page. Copyrights for third- party components of this work must be honored. For all other uses, contact the software has led to far more novel designs and research. the owner/author(s). Such a paradigm shift in TEEs will pave the way for low-cost EuroSys ’20, April 27–30, 2020, Heraklion, Greece use-case customization. It will allow the features and the © 2020 Copyright held by the owner/author(s). security model to be tuned for each hardware platform and ACM ISBN 978-1-4503-6882-7/20/04. https://doi.org/10.1145/3342195.3387532 EuroSys ’20, April 27–30, 2020, Heraklion, Greece D. Lee et al. use-case from a set of common software components, draw- self-paging, and more inside the enclave. For strengthening ing on ideas from modular kernel concepts [40, 60, 78]. This the security, our SM leverages any available configurable motivates the need for Customizable TEEs—an abstraction hardware to compose additional security mechanisms. We that allows entities that create the hardware, operate it, and demonstrate the potential of this with a highly configurable develop applications to configure and deploy various TEE cache controller to, in concert with PMP, transparently de- designs from the same base. Customizable TEEs promise in- fend against physical adversaries and cache side-channels. dependent exploration of gaps/trade-offs in existing designs, We built Keystone, the SM, two RTs (our native RT—Eyrie— quick prototyping of new feature requirements, a shorter and an off-the-shelf microkernel seL455 [ ]), and several mod- turn-around time for fixes, adaptation to threat models, and ules which together allow enclave-bound user applications usage-specific deployment. to selectively configure and use the above features (Figure 1). For realizing this vision, our first observation is the need We extensively benchmark Keystone on 4 suites with varying for a highly programmable trusted layer below the untrusted workloads: RV8, IOZone, CoreMark, and Beebs. We show- OS. Second, we must decouple our isolation mechanisms case use-case studies where Keystone can be used for secure from decisions of resource management, virtualization, and machine learning (Torch and FANN frameworks) and crypto- trust boundaries. We note that a hypervisor solution results graphic tasks (libsodium) on embedded devices and cloud in a trusted layer with a mix of security and virtualization re- servers. Lastly, we test Keystone on different RISC-V sys- sponsibilities, thus complicating the most critical component. tems: the HiFive Freedom Unleashed, 3 in-order cores and Similarly, firmware and micro-code are not programmable to 1 out-of-order core via FPGA, and a QEMU emulation—all a degree that satisfies our requirements. These two require- without modification. Keystone is fully open-source. ments ensure we avoid the mistake of using hardware with Contributions. We make the following contributions: a separation mechanism encumbered with a static bound- • Customizable TEEs. We define a new paradigm wherein ary between what is trusted and untrusted. Lastly, we draw the hardware manufacturer, hardware operator, and inspiration from proliferation of commercial (c.f. Intel SGX, the enclave programmer can tailor the TEE design. TrustZone) and non-commercial TEEs (c.f. Sanctum [36], • Keystone Framework. We present the first framework to Komodo [41]) which demonstrate the need for a common, configure, build, and instantiate customized TEEs. Our portable software base adaptable to ever-changing hardware principled way of ensuring modularity in Keystone capabilities and use-case demands. allows us to customize the design dimensions of TEE To this end, we propose Keystone—the first open-source instances as per the requirements. framework for building customized TEEs. We built Keystone • Open-source Implementation. We demonstrate advan- on unmodified RISC-V using its standard specifications17 [ ] tages of different Keystone TEE configurations that for physical memory protection (PMP)—a primitive which are tailored for minimizing the TCB, adapting to threat allows the programmable machine mode underneath the OS models, using hardware features, handling workloads, in RISC-V to specify arbitrary protections on physical mem- or providing rich functionality without any micro- ory regions. We use this machine mode to execute a trusted architectural changes. A typical Keystone instantiated security monitor (SM) to provide security boundaries without TEE design adds a total TCB of 12-15 K lines of code needing to perform any resource management. Critically, (LoC) to an enclave-bound application, of which the each enclave operates in its own isolated physical memory SM consists of only 1:6 KLoC added by Keystone. region and has its own supervisor-mode runtime (RT) compo- • Benchmarking & Real-world Applications. We evaluate nent to manage the virtual memory of the enclave and more. Keystone on 4 benchmarks: CoreMark, Beebs, and RV8 With this novel design, any enclave-specific functionality (< 1% overhead), and IOZone (40%). We demonstrate can be implemented cleanly by its RT while the SM manages real-world machine learning workloads with Torch in hardware-enforced guarantees. An enclave’s RT implements Eyrie (7:35%), FANN (0:36%) with seL4, and a Keystone- only the required functionality, communicates with the SM,