Enclave-Aware Compartmentalization and Secure Sharing with Sirius Zahra Tarkhani Anil Madhavapeddy University of Cambridge University of Cambridge Abstract Host App ! (3) Enclave " (4) (6) Hardware-assisted trusted execution environments (10) (1) shared Enclave #n Process #m memory (TEEs) are critical building blocks of many modern Threads applications. However, they have a one-way isolation (2) EL0 (5) (7) SEL0 model that introduces a semantic gap between a TEE and (9) (8) its outside world. This lack of information causes an ever- Normal world systems resources TEE systems resources increasing set of attacks on TEE-enabled applications OS Kernel & TEE driver (EL1) TEE Kernel/runtime (SEL1) that exploit various insecure interactions with the host OSs, applications, or other enclaves. We introduce Sirius, Security Monitor (EL3) the first compartmentalization framework that achieves strong isolation and secure sharing in TEE-assisted Figure 1: Simplified data flows in TZ-based TEE systems applications by controlling the dataflows within primary kernel objects (e.g. threads, processes, address spaces, files, sockets, pipes) in both the secure and normal worlds. Sirius replaces ad-hoc interactions in current However, in practice, TEEs are used in much more TEE systems with a principled approach that adds strong complex application architectures like web services [16, inter- and intra-address space isolation and effectively 40, 42], secure payments [1, 10, 12, 75], databases [64], eliminates a wide range of attacks. We evaluate Sirius on autonomous vehicle control [2], and privacy-preserving ARM platforms and find that it is lightweight (≈ 15K machine learning [37,39,40,60,68,79]. These applica- LoC) and only adds ≈ 10:8% overhead to enable TEE tions need fast bidirectional communications with their support on applications such as httpd, and improves the enclaves (e.g. via shared memory or RPC) and rely on performance of existing TEE-enabled applications such OS facilities for multithreading, networking, file opera- as the Darknet ML framework and ARM’s LibDDSSec tions, and IPC with other processes or enclaves. Despite by 0:05% − 5:6%. the fact that TEEs introduce a new secure kernel/runtime (Figure1), existing systems expect application devel- arXiv:2009.01869v3 [cs.CR] 23 Nov 2020 1 Introduction opers to manually bridge this semantic gap [76]. The resulting ad-hoc approaches have exposed TEE-enabled Hardware-assisted trusted computing primitives such as applications to severe attacks across these interaction lay- ARM TrustZone [13], Intel SGX [27], AMD SEV [56] or ers [41,51,70,71,76,84,91,92]. Even worse, mitigations RISC-V Keystone [46] exist to establish strong security based on sanitising the existing TEE interfaces are failing guarantees even in the presence of malicious privileged due to the wide attack surface [41, 76]. code. These TEEs1 assume a threat model in which only the CPU itself is trusted, and not the host applications, OS, We analysed 41 existing TEE-based open-source appli- or hypervisor. They expect in-enclave code to be small, cations to classify the vulnerabilities from these attacks verifiable, and need minimal external interactions [32]. (§A).We identified the following classes of vulnerabilities across the host/enclave boundary that leads to sensitive 1For simplicity, we use the terms TEE and enclave interchangeably. data compromise: 1 Category Num Thread Memory IPC/RPC Priv Reference monitor & Auditing 8 network sockets. Misusing these objects either directly Web apps 7 G# permits the earlier attacks, or increases the attack’s dam- Data analytics 5 Key management 4 G# age by propagating vulnerabilities or transferring ex- Attestation 2 G# G# G# Databases 4 G# G# tracted secrets to untrusted sources. 38 out of the 41 SSL/TLS 5 G# Blockchain 6 G# TEE-enabled applications we analysed depend on at least HPE [76] 95 G# three of these facilities. Therefore to comprehensively TEE-enabled Applications BOOMERANG [51] COIN Attacks [41] 10 G# mitigate these classes of vulnerabilities, we need a system CVE-2019-1010298 G# CVE-2018-11950 G# G# to selectively enable the protection and secure sharing of CVE-2017-8252 G# G# CVE-2017-8276 G# G# G# these objects across both the secure and normal worlds. CVE-2016-10297 G# G# CVE-2016-5349 G# G# We thus present Sirius, a framework for strongly com- CVE-2016-2431 G# G# TCB Vulnerabilities partmentalising and securely sharing systems resources CVE-2015-4422 G# and application data across normal world and enclave Object/feature is involved, partially involved userspaces. It does so by extending the normal world and Priv: Inadequate Privilege separation G# enclave kernels with interfaces that (i) allow applications Table 1: Summary of analyzing TEE-enabled applica- to compartmentalise sensitive data in both worlds; (ii) tions and known attacks. Sirius fully addresses these enforce intra-process protection such that these compart- threats or significantly weakens the damage. ments are protected even in multi-threaded applications with shared address space; and (iii) support flexible and mutually distrustful access control across systems objects Insecure threading and procedure calls: Attackers within both worlds. can launch malicious threads from within the host appli- Since TEE systems require running kernels on differ- cation process to exploit synchronisation vulnerabilities ent privilege levels, we need a decentralized approach such as TOCTTOU [90,91] or other types of concurrency for enabling mutually distrustful compartments in each attacks [41] on enclave interfaces (Figure1 (1,2,4)). This world. Sirius adopts decentralised information flow con- greatly limits the secure use of enclaves within a multi- trol (DIFC) [57] techniques for its unified access control threaded application; 56% of TEE-enabled applications to enforce mutual distrust, which provides application we analysed used multiple threads within the application programmers with a simple-to-use labeling interface to or enclave. control the flow of their private data. Our implementation of Sirius on ARM Trustzone runs efficiently on com- Memory vulnerabilities: Attackers take advantage of modity hardware (e.g. a Raspberry Pi). We have ported inadequate address space isolation in each world to complex multithreaded use cases such as the Apache web- launch ROP attacks [21, 47, 51] for extracting cryptog- server, the Darknet ML framework, and safety-critical raphy keys or bypass remote attestation (Figure1 (2,3)). applications such as autonomous vehicles or medical de- Additionally, insecure shared memory buffers are impor- vices that rely on secure data distribution services. Sirius tant attack vectors for extracting secrets or compromising framework provides a simple userspace API, and straight- RPC interfaces [41]. forward extensions to the Linux kernel and the TrustZone kernel. In return, the applications are comprehensively Insufficient privilege separation: A compromised protected from sensitive data leakage via the vulnerabili- or malicious third-party enclave can collect sensitive ties described earlier and with small performance over- data [52, 70] and leak them using OS facilities such heads on commodity hardware. as files, network sockets or pipes (Figure1 5–10). At- tackers can launch horizontal privilege escalation (HPE) 2 Background & Threat Model attacks [76] to compromise other process via a misbe- having enclave, or launch BOOMERANG attacks [51] to Secure enclaves are a hardware facility that allow for par- gain control of the host OS by tricking the secure world titioning sensitive data and associated computation away into modifying host kernel memory. No applications we from the “normal” world. Widespread TEE hardware studied implemented even a simple form of access con- includes ARM TrustZone (TZ) and Intel SGX, and they trol which would mitigate these vulnerabilities. both support running a separate kernel within a secure Table1 summarises the attack vectors used by the TEE world. Figure1 illustrates the architecture of a typical applications we studied. TEE applications use their OS (existing) TEE application on ARM TrustZone. interfaces to access system resources from userspace; This paper focuses on the changes required to the host e.g. threads, memory regions, IPCs or RPCs, files, and and ARM TZ-based enclaves to support Sirius’ stronger 2 security guarantees. We chose TZ for our prototype since receives client application (CA) data, processes it, and its secure world is more powerful than other enclaves— returns a result without retaining any data across invoca- and so exploited TZ enclaves can easily lead to a full OS tions. On the other hand, in stateful RPCs it persists some compromise. TZ also does not directly support attestation CA data across multiple invocations as a session state and as SGX does [27], which increases the possibility of global variables (.bss section) or inside persistent storage hosting malicious enclaves. Also, billions of embedded for larger objects. Typical secure storage in TZ is im- devices use TZ-based TEEs, which requires Sirius to plemented by encrypting memory blocks that are stored be resource-efficient. Porting Sirius to x86-64 requires in the NW’s file system. For Example, OPTEE uses a straightforward engineering that we discuss later (§7). secure storage key (SSK), enclave storage key (ESK), We will first describe how TZ hardware works (§2.1) and and file encryption key (FEK) for encrypting a persistent then elaborate on the threat model (§2.2). storage area in