CoSMIX: A Compiler-based System for Secure Memory Instrumentation and Execution in Enclaves Meni Orenbach, Technion; Yan Michalevsky, Anjuna Security; Christof Fetzer, TU Dresden; Mark Silberstein, Technion https://www.usenix.org/conference/atc19/presentation/orenbach This paper is included in the Proceedings of the 2019 USENIX Annual Technical Conference. July 10–12, 2019 • Renton, WA, USA ISBN 978-1-939133-03-8 Open access to the Proceedings of the 2019 USENIX Annual Technical Conference is sponsored by USENIX. CoSMIX: A Compiler-based System for Secure Memory Instrumentation and Execution in Enclaves Meni Orenbach Yan Michalevsky Christof Fetzer Mark Silberstein Technion Anjuna Security TU Dresden Technion Abstract compression [44], disaggregation [39, 75], distributed shared memory [36, 46] and heterogeneous memory support [37]. Hardware secure enclaves are increasingly used to run com- With the emergence of Software Guard Extensions (SGX) plex applications. Unfortunately, existing and emerging en- for Trusted Execution in Intel CPUs [16, 55], applications clave architectures do not allow secure and efficient implemen- are increasingly ported to be entirely executed in hardware- tation of custom page fault handlers. This limitation impedes enforced enclaves [58, 45, 23, 25]. The enclave hardware pro- in-enclave use of secure memory-mapped files and prevents tects them from attacks by a powerful privileged adversary, extensions of the application memory layer commonly used in such as a malicious OS or a hypervisor. A number of recent untrusted systems, such as transparent memory compression systems facilitate the porting to SGX by shielding unmodified or access to remote memory. applications in an enclave [21, 81, 18]. Unfortunately, these CoSMIX is a Compiler-based system for Secure Memory systems do not allow secure overriding of page fault handling Instrumentation and eXecution of applications in secure en- in enclave applications. This drawback complicates porting a claves. A novel memory store abstraction allows implementa- large class of applications that use memory-mapped files to tion of application-level secure page fault handlers that are SGX. Further, it prevents SGX applications from using secu- invoked by a lightweight enclave runtime. The CoSMIX com- rity and performance enhancements, such as efficient mem- piler instruments the application memory accesses to use one ory paging [61] and Oblivious RAM (ORAM) side-channel or more memory stores, guided by a global instrumentation protection [67, 11, 88] without intrusive application modifi- policy or code annotations without changing application code. cations. Our goal is to eliminate these constraints. The CoSMIX prototype runs on Intel SGX and is com- For example, consider the task of running an SQLite patible with popular SGX execution environments, including database that uses memory-mapped files in the enclave. The SCONE and Graphene. Our evaluation of several production database file must be encrypted to ensure data confidentiality. applications shows how CoSMIX improves their security and Enabling in-enclave execution of SQLite therefore requires performance by recompiling them with appropriate memory support for encrypted memory-mapped files, which in turn stores. For example, unmodified Redis and Memcached key- implies that the page fault handler must be executed securely value stores achieve about 2× speedup by using a self-paging as well. Unfortunately, hardware enclaves available today memory store while working on datasets up to 6× larger do not support secure page faults. Instead, existing solutions than the enclave’s secure memory. Similarly, annotating a use workarounds, such as eagerly reading and decrypting the single line of code in a biometric verification server changes whole mapped file region into trusted enclave memory [18]. it to store its sensitive data in Oblivious RAM and makes it This solution does not scale to large files and lacks the perfor- resilient against SGX side-channel attacks. mance benefits of on-demand data access. We argue that the problem is rooted in the fundamental lim- 1 Introduction itation of SGX architecture, which does not provide the mech- anism to define secure page fault handlers. The upcoming Virtual Memory is integral to modern processor architectures. SGX-V2 [54, 86, 43] will not solve this problem either. More- In addition to its primary role in physical memory manage- over, we observe that existing and emerging secure enclave ment, it empowers developers to extend the standard mem- architectures [28,4, 34] suffer from similar limitations(§2). ory layer with custom data storage mechanisms in software. In this work, we build CoSMIX, a compiler and a For example, the memory-mapped file abstraction, which is lightweight enclave runtime that overcomes the SGX architec- broadly used, e.g., in databases [10, 5], relies on the OS’s tural limitations and enables secure and efficient extensions page fault handler to map a frame and populate it with the con- to the memory layer of unmodified applications running in tents of a file. Replacing accesses to physical memory with enclaves. We introduce a memory store, (mstore), a program- file accesses requires no application code changes. Therefore, ming abstraction for implementing custom memory manage- the ability to override page fault behavior has been essential ment extensions for enclaves. The CoSMIX compiler auto- for implementing a range of system services, such as memory matically instruments application code to allocate the selected USENIX Association 2019 USENIX Annual Technical Conference 555 variables and memory buffers in the mstore, replacing the ac- well as on unmodified production servers: memcached, Redis, cesses to enclave memory with the accesses to the mstore. SQLite, and a biometric verification server [61]. The compiler The mstore logic runs in the enclave as part of the application. is able to correctly instrument all of these applications, some The CoSMIX runtime securely invokes the mstore memory with hundreds of thousands of lines of code (LOC), without management callbacks, which include custom page fault han- the need to manually change the application code. dlers. The page faults are semantically equivalent to hardware Our microbenchmarks using Phoenix with SUVM and se- page faults yet are triggered by the CoSMIX runtime. cure mmap mstores show that CoSMIX instrumentation results An mstore can implement the missing functionalities that in a low geometric mean overhead of 20%. require secure page fault handlers. For example, it may pro- For the end-to-end evaluation, we run memcached and Re- vide the secure mmap functionality by implementing the page dis key value stores on 600 MB datasets – each about 6× fault handler that accesses the file and decrypts it into the the size of the secure physical memory available to SGX en- application buffer. A more advanced mstore may add its own claves. In this setting, SGX hardware paging significantly in-memory cache analogous to an OS page cache, to avoid affects the performance. The SUVM [61] mstore aims to opti- costly accesses to the underlying storage layer. CoSMIX sup- mize exactly this scenario. To use it, we only annotate the item ports several types of mstores, adjusting the runtime to handle allocator in memcached (a single line of code) and compile it different mstore behaviors while optimizing the performance. with CoSMIX. Redis is compiled without adding annotations. CoSMIX allows the use of multiple mstores in the same The instrumented versions of both servers achieve about 2× program. This can be used, for example, to leverage both speedup compared to their respective vanilla SGX baselines. secure mmap mstore and an ORAM mstore for side-channel In another experiment, we evaluate a biometric verification protection. Additionally, CoSMIX supports stacking of mul- server with a database storing 256 MB of sensitive data. We tiple mstores to enable their efficient composition and reuse. use the ORAM mstore to protect it from SGX controlled side- We design and prototype three sophisticated mstores in §3.2.5, channel attacks [87] that may leak sensitive database access and demonstrate the benefits of stacking in §4.5. statistics. We annotate the buffers containing this database CoSMIX’s design focuses on two primary goals: (1) mini- (one line of code) to use ORAM. The resulting ORAM- mizing the application modifications to use mstores and (2) enhanced application provides security guarantees similar reducing the instrumentation performance overheads. We in- to other ORAM systems for SGX, such as ZeroTrace [67], yet troduce the following mechanisms to achieve them: without modifying the application source code. ORAM sys- Automatic inference of pointer types. CoSMIX does not tems are known to result in dramatic performance penalties of require annotating every access to a pointer. Instead, it uses several orders of magnitude [26]. However, our hardened ap- inter-procedural pointer analysis [17] to determine the type of plication is only 5:8× slower than the vanilla SGX thanks to the mstore (or plain memory) to use for each pointer. When the benefits of selective instrumentation enabled by CoSMIX. the static analysis is inconclusive, CoSMIX uses tagged point- To summarize, our contributions are as follows: ers [47, 74, 13] with the mstore type encoded in the unused • Design of a compiler and an mstore abstraction for trans- Most Significant Bits, enabling runtime detection (§3.3.1). parent secure memory instrumentation