Varys: Protecting SGX enclaves from practical side-channel attacks Oleksii Oleksenko, Bohdan Trach, Robert Krahn, and André Martin, TU Dresden; Mark Silberstein, Technion; Christof Fetzer, TU Dresden https://www.usenix.org/conference/atc18/presentation/oleksenko This paper is included in the Proceedings of the 2018 USENIX Annual Technical Conference (USENIX ATC ’18). July 11–13, 2018 • Boston, MA, USA ISBN 978-1-939133-02-1 Open access to the Proceedings of the 2018 USENIX Annual Technical Conference is sponsored by USENIX. Varys Protecting SGX Enclaves From Practical Side-Channel Attacks Oleksii Oleksenko†, Bohdan Trach†, Robert Krahn†, Andre Martin†, Christof Fetzer†, Mark Silberstein‡ †TU Dresden, ‡Technion Abstract cludes side channels from the SGX threat model, SCAs Numerous recent works have experimentally shown that effectively circumvent the SGX confidentiality guarantees Intel Software Guard Extensions (SGX) are vulnerable to and impede SGX adoption in many real-world scenarios. cache timing and page table side-channel attacks which More crucially, a privileged adversary against SGX can could be used to circumvent the data confidentiality guar- mount much more powerful SCAs compared to the un- antees provided by SGX. Existing mechanisms that pro- privileged one in canonical variants of the attacks. For ex- tect against these attacks either incur high execution costs, ample, a malicious OS can dramatically reduce the noise are ineffective against certain attack variants, or require levels in cache timing attacks via single-stepping [24] or significant code modifications. by slowing down the victim. We present Varys, a system that protects unmodified In this paper, we investigate practical ways of protect- programs running in SGX enclaves from cache timing ing SGX programs from page table and cache timing and page table side-channel attacks. Varys takes a prag- SCAs. Specifically, we focus on the case where unmodi- matic approach of strict reservation of physical cores to fied general-purpose applications are executed in enclaves security-sensitive threads, thereby preventing the attacker in order to protect their secrets, as it is the case with from accessing shared CPU resources during enclave exe- Haven [4], Graphene-SGX [11], or SCONE [3]. cution. The key challenge that we are addressing is that We postulate that a practical solution should have of maintaining the core reservation in the presence of an low performance overheads, require no modifications to untrusted OS. application source code, and impose no restrictions on Varys fully protects against all L1/L2 cache timing the application’s functionality (such as restricting multi- attacks and significantly raises the bar for page table side- threading). We assume that recompilation of source code channel attacks—all with only 15% overhead on average is acceptable as long as it does not require code changes. for Phoenix and PARSEC benchmarks. Additionally, we Existing mitigation techniques, however, fall short of propose a set of minor hardware extensions that hold the satisfying our requirements. Traditional hardening tech- potential to extend Varys’ security guarantees to L3 cache niques against cache timing attacks [5, 23] require rewrit- and further improve its performance. ing the application; recent defenses [22] based on Intel TSX technology also require code changes; memory ac- 1 Introduction cesses obfuscation approaches, such as DR.SGX [8], in- Intel Software Guard Extensions (SGX) enclaves provide cur high performance cost (at least 3× and up to 20×); a shielded environment to securely execute sensitive pro- T-SGX [40] prevents controlled OS attacks, but it is in- grams on commodity CPUs in the presence of a privileged effective against concurrent attacks on page tables and adversary. So far, no successful direct attack on SGX caches. Déjà Vu [14] protects only against page table has been reported, i.e., none that compromises SGX’s attacks and is prone to false positives. security guarantees. However, numerous works demon- In Varys, we strive to achieve both application perfor- strate that SGX is vulnerable to several types of side mance and user convenience while making page table and channel attacks (SCAs), in particular, traditional cache cache timing SCAs on enclaves much harder or entirely timing and page table SCA that reveal page-level memory impossible to mount. The basic idea behind Varys design accesses [9, 39, 21, 47, 43, 45, 24], as well as specula- is trust but verify. A potentially malicious OS is requested tive attacks [29, 12] that use the side channels as a way to execute the enclave in a protected environment that pre- of retrieving information. Although Intel explicitly ex- vents all known cache timing and page fault attacks on USENIX Association 2018 USENIX Annual Technical Conference 227 SGX enclaves. However, the Varys trusted runtime inside quested to co-locate these pairs on the same physical CPU the enclave verifies that the OS fulfills the request. core. The trusted application threads are instrumented Our main observation is that all published page table (via a compiler pass) to periodically verify that they are in- and L1/L2 cache timing attacks on SGX require either deed co-scheduled and running together on the same core. (1) a high rate of enclave exits, or (2) control of a sib- Varys terminates the enclave if co-scheduling is violated ling hyperthread on the same CPU core with the victim. or if any of the threads in the pair gets preempted too of- Consequently, if an enclave is guarded against frequent ten. To reduce the frequency of legitimate exits and lower asynchronous exits and executes on a dedicated CPU core the false positives, Varys uses exitless system calls [3] without sharing it with untrusted threads, it would be pro- and in-enclave thread scheduling such that multiple appli- tected against the attacks. The primary challenge that cation threads can share the same OS thread. Moreover, Varys addresses is in maintaining such a protected envi- Varys configures the OS to reduce the frequency of in- ronment in face of a potentially malicious OS. It achieves terrupts routed to the core in order to avoid interference this goal via two mechanisms: asynchronous enclave exits with attack-free program execution. However, if the OS monitoring and trusted reservation. ignores the request, this will effectively lead to denial of First, Varys monitors when asynchronous enclave exits service without compromising the enclave’s security. (AEX) occur (e.g., for scheduling another process on the Varys primarily aims to protect multi-threaded pro- core or handling an exception) and restricts the frequency grams by reserving complete cores and scheduling the of such exits, terminating the enclave once the AEX fre- application threads on them, i.e., protection against SCAs quency bound is exceeded. Varys sets the bound to the translates into an allocation policy that allocates or frees values that render all known attacks impossible. Notably, computing resource with a granularity of one core. We the bound is much higher than the frequency of exits in believe that Varys exercises a reasonable trade-off be- an attack-free execution, thereby minimizing the chances tween security and throughput for services that require of false positives as we explain in §4. the computational power of one or more cores. For single- Second, Varys includes a mechanism for trusted core threaded applications Varys pairs the application thread reservation such that the attacker cannot access the core with a service thread to reserve the complete core. resources shared with the enclave threads while they are Due to the lack of appropriate hardware support in running, nor can it recover any secrets from the core’s today’s SGX hardware, Varys remains vulnerable to tim- L1 and L2 caches afterward. For example, consider an ing attacks on Last Level Cache (LLC) as we explain in-enclave execution of a multi-threaded application with in §8. We suggest a few minor hardware modifications two threads. Assuming a standard processor with hyper- that hold the potential to solve this limitation and addi- threading (SMT), all it takes to prevent concurrent attacks tionally, eliminate most of the runtime overhead. These on L1/L2 caches is to guarantee that the two enclave extensions allow the operating system to stay in control threads always run together on the same physical core. of resource allocations but permit an enclave to determine As a result, the threads occupy both hardware threads of if its resource allocation has changed. the core, and the attacker cannot access the core’s caches. Our contributions include: Note that this simple idea prevents any concurrent attacks • Analysis of attack requirements. on core’s resources shared between its hyperthreads, such • A set of measures that can be taken to protect against as branch predictor and floating point unit. It also prevents these attacks using existing OS features. exit-less SCAs on page table attributes [43] because they • Varys, an approach to verifying that the OS correctly require attacker’s access to the core’s TLB—available serves our request for a protected environment. only if the attacker thread is running on that core. Addi- • Implementation of Varys with 15% overhead across tionally, to ensure that the victim leaves no traces in the PARSEC [7] and Phoenix [38] benchmark suites. caches when de-scheduled from the core, Varys explicitly • Proposal for hardware extensions that improve evicts the caches when enclave threads are preempted. Varys’s security guarantees and performance. While conceptually simple, the implementation of the trusted reservation mechanism is a significant challenge. 2 Background An untrusted OS may ignore the request to pin two en- clave threads to the same physical core, may re-enable 2.1 Intel SGX CPU hyperthreading if disabled prior to enclave execution, Intel Software Guard Extensions is an ISA extension that and may preempt each of the enclave threads separately adds hardware support for Trusted Execution Environ- in an attempt to break Varys’s defense.