Defeating Speculative-Execution Attacks on SGX with HYPERRACE Guoxing Chen Mengyuan Li Fengwei Zhang Yinqian Zhang The Ohio State University The Ohio State University Southern University of The Ohio State University [email protected] [email protected] Science and Technology [email protected] [email protected] Abstract—Speculative-execution attacks, such as SgxSpectre, In this paper, we aim to address speculative-execution at- Foreshadow, and MDS attacks, leverage recently disclosed CPU tacks against Intel SGX. Existing solutions typically proposes hardware vulnerabilities and micro-architectural side channels to to either prevent speculative execution of suspicious memory breach the confidentiality and integrity of Intel Software Guard eXtensions (SGX). Unlike traditional micro-architectural side- loads or close cache side channels [17]–[19], but our solution channel attacks, speculative-execution attacks extract any data is software-based. It (1) requests the untrusted operating sys- in the enclave memory, which makes them very challenging to tem (OS) to create a special execution condition for enclaves, defeat purely from the software. However, to date, Intel has not under which speculative-execution attacks are impossible; (2) completely mitigated the threats of speculative-execution attacks verifies such execution condition is met dynamically at runtime from the hardware. Hence, future attack variants may emerge. and provides a proof through remote attestation; (3) utilizes This paper proposes a software-based solution to speculative- execution attacks, even with the strong assumption that con- an extended SGX feature that is implementable in microcode fidentiality of enclave memory is compromised. Our solution to guard the attestation keys from memory leaks. extends an existing work called HyperRace, which is a compiler- The base of our solution is HYPERRACE, which is a soft- assisted tool for detecting Hyper-Threading based side-channel ware framework designed to eliminate all micro-architectural attacks against SGX enclaves, to thwart speculative-execution side-channel threats due to Hyper-Threading and Asyn- attacks from within SGX enclaves. It requires supports from the untrusted operating system, e.g., for temporarily disabling chronous Enclave eXit (AEX) [20]. Particularly, it creates an interrupts, but verifies the OS’s behaviors. Additional microcode auxiliary enclave thread to occupy the sibling hyper thread co- upgrades are required from Intel to secure the attestation flow. located on the same physical core. Since thread scheduling is Index Terms—Intel SGX, speculative-execution attacks, remote performed by the OS, which is untrusted, HYPERRACE runs attestation a statistical tests to verify the threads’ co-location. To guard against speculative-execution attacks that leak I. INTRODUCTION any data in the enclave memory, HYPERRACE itself is not enough. In this paper, we propose to disallow interrupts during Intel Software Guard eXtensions (SGX) is introduced in the invocation of enclave ECall functions such that secrets recent Intel processors aiming to protect data and code within are only allowed to be unsealed into the memory during a secure enclave against its untrusted host operation systems invocations. HYPERRACE is used to detect the concurrent (OS) or even rogue system administrators. Due to its promise use of hyper threads so that speculative-execution attacks that of shielded execution, both researchers and practitioners have leak data from Hyper-Threading is prevented. Eliminating built various software tools and applications with these fea- interrupts during ECall function invocations require mod- tures, e.g., [1]–[11]. ification of OS kernel to suppress interrupts momentarily; The recently disclosed speculative-execution attacks, i.e., the enclave code needs to verify the occurrence of interrupts Meltdown and Spectre attacks enable a malicious program to at the end of it using techniques proposed by Cloak [21], read memory content outside its security domain (e.g., reading which is also used in HYPERRACE to detect AEX. As such, kernel data from userspace). Their variants, SgxPectre [12] and no speculative-execution attacks is possible under such an Foreshadow [13], specifically target Intel SGX to read enclave execution condition. memory content, completely breaking the confidentiality guar- We performed a security analysis of the proposed solution, antee of SGX. The newly disclosed Microarchitectural Data implemented the kernel components and enclave modules that Sampling (MDS) hardware vulnerabilities [14]–[16] could create the interrupt-free execution windows, and evaluated its also enable the adversary to read enclave memory on-the- performance. The main contribution of this work is that it is fly. Though microcode patches have been released to mitigate the first attempt to mitigate the threats of speculative-execution these attacks, the fix does not remove the root cause of the attacks from software, even assuming the possibility of new vulnerabilities—speculative executed instructions beyond se- speculative-execution attack variants discovered in the future. curity boundary check and legitimate control flows. Therefore, This idea follows with Intel’s philosophy of TCB recovery [22] new variants of such attacks may be discovered in the future. that aims to minimize the security risks enclave data in the events of SGX compromises. enclave executes the EGETKEY instruction to derive its report key to generate a CMAC of the report data and compare it with II. BACKGROUND AND RELATED WORK the one in the report. If they match, the verification passes. The A. Intel SGX derivation of report keys uses both the root provisioning key and the root seal key, so that the local attestation report could Intel Software Guard eXtensions (SGX) is a new hard- only be verified locally on the same platform. ware feature introduced on recent Intel processors, aiming to improve the security of application code and data. Sensitive Sealing. Sealing is a process used by enclaves to encrypt and information can be processed with a shielded execution envi- integrity-protect some secrets to be stored outside the enclave. ronment called enclave, where the code and data are stored The encryption is performed using a seal key, which is derived in Processor Reserved Memory (PRM), a dedicated region of via the EGETKEY instruction. Both the root provisioning the DRAM. Any access to an enclave’s memory within the key and the root seal key will be used in the derivation. PRM from any software outside of the enclave, even from the Additionally, a 64-bit KeyID can be specified during the privileged softwares, will be denied. derivation, enabling an enclave to derive a large number of different seal keys. The derivation of seals keys uses both the Asynchronous Enclave eXit (AEX). To prevent sensitive root provisioning key and the root seal key, so that the sealed information from leaking when interrupts or exceptions occur data could only be unsealed locally on the same platform. during the enclave’s execution, an event called Asynchronous Enclave eXit (AEX) will be triggered. Particularly, before Trusted Platform Services. Intel SGX provides trusted plat- transferring control to the OS, the processor saves the enclave’s form services, i.e., trusted time and monotonic counters. These execution state in a specific enclave memory area called State services are enabled by the Intel Converged Security and Save Area (SSA). When resuming the enclave’s execution after Management Engine (CSME). An Intel issued privileged en- the interrupts or exceptions are handled, the processor restores clave, called Platform Service Enclave (PSE) is responsible for the enclave’s execution state from SSA. communicating with the CSME and supports these services. Remote Attestation. To establish trust between enclaves and Intel Hyper-Threading. Intel Hyper-Threading is Intel’s their remote users or clients, remote attestation is introduced proprietary implementation of simultaneous multithreading to prove to the remote party that the enclave is running inside (SMT). When Hyper-Threading is enabled, a single physical an SGX enabled platform. In current SGX remote attestation core could execute two separate code streams (called hyper design, an anonymous signature scheme, called Intel Enhanced threads) concurrently. These two hyper threads, regardless Privacy ID (EPID), is adopted [23]. Particularly, an EPID of running in the SGX enclave mode or not, share various private key is provisioned by Intel Provisioning Service to resources on the same physical core, e.g., line fill buffers, each SGX platform for generating attestation signatures, which store buffers, branch prediction units (BPU) and translation could be verified by Intel Attestation Service. A privileged lookaside buffers (TLB). enclave, called Quoting Enclave (QE) is issued by Intel to manage the attestation key and sign attestation data. B. Speculative-Execution Attacks and Defenses Root of Trust. A root provisioning key is generated at Intel’s Modern CPUs use speculative execution techniques to in- internal key generation facility, and burnt into the e-fuse of crease instruction-level parallelism and hence improve CPU SGX platforms. The root provisioning key, as a shared secret, performance. Instructions may be executed speculatively with establishes the root of trust between Intel and the underlying regard to their program order but are required to retire in order. SGX platform. Besides the root