A Software Approach to Defeating Side Channels in Last-Level Caches Ziqiao Zhou Michael K. Reiter Yinqian Zhang University of North Carolina University of North Carolina Ohio State University Chapel Hill, NC, USA Chapel Hill, NC, USA Columbus, OH, USA ABSTRACT do not require preemption of the victim to extract fine- grained information from it (e.g., [35, 37, 12, 21]). We present a software approach to mitigate access-driven Two varieties of LLC-based side channels capable of side-channel attacks that leverage last-level caches extracting fine-grained information from a victim have (LLCs) shared across cores to leak information between been demonstrated. The first such attacks were of the security domains (e.g., tenants in a cloud). Our approach FLUSH-RELOAD variety [35, 37], which requires the at- dynamically manages physical memory pages shared be- tacker to share a physicalmemory pagewith the victim— tween security domains to disable sharing of LLC lines, a common situation in a modern OS, due to shared li- thus preventing “FLUSH-RELOAD” side channels via brary, copy-on-write memory management and memory LLCs. It also manages cacheability of memory pages deduplication mechanisms that aim for smaller mem- to thwart cross-tenant “PRIME-PROBE” attacks in LLCs. ory footprints. The attacker first FLUSHes a cache-line We have implemented our approach as a memory man- sized chunk of the shared page out of the cache using agement subsystem called CACHEBAR within the Linux processor-specific instructions (e.g., clflush in x86 kernel to intervene on such side channels across con- processors) and later measures the time to RELOAD (or tainer boundaries, as containers are a common method re-FLUSH [8]) it to infer whether this chunk was touched for enforcing tenant isolation in Platform-as-a-Service (and thus loaded to the shared cache already) by the (PaaS) clouds. Through formal verification, princi- victim. More recently, so-called PRIME-PROBE attacks pled analysis, and empirical evaluation, we show that have been demonstrated via LLCs [12, 21]; these do not CACHEBAR achieves strong security with small perfor- require page sharing between the attacker and victim. mance overheads for PaaS workloads. Rather, PRIME-PROBE attacks can be conducted when the two programsshare the same CPU cache sets. The at- 1 INTRODUCTION tacker PRIMEs the cache by loadingits own memory into An access-driven side channel is an attack by which certain cache sets. Later it PROBEs the cache by measur- an attacker computation learns secret information about ing the time to load the same memory into the cache sets a victim computation running on the same computer, and inferring how many cache lines in each cache set are not by violating the logical access control implemented absent due to conflicts with the victim’s execution. by the isolation software (typically an operating sys- In this paper we propose a software-only defense tem (OS) or virtual machine monitor (VMM)) but rather against these LLC-based side-channel attacks, based on by observing the effects of the victim’s execution on two seemingly straightforward principles. First, to defeat arXiv:1603.05615v1 [cs.CR] 17 Mar 2016 microarchitectural components it shares with the at- FLUSH-RELOAD attacks, we propose a copy-on-access tacker’s. Overwhelmingly, the components most often mechanism to manage physical pages shared across mu- used in these attacks are CPU caches. Early cache-based tually distrusting security domains (i.e., processes, con- side channels capable of leaking fine-grained informa- tainers2, or VMs). Specifically, temporally proximate ac- tion (e.g., cryptographickeys) across security boundaries cesses to the same physical page by multiple security do- used per-core caches (e.g., [30, 9, 36]), though the need mains results in the page being copied so that each do- for the attacker to frequently preempt the victim to ob- main has its own copy. In this way, a victim’s access to serve its effect on per-core caches renders these attacks its copy will be invisible to an attacker’s RELOAD in a relatively easy to mitigate in software (e.g., [39, 31]).1 FLUSH-RELOAD attack. When accesses are sufficiently Of more concern are side channels via last-level caches spaced in time, the copies can be deduplicated to return (LLCs) that are shared across cores and, in particular, the overall memory footprint to its original size. Sec- ond, to defeat PRIME-PROBE attacks, we design a mech- 1Hyper-threading can enable the attacker to observe the victim’s anism to manage the cacheability of memory pages so as effects on per-core caches without preempting it, if both are simultane- to limit the number of lines per cache set that an attacker ously scheduled on the same core. So, potentially adversarial tenants are generally not scheduled together (or hyper-threading is disabled) in cloud environments, for example. 2https://linuxcontainers.org/ may PROBE. In doing so, we limit the visibility of the at- (e.g., [3, 4]) to application-specific side-channel-free im- tacker into the victim’s demand for memory that maps to plementations (e.g., [16]). These techniques can intro- that cache set. Of course, the challenge in these defenses duce substantial runtime overheads, however, and these is in engineering them to be effective in both mitigating overheads tend to increase with the generality of the tool. LLC-based side-channels and supporting efficient execu- It is for this reason that we believe that systems-level tion of computations. (i.e., OS- or VMM-level) defenses are the most plausible, To demonstrate these defenses and the tradeoffs be- general defense for deployment in the foreseeable future. tween security and efficiency that they offer, we detail With attention to cache-based side-channels specifically, their design and implementation in a memory manage- several works provide to each security domain a limited ment subsystem called CACHEBAR (short for “Cache number of designated pages that are never evicted from Barrier”) for the Linux kernel. CACHEBAR sup- the LLC (e.g., [14, 19]), thereby rendering their con- ports these defenses for security domains represented tents immune to PRIME-PROBE and FLUSH-RELOAD as Linux containers. That is, copy-on-access to de- attacks. These approaches, however, require the appli- fend against FLUSH-RELOAD attacks makes copies of cation developer to determine what data/instructions to pages as needed to isolate temporally proximate ac- protect and then to modify the application to organize cesses to the same page from different containers. More- the sensitive content into the protected pages; in contrast, over, memory cacheability is managed so that the pro- CACHEBAR seeks to protect applications holistically and cesses in each container are collectively limited in the requires no application modifications. CACHEBAR also number of lines per cache set they can PROBE. This differs in several design choices that free it from limi- implementation would thus be well-suited for use in tations of prior approaches (e.g., the limitation of only Platform-as-a-Service (PaaS) clouds that isolate cloud one protected page per core [14] or dependence on rel- customers in distinct containers; indeed, cross-container atively recent, Intel-specific cache optimizations [19]). LLC-based side channels have been demonstrated in Other systems-level solutions manage memory so as to such clouds in the wild [37]. Our security evalua- partition the use of the LLC by different security do- tions show that CACHEBAR mitigates cache-based side- mains (e.g., [26, 27]), though these approaches preclude channel attacks, and our performance evaluation indi- memory-page and CPU-cache sharing entirely and hence cates that CACHEBAR imposes very modest overheads can underutilize these resources considerably. on PaaS workloads. LLC-based side channels are a particular instance of To summarize, we contribute: timing side channels, and so defenses that seek to elim- inate timing side channels are also relevant to our prob- • A novel copy-on-access mechanism to manage phys- lem. Examples include fuzzing real-time sources on the ical memory pages shared by distrusting tenants to computer (e.g., [32]), though this impinges on legitimate prevent FLUSH-RELOAD side-channel attacks, and uses of real time. Since real-time counters are not the its formal verification using model checking. only way to time memory fetches [34], other efforts have • A novel mechanism to dynamically maintain queues sought to eliminate side-channel risks more holistically of cacheable memory pages so as to limit the cache via altering the CPU scheduler (e.g., [28, 18]) and man- lines a malicious tenant may access in PRIME-PROBE aging how tenants co-locate (e.g., [17, 38, 10, 2, 18]). In attacks, and a principled derivation of its parameters contrast, here we focus specifically on LLC-based side to balance security and performance. channels (vs. a larger subset of timing side-channels)— • Implementation of both mechanisms in a mainstream which again are arguably the most potent known side- Linux operating system kernel and an extensive secu- channel vectors [35, 37, 12, 21]—and restrict our modi- rity and performance evaluation for PaaS workloads. fications to the memory management subsystem. 2 RELATED WORK 3 COPY-ON-ACCESS FOR FLUSH- Numerous proposals have sought to mitigate cache- RELOAD DEFENSE based side channels with low overhead through redesign The FLUSH-RELOAD attack is a highly effective LLC- of the cache hardware, e.g., [24, 15, 33, 13, 20]. Un- based side channel that was used, e.g., by Zhang et fortunately, there is little evidence that mainstream CPU al. [37] to mount fine-grained side-channel attacks in manufacturers will deploy such defenses in the foresee- commercial PaaS clouds. It leverages physical mem- able future, and even if they did, it would be years be- ory pages shared between an attacker and victim se- fore these defenses permeated the installed computing curity domains, as well as the ability to evict those base.