How to Shadow Every Byte of Memory Used by a Program Nicholas Nethercote Julian Seward National ICT Australia, Melbourne, Australia OpenWorks LLP, Cambridge, UK [email protected] [email protected] Abstract 1.1 What is Shadow Memory? Several existing dynamic binary analysis tools use shadow mem- Programming tools such as profilers and checkers make program- ory—they shadow, in software, every byte of memory used by a ming easier and improve software quality. Dynamic binary analy- program with another value that says something about it. Shadow sis (DBA) tools are one class of such tools. They analyse a client memory is difficult to implement both efficiently and robustly. program at the level of machine code as it runs. They can be built Nonetheless, existing shadow memory implementations have not from scratch, but nowadays are usually implemented using a dy- been studied in detail. This is unfortunate, because shadow mem- namic binary instrumentation (DBI) framework such as Pin [9] or ory is powerful—for example, some of the existing tools that use it Valgrind [16]. detect critical errors such as bad memory accesses, data races, and This paper focuses on a class of DBA tools that use shadow uses of uninitialised or untrusted data. memory, i.e. they shadow, in software, every byte of memory used In this paper we describe the implementation of shadow mem- by a program with a shadow memory value that says something ory in Memcheck, a popular memory checker built with Valgrind, a about it. We call these tools shadow memory tools.Ashadow dynamic binary instrumentation framework. This implementation memory value may describe the value within a memory location has several novel features that make it efficient: carefully chosen (e.g. is it from a trusted source?), or it may describe the memory data structures and operations result in a mean slow-down factor of location itself (e.g. how many times has it been accessed?). only 22.2 and moderate memory usage. This may sound slow, but The analysis code added by the tool updates the shadow mem- we show it is 8.9 times faster and 8.5 times smaller on average than ory in response to memory accesses, and uses the shadow mem- a naive implementation, and shadow memory operations account ory to report information to the programmer. The granularity of the for only about half of Memcheck’s execution time. Equally impor- shadowing can vary, but usually every used memory byte or word tantly, unlike some tools, Memcheck’s shadow memory implemen- has a shadow memory value, and each shadow memory value may tation is robust: it is used on Linux by thousands of programmers itself be one bit, a few bits, one byte, or one word, for example. on sizeable programs such as Mozilla and OpenOffice, and is suited Some tools that use shadow memory also shadow every register to almost any memory configuration. with an extra value. Shadow registers are challenging to implement This is the first detailed description of a robust shadow mem- in their own right [16] but their implementation details are beyond ory implementation, and the first detailed experimental evaluation the scope of this paper. of any shadow memory implementation. The ideas within are ap- plicable to any shadow memory tool built with any instrumentation framework. 1.2 Shadow Memory is Useful Shadow memory lets a tool remember something about the history Categories and Subject Descriptors D.2.5 [Software Engineer- of every memory location and/or value in memory. Consider the ing]: Testing and Debugging—debugging aids, monitors; E.1 following motivating list of shadow memory tools; the descriptions [Data Structures] are brief but demonstrate that shadow memory (a) is powerful, and General Terms Design, Reliability, Performance, Experimenta- (b) can be used in a wide variety of ways. tion Memcheck [21, 12] is a memory checker. It remembers what Keywords Shadow memory, Valgrind, Memcheck, dynamic bi- allocation/deallocation operations have affected each memory lo- nary instrumentation, dynamic binary analysis cation, and can thus detect accesses of unaddressable memory. It also remembers which values are undefined (uninitialised or de- rived from undefined values) and can therefore detect dangerous 1. Introduction uses of undefined values. Purify [6] is a similar tool. This paper describes how to create dynamic analysis tools that use TaintCheck [17] is a security tool. It remembers which values shadow memory—tools that shadow every byte of memory used by are from untrusted (tainted) sources and which values were subse- a program with another value, in software—that are both efficient quently derived from them, and can thus detect dangerous uses of and robust. tainted values. TaintTrace [4] and LIFT [18] are similar tools. Eraser [20] is a data race detector. It remembers which locks are held when each memory location is accessed, and can thus detect when a memory location is accessed without a consistent lock-set, Permission to make digital or hard copies of all or part of this work for personal or which may imply a data race. VisualThreads [5] and Helgrind [10] classroom use is granted without fee provided that copies are not made or distributed are similar. DRD [19] is another race detector that uses a different for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute race-detection algorithm. to lists, requires prior specific permission and/or a fee. Hobbes [3] is a run-time type checker. It remembers what oper- VEE 2007 June 13–15, San Diego, California, USA. ations have been performed on each value, and can thus detect any Copyright c 2007 ACM 978-1-59593-630-1/07/0006. $5.00 later operations that are inappropriate for a value of that type. 65 Annelid [13] is a bounds checker. It remembers which values • First detailed description of Memcheck’s shadow memory. are array pointers, and can thus detect bounds errors. Memcheck is a widely-used tool, and this is the first detailed de- Redux [14] is a dataflow visualisation tool. It remembers which scription of its shadow memory implementation. Previous pub- operation created each value, and its inputs, and records these lications [21, 12, 16] have discussed in detail every significant in a dynamic dataflow graph which can be viewed at program aspect of Memcheck except its shadow memory implementa- termination. tion. pinSEL [11] automatically extracts system call side-effects from • First detailed description of any robust shadow memory im- benchmarks so that architectural simulators do not have to emulate plementation. This is also the first detailed description of any system calls when running those benchmarks. It shadows each robust shadow memory implementation. This is more important memory location with a copy of itself, and does a “memory diff” than it may seem, because shadow memory is a topic where between original and shadow memory values after each system details matter. High-level descriptions are not sufficient; lower- call executes in order to determine how the system call affected level implementation details such a data representations are cru- memory. cial, as they make the difference between a toy and a real-world All of these tools rely crucially on shadow memory. Eraser, tool. Most published descriptions of shadow memory imple- DRD and pinSEL use shadow memory but not shadow registers, the mentations have been minimal, and the three that have been others use both shadow memory and shadow registers. The shadow discussed in detail are not robust enough, in our opinion, for memory is implemented entirely in software and so these tools run use in a widely used tool like Memcheck. on stock hardware. • First experimental evaluation of shadow memory. This is the 1.3 Shadow Memory is Hard to Implement Well first paper that has evaluated and compared multiple versions of Speed. The speed of a shadow memory implementation is impor- a shadow memory implementation. tant. Although programmers will use slow tools if the benefits are The fourth and final contribution advances the state-of-the-art in high enough, they prefer fast tools. shadow memory implementations. Shadow memory is inherently expensive. Large amounts of extra state must be maintained; one shadow byte per byte of live • Novel shadow memory optimisations. Memcheck’s basic shadow original memory is typical. Most or all loads and stores must be memory data structure is similar to that used in several other instrumented to keep the shadow memory state up-to-date, as must shadow memory tools. However, Memcheck adds several novel operations that affect large regions of memory, such as allocations optimisations that speed up common cases, and compress and deallocations (on the heap, stack or via system calls such as shadow memory at coarse-grained (per-64KB chunk) and fine- mmap), reads/writes of large areas by system calls, and the loading grained (per-byte) levels. Together they reduce Memcheck’s of the program image into memory at start-up. mean slow-down factor by 4.0–13.6x and shrink its mean These requirements unavoidably increase the total amount of shadow memory size by a factor of 4.5–213.4 over a naive code that is run, increase a program’s memory footprint, and de- implementation. The reduction in shadow memory size also grade the locality of its memory accesses. Shadow memory tools improves robustness because it allows programs with larger thus typically slow down programs by a factor of 10–100, and memory footprints to be run in the same amount of address shadow memory operations cause much more of this slow-down space.