ReTrace: Collecting Execution Trace with Virtual Machine Deterministic Replay Min Xu Vyacheslav Malyugin Jeffrey Sheldon Ganesh Venkitachalam Boris Weissman VMware Inc. Abstract means collecting run time information (herein called ex- ecution traces) from a workload. Offline, the execu- Execution trace is an important tool in computer archi- tion traces can be analyzed in more detail, such as per- tecture research. Unfortunately, existing trace collection forming cache simulations. Execution traces vary from techniques are often slow (due to software tracing over- sparse traces (e.g., collecting context switches) to de- heads) or expensive (due to special tracing hardware re- tailed traces (e.g., collecting memory references). Not quirements). Regardless of the method of collection, de- surprisingly, it is the detailed execution traces that are tailed trace files are generally large and inconvenient to the most challenging to collect. store and share. In their survey, Uhlig and Mudge noted the following We present ReTrace, a trace collection tool based on the challenges in trace collection [25]. deterministic replay technology of the VMware hyper- visor. ReTrace operates in two stages: capturing and • Maximizing trace completeness and the detail level. expansion. ReTrace capturing accumulates the mini- For example, a memory reference trace is more com- mal amount of information necessary to later recreate plete if it includes both operating system (OS) and a more detailed execution trace. It captures (records) application memory references. Similarly, a memory only non-deterministic events resulting in low time and reference trace is more detailed if it includes the an- space overheads (as low as 5% run-time overhead, as low notation of virtual address to physical address map- as 0.5 byte per thousand instructions log growth rate) ping for each reference. on supported platforms. ReTrace expansion uses the in- formation collected by the capturing stage to generate a • Minimizing trace distortion. In other words, the complete and accurate execution trace without any data trace collected should faithfully represent the pro- loss or distortion. ReTrace is an experimental feature gram being traced. For example, trace collection of VMware Workstation 6.0 currently available in Win- itself should not introduce extra memory references dows and Linux flavors for commodity IA32 platforms. in a memory trace. Perhaps more subtly, tracing No special tracing hardware is required. should minimize time dilation and memory dilation, which occurs when tracing causes a program to run We have three key results. First, we find that trace col- slower (from its own perspective) or to consume lection can be done both efficiently and inexpensively. more memory (in its own address space). Second, deterministic replay is an effective technique for compressing large trace files. Third, performing the trace • Reducing trace file size. The trace collecting tool collection at the hypervisor layer is minimally invasive should produce highly compact trace files, which to the collected trace while enabling tracing of the entire are easier to store and share with other researchers. system (user/supervisor level, CPU, peripheral devices). Compact trace files also enable long execution traces to be collected for different analysis needs. ReTrace is a rapidly evolving technology. We would like to use this paper to solicit feedback on the applicability • Fast, inexpensive and easy to operate. of ReTrace in computer architecture research to help us refine our future development plans. To the best of our knowledge, none of the existing trace collection techniques sufficiently solves all these chal- lenges. We briefly survey these techniques in Section 6. 1 Introduction This paper presents a deterministic replay based execu- tion trace technique, ReTrace, which better solves all When computer architects want to peek into a running these challenges. ReTrace decomposes traditional trace system, they often use trace-driven techniques, which collection process into two steps: ReTrace capturing and ReTrace expansion. This decomposition is the key to devices, to run a guest OS. The Virtual Machine Moni- meet the four challenges above. During ReTrace captur- tor (VMM) is a thin layer of software that sits between ing, minimum information on execution nondeterminism the physical hardware and the guest OS to enable shar- is logged. This reduces the necessary trace file size and ing of the physical CPU, memory and I/O devices among trace distortion, because the capturing overhead is min- multiple VM. Until recently, the IA32 architecture has imal. During ReTrace expansion, maximum information not permitted the classical trap-and-emulate virtualiza- on all aspects of the execution can be collected. The tion. Adams and Agesen present an up to date study on resulting trace is complete and detailed without added virtualizing IA32 [1]. trace distortion. By building on virtual machine tech- Deterministic replay [3, 10, 17, 18, 20, 23, 28] creates an nology, ReTrace is fast, inexpensive and easy to operate. execution that is logically equivalent to an original exe- Deterministic replay is the key technology that enables cution of interest. Two executions are logically equiva- the decomposition of ReTrace capturing and ReTrace lent if they contain the same set of dynamic instructions, expansion. Our deterministic replayer is built on top each dynamic instruction computes the same result in of VMware’s virtual machine monitor [1, 24]. A vir- the two executions, and the two executions compute the tual machine monitor enables multiple operating systems same final state. A deterministic replayer is a software (guests) to share the same physical hardware (host) [21]. or hardware component that records an execution and Our deterministic replayer can record and replay the replays it deterministically. Our deterministic replayer guest execution (including both guest OS and guest is based on VMware’s VMM [1, 24], which is an ideal applications) at the IA32 Instruction Set Architecture place to record and replay the guest execution as we will (ISA) layer. During recording, only minimal information show. Our replayer supports full-system recording and to capture the nondeterminism is recorded, thereby, min- replay, i.e., the entire VM execution, including guest OS imizing the trace distortion and trace file size. Because and guest applications, is recorded and replayed. of the low overhead in recording, ReTrace is accurate, During recording, all sources of nondeterminism from fast and easy to operate. During replaying, complete outside the virtual machine are captured and logged in and detailed execution traces can be selectively collected a log file. These include data and timing of inputs to in batch mode. Even though replaying is much slower all devices, including virtual disks, virtual network in- than recording, no additional trace distortion is intro- terface cards (NIC), etc. A combination of techniques, duced because of deterministic replay. Finally, ReTrace such as device emulation and binary translation, are used is implemented completely in software and is available to ensure deterministically replay as long as the recorded as an experimental feature in VMware Workstation 6.01 device input data are replayed at right time. without requiring special tracing hardware. We solve two challenging problems in this replay system. The rest of the paper is organized as follows. Section 2 First, we efficiently keep track of number of instructions briefly introduces our deterministic replayer, which is executed in the recorded execution, so that we can re- the key enabling technology of ReTrace. Section 3 and inject the same nondeterministic events at the same ex- Section 4 describe in detail the two steps of using Re- ecution points during replay. Second, we ensure the VM Trace – ReTrace capturing and ReTrace expansion. Sec- is deterministic, even though underlying real machine is tion 5 presents evaluation results of ReTrace using SPEC nondeterministic. CPU2006 [12] and Apache Benchmark [2]. Finally, we briefly discuss existing work in Section 6 and conclude Our deterministic replayer is currently limited to unipro- in Section 7. cessor VMs and it supports popular IA32 platforms, such as Pentium 4, Opteron and Core 22. 2 Deterministic replay of virtual 3 ReTrace capturing machines The first step in using ReTrace is trace capturing. The Virtualization allows multiple operating systems (OS) resulting log file is called the replay log. The replay log to simultaneously execute on a single physical hardware contains multiple types of log entries. The log entries platform [21]. Virtualization provides many benefits, are sequentially stored in the execution order. Some en- such as server consolidation and security isolation, to tries records the nondeterministic input values needed computer users. A Virtual Machine (VM) consists of during replay. Others records both timing and values of necessary state, including the CPU, memory and I/O nondeterministic events. 1VMware Workstation 6.0 is available to researchers through 2Opteron and Core 2 are supported without hardware acceler- VMAP [13]. ation in the released VMware Workstation 6.0 software. To use ReTrace, users need to execute the target work- virtual machine. During capturing, there is minimal im- load in a virtual machine environment. The virtual ma- pact on the interactive responsiveness of a workload, due chine environment is almost identical to its native coun- to