TimeKeeper: A Lightweight Virtual Time System for Linux Jereme Lamps, David M. Nicol, Matthew Caesar University of Illinois at Urbana-Champaign {lamps1, dmnicol, caesar}@illinois.edu ABSTRACT time variable? In most systems the system clock will give We present TimeKeeper: a simple lightweight approach to the time, but the system clock in a VMM typically reflects embedding Linux containers (LXC) in virtual time. Each time advancement of the VMM, not its VMs. A Xen VM container can be directed to progress in virtual time either has an associated domain time, which reflects the amount of more rapidly or more slowly than the physical wall clock wall-clock time that the VM has received. The domain time time. As a result, interactions between an LXC and physi- advances with the system clock while the VM is served, and cal devices can be artificially scaled, e.g., to make a network stops advancing when it no longer has CPU service. How- appear to be ten times faster with respect to the software ever, domain time is used in Xen for scheduling, and not as within the LXC than it actually is. Our approach also sup- a measure of virtual time. ports synchronized (in virtual time) emulation, by grouping The idea for embedding Xen in virtual time was originally LXCs together into an experiment where the virtual times of expressed in the context of testing distributed applications containers are kept synchronized, even when they advance at [7]. The basic idea is to make virtual time in a VM advance different speeds. This has direct application to the integra- more slowly than real time, in order to make the (real) net- tion of emulation and simulation within a common frame- work connected to the VM appear to be performing faster. work. The approach associates with each VM an integer-valued time dialation factor,orTDF.ATDFofn reduces the ad- vancement rate of a VM in time by a factor of n; for example, Categories and Subject Descriptors a TDF of 2 makes virtual time in the VM advance at half the C.2.4 [Computer-Communication Networks]: Distributed rate of wall-clock time. This approach (and subsequent ones Systems—distributed applications; D.4.4 [Operating Sys- [6]) rescale a VM’s notion of time with reference to a physical tems]: Communications Management—message sending, com- network, in order to emulate a seemingly accelerated rate of munication management; D.4.8 [Operating Systems]: Per- interaction between the VM and the network. formance—measurements, simulation; I.6.3 [Simulation and We are motivated by a different objective, to virtualize Modeling]: Applications—Miscellaneous time in a VM in order integrate its behavior with a network simulator such as S3F [11] or ns-3 [13]. This goal raises Keywords new considerations. A network simulator can represent a much larger infrastructure than a real network in a typi- Simulation, Emulation, LXCs, Virtualization, Time Dila- cal lab, which creates the need to emulate in virtual time tion, CORE, Linux Kernel many VMs; this in turn raises the importance of minimiz- ing overhead. In particular, we want to have VMs “jump” 1. INTRODUCTION over epochs in virtual time where nothing of interest occurs, Virtual machine managers (VMM) multiplex the execu- rather than rescale time and have the VM crawl through tion of virtual machines (VM), i.e., software stacks, in such the epoch just to advance its clock to its next interaction a way that the VMs behave as though they are running on with the network simulator. An example is a web-server, individual pieces of hardware. A question of great interest whose behavior is to wait for a request, formulate and issue to us is how the advancement of time in a VM is perceived. a database query, wait for the IO system’s response, report For example, suppose that an application in one VM sends the response and then wait for another request. Depending a message to another, and includes in that message the time on what the experiment is measuring, the VM might be di- at which the message was sent. What is the value of that rected to reset its virtual time to the time of a request or IO completion and completely bypass epochs where the pro- cess is suspended. Another new consideration is the need to Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed advance a group of VMs through virtual time so that their for profit or commercial advantage and that copies bear this notice and the full cita- virtual clocks are closely synchronized, even if they advance tion on the first page. Copyrights for components of this work owned by others than those clocks at different rates by virtue of different TDFs. ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- A final goal is to bring virtual time to the Linux kernel in publish, to post on servers or to redistribute to lists, requires prior specific permission a minimally invasive way, that exposes an API to support and/or a fee. Request permissions from [email protected]. our motivating problem of integrating emulation and simula- SIGSIM-PADS’14, May 18–21, 2014, Denver, CO, USA. Copyright 2014 ACM 978-1-4503-2794-7/14/05 ...$15.00. tion, and is general enough to support other uses of virtual http://dx.doi.org/10.1145/2601381.2601395 . 179 time. These considerations have led us to develop Time- other users may be running experiments on the same nodes Keeper, a small set of modifications to the Linux kernel that simultaneously. allows for the creation of LXCs, each with their own vir- There also exist hybrid solutions, supporting both simu- tual clock. There are two main distinctions separating this lation and emulation, such as the Common Open Research project from previous ones. First, our approach becomes an Emulator (CORE) [1] and ns-3 [13]. This allows the simula- integral (and very small) part of the Linux kernel, giving it tor to be interfaced with real-world communication systems the potential to become mainstream. Second, our approach for more realistic measurements. is much lighter weight than Xen [2] or OpenVZ [16] (admit- tedly at the cost of less generality than Xen). Finally, our 2.2 Virtual Time Systems approach to virtual time synchronization is more sophisti- There have been many recent papers dealing with giv- cated than that which has been applied to Xen’s to date, ing systems a sense of virtual time, e.g., [3, 5, 6, 19, 22]. and has greater flexibility of interaction between emulation DieCast [6] makes modifications to the Xen hypervisor to and simulation than previous solutions based on OpenVZ. give VM’s a concept of virtual time. DieCast also scales the performance of physical hardware components. This is a useful option if you want to create an experiment where 2. RELATED WORK the number of nodes in the experiment is greater than the Related work exists in the area of simulation/emulation, number of nodes in your testbed. SVEET! [3] is a perfor- and in the area of virtual time. We discuss these separately. mance evaluation testbed running on Xen-based VMs that implements time virtualization techniques if the simulation 2.1 Simulation/Emulation is overloaded. It sets a static TDF to slow down both the In simulation computer systems are modeled entirely in simulator and the VMs. Our work differs from both DieCast software. Simulation has the attractive property of being and Sveet, as we use lightweight LXCs with Linux kernel scalable and repeatable. There are various simulation tools modifications instead of Xen-based VMs. We can also dy- available today, such as J-Sim [9], ns-2 [12], OMNeT++ namically change TDFs, as well as support the synchroniza- [14], and ns-3 [13]. J-Sim is a component-based simulation tion of LXCs virtual times, even if they have different TDFs. environment, where each link, node, and protocol is a com- In some ways our approach resembles that of Zheng et al., ponent. Components have ports associated with them, and [22] who developed a virtual time system for simulation and a component contract describes how data should be han- emulation using OpenVZ. Like our solution, they modified dled if it arrives at a specific port. Event executions are time-related system calls to return virtual time as opposed in real-time, thus improving the fidelity of the simulation. to the system time. However, our work is different, as it OMNet++ and ns-2 are both popular discrete event sim- uses LXCs, and brings the notion of a time dialation fac- ulators. Both simulators are written in C++, while ns-2 tor to the forefront. The OpenVZ system scales measured provides the interface through OTcl. In ns-2, models are elapsed time as we do, but that scaling factor is fixed. flat, meaning that creating subnetworks is not possible. On the other hand, OMNeT++ supports a hierarchical module structure which makes it easier to develop complex mod- 3. DESIGN els in a methodical manner. Many papers have compared We designed TimeKeeper with three objectives. First, we the performance of ns-2 and OMNeT++, concluding that wanted to develop a lightweight solution. This minimizes ns-2 is not nearly as scalable or easy to use as OMNeT++ overhead for time-dilated processes. Next, we wanted a sim- [10, 15, 20]. Also, there is ns-3, a discrete event simulator ple solution which would allow researchers to create and test that is aimed to overcome ns-2’s shortcomings.