Supporting Docker in Emulab-Based Network Testbeds David Johnson Elijah Grubb Eric Eide University of Utah {johnsond, grubb, eeide}@cs.utah.edu Abstract that allows a person to package a collection of software Researchers conduct experiments in a variety of comput- into an image and then instantiate virtual devices based ing environments, including dedicated testbeds and com- on that image. Docker images are portable across multi- mercial clouds, and they need convenient mechanisms for ple infrastructures because they are based on containers, deploying their software within these disparate platforms. which run atop an infrastructure-managed OS kernel and To address this need, we have extended Emulab so that it hardware device. The Docker ecosystem includes public can instantiate and configure container-based virtual de- repositories of popular “virtual appliances” and tools that vices using Docker images. Docker is a de facto standard allow people to create and publish their own images. for packaging and deploying software in cloud environ- We have extended the Emulab testbed-management ments; now, researchers can use Docker to package and software to support the allocation and configuration of deploy software within Emulab-based testbeds as well. container-based devices using Docker images. The main We describe how Emulab incorporates Docker and how benefit is to make it easier for experimenters to move ac- it extends Docker images to support the interactivity that tivities into Emulab-based testbeds—including the Utah researchers expect within a testbed. We show that Emulab Emulab site, CloudLab, PhantomNet, and POWDER2— can use many popular Docker images to create testbed and out to production or other environments. An addi- experiments. We expect that Emulab’s support for Docker tional and important benefit is scale. Because containers will make it easier for researchers to move their activities are a lightweight virtualization technology, testbed users freely, both into the testbed and out into production. can run experiments that involve large numbers of vir- tual nodes while using only modest physical resources. 1 Introduction Obtaining these benefits was not straightforward. Our Reliably initializing and configuring devices is essen- goal was to add Docker support to Emulab in a way that tial for conducting repeatable experiments in a testbed. preserves both the Emulab features that testbed users ex- Since its inception, the Emulab testbed-management sys- pect and the Docker features that users of that ecosystem tem [17] has used a custom disk-imaging subsystem for expect. Smooth integration requires that existing Docker installing operating systems and other software on de- images “just work” as the bases of devices in Emulab. vices. That subsystem, called Frisbee [11], was built and This paper describes how we met this challenge. optimized for distributing disk images within a controlled We make two contributions. First, we present the de- cluster of machines—and only within a cluster. Today, sign and implementation of Docker support within Em- however, many Emulab users rely on multiple platforms ulab. Emulab automatically adapts Docker images for to conduct experiments over the course of a full study. use in a testbed: this allows container-based virtual de- In addition to using multiple clusters (e.g., Utah Emu- vices to (1) be “first-class citizens” within experiments lab, CloudLab, and Chameleon)1 to run tests atop myriad and (2) support features that researchers expect of devices hardware, a cybersecurity researcher might debug an ex- within Emulab, such as interactive login. Second, we periment by scaling it down to run in VMs on a laptop, show that Emulab’s Docker support works with existing, and perform “production testing” by scaling it up for a third-party Docker images. Emulab can automatically commercial cloud. An experimenter who moves among adapt 52 of the 60 most popular images from Docker Hub multiple platforms needs to deploy software and config- into the Emulab environment and instantiate containers ure devices in each of these environments—a task that from them (§5.1). The process also works for images from Frisbee was never designed to tackle. research projects that we selected as case studies (§5.2). Happily for testbed users, the DevOps community has The time to adapt a Docker image for Emulab is suitable already met the challenge. Docker [6] is a popular tool for on-demand conversion, and Emulab can quickly create large experiments using Docker images (§5.3). 1https://www.emulab.net/, https://cloudlab.us/, https: //www.chameleoncloud.org/ 2https://phantomnet.us/, https://powderwireless.net/ 2 Background: Emulab are visible to the processes running inside it. These may be very different from the resources that are available to Emulab is testbed-management software that supports processes that run on the host outside of the container; controlled and repeatable experiments in systems, net- indeed, through the careful selection of resources, con- working, cybersecurity, and other areas of computing. tainers are generally configured to isolate the processes The Utah Emulab testbed manages hundreds of bare-metal that run within them. Docker deploys containers along servers that have multiple network interfaces to a variety with filesystem images: the user-space software running of layer 1 and 2 switches. It provisions subsets of those within the container is therefore independent of the soft- resources into isolated, emulated networks of physical ware provided by the container’s host. and virtual nodes for users on demand. Emulab resembles Prior to Docker, a variety of other systems (e.g., lxc, the control software of today’s clouds, which provision OpenVZ, Linux vservers, and FreeBSD jails) provided virtual machines and networks atop physical substrate, al- similar container experiences. The overwhelming success though Emulab in general offers more kinds of resources, of Docker can be attributed to its overall accessibility lower-level resource control, and stronger isolation. to users, especially its image-management toolchain and To allocate resources in Emulab, a user writes a profile, central, public image repository. Any system that seeks which is a description of everything needed to build an em- to build on top of Docker, therefore, must be careful to ulated, networked research environment: physical and/or preserve the user experience that makes Docker attractive. virtual hardware (servers, storage, switches), network con- The primary use case envisioned by Docker’s devel- figuration (links and LANs, IP addresses, traffic shaping), opers was not that a Docker container would serve as a and software (operating systems, packages, scripts). The kind of node in a testbed. Still, we assert that Docker can user then instantiates the profile to create an experiment. be successfully integrated into testbeds, and that it can Emulab provisions the experiment with the hardware and provide useful provenance and reproducibility features software described in the profile. The profile may request to experimenters. Below, we examine the possibility of specific hardware and configuration, such as requesting utilizing Docker as a type of network testbed node in Em- specific physical machines, or asserting that some virtual ulab and discuss the challenges (C1–C4) we encountered machines should run on specific physical host nodes. Al- while designing our integration. Our goals were not only ternatively, the user can leave some or all of the node and to provide a testbed-oriented container orchestration ser- network configuration details to Emulab, which will find vice, but also to allow users to seamlessly use the Docker a best-fit mapping onto its available physical resources. toolchain in concert with Emulab’s services and features. Once an experiment is allocated, its creator has full con- trol over its resources. Users interact with nodes over a 3.1 Broadening the Process Model (C1) remotely accessible control network. In addition to simply logging into nodes and running installed software, a user The primary use case that Docker supports involves a can change software and configuration, capture disk im- large collection of containers, potentially spanning many ages, change traffic-shaping parameters, take down links, machines, in which each container runs a single applica- reboot nodes, and so on. tion (one or more processes launched by a single parent) that provides a specific service (i.e., a microservice ar- 3 Leveraging Docker in a Network Testbed chitecture). For instance, there are popular Docker im- ages that run memcached and redis3 as services. In this The core abstraction that a network testbed provides to model, container images are tailored to support only their its users is a collection of physical or virtual nodes, con- single service, unencumbered by unnecessary software nected into networks of links and LANs. A user provides or configuration. Many Docker images do not include an the testbed with a description of the environment he or init daemon to launch and monitor processes, or even a she requires: the hardware, network, software, and au- basic set of user-space tools and libraries. tomation needed to run the user’s experiment(s). The The Docker community discourages whole-OS-like testbed maps this description to a subset of its resources, containers that run an init and the traditional litany of configures them, and makes them available to the user. supporting services, e.g., syslogd, sshd, and crond [9]. Docker enables its users to create, deploy, and manage The rationale is that these services are not needed in ev- containers, which are process-level “virtual machines” fa- ery container: in a large-scale deployment, if a single cilitated by the underlying operating system abstractions. service container crashes, orchestration software can sim- A container is an execution environment that sits atop a ply spawn another container to fill the void. Docker 1.13 host’s operating system kernel and hardware resources; (API version 1.25) added the ability to run a containerized a collection of processes runs within the container.