2017 IEEE 37th International Conference on Distributed Computing Systems Voyager: Complete Container State Migration Shripad Nadgowda, Sahil Suneja, Nilton Bila and Canturk Isci IBM T.J. Watson Research Center Abstract—Due to the small memory footprint and ternatives that either provide memory-only migration, or fast startup times offerred by container virtualization, rely on specific filesystems to migrate persistent storage made ever more popular by the Docker platform, con- (Section 2). Voyager performs just-in-time live container tainers are seeing rapid adoption as a foundational capability to build PaaS and SaaS clouds. For such migration with minimal downtime, by combining the data container clouds, which are fundamentally different federation capabilities of union mounts together with from VM clouds, various cloud management services CRIU-based memory migration. With a union view of data need to be revisited. In this paper, we present our between the source and target hosts, Voyager containers Voyager - just-in-time live container migration service, can resume operation instantly on the target host, while designed in accordance with the Open Container Ini- tiative (OCI) principles. Voyager is a novel filesystem- performing disk state transfer either on-demand (Copy- agnostic and vendor-agnostic migration service that on-Write) or through lazy replication. Our experiments provides consistent full-system migration.Voyager com- show Voyager’s federation framework imposes no overhead bines CRIU-based memory migration together with during data updates/writes, and ≈ 1% overheads for reads the data federation capabilities of union mounts to and upto 10% for scans. minimize migration downtime. With a union view of data between the source and target hosts, Voyager We have open-sourced an initial version of our data mi- containers can resume operation instantly on the target gration framework[8], although it works specifically with host, while performing disk state transfer lazily in the docker containers and does not support live migration. We background. intend to open-source the enhancements described in this I. Introduction paper as well, which include in-memory state migration, OCI compliance, and support for multiple data storage Container virtualization has existed since two decades types (rootfs, local and network attached data volumes). in the form of FreeBSD Jails[18], Solaris Zones[26], IBM AIX Workload Partitional WPAR[5], and LXC for II. Background and Related Work Linux, amongst others. But these have recently started gaining acceptance as a lightweight alternative to vir- The objective in this paper is not to compare and tual machines (VMs), owing to technology maturity and establish performance advantages between VM and con- popularization by platforms like Docker[14], CoreOS’s tainer migration techniques. Containers are fundamentally rocket[12], Cloud Foundry Warden[9]. Containers are be- different than VMs in terms of their system resource ing adopted as a foundational vitualization capability in requirements, high density and agility. We believe a more building Platform-as-a-Service (PaaS) and Software-as-a- optimal migration service can be designed for container Service (SaaS) cloud solutions, e.g. Amazon Container clouds than the prevalent ones in VM clouds. service[6], Google Container Engine [15] and IBM’s Con- tainer Cloud[17]. For such container clouds, which are A. VM Migration fundamentally different from VM clouds, various cloud Migration has extensively been studied primarily for management services need to be revisited. In this paper, VM Clouds[7][25][24][16][22]. Different vendors use differ- we target one such service- container migration. ent virtual disk (vDisk) formats to encapsulate a VM’s An efficient migration solution becomes essential as persistent state (e.g. vmdk, VHD, qcow2), which are containers start running production workloads. Borrowing migrated via proprietary hypervisor-assisted services like the scenarios from their VM counterparts, migrations are vMotion[29] and Hyper-V Live Migration[27], third party required during host maintenance, load balancing, server tools [7], [25], [13], or via explicit vDisk conversions[28][19] consolidation and movement between availability zones across hypervisors. Containers, on the other hand, are (e.g., a Silver zone with HDD storage and 100 IOPS vs. (being) standardised by the Open Container Initiative a Gold zone with SSD storage with 1000 IOPS). (OCI)[3] which specifies an industry standard for container In this paper, we present our Voyager container migra- image format- a filesystem bundle or rootfs, and multiple tion service, tailored specifically for containers in accor- data volumes (Section III). These being essentially direc- dance with OCI principles. Voyager is a novel filesystem- tories on the host filesystem, a generic file-based migration agnostic and vendor-agnostic migration service that pro- solution can be designed for containers. There also exist vides consistent full-system migration, unlike existing al- vendor-agnostic file-based migration solutions for VMs like 1063-6927/17 $31.00 © 2017 IEEE 2137986 DOI 10.1109/ICDCS.2017.91 I2Map[22] and racemi[4], but there require extra agents to Picocenter[30] on the other hand is a system that en- be installed inside the systems. ables swapping-out in-active containers in cloud to object Some container clouds[6][15] provision VMs to host con- store (Amazon S3) and swapping-in on-demand. It uses tainers, primarily to mitigate security and isolation con- CRIU to capture and migrate memory-state and btrfs cerns for containers, while others run containers directly filesystem snapshots for persistent state. It also proposes on cloud hosts [17]. This paper shows how to migrate the feature of ActiveSet for memory in which memory pages small execution state that is part of a container without are restored for container on-access and lazily. This is resorting to installing and migrating VMs that impose again a filesystem specific solution and not optimized for different constraints on their hosts. Installing VMs for container migration use-case. the purpose of migration defeats the advantages of using III. Design and Implementation containers for application deployment. As per OCI specifications, every container image is B. Container Migration stored in a filesystem bundle, which after unpacking be- Containers have been popular with the microservice comes just another directory– rootfs– on the host filesys- architecture. Although container migration may seem re- tem. On container instantiation, all runtime environment dundant for stateless containerized applications, but it changes (e.g., new package installations) and data changes is still pertinent to several stateful microservice applica- (e.g., application state and logs) are persisted in rootfs tions like databases (e.g. Mysql, cassandra), message bro- by default. Since runtime state changes actively, hosting kers (kafka), and state-coordination service (zookeeper), rootfs at the local filesystem also complies from a per- amongst others. This is being acknowledged and supported formance viewpoint. Additionally, any directory from the in standard frameworks like Kubernetes’ ‘StatefulSet’. host filesystem can also be bind mount inside a container Portability of stateful containers is also explored in exist- as data volumes. Such a volume can also be a network at- ing solutions like ClusterHQ’s Flocker[10], Virtuozzo[20] tached filesystem mounted on the host. Similarly, although and Picocenter[30]. rare, any block devices on the host can also be mapped Flocker[10] primarily is a data management solution inside container. As for volatile state, containers have their specifically for docker containers. It supports migration memory share from the host controlled via cgroups. for network-attached storage backends like Amazon EBS, Thus, migrating a container involves discovering all Openstack Cinder, VMware vSphere etc., by re-attaching data end-points of a container, and moving their states, these network storage for containers. Local attached vol- in addition to its memory state, consistently from the ume migration is supported only for ZFS filesystem. On source to the target host. Migration of in-memory state the other hand, Voyager is a generic, filesystem-agnostic can be achieved in userspace via CRIU. Also, any net- and vendor-agnostic migration solution. work attached storage can be migrated by un-mounting Virtuozzo is a bare-metal virtualization solution that it from source host and mounting it on target host. includes container virtualization. It facilitates Zero- In addition, Voyager provides userspace-level filesystem- downtime live migration for containers[23][20]. During this agnostic migration of locally persistent container state, migration it first transfer container’s filesystem and virtual while ensuring consistency across all these states, as memory to target host. Once transfer is finished, it freeze well as minimum application downtime. Fig.1 shows the all container processes and disable networking. It then complete orchestration framework of Voyager. Primarily, dump this memory state to file and copies these dump file we are migrating container state across three different to target host. Any changed memory and disk blocks since data stores, namely in-memory, local filesystem, and the last transfer are then migrated to target host and then network filesystem. We discuss each of these migration container is resumed. It has an underlying assumption capabilities