This is a repository copy of Cider: a Rapid Docker Container Deployment System through Sharing Network Storage. White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/166907/ Version: Accepted Version Proceedings Paper: Du, L, Wo, T, Yang, R et al. (1 more author) (2018) Cider: a Rapid Docker Container Deployment System through Sharing Network Storage. In: Proceedings of the 2017 IEEE 19th International Conference on High Performance Computing and Communications; IEEE 15th International Conference on Smart City; IEEE 3rd International Conference on Data Science and Systems (HPCC/SmartCity/DSS). 2017 IEEE 19th International Conference on High Performance Computing and Communications; IEEE 15th International Conference on Smart City; IEEE 3rd International Conference on Data Science and Systems (HPCC/SmartCity/DSS), 18-20 Dec 2017, Bangkok, Thailand. IEEE , pp. 332-339. ISBN 978-1-5386-2589-7 https://doi.org/10.1109/hpcc-smartcity-dss.2017.44 © 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Reuse Items deposited in White Rose Research Online are protected by copyright, with all rights reserved unless indicated otherwise. They may be downloaded and/or printed for private study, or other acts as permitted by national copyright laws. The publisher or other rights holders may allow further reproduction and re-use of the full text version. This is indicated by the licence information on the White Rose Research Online record for the item. Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing [email protected] including the URL of the record and the reason for the withdrawal request. [email protected] https://eprints.whiterose.ac.uk/ Cider: a Rapid Docker Container Deployment System Through Sharing Network Storage Lian Du, Tianyu Wo, Renyu Yang*, Chunming Hu State Key Laboratory of Software Development Environment BDBC, Beihang University, Beijing, China Email: {dulian, woty, hucm}@act.buaa.edu.cn; [email protected]* Abstract — Container technology has been prevalent and sourced project Harbor [19] is an enterprise-class container widely-adopted in production environment considering the huge registry server and it also integrates the decentralized image benefits to application packing, deploying and management. distribution mechanism. However, application’s images still However, the deployment process is relatively slow by using need to be fully downloaded to worker nodes, resulting in an conventional approaches. In large-scale concurrent deployments, extended latency derived from transferring large amounts of resource contentions on the central image repository would data through local network. Harter et al. [18] propose a lazy aggravate such situation. In fact, it is observable that the image fetching strategy for container data on single node and the pulling operation is mainly responsible for the degraded method can effectively reduce the container provision time. performance. To this end, we propose Cider - a novel deployment Nevertheless, large-scale and concurrent deployment system to enable rapid container deployment in a high experiments have not yet been conducted, leading to concurrent and scalable manner at scale. Firstly, on-demand image data loading is proposed by altering the local Docker difficulties in validating their effectiveness and understanding storage of worker nodes into all-nodes-sharing network storage. their operational intricacies at scale. In reality, these scenarios Also, the local copy-on-write layer for containers can ensure are commonly-used and scalability bottleneck might manifest Cider to achieve the scalability whilst improving the cost- very frequently. Therefore, it is particularly desirable for a effectiveness during the holistic deployment. Experimental container deployment system to rapidly provision container results reveal that Cider can shorten the overall deployment time especially at large scale. by 85% and 62% on average when deploying one container and In this paper, we present Cider - an innovative deployment 100 concurrent containers respectively. system for Docker containers, which can enable rapid container Keywords — container; network storage; copy-on-write; deployment in a high concurrent and scalable manner. application deployment Specifically, we alter the local Docker storage of worker nodes into all-nodes-sharing network storage, allowing for on- demand image data loading when deploying containers. This I. INTRODUCTION approach can dramatically reduce the transferred data, thereby Recent years have witnessed the prosperity of container considerably accelerating the single container deployment. technique and Docker is undoubtedly the most representative Additionally, to achieve higher scalability and efficiency of among them [1]. Compared with virtual machine, container can memory usage, we design and implement a local copy-on-write provision performance and user-space isolation with extremely (COW) layer for setting-up containers. The approach assigns low virtualization overheads [2]. The deployment of concurrent COW request flooding in network storage into local applications (especially in large clusters) increasingly tends to worker nodes. Experimental results show that compared with depend on containers, driven by recent advances in packing conventional approach, Cider is able to shorten the average and orchestration. Massive-scale container deployment is deployment time by 85% in single container case. A 62% becoming increasingly important for micro-service compos- reduction can still be achieved when concurrently deploying ition and execution. Unfortunately, the enlarged system latency 100 containers, with no conspicuous runtime overheads. In introduced by slow container deployment is becoming a non- particular, the major contributions in this paper can be summa- negligible problem, which is critical in scenarios such as rized as follows: bursting traffic handling, fast system component failover etc. According to our in-depth investigation of conventional Introducing a rapid and cost-effective container deployment approaches, we found that the mean deployment deployment system based on sharing network storage, time of the top 69 images in Docker Hub is up to 13.4 seconds which can reduce the amount of transferred data during and 92% of the holistic time is consumed by transferring image the deployment. data through network. Even worse, in large-scale concurrent Combining network storage with the local file-level deployments, resource contentions on the central image COW layer to guarantee the system scalability and repository would aggravate such situation. memory efficiency in face of jobs of high concurrent To cope with slow container deployment and improve the deployment. launch efficiency, cluster management systems (such as Borg The remaining sections are structured as follows: Section II [12], Tupperware[13], Fuxi[25] etc.) adopt a P2P-based introduces the background and our motivation; Section III method to accelerate the package distribution. VMware open- node1 image in the worker node is managed by the storage driver such as metadata Aufs [3], Devicemapper [4], Overlay or Overlay2 [5]. After the image is pulled, the driver will setup another COW layer on local COW storage top of the image for the container. containers In effect, current deployment approaches [8] are far from 2. COW enough, especially in cases that highly require rapid nginx postgres deployments. To illustrate this phenomenon, we evaluate the debian deploy time of the top 69 images from Docker Hub [6] in node2 sequence (Images are ordered by pull times and the selected 1. pull images are pulled by more than 10 million times; the image size is 347 MB on average). The evaluation is actually registry image conducted through setting up a private registry and deploying metadata containers on another node which resides in the same LAN of the registry. The result demonstrates that the mean deploy time deploy nodeN image layers can be up to 13.4 seconds. Furthermore, there are a number of scenarios that require low deploy latency. For instance, services should scale-out instantly to tackle bursting traffic Fig. 1. Conventional container deployment architecture. whilst applications should recover as soon as possible upon the arrival of failover event to retain the system availability [26]. presents the core idea of design and implementation in Cider; Developers might intend to acquire the high efficiency and Section IV shows the experimental results and Section V productivity in Continuous Integration/Continuous discusses the related work; Finally, Section VI presents the Deployment work flow. All these requirements motivate us to conclusion followed by the discussion of future work. boost the performance of container deployment. II. BACKGROUND AND MOTIVATION B. Deep Dive into Container Deployment Process Container technique has been widely adopted in the In order to investigate the root cause of slow deployment, production environment over a decade[12][13] as it provides we conduct a fine-grained analysis into the deployment several critical capabilities for large-scale