applied sciences Article A Comprehensive Feature Comparison Study of Open-Source Container Orchestration Frameworks Eddy Truyen * , Dimitri Van Landuyt, Davy Preuveneers , Bert Lagaisse and Wouter Joosen imec-DistriNet, KU Leuven, 3001 Leuven, Belgium; [email protected] (D.V.L.); [email protected] (D.P.); [email protected] (B.L.); [email protected] (W.J.) * Correspondence: [email protected]; Tel.: +32-163-735-85 Received: 23 November 2018; Accepted: 25 February 2019; Published: 5 March 2019 Featured Application: Practitioners and industry adopters can use the descriptive feature comparison as a decision structure for identifying the most suited container orchestration framework for a particular application with respect to different software quality properties such as genericity, maturity, and stability. Researchers and entrepreneurs can use it to check if their ideas for innovative products or future research are not already covered in the overall technological domain. Abstract: (1) Background: Container orchestration frameworks provide support for management of complex distributed applications. Different frameworks have emerged only recently, and they have been in constant evolution as new features are being introduced. This reality makes it difficult for practitioners and researchers to maintain a clear view of the technology space. (2) Methods: we present a descriptive feature comparison study of the three most prominent orchestration frameworks: Docker Swarm, Kubernetes, and Mesos, which can be combined with Marathon, Aurora or DC/OS. This study aims at (i) identifying the common and unique features of all frameworks, (ii) comparing these frameworks qualitatively and quantitatively with respect to genericity in terms of supported features, and (iii) investigating the maturity and stability of the frameworks as well as the pioneering nature of each framework by studying the historical evolution of the frameworks on GitHub. (3) Results: (i) we have identified 124 common features and 54 unique features that we divided into a taxonomy of 9 functional aspects and 27 functional sub-aspects. (ii) Kubernetes supports the highest number of accumulated common and unique features for all 9 functional aspects; however, no evidence has been found for significant differences in genericity with Docker Swarm and DC/OS. (iii) Very little feature deprecations have been found and 15 out of 27 sub-aspects have been identified as mature and stable. These are pioneered in descending order by Kubernetes, Mesos, and Marathon. (4) Conclusion: there is a broad and mature foundation that underpins all container orchestration frameworks. Likely areas for further evolution and innovation include system support for improved cluster security and container security, performance isolation of GPU, disk and network resources, and network plugin architectures. Keywords: container orchestration frameworks; middleware for cloud-native applications; commonality and variability analysis; maturity of features; feature deprecation risk; genericity 1. Introduction In recent years, there has been a strong industry adoption of Docker containers due to its easy-to-use approach for distributing and bootstrapping container images. Moreover, in comparison to virtual machines, Linux containers have a lower memory footprint and allow for flexible resource Appl. Sci. 2019, 9, 931; doi:10.3390/app9050931 www.mdpi.com/journal/applsci Appl. Sci. 2019, 9, x FOR PEER REVIEW 2 of 81 1. Introduction In recent years, there has been a strong industry adoption of Docker containers due to its easy- Appl.to-use Sci. approach2019, 9, 931 for distributing and bootstrapping container images. Moreover, in comparison2 of 76to virtual machines, Linux containers have a lower memory footprint and allow for flexible resource allocation to improve server consolidation [1]. The popularity of Docker has also changed the way in allocationwhich application to improve software server consolidationcan be packaged [1]. Theand popularitydeployed: ofcontainer Docker images has also are changed self-contained the way incomponents which application that can softwarebe tagged can with be packagedversion nu andmbers deployed: and are containermade available images for are download self-contained from componentsprivate or public that can Docker be tagged regist withries. versionMoreover, numbers container and areimages made are available portable for downloadacross different from privateoperating or publicsystems Docker and different registries. cloud Moreover, provider container stacks [2]. images are portable across different operating systemsContainer and different orchestration cloud provider (CO) frameworks, stacks [2]. such as Docker Swarm, Kubernetes, and the Mesos- basedContainer Marathon, orchestration provide support (CO) for frameworks, deploying and such managing as Docker a multi-tiered Swarm, distributed Kubernetes, application and the Mesos-basedas a set of containers Marathon, on provide a cluster support of nodes for deploying[3]. Container and managingorchestration a multi-tiered frameworks distributed have also applicationincreasingly as been a set used of containers to run production on a cluster work of nodesloads [ 3as]. Containerfor example orchestration demonstrated frameworks in the annual have alsoOpenStack increasingly user survey been used [4–6]. to run production workloads as for example demonstrated in the annual OpenStackHowever, user there survey have [4– 6been]. several high paces of feature additions among the most popular CO frameworksHowever, as thereillustrated have by been Figure several 1, which high pacesshows ofthe feature number additions of feature among additions the mostbetween popular June CO2013 frameworks and June 2018. as illustrated As shown, by there Figure was1, whicha first peak shows of the feature number additions of feature between additions June between2014 and JuneJanuary 2013 2015 and because June 2018. Mesos As v0.20.0 shown, [7] there added was supp a firstort for peak Docker of feature containers additions and betweenGoogle open- June 2014sourced and Kubernetes January 2015 v0.4.0 because [8] Mesosthat from v0.20.0 its [ 7inception] added support offered for support Docker for containers Docker andcontainers. Google open-sourcedMoreover, Kubernetes Kubernetes v0.6.0 v0.4.0 inclu [8ded] that several from innovating its inception features offered such support as container for Docker IP and containers. service Moreover,IP networking Kubernetes [9], pods v0.6.0 [10], included and persistent several innovatingvolumes [11]. features This such caused as container a ripple IPeffect and of service feature IP networkingadditions across [9], pods the [other10], and CO persistent frameworks. volumes For exam [11]. Thisple, support caused afor ripple persistent effect of volumes feature additionshas been acrossadded theto Docker other COv1.7 frameworks. [12] in June 2015. For example,By August support 2016, Docker’s for persistent architecture volumes for haspersistent been added volumes to Dockerhas also v1.7 been [12 ]supported in June 2015. by ByMesos August v1.0.0 2016, [13]. Docker’s As another architecture example, for persistent support volumesfor container has also IP beennetworking supported has bybeen Mesos added v1.0.0 to [Mesos13]. As v0.25.0 another [14] example, and Docker support Swarm for container stand-alone IP networking v1.0.0 [15] has by beenJanuary added 2016. to Mesos v0.25.0 [14] and Docker Swarm stand-alone v1.0.0 [15] by January 2016. Figure 1. Density of feature additions over time (common featuresfeatures only).only). ThisThis highhigh pacepace of feature additionsadditions hashas beenbeen aa challengechallenge forfor companies toto (a) keep an up-to-date understandingunderstanding ofof whatwhat constitutesconstitutes thethe conceptualconceptual foundationfoundation ofof thethe overalloverall domain,domain, toto (b)(b) determine whichwhich COCO frameworkframework matchesmatches mostmost closelyclosely withwith theirtheir requirementsrequirements andand toto (c)(c) determinedetermine whichwhich framework is most mature with respect to these requirements. This is both a risk for companies who start using container orchestration technology and companies who consider migrating from one CO framework to another framework. They are also faced with (d) feature deprecation risks, i.e., there is strong dependence on a feature that will not be supported anymore by future versions of the employed Appl. Sci. 2019, 9, 931 3 of 76 CO framework. Finally, (e) academic researchers and entrepreneurs are also faced with the challenge that their innovative idea for a product or research prototype may become obsolete when a new version of the CO framework has been released. An illustration of these challenges from the entrepreneur side is the story of ClusterHQ, a company that pioneered in 2014 with the container data management service Flocker [16]. Flocker initially gained a lot of traction and the company raised 12 $ million in 2015 [17] and there was a well-working integration [18] with Kubernetes, Mesos and Docker Swarm. However, by the end of 2016, the company stopped all its activities because of reportedly “self-inflicted wounds” [19]. Actually, by that time all major CO frameworks provided also built-in support for external persistent volumes. A final challenge is to (f) keep track and interpret ongoing standardization efforts in this space. For example, the Cloud Native Computing Foundation