Technical report A comprehensive feature comparison study of open- source container orchestration frameworks Eddy Truyen *, Dimitri Van Landuyt, Davy Preuveneers, Bert Lagaisse and Wouter Joosen 1 imec-DistriNet, KU Leuven; [email protected] (D.V.L); [email protected] (D.P.); [email protected] (B.L); [email protected] (W.J.) * Correspondence: [email protected]; Tel.: +32-163-735-85 Featured Application: Practitioners and industry adopters can use the descriptive feature comparison as a decision structure for identifying the most suited container orchestration framework for a particular application with respect to different quality attributes such as genericity, maturity and stability. Researchers and entrepreneurs can use it to check if their ideas for innovative products or future research are not already covered in the overall technological domain. Abstract: 1) Background: Container orchestration frameworks provide support for management of complex distributed applications. Different frameworks have emerged only recently, and they have been in constant evolution as new features are being introduced. This reality makes it difficult for practitioners and researchers to maintain a clear view on the technology space. 2) Methods: we present a descriptive feature comparison study of the three most prominent orchestration frameworks: Docker Swarm, Kubernetes and Mesos that can be combined with Marathon, Aurora or DC/OS. This study aims at (i) identifying the common and unique features of all frameworks, (ii) comparing these frameworks qualitatively ánd quantitatively with respect to genericity in terms of supported features, and (iii) investigating the maturity and stability of the frameworks as well as the pioneering nature of each framework by studying the historical evolution of the frameworks on GitHub. 3) Results: (i) we have identified 124 common features and 54 unique features that we divided into a taxonomy of 9 functional aspects and 27 functional sub-aspects. (ii) Kubernetes supports the highest number of accumulated common and unique features for all 9 functional aspects; however no evidence has been found for significant differences in genericity with Docker Swarm and DC/OS. (iii) Very little feature deprecations have been found and 15 out of 27 sub- aspects have been identified as mature and stable. These are pioneered in descending order by Kubernetes, Mesos and Marathon. 4) Conclusion: there is a broad and mature foundation that underpins all container orchestration frameworks. Likely areas for further evolution and innovation include system support for improved cluster security and container security, performance isolation of GPU, disk and network resources and network plugin architectures. Keywords: Container orchestration frameworks; Middleware for cloud-native applications; Commonality and variability analysis; Maturity of features; Feature deprecation risk; Genericity. 2 of 124 Contents 1. Introduction ..................................................................................................................................... 4 1.1 Motivation ................................................................................................................................. 6 1.2 Contribution statement ............................................................................................................. 7 1.3 Structure of the article ............................................................................................................ 10 2. Related work ................................................................................................................................. 11 3. Research method .......................................................................................................................... 12 3.1 Qualitative assessement of genericity requirement ............................................................... 12 3.1.1 Identifying features in documentation of CO frameworks .............................................. 12 3.1.2 Discovering common and unique features ...................................................................... 13 3.1.3 Organizing features in functional aspects and sub-aspects ............................................. 13 3.2 Quantitative analysis with respect to genericity ..................................................................... 14 3.3 Study of maturity ..................................................................................................................... 14 3.4 Assessment of stability ............................................................................................................ 15 3.5 Involvement and feedback from industry ............................................................................... 15 3.6 Dealing with continuous evolution of CO frameworks during the research ........................... 15 4. Qualitative assessment with respect to genericity ....................................................................... 16 4.1 Cluster architecture and setup ................................................................................................ 17 4.1.1 Common features ............................................................................................................. 17 4.1.2 Unique features ................................................................................................................ 22 4.2 CO framework customization .................................................................................................. 22 4.2.1 Common features ............................................................................................................. 22 4.2.2 Unique features ................................................................................................................ 24 4.3 Container networking.............................................................................................................. 25 4.3.1 Common features ............................................................................................................. 25 4.3.2 Unique features ................................................................................................................ 31 4.4 Application configuration and deployment ............................................................................ 32 4.4.1 Common features ............................................................................................................. 32 4.4.2 Unique features ................................................................................................................ 38 4.5 Resource quota management ................................................................................................. 40 4.5.1 Common features ............................................................................................................. 40 4.5.2 Unique features ................................................................................................................ 41 4.6 Container QoS management ................................................................................................... 41 3 of 124 4.6.1 Common features ............................................................................................................. 41 4.6.2 Unique features ................................................................................................................ 45 4.7 Securing clusters ...................................................................................................................... 46 4.7.1 Common features ............................................................................................................. 46 4.7.2 Unique features ................................................................................................................ 49 4.8 Securing containers ................................................................................................................. 49 4.8.1 Common features ............................................................................................................. 49 4.8.2 Unique features ................................................................................................................ 51 4.9 Application and cluster management ..................................................................................... 51 4.9.1 Common features ............................................................................................................. 52 4.9.2 Unique features ................................................................................................................ 57 5. Quantitative analysis with respect to genericity ........................................................................... 57 6. Assessment of Maturity ................................................................................................................ 69 6.1 Cluster architecture and setup ................................................................................................ 70 6.2 Container orchestration framework customization ................................................................ 70 6.3 Container networking.............................................................................................................. 70 6.4 Application configuration and deployment ............................................................................ 72 6.5 Resource quota