A Buyer's Guide to Enterprise Kubernetes Solutions

A Buyer's Guide to Enterprise Kubernetes Solutions A Comparison of Pivotal PKS, Platform9 Managed Kubernetes, Rancher, and Red Hat OpenShift A Buyer's Guide to Enterprise Kubernetes Solutions A Comparison of Pivotal PKS, Platform9 Managed Kubernetes, Rancher, and Red Hat OpenShift

In just a few years, Kubernetes has rapidly emerged as the de-facto open source standard for container orchestration. Numerous Kubernetes solutions and products have emerged from startups to established traditional vendors thus making it difficult to compare and contrast the various offerings. This guide identifies 18 technical and operational capabilities to consider while evaluating various vendor offerings and then provides a detailed comparison of the level of completeness of these capabilities for four leading solutions in this market: Pivotal PKS, Red Hat OpenShift, Rancher, and Platform9 Managed Kubernetes.

What are the key features to consider while evaluating an enterprise Kubernetes platform?

1. Provisioning of Kubernetes Clusters, High • Hosted Kubernetes as a service (KaaS) - A vendor Availability and Healing will offer Kubernetes as a service on top of Kubernetes does not offer deployment of highly infrastructure that’s hosted by the provider available clusters out of box and must be configured by the Kubernetes administrator. It is recommended • Hybrid Cloud Kubernetes as a Service that at least three master nodes are configured - Kubernetes is offered as a service on the behind a load balancing solution with integrated infrastructure of your choice, your own or independent clustered deployment of etcd that infrastructure or public cloud stores all the cluster state information. Any high availability solution must also account for failure 3. Prerequisites and Operating System Requirements scenarios and auto-repair and recovery. The prerequisites of an enterprise Kubernetes solution define what infrastructure requirements you 2. Deployment Model(s) Supported need to satisfy before you can get up and running The deployment model of a Kubernetes solution with Kubernetes. Some solutions require an expensive defines how it will integrate within your enterprise licensing purchase of underlying infrastructure, environment and what level of support service such as a hypervisor, or an investment in a hosted level agreement (SLA) it can provide for day 2 Kubernetes solution. operations. The top three deployment models for Kubernetes solutions are: 4. Monitoring and Operations Management A production Kubernetes cluster must be monitored • Traditional on-premises deployment - users at all times to handle any issues and outages download and deploy Kubernetes on their without severely affecting cluster and application infrastructure on their own or using professional availability to users. An enterprise Kubernetes services and support from a vendor solution must provide this capability out of box.

2 5. Cluster Upgrades strategy by investing in multiple cloud solutions. Kubernetes has a large community of contributors Having multiple private and/or public clouds as part and a new version is available every 3 months. An of your cloud strategy ensures that you aren’t getting enterprise-class solution will support rolling upgrades locked into a single provider with no leverage on pricing, of clusters, such that the cluster and the cluster API to have high availability across your infrastructure is always available even while the cluster is being overall, and to satisfy your unique business policies. upgraded. Additionally, it will provide the ability to rollback to previous stable version upon failure. 11. Enterprise Grade User Experience Enterprise grade user experience is all about having 6. Multi-cluster Management a polished user interface that enables enterprises to A single Kubernetes cluster can scale horizontally to manage their hybrid environments though a single support large sets of workloads. However, running UI. This delivers complete visibility simplifying Kubernetes in production requires being able to run communications across the environment. This UI multiple Kubernetes clusters, as you will want to fully isolate should allow operations that span multiple clusters: your dev/test/staging applications from production for example, globally searching for workloads of a applications by deploying them on a separate cluster. specific type or tagged with a specific label across all clusters running on different regions, data 7. Multi-tenancy, Role-based Access Control and centers and cloud providers. Single Sign-on Support Kubernetes supports multi-tenancy at the cluster 12. Application Lifecycle Management - level using the namespace abstraction. However, in Application Catalog a multi-cluster environment, you need a higher level Application catalog provides easy one-click deployment multi-tenancy abstraction to supplement Kubernetes for a set of pre-packaged applications on top of multi-tenancy and provide the right level of isolation Kubernetes. It also provides end users a vehicle to build across different teams of users. It should integrate and publish their own applications via the catalog for others with Single-Sign On (SSO) solutions most commonly in their team or their organization to deploy in a one click used by enterprises such as Active Directory or manner. The application catalog enables organizations to ADFS, Okta, and other popular SAML providers. standardize on a set of application deployment recipes or blueprints, avoiding sprawl of configurations. 8. Load Balancing Kubernetes automatically load balances requests to 13. Production Grade Service Level Agreements (SLA) application services inside of a Kubernetes cluster. As more and more organizations are running their business However, some services need to be exposed externally on Kubernetes, IT must ensure that it can support the SLAs for consumption by outside clients. Kubernetes does that the business requires. IT must ensure that Kubernetes not provide an out-of-the box load balancing solution is available to developers and the business to support key for that type of services. An enterprise Kubernetes initiatives. Most organizations require 99.9% uptime. product should include a robust external load balancing solution, or integrate seamlessly with existing commercial 14. Ease of Setup, Installation, Continuous load balancers. Use, Management, and Maintenance A successful Kubernetes platform must be easy to 9. Private Registry Support and implement and maintain so organizations can leverage Image Management containers continuously. This alone is a major barrier Running containerized applications on Kubernetes clusters that many organizations do not overcome. requires having access to a container registry where your application images will be stored. A large enterprise 15. Networking Support and Integrations organization will typically want a secure private container Networking integration is a critical component of registry to store their proprietary application images. An running Kubernetes clusters in production and at scale. enterprise Kubernetes solution should provide image An enterprise will typically want Kubernetes to integrate management capability out of box. with a Software-Defined-Networking (SDN) solution of their choice that they currently standardize on or a 10. Hybrid Cloud Integrations and APIs container native solution such as calico or weave that Every enterprise today wants to build a cloud neutral gives them more options around isolation.

3 16. Storage Support and Integrations Similar to networking, integration with enterprise grade storage is an essential component of running Kubernetes clusters in production. Kubernetes provides an abstraction called Persistent Volumes to hold data persisted by stateful applications. It is important for a Enterprise Kubernetes product to map PVs to an actual highly-available storage technology. Enterprises will typically want their Kubernetes deployment to integrate with storage solutions that they have already deployed such as NetApp, Pure, SolidFire, etc. or they may want to integrate with a container native storage technology such as Portworx.

17. Self Service Provisioning Developers must have self-service access to one or more Kubernetes clusters with right levels of isolation in place so only members with right privileges can access production workloads.

18. Built-in CI/CD Support One of the most critical workloads run by the developers is Continuous Integration / Continuous Delivery. A robust CI / CD pipeline is critical to ensure agile development and rapid delivery of new software releases to customers.

Rancher, and Red Hat OpenShift

Platform9 Managed Kubernetes,

A Comparison of Pivotal PKS,

Solutions

Kubernetes

to Enterprise A Buyer's Guide Comparison Scorecard

PLATFORM9 RED HAT FEATURES MANAGED RANCHER PIVOTAL PKS OPEN SHIFT KUBERNETES

Provisioning of Kubernetes Clusters

High Availability and Healing

Deployment Model(s) Supported

Prerequisites and Operating System Requirements

Monitoring and Operations Management

Cluster Upgrades

Multi-cluster Management

Multi-tenancy, Role-based Access Control, and Single Sign-on Support

Load Balancing

Private Registry Support and Image Management

Hybrid Cloud Integrations and APIs

Enterprise Grade User Experience

Application Lifecycle Management - Application Catalog

Production Grade Service Level Agreement

Ease of Setup, Installation, Continuous Use, Management, and Maintenance

Networking Support and Integrations

Storage Support and Integrations

Self Service Provisioning

Built-in CI/CD Support

5 Detailed Comparison Table

PLATFORM9 MANAGED RED HAT PRODUCT RANCHER PIVOTAL PKS KUBERNETES OPEN SHIFT

Provisioning of Fully automated provi- Fully automated Fully automated Fully automated Kubernetes Clusters sioning of clusters provisioning of clusters provisioning of clusters provisioning of clusters

High Availability »»Built-in support »»Leverages native »»Supports a highly »»Includes Cloud and Healing for highly available Kubernetes features to available cluster Foundry Ops clusters out of the box deliver HA and healing deployment Manager Provides BOSH which is a vital »»Clusters of 1/3/5 »»The default HAProxy component within PKS masters are supported load balancer can for quorum be used to create »»BOSH monitors the a multi-master and health of clusters and »»Built-in etcd high multi-etcd cluster enables self-healing availability support environment - with etcd to optimize capacity »»Supports full repair or nodes either forming Unhealthy nodes recovery of etcd upon their own cluster or are automatically failure deployed on the same detected and node as the master resurrected without downtime

Deployment Model(s) One deployment »»Traditional deployment Three deployment One deployment model Supported model offered: and support model models offered: offered across three with software platforms. The product »»SaaS-managed »»Hosted Public downloaded files are downloaded for offering - ideally suited Cloud Offering - trial and installed Pivotal Ops Manager for enterprise hybrid environment only and PKS and the end- clouds, includes sup- »»Support for on-prem »»OpenShift Dedicated user performs the install. port, upgrades, remote and public cloud - Single-tenant, VMware vSphere, AWS, management, etc. deployments high-availability and GCP supported OpenShift clusters hosted on Amazon Web Services Delivered as a hosted service »»OpenShift Container Platform - Enterprise offering available for large customers with on-prem and/or hybrid infrastructure

Prerequisites and Supports all popular Supports all popular »»Supports Red Hat PKS has an extensive Operating System enterprise Linux distribu- enterprise Linux distribu- Linux only list of prerequisites: tions - Red Hat, CentOS, tions - Red Hat, CentOS, Requirements »»RHEL subscription »»It can not be installed Ubuntu Ubuntu is required and on any linux systems included as part of the »»Requires VMware OpenShift bundled vSphere Enterprise product subscription Plus Edition or vSphere with Operations Management Enterprise Plus minimum

6 PLATFORM9 MANAGED RED HAT PRODUCT RANCHER PIVOTAL PKS KUBERNETES OPEN SHIFT

Monitoring and »»24 x 7 live monitoring »»Performs health »»Diagnostic tools via »»Does not provide any checks on all command line for out of the box live Operations »»99.9% guaranteed SLA Kubernetes clusters, health statistics monitoring for your k8s Management »»Proactive repair and presents resource clusters, Prometheus »»Environmental health consumption statistics support, etc. »»Automated email check information notifications for any »Sends cluster-level »Traditional support » »»Prometheus available » issues alerts for Kubernetes ticketing system but requires lot of system components »»Automated support manual configuration (e.g., etcd, DNS, ticket creation and for storage, alerting etc.) -Customizable triaging of issues etc resource alerts such as CPU, memory etc. »»Traditional support ticketing process for issues »»Automatically deploys and configures Prometheus for monitoring

Cluster Upgrades »»Fully automated »»Providers an easy built Administrators need to »»Patching and upgrades cluster upgrades in cluster upgrade manually initiate upgrades of Kubernetes nodes delivered seamlessly, experience to clusters upon installing can be managed from with no interruption new version of OpenShift the PKS platform in »»Upgrade one cluster to the environment a centralized fashion, at a time or multiple without impact to »»Zero-downtime clusters simultaneously running applications upgrade via BOSH »»Customers are guaranteed to always have the latest Kubernetes version available to them »»PKS maintainst compatibility to Google Container Engine (GKE)

Multi-cluster »»Built in multi-cluster »»Provides unified »»A typical deployment »»Supports multi-cluster Management support. Create any management of creates a single management. Create number of clusters multiple clusters Kubernetes cluster that one or more clusters is designed to scale up using PKS CLI, then »»Admins can manage »»Build clusters on to 2000 nodes list the clusters, view multiple clusters across public cloud providers cluster details and different regions, data like GKE, EKS, AKS or »»All users of that delete clusters using centers and clouds on data centers deployment are the CLI expected to share »»Able to discover that single cluster existing clusters and achieve isolation pre-created via a combination of Kubernetes namespaces, and OpenShift multi-tenancy »»Multiple clusters achieved through multiple OpenShift deployments

7 PLATFORM9 MANAGED RED HAT PRODUCT RANCHER PIVOTAL PKS KUBERNETES OPEN SHIFT

Multi-tenancy, »»Support for multi-re- »»Provides centralized »»Delivers multi-tenancy »»Supports multi-ten- Role-based Access gion management. authentication through projects, ancy in a limited Control, and Single Built in multi-tenancy (GitHub, AD/ called Kubernetes form support LDAP, SAML, etc.) namespaces Sign-on Support »»On VMware vSphere across RKE or cloud »»Kubernetes RBAC is »»Kubernetes RBAC with NSX-T, PKS Kubernetes services fully supported is utilized to define uses VMware NSX-T »Allows admins to granular access to isolate different »»Full support for » define Kubernetes policies for users clusters using NSX Single-Sign On (SSO). RBAC policies and security policies Integrate with a »»There is no cross network and pod SAML-based provider cluster multi-tenancy »»Kubernetes clusters security policies cen- that your organization can be deployed into trally and apply them uses such as Okta, different vSphere across any cluster ADFS, Ping Identity, clusters and config- etc. ured to use different datastores »»Multi-tenancy is not available when deployed without NSX-T or on public clouds

Load Balancing »»Out of the box »»Leverages native »»Out of the box support »»When deployed on support for cluster Kubernetes features to for cluster and service vSphere with NSX-T, and service level deliver load balancing level load balancing PKS can leverage load load balancing with with default HAProxy balancing feature MetalLB load balancer load balancer from NSX-T to provide cluster and service »»Can integrate with level load balancing customer specific load balancers(AVI and »»When deployed on others) public clouds like GCP or AWS, PKS leverages the cloud native load balancing capability

Private Registry »»Does not provide out »»Does not provide The internal integrated »»IncludesProject Support and Image of the box support for out of the box Docker registry can Harbor which is Management private registries support for private be deployed in the an open source registry deployment. OpenShift environment enterprise container »»Registries and Users have to deploy to locally manage images. registry. Harbor secrets required to their own public or OpenShift does not han- simplifies image authenticate with the private registries dle DockerHub well with management with registries need to their private registries. distribution, be managed by the »»Does support storing Because of complex replication and security customer separately of secrets required security requirements and mechanisms to pull images from configs specific to open- private registries »»The Harbor registry shift, a user is prevented uses a logical construct from pulling a docker called Project image. A user would have to build image streams, »»This is used to group secrets management, users and repositories and built to image in a to enable fine-grained lot of cases, unlike pure access control kubernetes which is much less complex.

8 PLATFORM9 MANAGED RED HAT PRODUCT RANCHER PIVOTAL PKS KUBERNETES OPEN SHIFT

Hybrid Cloud »»Includes the most »»Automates cluster »»Provides a managed »»Supports on-premises Integrations and APIs native integration with creation on bare metal deployment on AWS and public cloud all major private data servers, VMware, or based deployments »»A joint collaboration center/private cloud any IaaS cloud has been announced »»On-premises offerings and major »»Clusters on IaaS with Microsoft to requires vSphere or public cloud providers clouds are created support OpenShift vSphere with NSX-T »Integrates natively with cloud specific deployment on Azure » »»Does not support with VMware version of Kubernetes pure bare metal vSphere, Linux/KVM, (EKS for Amazon AWS deployment, or OpenStack cloud, AKS for Azure Linux/KVM virtualized cloud, etc.) »»Clusters on public environment clouds are created »»Rancher is subject with the public cloud’s to any limitations IaaS layer to provide the cloud providers a native Kubernetes expose around cluster experience versioning, update, HA, etc.

Enterprise Grade User »»Provides and enterprise »»Includes an intuitive UI »»Provides a native »»Lags behind signifi- Experience class UI and user that makes it easy for UI that enables cantly in this category experience users to deploy services management of your »»There is no UI for on Kubernetes and get Kubernetes resources »»The clarity UI provides PKS. All Kubernetes complete visibility and the catalog a single pane of glass operations such as across bare metal, »»Common configuration »»This is a web console cluster creation and virtualized and options directly from the only and does allow for management happen containerized UI for defining scheduling customization purely via CLI workloads rules, health checks, »»PKS does enable ingress controllers, »»Unify all your data the Kubernetes secrets, storage and centers, private dashboard other key configuration clouds, and public choices are offered clouds under single UI

Application Lifecycle »»Built in support for »»Built in support for »»An extensive »»Does not ship with a Management - Application catalog Application catalog application catalog built-in Kubernetes Application Catalog that’s populated with that’s populated with and PaaS layer helps application catalog public Helm chart public Helm chart with building and »»It is able to deploy applications applications deploying apps Helm charts »»Administrators can »»‘Rancher certified’ »»The service catalog is provide users access applications provided based on Open Service to applications that in the catalog that are Broker API are private to the tested and certified »»It ships with two service organization by Rancher brokers, one to enables »»Support for applications from their managed apps built in app template library, the other is an ansible broker »»The templated applications support - Rails (Ruby), Django (Python), Node.js, CakePHP (PHP), and Dancer (Perl) »»The Ansible broker supports integration w/ Ansible Playbook Bundles (APB) »»The service catalog offers Prometheus, EFK, Jenkins etc.

9 PLATFORM9 MANAGED RED HAT PRODUCT RANCHER PIVOTAL PKS KUBERNETES OPEN SHIFT

Production Grade »»Platform9 contractu- »»Provides a traditional »»Provides a traditional »»Provides a traditional Service Level ally promises 99.9% enterprise class enterprise class enterprise class Agreement cluster uptime and support model support model support model high availability »»Troubleshooting »»Troubleshooting »»Troubleshooting »»Provides self healing, is handled via is handled via is handled via problem resolution support tickets support tickets support tickets through the service »»Customers drive the »»Customers drive the »»Customers drive the manual upgrades and any manual upgrades and any manual upgrades and issues require support issues require support any issues require support team engagement team engagement team engagement

Ease of Setup, »»Platform9’s SaaS »»Simple setup, run »»Installing and config- »»The installation of PKS Installation, managed gets a single docker uring OpenShift is a is manual and requires Continuous Use, Kubernetes up and command on a linux manual process which Pivotal Ops Managed Management, and running in minutes machine and you are is ansible-based Several to be configured for up and running ansible playbooks are on-prem and public Maintenance »»Create a simple required during the cloud use Kubernetes cluster »»Provides an intuitive UI installation using on-prem servers, to help with rest of the »»Any problems require VMs or public cloud setup and Kubernetes logging a support resources in minutes cluster creation ticket »»Manage clusters with one-click UI-based upgrades and troubleshooting

Networking Support »»Provides full CNI »»Rancher provides CNI »»OpenShift provides »»Supports integration and Integrations support support CNI support and can with VMware NSX-T for integrate with any CNI advance networking »»Integrates OOB with »»Out of the box based SDN on VMware. Flannel, Calico, Weave support provided for and OpenContrail canal, calico, flannel »»By default OpenShift »»It also support SDN is deployed, which flannel, calico, »»Other CNI compatible configures an overlay nuage, OVN and integrations possible network using Open kube-router on customer request vSwitch (OVS) and supports 3 modes:

1. Flat network model with ovs-subnet plugin where every pod can communicate with every other pod

2. Project level isolation for pods

3. Services using ovs-multitenant plugin, and which enables administrators to configure their own isolation policies using Network Policy objects with ovs-network policy plug-in

»»Out of the box third party CNI plugins supported: Flannel, Nuage and Kuryer

10 PLATFORM9 MANAGED RED HAT PRODUCT RANCHER PIVOTAL PKS KUBERNETES OPEN SHIFT

Storage Support and »»Supports integration »»Rancher supports »»Supports integration »»Requires k8s vSphere Integrations with any flexvolume flexvolume driver with network based cloud provider so that drivers level integration with persistent storage vSphere persistent storage providers using the Kubernetes storage and data »»Integrates with any persistent volume services are exposed at cinder supported storage framework container volume level backend. - Supports integration with all »»Supports a wide »»Cluster admins can popular storage variety of persistent create storage classes backends such as storage endpoints that can map to a NetApp, Pure Storage, etc. such as NFS, specific underlying GlusterFS, OpenStack vSphere datastore or Cinder, FlexVolume, a VSAN datastore with VMware vSphere etc desired configuration, or by referencing a vSphere storage policy name »»For deployment on public clouds, PKS integrates with cloud specific persistent storage offerings

Self Service »»Complete self-service Provides a complete Provides a self-service »»Does not include a UI Provisioning provisioning enabled self-service provisioning UI that is separate from for Kubernetes, which via Platform9’s clarity UI UI for end users and the default Kubernetes means there is no self admins to create dashboard UI to enable service capabilities »»Users log into the UI workloads on top of self-service for develop- for Administrators as part of a specific Kubernetes ers and administrators to manage their k8s Tenant (eg dev/test/ clusters production) and are able to access clusters »»End-users cannot log in provided they have and deploy or manage been granted access their workloads »»Quick deployment of »»The Kubernetes pods, deployments and dashboard UI can be services via a wizard used out-of-the box for cluster creation

11 PLATFORM9 MANAGED RED HAT PRODUCT RANCHER PIVOTAL PKS KUBERNETES OPEN SHIFT

Built-in CI/CD Support »»Provides Spinnaker and »»Includes integrated CI/ »»Pipelines and Build Does not ship with a CI/ Jenkins via the Helm CD, making it easy for Strategies simplifies CD tool by default application catalog teams using Kubernetes, the creation and to quickly integrate it automation of with their development, dev/test and testing and release production pipelines management process »»Ships out of the box »»Users can easily with a Jenkins build point Rancher at any strategy and client git repo and it will plugin to create a automatically run builds Jenkins pipeline. on Kubernetes, deploy However, the setup to test environments, and create and configure move product production pipelines is manual and time consuming. »»The pipeline build configuration creates a Jenkins master pod (if one doesn’t exist) and then automatically creates slave pods to scale jobs & assign different pods for jobs with different runtimes

12 Platform9 Managed Kubernetes Platform9 Managed Kubernetes is the industry’s only SaaS-based continuously managed Kubernetes service that provides:

»»Guaranteed 99.9% uptime SLA »»Remote Monitoring and Healing »»Remote Upgrading and Instant Security Patching »»Central management across on-prem data centers, public clouds, and at the Edge. »»Upstream open source Kubernetes

To learn more, please visit platform9.com/managed-kubernetes

About Platform9 Platform9 (platform9.com) delivers a SaaS-managed hybrid cloud solution that turns existing infrastructure into a cloud, instantly. We help enterprises drive digital transformation by enabling them to manage VMs, Containers and Serverless Functions on ANY infrastructure — on-premises, in public clouds, or at the edge – with a self-service, simple and unified experience. Customers such as Cadence, Autodesk, Veritas, Nanometrics, EBSCO, Bitly, LogMeIn, and Aruba see upwards of 300 percent improvement in IT efficiency, 33 percent faster time to market, and 50-80 percent improvement in data center utilization and cost reduction. The company is headquartered in Sunnyvale, CA, and is backed by Redpoint Ventures, Menlo Ventures, Canvas Ventures, and HPE.