BOSH: Running Platforms So You Can Run Platforms on Your Platforms

BOSH: running platforms so you can run platforms on your platforms

Paul Czarkowski / @pczarkowski Principal Technologist Pivotal software

Sucessful Patterns for Deploying [and Operating] Platforms.

Paul Czarkowski / @pczarkowski Principal Technologist Pivotal software

https://en.wikipedia.org/wiki/Computing_platform A modern software platform provides API driven compute resources. “Every IT team is a Platform Team.” http://slides.eightypercent.net/platform-platform/index.html#1 https://twitter.com/kelseyhightower/status/935252923721793536 Generic Platform

Users

API

Artifacts Database Storage Compute Network Access Enterprise IT

Users

API

Storage Systems Network DBA Security QA Admin Admin Engineer

“You don’t have a platform problem, you have a culture problem.” Evolve your IT teams!

Take business requirements and turn Application App1 App2 App3 Team them into features

Messaging Creds/Certs Middleware Platform App ML ELK DBaaS Build common services Team Platform Container Services Container Hosts | Kubernetes for App Teams ? ? ?

Infrastructure Team

Abstract infrastructure complexity with easy IaaS consumption Infra Services Virtual Infrastructure

Physical Infrastructure People build Platforms People are the most People build Apps important component Apps run on Platforms of any platform. What Abstraction is the Right One for You ?

More Control Less Complexity

Build App Build App App → to the Artifact Container(s) Platform

Traditional Build Config CF API Application Ticket Management

Based Dependencies PaaS PaaS Human Application Platform Infrastructure K8s API Toil As Code CaaS ContainerCaaS Orchestrator IaaS API

Container Runtime PXE boot ? ContainerIaaS Hosts

HardwareIaaS

18 FaaS

PaaS

Higher flexibility and Lower development less enforcement of complexity and higher CaaS standards operational efficiency

IaaS

Hardware

Strategic goal: Push as many workloads as technically feasible to the top of the platform hierarchy Consumers of the abstraction

App Team Platform Team FaaS Infrastructure Team App Team PaaS Platform Team

Platform Team Infrastructure Team CaaS App Team

Infrastructure Team IaaS Platform Team

Infrastructure Team Hardware “In general, taking something that’s already working somewhere and expanding its usage (capabilities) is far more likely to succeed than building these capabilities from scratch” WE build the container YOU build the container

Continuous Application Code & Frameworks delivery Buildpacks | Spring Boot | Spring Cloud | Steeltoe

Github

Public Cloud Pivotal Services Network Pivotal Application Pivotal Container Pivotal Services Customer Marketplace Managed Concourse Service (PAS) Service (PKS) Services

>cf push >kubectl run Pivotal and Open Service Broker API v1 Partner Products CVEs Java | .NET | NodeJS Elastic | Packaged Software | Spark Product Updates

v2 “3Rs”

v3 Embedded OS Repair Repave Rotate ... NSX-T — CVEs — Credhub (Windows & Linux)

Concourse

CPI (15 methods)

Azure & vSphere Openstack AWS Google Cloud Azure Stack

The Google Problem

x 1,000,000

K8s Contr Master Etcd oller Kube Mana Scheduler ger API Server

K8s K8s K8s Worker Worker Worker Kubelet Kubelet Kubelet Pod Pod Pod Pod Pod Pod Docker C Pod Docker Pod Docker Pod Kube-pro Kube-pro Kube-pro xy xy xy CNI CNI CNI

K8s Master Etcd Controller Kube Scheduler Google Compute Engine Manager (hidden from user) API Server

Google Compute Engine

K8s Worker K8s Worker K8s Worker

Kubelet Kubelet Kubelet Pod Pod Pod Pod Pod Pod Docker C Pod Docker Pod Docker Pod

Kube-proxy Kube-proxy Kube-proxy

CNI CNI CNI

What Abstraction is the Right One for You ?

More Control Less Complexity

Build App Build App App → to the Artifact Container(s) Platform

Traditional Build Config CF API Application Ticket Management

Based Dependencies PaaS PaaS Human Application Platform Infrastructure K8s API Toil As Code CaaS ContainerCaaS Orchestrator IaaS API

Container Runtime PXE boot ? ContainerIaaS Hosts

HardwareIaaS

BOSH - Component Architecture

IaaS Cloud Provider CLI IaaS Interface API

Registry

Health BOSH provides the means to go Monitor Director from deployment configuration to Agent VM creation and management. It NATS Agent includes interfaces for Azure, vSphere, AWS, GCP, and Agent Postgres Blob Agent OpenStack. Additional CPI can be DB Store written for alternative IaaS providers. BOSH - Service Deployment

Deployment

Manifest

Release

w Stemcell BOSH - Stemcell

▪ Secured, Hardened, and Versioned Operating System image wrapped with IaaS specific packaging ▪ ▪ Contains a bare minimum OS skeleton with a few common utilities pre-installed, a BOSH Agent, and a few configuration files to securely configure the OS by default. Stemcell ▪ ▪ Images come in two flavors Ubuntu 14.04 and CentOS 7 for all IaaS’ supported ▪ ▪ Maintained by BOSH team and available at http://bosh.io/stemcells BOSH - Release

Elements: Organization: Dependency Tree:

Jobs - Pieces of the service or application you are releasing, Job(s) including how to compile & run them Monit

Packages - Provide source code and dependencies to jobs Spec

Src - Non-binary files which is provided to packages Blobs Blobs - Provide binary files (other than those checked into a source code repository) to packages Package(s)

Monit - Script utilized to start/stop/restart the job Packaging Spec Packaging - Script utilized to compile the source needed by a job Src Spec - Key/Value file which stores all configuration properties which can be set externally

Green - Script (erb or bash) Orange - Properties (yml) BOSH - Manifest

Provides the ability to customize BOSH releases (your service)

YAML - Primer found at end of presentation

Required Blocks:

▪ Deployment Identification: A name for the deployment and the UUID of the Director managing the deployment

▪ Releases Block: Name and version of each release in a deployment

▪ Stemcells Block: Name and version of each stemcell in a deployment

▪ Update Block: Defines how BOSH updates instances during deployment

▪ Instance Groups Block: Configuration and resource information for instance groups (Jobs)

▪ Properties Block: Describes global properties and generalized configuration information

▪ BOSH - Service Deployment

Cloud Provider CLI IaaS Interface API Stemcell Registry

Manifest Config Health Monitor Release Director Agent NATS Agent

Postgres Blob Agent DB Store Agent

* * BOSH - Process High Availability

IaaS

Alert Sent!

Proc-1 Health Restart! Monitor Agent Director Proc-1 NATS Agent BOSH - VM High Availability

Manifest - Desired State Cloud IaaS Provider IaaS Interface API Proc-1

Alert Sent! Agent

Proc-1 Health Proc-1 Monitor Agent Agent Director Proc-1 = NATS Agent

Proc-1

Agent BOSH - Canary Upgrades

Any update error causes the deployment to stop. Since only canaries are affected before an update stops, problem jobs Manifest - and packages are prevented from taking over all instances.

Code Update 4.04! Agent

Agent

Agent BOSH - Day 2 Ops

Consistent, Reliable, Scalable, Secure

Deploy Provision and ▪ Checks against “desired state to return Configure consistency ▪ No ad hoc automation burden ▪ Manage services, not servers Current State ▪ 4 layers of Self Healing Desired State

Monitor Re-mediate and Detect

PKS “How it Works” #pks# pks create-cluster create-cluster K8s-1K8s-2K8s-3 --plan -n small3 Cluster3 The value of BOSH

API Node V API M Kubernetes PKS Control Plane Node V Cluster Services M PKS includes: Node V Cluster1 M • PKS Control Plane, CFCR

• NSX-T, Harbor, GCP Broker Kubernetes Node V CNI API M (As a Bosh Release) NSX-T • BOSH Release for Kubernetes Kubernetes Node V Cluster Services M • Configures Day 1 of - CFCR BOSH CPI vSphereIaaS Node V Cluster2 M - vSphere (Deploys/Manages VMs) - NSX Integration API Node V - Harbor M Kubernetes • Manages Day 2 of Kubo CFCR Cluster Services Node V Cluster Services M Kubernetes Clusters V Node V - Scaling Cluster3 Node M Harbor GCP M - Patching Private Container Service Node V - Upgrades Registry Broker M - Failures Node V M

AWS Lambda, Azure Functions, OpenWhisk, FaaS kubeless, PFS PCF, Azure App PaaS Service, Heroku, Deis PKS, GKE, OpenShift, AWS CaaS Fargate, Kubernetes AWS, Azure, IaaS GCP, Openstack, VMWare HPE, Dell, Hardware IBM, Lenovo

http://www.bsielearning.com.au/keep-simple-stupid/ Full Opensource DIY

● ??? pxe ● Openstack Ansible ● Kubespray ● Ansible Hardening

https://github.com/openstack/openstack-ansible https://github.com/openstack/ansible-hardening https://github.com/kubernetes-incubator/kubespray !!! How We Think about the Business Case

40-60%* 25-50%*

More Projects With Fewer Support $ Same Staff Incidents REPLATFORM > MODERNIZE > OPTIMIZE

40%*

Faster Patching SPEED & AGILITY STABILITY % Delivery @ Zero

Downtime

$ ESTABLISH, MEASURE AND UPDATE SECURITY Millions -90%* Annual Savings on KEY OBJECTIVES AND RESULTS (OKRs) Time to Scale $ HW, SW and Support

SCALABILITY SAVINGS

PLATFORM VALUE STREAM AND METRICS THE END Questions ?