Breakout Operators Robert Bohne SR. SPECIALIST SOLUTION ARCHITECT | OPENSHIFT Twitter: @RobertBohne 1 What is an Operator? Operator is a automated software manager that deals with the installation and life cycle of an applications on top of Kubernetes/OpenShift. 2 Controller Custom Resource Definition (CRD) Piece of software that deals OpenShift API extension to interact and with the installation and life cycle of communicate with the Controller. an applications on top of OpenShift. 3 Custom Resource Definition (CRD) Custom Resource (CR) apiVersion: apiextensions.k8s.io/v1beta1 apiVersion: openshift.pub/v1 kind: CustomResourceDefinition kind: Car metadata: metadata: name: cars.openshift.pub name: bmw spec: spec: group: openshift.pub date_of_manufacturing: "2014-07-01T00:00:00Z" names: engine: N57D30 kind: Car listKind: CarList plural: cars singular: car scope: Namespaced subresources: status: {} version: v1 4 Custom Resource Definition (CRD) [..snipped..] additionalPrinterColumns: - JSONPath: .status.conditions[?(@.type=="Succeeded")].status name: Succeeded type: string - JSONPath: .status.conditions[?(@.type=="Succeeded")].reason name: Reason type: string - JSONPath: .spec.date_of_manufacturing name: Produced type: date - JSONPath: .spec.engine name: Engine type: string priority: 1 5 apiVersion: apps/v1 kind: Deployment metadata: name: operator spec: replicas: 1 selector: matchLabels: name: operator template: Controller metadata: labels: Piece of software running name: operator on top of OpenShift spec: containers: - name: operator Image: "quay.io/org/operator:v0.0.1" 6 { } Custom Resource 7 { } Custom Resource 8 k8s API Operator Pattern My Application Custom Resource MyApplication Kubernetes Resources State for My App Controller Watch Events Reconciliation 9 How to create an Operator? 10 Operator Framework in Action OPERATOR SDK “create new operator” DEVELOPER + = KUBERNETES OPERATOR implemented as a container image plus: scaffolding custom logic Deployment Role ClusterRole RoleBinding ClusterRoleBinding ServiceAccount 11 CustomResourceDefinition Operator Adoption Patterns Ansible Playbooks Helm Chart APBs Helm SDK Ansible SDK Go SDK Build operators from Build operators from Build advanced operators Helm chart, without any Ansible playbooks and for full lifecycle coding APBs management 12 Types of Operators 13 Use Cases ideal for operator development ● stateful applications (such as databases) ● clustered or high-availability applications (clustered databases, key value stores such as etcd, in-memory cache clusters such as redis) ● multiservice applications (an application which needs dependent services to come online first) ● microservices (numerous small components that together make up a cohesive product or app) Use cases less ideal for operator development ● stateless apps (most web apps or front-ends) ● infrastructure/host agents (monitoring agents or log forwarders) 14 Demo https://examples.openshift.pub/operator#ansible-operator-example 15 k8s API Ansible Operator My Application Custom Resource Kubernetes Resources for My App MyApplication State Ansible Operator Operator-S Ansible DK Role Ansible k8s modules will be File Mapping used to create resources in kubernetes 16 What you need to create an Ansible Operator ● A CustomResourceDefinition (CRD) ● An Ansible Playbook or Role ● A mapping from CRD to Ansible playbook / roles ● operator-sdk 17 Create the Operator with the SDK $ operator-sdk new memcached-operator \ --api-version=cache.example.com/v1alpha1 \ --kind=Memcached --type=ansible Creates: ● Ansible Role ● Mapping File (watches.yaml) ● Custom Resource Definition ● Deploy manifest for the new Operator 18 Custom Resource (CR) Ansible Operator apiVersion: <Group/Version> kind: <kind> metadata: Spec values will be translated to Ansible extra vars. name: <name> spec: <key>: <value> …. status: <key>: <value> Status will be a generic status defined …. by the operator. This will use ansible runner output to generate meaningful output for the user. 19 Ansible Role memcached/ ├── defaults │ └── main.yml ├── files ├── handlers │ └── main.yml ├── meta Create a Role that deploys and │ └── main.yml ├── README.md manages your application ├── tasks │ └── main.yml ├── templates ├── tests │ ├── inventory │ └── test.yml └── vars └── main.yml 20 Mapping between CRDs and Ansible Maps a Group Version Kind (GVK) to a role or playbook. # watches.yaml --- - version: v1alpha1 - group: cache.example.com kind: Memcached playbook: /path/to/playbook 21 Build the Operator with the SDK $ operator-sdk build memcached-operator:v0.0.1 Creates: ● A Dockerfile that creates the Operator ● Builds the container on top of ansible-runner image 22 What is an Operator? ● Container Image ( contains the software manager ) ● Deployment ● Custom Resource Definition (CRD) Metadata ● Role Bindings ● ... 23 What is an Operator? OPERATOR SDK “create new package DEVELOPER operator” KUBERNETES + = OPERATOR + scaffolding custom logic metadata 24 How to ship an Operator? Catalog Source quay.io / OPERATOR Image Registry OPERATOR LIFECYCLE SDK MANAGER “create new package DEVELOPER operator” “list packages” KUBERNETES + = OPERATOR + scaffolding custom logic metadata ADMINISTRATOR Package Discovery: $ oc get packagemanifests 25 How to ship an Operator? Catalog Source quay.io / OPERATOR Image Registry OPERATOR LIFECYCLE SDK MANAGER “create new package DEVELOPER operator” “list packages” KUBERNETES + = OPERATOR + scaffolding custom logic metadata ADMINISTRATOR Package Discovery: $ oc get packagemanifests 26 27 Deployment Role OPERATOR YourOperator v1.1.2 ClusterRole Bundle LIFECYCLE MANAGER RoleBinding Operator Deployment ClusterRoleBinding Custom Resource Definitions RBAC ServiceAccount API Dependencies Update Path CustomResourceDefinition Metadata 28 OperatorHub Deployment Role OPERATOR LIFECYCLE MANAGER ClusterRole RoleBinding ClusterRoleBinding ServiceAccount CustomResourceDefinition 29 OperatorHub data sources Operator Metadata from quay.io ● Backend for all default sources, cluster needs to be online ● Supplies Red Hat Operators, ISV Operators and Community Operator Operator Package Metadata ● Custom sources supported in customer-owned quay.io Package name namespaces e.g. prometheus Channel name Operator Operator e.g. stable Bundle 1 Bundle 2 ... Operator Metadata in container images ● Already used internally used by OLM Channel name Operator e.g. tech-preview Bundle 3 ● Operator package data is served from a SQlite database, bundled up in a container image ● Custom sources supported in customer-owned image registries ● Cluster can be disconnected / air-gapped 30 ● Certified Operators submitted through Red Hat Connect - listed in OpenShift OperatorHub ● Community Operators submitted through GitHub - listed in both OpenShift and OKD OperatorHub ● Upstream Community Operators also submitted through GitHub - listed in the OperatorHub.io community website 31 CONFIDENTIAL INTERNAL USE How Operator Catalogs are downloaded Operator Metadata from quay.io apiVersion: operators.coreos.com/v1 ● Backend for all default sources, cluster needs to be online kind: OperatorSource metadata: ● Supplies Red Hat Operators, ISV Operators and Community name: johndoe-operators namespace: marketplace Operator spec: type: appregistry ● Custom sources supported in customer-owned quay.io endpoint: https://quay.io/cnr registryNamespace: johndoe namespaces, data uploaded via operator-courier Operator Metadata in container images ● Already used internally used by OLM apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource ● Operator package data is served from a SQlite database, metadata: name: johndoe-operators bundled up in a container image (created via operator-registry) namespace: olm spec: ● Custom sources supported in customer-owner image registries sourceType: grpc ● Cluster can be disconnected / air-gapped image: johndoe-catalog:latest 32 Operator Framework in Action OPERATOR LIFECYCLE MANAGER Subscription for Catalog containing Release Channel for Instance of YourOperator YourOperator YourOperator YourOperator To install an Operator an administrator... apiVersion: operators.coreos.com/v1alpha1 kind: Subscription 1. Picks an Operator from the Catalog metadata: 2. (Selects a distribution channel from the Operators package) name: johns-subsription namespace: default 3. (Selects a version of the Operators from the channel) spec: channel: stable 4. Creates a Subscription pointing to the Catalog, Operator, Version and Channel name: my-operator a. If no channel is specified, the default channel is used source: johndoe-operators sourceNamespace: olm b. If no version is specified, the latest is used 33 Operator Updates Operator Catalog Version YourOperator v1.2.2 YourOperator v1.2.0 OPERATOR LIFECYCLE MANAGER YourOperator v1.1.3 Subscription for YourOperator YourOperator v1.1.2 Time 34 Proxy Support OpenShift 4.2 Cluster OPERATOR LIFECYCLE MANAGER Cluster Proxy spec: Config containers: Operator - name: my-container image: ... env: - name: HTTP_PROXY Pod value: "..." - name: HTTPS_PROXY value: "..." 35 4.2 Automated Dependency Resolution Operator Framework Dependency Graphs resolves to installed by requires jaeger.jaegertracing.io/v1 Jaeger Operator OPERATOR LIFECYCLE MANAGER requires YourOperator v1.1.2 resolves to installed by cockroachdb.charts.helm.k8s.io/v1alpha1 CockroachDB Operator 36 Namespace / Project SA Role Allow regular users to install Operators ● cluster-admin Service In 4.1: only users carrying roles are allowed Account to install Operators cannot be greater than be greater cannot