Openshift 4 Operating

1 V0000000 OpenShift 4 Operating

Alfred Bach Partner Enablement Manager Cloud

2 V0000000 About me Alfred Bach Tech. Partner Enablement Manager Cloud 4 years with Rat Hat

abach@redhat .com

3 V0000000 OpenShift Installation

4 V0000000 OpenShift 4 Architectural Principles

DAY 1 & 2 IMMUTABLE OPERATOR OPERATIONS INFRASTRUCTURE FRAMEWORK

Installer + bootstrapping Red Hat Enterprise Linux CoreOS SDK & testing tools Autoscale out of the box Discourage SSH/node mutation OperatorHub for discovery MachineSet node pools Ignition for Machine conﬁg OLM delivers upper stack services

5 V0000000 OpenShift 4 Architectural Principles

DAY 1 & 2 IMMUTABLE OPERATOR OPERATIONS INFRASTRUCTURE FRAMEWORK

6 V0000000 OPENSHIFT CONTAINER PLATFORM | Installation

Installation Paradigms OPENSHIFT CONTAINER PLATFORM HOSTED OPENSHIFT

Red Hat OpenShift on IBM Cloud * Full Stack Automated (IPI) Pre-existing Infrastructure (UPI) Deploy directly from the IBM Cloud Simpliﬁed opinionated “Best Customer managed resources & console. An IBM service, master nodes Practices” for cluster provisioning infrastructure provisioning are managed by IBM Cloud engineers.

Fully automated installation and Plug into existing DNS and security Azure Red Hat OpenShift **

updates including host container boundaries Deploy directly from the Azure console. OS. A MSFT service, jointly managed by Red Hat and Microsoft Azure engineers.

OpenShift Dedicated **

Get a powerful cluster, fully managed by Red Hat engineers and support; a Red Hat service.

7 * Based on OCP v4.3 GA slated for March; public beta available now ** Entitlements of OCP obtained through a Cloud Pak purchase are not transferable to these environments V0000000

OPENSHIFT PLATFORM What's new in OpenShift 4.3

4.3 Supported Providers

Full Stack Automation (IPI) Pre-existing Infrastructure (UPI)

* Support planned for an upcoming 4.3 z-stream release

Generally Available PMs: Katherine Dubé (AWS, Azure, GCP), Maria Bracho (BM UPI, VMware, Upgrades), Peter Lauterbach (RHV), Ramon Acedo Rodriguez (OSP, BM IPI), Mike Barrett (IBM Z & Power) Provider Roadmap & Minimum Supported Version Full Stack Automation Pre-existing Infrastructure Provider (Installer provisioned infra) (User provisioned infra)

4.1 4.1

4.2 4.3+ (z-stream)

Bare Metal 4.4 (TBD) 4.1

4.2 4.2

4.2 4.4

4.4 4.4

4.4 4.1

- 4.2+ (z-stream)

IBM Power Systems - 4.3+ (z-stream)

9 4.5 -

PMs: Katherine Dubé (AWS, Azure, GCP), Maria Bracho (BM UPI, VMware, Alibaba), Peter Lauterbach (RHV), Ramon Acedo Rodriguez (OSP, BM IPI), Mike Barrett (IBM Z & Power) OpenShift Architecture

Red Hat® OpenShift® Kibana | Elasticsearch Kibana | Elasticsearch services

Infrastructure Registry Registry services

Router Router Developers Kubernetes services

Prometheus | Grafana Prometheus | Grafana Alertmanager Alertmanager etcd

Monitoring | Logging | Tuned Monitoring | Logging | Tuned

Admins SDN | DNS | Kubelet SDN | DNS | Kubelet

MASTER WORKER WORKER

COMPUTE NETWORK STORAGE

10 V0000000

10 OCP Cloud Layout

Virtual Environments

11 V0000000 12 V0000000 OpenShift 4 Installer

13 V0000000 OPENSHIFT CONTAINER PLATFORM | Installation

Full-stack Automated Installation (aka IPI)

User managed

Operator managed Control Plane Worker Nodes

OCP Cluster Resources

OCP Cluster openshift-install deployed RH CoreOS RH CoreOS RHELRH CoreOS CoreOS RHELRH CoreOS CoreOS

Cloud Resources Cloud Resources

14 V0000000 OPENSHIFT PLATFORM Full Stack Automated Deployments

Simplified Cluster Creation Designed to easily provision a “best practices” OpenShift $ ./openshift-install --dir ./demo create cluster cluster ? SSH Public Key /Users/demo/.ssh/id_rsa.pub ● New CLI-based installer with interactive guided workflow ? Platform aws ? Region us-west-2 that allows for customization at each step ? Base Domain example.com ? Cluster Name demo ● Installer takes care of provisioning the underlying ? Pull Secret [? for help] Infrastructure significantly reducing deployment complexity ************************************************************* INFO Creating cluster... ● Leverages RHEL CoreOS for all node types enabling full INFO Waiting up to 30m0s for the Kubernetes API... INFO API v1.11.0+c69f926354 up stack automation of installation and updates of both INFO Waiting up to 30m0s for the bootstrap-complete event... platform and host OS content INFO Destroying the bootstrap resources... INFO Waiting up to 10m0s for the openshift-console route to be created... INFO Install complete! Faster Install INFO Run 'export KUBECONFIG=/auth/kubeconfig' to manage the cluster with 'oc', the OpenShift CLI. The installer typically finishes within 30 minutes INFO The cluster is ready when 'oc login -u kubeadmin -p ' ● Only minimal user input needed with all non-essential succeeds (wait a few minutes). INFO Access the OpenShift web-console here: install config options now handled by component operator https://console-openshift-console.apps.demo.example.com CRD’s INFO Login to the console with user: kubeadmin, password: ● See the OpenShift documentation for more details 15 V0000000 OpenShift Installation

How everything deployed comes under management

Masters (Special) ● Terraform provisions initial masters* ● Machine API adopts existing masters post-provision ● Each master is a standalone Machine object ● Termination protection (avoid self-destruction) Workers ● Each Machine Pool corresponds to MachineSet ● Optionally autoscale (min,max) and health check (replace if not ready > X minutes) Multi-AZ ● MachineSets scoped to single AZ ● Installer stripes N machine sets across AZs by default ● Post-install best effort balance via cluster autoscaler

16 V0000000 INSTALL A OPENSHIFT CLUSTER CONFIDENTIAL Designator

Deployment Server RHEL 8 or Centos HAProxy BIND or DNSMasq

Control Plane Worker Nodes CNS (Optional)

CoreOS CoreOS or Container Native optional RHEL 7 Storage (CEPH) 3 Master Nodes CoreOS 17 Boot Strap Registry DIrect att. Disk V0000000 CoreOS Hubscribe an OpenShift 4 cluster CONFIDENTIAL Designator

OCS 2 OCS 2 CNS CORE CORE

RHEL OCS 2 OCS 2 CNS CORE CORE Deployment Server RHEL 8 or Centos OCS 2 HAProxy CORE BIND or DNSMasq

Control Plane Worker Nodes CNS (Optional)

CoreOS CoreOS or Container Native Optional RHEL 7 Storage (CEPH) 3 Master Nodes CoreOS 18 Boot Strap Registry DIrect att. Disk V0000000 CoreOS OPENSHIFT CONTAINER PLATFORM | Installation

Pre-existing Infrastructure Installation (aka UPI)

User managed

Operator managed Control Plane Worker Nodes

OCP Cluster Resources openshift-install deployed OCP Cluster

Note: Control plane nodes must run RHEL CoreOS! RH CoreOS RHEL RHELRH CoreOS CoreOS RHEL 7 CoreOS Customer deployed Cloud Resources Cloud Resources

19 V0000000 OPENSHIFT CONTAINER PLATFORM | Installation

Comparison of Paradigms Full Stack Automation Pre-existing Infrastructure

Build Network Installer User

Setup Load Balancers Installer User

Conﬁgure DNS Installer User

Hardware/VM Provisioning Installer User

OS Installation Installer User

Generate Ignition Conﬁgs Installer Installer

OS Support Installer: RHEL CoreOS User: RHEL CoreOS + RHEL 7

Node Provisioning / Autoscaling Yes Only for providers with OpenShift Machine API support

20 V0000000 Disconnected “Air-gapped” Installation & Upgrading

# mirror update image: $ oc adm -a release mirror \ --from=quay.io// \ --to=/ \ --to-release-image=/ # provide cluster with update image to update to: $ oc adm upgrade --to-mirror= Customer Cluster Local Copy of Red Hat sourced Update Image Update Image

Disconnected Local Container Quay.io OpenShift Cluster Cluster Registry Container Mirrored to updated Registry local registry locally Admin

Overview Installation Procedure ● 4.2 introduces support for installing and updating OpenShift ● Mirror OpenShift content to local container registry in the disconnected environment clusters in disconnected environments ● Generate install-conﬁg.yaml: $ ./openshift-install create install-conﬁg --dir

● Requires local Docker 2.2 spec compliant container registry to ○ Edit and add pull secret (PullSecret), CA certificate (AdditionalTrustBundle), host OpenShift content and image content sources (ImageContentSources) to install-config.yaml ● Designed to work with the user provisioned infrastructure ● Set the OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE environment variable deployment method during the creation of the ignition configs ○ Note: Will not work with Installer provisioned ● Generate the ignition configuration: $ ./openshift-install create ignition-configs --dir infrastructure deployments ● 21 Use the resulting ignition files to bootstrap the cluster deployment V0000000 OpenShift 4 Cluster Management Powered by Operators, OpenShift 4 automates many cluster management activities

22 V0000000 OpenShift Architecture Over-the-air updates

Release Payload Info

machine-config-operato r Machine Cluster Machine Machine machine-os-content Rolling ConfigMachine Version Config Config OperatorConfig ... Operator Operator Operator Daemons ...

Machine Machine Conﬁg Conﬁg Daemon Daemon

Download and Update host mount update using mounted content into host content

23 V0000000 OpenShift Architecture Cloud API

MachineDeployment MachineSet Machine

Machine Machine Set Machine Deployment Controller Controller Controller

Future Cloud

Node

Bootstrap Instance NodeLink Controller

24 V0000000 OpenShift Security

Features, mechanisms and processes for container and platform isolation

25 V0000000 OPENSHIFT SECURITY | Comprehensive features

CONTROL Container Content CI/CD Pipeline Application Security Container Registry Deployment Policies

Container Host Container Platform Multi-tenancy DEFEND Infrastructure Network Isolation Storage

Audit & Logging API Management

EXTEND Security Ecosystem

26 V0000000 OPENSHIFT SECURITY | Comprehensive features

Extended Depth of Protection

Feature Transfer (upstream)

Security Pod Context Security Constraint Preset (PSP) (SCC)

Feature Development (joint)

27 V0000000 OPENSHIFT SECURITY | Comprehensive features

Certificates and Certificate Management

● OpenShift provides its own internal CA ✓ MASTER ● Certificates are used to provide secure connections to ✓ ETCD ○ master (APIs) and nodes ✓ NODES ○ Ingress controller and registry ○ etcd INGRESS ✓CONTROLLER ● Certificate rotation is automated ✓ CONSOLE ● Optionally configure external endpoints to use custom certificates ✓ REGISTRY

28 V0000000 OPENSHIFT SECURITY | Comprehensive features

Service Certificates

service.beta.openshift.io / inject-cabundle="true"

service-ca.crt

ConﬁgMap service.alpha.openshift.io/ serving-cert-my CONTAINER

My Service POD

serving-cert-my

tls.crt tls.key

Secret ConﬁgMap Service Serving CAbundle Injector Cert Signer

29 V0000000 OPENSHIFT SECURITY | Comprehensive features

Identity and Access Management

(6) Token LDAP Google

Keystone OpenID

Master GitHub Request Header (3) Validate (API) Credentials GitLab Basic (1) Referral

(2) Credentials

userXX (4) Create & Map Identity User (5) Token Identity OAuth Server

V0000000 OPENSHIFT SECURITY | Comprehensive features

Fine-Grained RBAC ● Project scope & cluster scope available ● Matches request attributes (verb,object,etc) ● If no roles match, request is denied ( deny by default ) ● Operator- and user-level roles are deﬁned by default ● Custom roles are supported

31 V0000000 OpenShift Monitoring

An integrated cluster monitoring and alerting stack

32 V0000000 OPENSHIFT MONITORING | Solution Overview

OpenShift Cluster Monitoring

Metrics collection and Alerting/notiﬁcation via Metrics visualization via storage via Prometheus, an Prometheus’ Alertmanager, an Grafana, the leading metrics open-source monitoring open-source tool that handles visualization technology. system time series database. alerts send by Prometheus.

33 V0000000 OPENSHIFT MONITORING | Operator & Operand Relationships

Grafana node-exporter

kube-state-metrics openshift-state-metrics (4.2) cluster-monitoring-operator

prometheus-adapter telemeter-client

Prometheus Alertmanager prometheus-operator

34 V0000000 OPENSHIFT MONITORING | Prometheus, Grafana and Alertmanager Wiring

Grafana Prometheus Alertmanager

Control Plane (API) kube-state-metrics

node-exporter node-exporter

Node (kubelet) Node (kubelet)

Infra/Worker (“hardware”) Worker (“hardware”)

35 V0000000 OpenShift Logging

An integrated solution for exploring and corroborating application logs

36 V0000000 OPENSHIFT LOGGING | Solution Overview

Observability via log exploration and corroboration with EFK

Components ･ Elasticsearch: a search and analytics engine to store logs ･ Fluentd: gathers logs and sends to Elasticsearch. ･ Kibana: A web UI for Elasticsearch.

Access control ･ Cluster administrators can view all logs ･ Users can only view logs for their projects

Ability to forward logs elsewhere ･ External elasticsearch, Splunk, etc

37 V0000000 OPENSHIFT LOGGING | Operator & Operand Relationships

ElasticSearch ElasticSearch Operator Cluster

Cluster Logging Kibana Operator

Curator CronJob ...

Fluentd (per node)

38 Curator V0000000 OPENSHIFT LOGGING | Architecture

Log data flow in OpenShift

Fluentd

TLS Fluentd TLS Node Elasticsearch Kibana Fluentd Node Application Logs Node

39 V0000000 OPENSHIFT LOGGING | Architecture

Log data flow in OpenShift

stdout stderr

Fluentd TLS Elasticsearch

CRI-O

OS DISK journald

kubelet

Node (OS)

40 V0000000 A broad ecosystem of workloads

Operator-backed services allow for a SaaS experience on your own infrastructure

Big Data Relational DBs

Monitoring

Security NoSQL DBs

DevOps

AL/ML Messaging Storage

41 V0000000 Kubernetes-native day 2 management

Flexible app No reinvention architectures of core concepts

Uniform deploy Truly hybrid and debug

Operators codify operational knowledge and workﬂows to automate life-cycle management of containerized applications with Kubernetes

42 V0000000 Kubernetes Operator

Custom Kubernetes Controller

Watch Events

Reconciliation

+ 43 Custom Resource Deﬁnition V0000000 K8s API Developer / Kubernetes Operator OpenShift User

Custom Resource Custom Kubernetes Controller

kind: ProductionReadyDatabase apiVersion: database.example.com/v1alpha1 Watch Events metadata: name: my-important-database spec: Reconciliation connectionPoolSize: 300 readReplicas: 2 version: v4.0.1

+ 44 Custom Resource Deﬁnition V0000000 K8s API Developer / Kubernetes Operator Native Kubernetes OpenShift User Resources

Custom Resource Custom Kubernetes Controller

kind: ProductionReadyDatabase apiVersion: database.example.com/v1alpha1 Watch Events metadata: name: my-important-database spec: Reconciliation Deployments connectionPoolSize: 300 StatefulSets readReplicas: 2 Autoscalers version: v4.0.1 Secrets Conﬁg maps + PersistentVolume 45 Custom Resource Deﬁnition V0000000 ● Operator Lifecycle Manager - Helps you to install, and update, and generally manage the lifecycle of all of the Operators (and their associated services) running across your clusters

● Operator Hub - Provides access to Operators ready to use

● Operator Metering - Enable usage reporting for Operators and resources within Kubernetes

● Operator SDK - Allows developers to build, package and test an Operator based on your expertise without requiring all the knowledge of Kubernetes API complexities

46 V0000000 TYPES OF OPERATORS

Ansible Playbooks Helm Chart APBs

Helm SDK Ansible SDK Go SDK

Build operators from Build operators from Build advanced operators Helm chart, without any Ansible playbooks and for full lifecycle coding APBs management

47 V0000000 CATEGORIES OF OPERATORS

Services Operators Platform Operators Resource Operators Full solutions

“How do I scale services without Software to conﬁgure, audit and Software to manage precious A complete software solution scaling humans?” and “As a remediate changes to the platform resources, such as network adapters, can combine all of these types vendor, how do I easily deploy my you run GPUs, ... together, app the same way across X customers in Y environments?”

Examples: MongoDB Operator, Examples: Aggregated Logging, Examples: CNI Operators, Hardware Examples: an AI/ML product Spark Operator, Nginx Operator Security Scanning, Namespace Management Operators, with an Operator to manage Management Telco/Cellular Radios GPU resources

48 V0000000 CAPABILITY LEVELS FOR OPERATORS

49 V0000000 OperatorHub and certified Operators

● OperatorHub.io launched by Red Hat, AWS, Microsoft and Google ● OpenShift Operator Certiﬁcation ● OperatorHub integrated into OpenShift 4

COMMUNITY OPERATORS

OPENSHIFT CERTIFIED OPERATORS

50 V0000000 BROAD ECOSYSTEM OF WORKLOADS

Operators as a First-Class Citizen

Deployment

Role

YourOperator v1.1.2 ClusterRole Bundle RoleBinding

Operator Deployment ClusterRoleBinding Custom Resource Deﬁnitions RBAC ServiceAccount API Dependencies Update Path CustomResourceDefinition Metadata

51 Product Manager: Daniel Messer Generally Available V0000000 BROAD ECOSYSTEM OF WORKLOADS

Operator Lifecycle Management

Operator Catalog Version

YourOperator v1.2.2

YourOperator v1.2.0

YourOperator v1.1.3

Subscription for YourOperator YourOperator v1.1.2

Time

52 Product Manager: Daniel Messer Generally Available V0000000

BROAD ECOSYSTEM OF WORKLOADS Build Operators for your apps

Ansible Playbooks Helm Chart APBs

Helm SDK Ansible SDK Go SDK

Build operators from Build operators from Build advanced operators Helm chart, without any Ansible playbooks and for full lifecycle coding APBs management

Product Manager: Daniel Messer Generally Available V0000000 BROAD ECOSYSTEM OF WORKLOADS Depend on other Operators

Operator Framework Dependency Graphs

resolves to installed by

requires jaeger.jaegertracing.io/v1 Jaeger Operator

requires

YourOperator v1.1.2 resolves to

installed by

cockroachdb.charts.helm.k8s.io/v1alpha1 CockroachDB Operator

Product Manager: Daniel Messer Generally Available V0000000 Interesting links for you: Get a free account on cloud.redhat.com https://developer.redhat.com

Red Hat OCP Install portal cloud.redhat.com

Install OCP on IBM Z https://docs.openshift.com/container-platform/4.2/installing/installing_ibm_z/installing-ibm-z.html

Learn OpenShift

https://learn.openshift.com

55 V0000000 Thank you linkedin.com/company/red-hat

youtube.com/user/RedHatVideos Red Hat is the world’s leading provider of enterprise

open source software solutions. Award-winning facebook.com/redhatinc support, training, and consulting services make

Red Hat a trusted adviser to the Fortune 500. twitter.com/RedHat

56 V0000000