Cisco Container Platform for Infrastructure Teams
Justin Barksdale Technical Solutions Architect @3Pings
BRKCLD-2005 Cisco Webex Teams
Questions? Use Cisco Webex Teams to chat with the speaker after the session How 1 Find this session in the Cisco Events Mobile App 2 Click “Join the Discussion” 3 Install Webex Teams or go directly to the team space 4 Enter messages/questions in the team space
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3 Deploying production Kubernetes clusters does not have to be hard.
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 5 Agenda
• Containers 101 (Docker)
• Kubernetes
• CCP Architecture
• Container Networking • ACI-CNI
• Hyperflex-CSI
• Cloud
• HX-AP
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 6 Agenda
• Containers 101 (Docker)
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7 Container Images
Application Code
Container Image
Application Dependencies
Think of Container Images as “golden” templates from which we deploy container instances
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 8 Dockerfile #Base Image FROM nginx:mainline-alpine
# Remove Base conf.d file RUN rm /etc/nginx/conf.d/*
# Add new conf.d file ADD hello.conf /etc/nginx/conf.d/
# Add the index ADD index.html /usr/share/nginx/html/
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9 Docker Image
~ $ docker build -t 3pings/nginx-hello . Sending build context to Docker daemon 214.5kB Step 1/4 : FROM nginx:mainline-alpine mainline-alpine: Pulling from library/nginx 4167d3e14976: Pull complete db94a93dfca0: Pull complete Digest: sha256:9e81b8f9cef5a095f892183688798a5b2c368663276aa0f2be4b1cd283ace53d Status: Downloaded newer image for nginx:mainline-alpine ---> 48c8a7c47625 Step 2/4 : RUN rm /etc/nginx/conf.d/* ---> Running in 88f85486bb0f Removing intermediate container 88f85486bb0f ---> bd8801ea4219 Step 3/4 : ADD hello.conf /etc/nginx/conf.d/ ---> 8f8b37538099 Step 4/4 : ADD index.html /usr/share/nginx/html/ ---> c487928011dc Successfully built c487928011dc Successfully tagged 3pings/nginx-hello:latest
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 Container
~ $ docker history 3pings/nginx-hello IMAGE CREATED CREATED BY SIZE COMMENT c487928011dc 23 seconds ago /bin/sh -c #(nop) ADD file:3ed87b7f3e1501306… 7.19kB 8f8b37538099 23 seconds ago /bin/sh -c #(nop) ADD file:28b8354e8f97bc7e2… 373B bd8801ea4219 24 seconds ago /bin/sh -c rm /etc/nginx/conf.d/* 0B 48c8a7c47625 2 days ago /bin/sh -c #(nop) CMD ["nginx" "-g" "daemon… 0B
~ $ docker run -P -d 3pings/nginx-hello 82dae961268c30c91fa81978bc79c5da436e6e1041843b7b1fd7ce16910ee8d0
~ $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 82dae961268c 3pings/nginx-hello "nginx -g 'daemon of…" 7 seconds ago Up 6 seconds 0.0.0.0:32770->80/tcp determined_tu
82dae961268c Writeable Container Layer R/W c487928011dc image layer: ADD index.html /usr/share/nginx/html/ 8f8b37538099 image layer: ADD hello.conf /etc/nginx/conf.d/ Image bd8801ea4219 image layer: RUN rm /etc/nginx/conf.d/* 48c8a7c47625 base image: nginx:mainline-alpine
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 11 Agenda
• Containers 101 (Docker)
• Kubernetes
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12 Comparing Operational Models
10.1.1.5 10.1.1.6 10.1.1.5 10.1.1.6 Nginx Mysql Container: Container: Tomcat Java Nginx Mysql Container: Container: Ubuntu CentOS Tomcat Java VM VM Pod Pod
ESXi Host Kubernetes Node
VMware vSphere Kubernetes
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13 Single Docker Host
Container: Container: Nginx Mysql Container: Container: Tomcat Java Container Runtime
Host OS Host Hardware
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 14 The need for Orchestration
Container: Container: Container: Container: Container: Container: Nginx Mysql Nginx Mysql Nginx Mysql Container: Container: Container: Container: Container: Container: Tomcat Java Tomcat Java Tomcat Java Container Runtime Container Runtime Container Runtime
Host OS Host OS Host OS Host Hardware Host Hardware Host Hardware
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15 Orchestration Wars Circa 2015-2016
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 16 Docker vs Kubernetes
Packaging an Application Running an Application
Day 1 Day 2 and Beyond
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 17 What is Kubernetes
Kubernetes provides an API for managing application
• Service discovery and load balancing • Storage orchestration • Automated rollouts and rollbacks • Automatic bin packing • Self-healing • Secret and configuration management
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 18 Kubernetes does not:
• Deploy Source Code • Dictate Logging and Monitoring or alerting solutions • Come with networking • Provide Middleware
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19 Kubernetes Architecture
Worker Node Kubelet Kube Proxy Kubectl Container Runtime
Master Node Pod Pod Pod
K8s API Server Dashboard Worker Node Scheduler Kubelet Kube Proxy Container Runtime REST etcd Pod Pod Pod API
Controllers Worker Node Kubelet Kube Proxy Container Runtime • Kubectl: CLI-based remote management of Pod Pod Pod K8s cluster • K8s Dashboard: Native K8s UI
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20 Agenda
• Containers 101 (Docker)
• Kubernetes
• CCP Architecture
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21 Development
Orchestrator Container Runtime Host
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 Production Kubernetes is more than just an Orchestrator Development Production
CI/CD Repository Service Mesh Addons* Logging Monitoring Load Balancing CCP CNI Orchestrator Container Runtime Orchestrator OS Container Runtime IaaS Host Storage
* Addons are available at no additional charge but are not required to be installed. They are supported as part of the platform if they are installed
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 23 Software Layering and CCP Security Scope
End-user Application responsibility
End-user CCP Application Addons Addons Applications CCP packaging & Kubernetes, Docker, Container Kubernetes, Docker, Container Security infra plugins infra plugins responsibility
VMs, Instances, Node OS VMs, Instances, Node OS
Physical Infra Hypervisor, Virtualization infra e.g. Hypervisor, Virtualization infra e.g. separate vSphere vSphere setup + responsibility Physical Compute, Network, Physical Compute, Network, Storage Storage Control Plane Cluster Tenant Cluster BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 24 Cisco Container Platform
Control Plane Cluster Tenant Clusters
Pod Pod
Pod Pod Ops Ops Cluster 1 Cluster Cluster 2 Cluster Cluster 1 Cluster Cluster 2 Cluster Workloads Pod Workloads Pod HX Connect Cluster/ Operations Automation Machine Orchestration Controllers Cluster 1 Kubernetes Cluster 2 Kubernetes
M M M M Control Plane Kubernetes VM VM VM VM VM VM
M VM VM VM VM VM VM VM
Storage (HyperFlex / VMware) Hypervisor Layer (HyperFlex / VMware)
Compute Hardware (UCS)
Networking (Nexus 9K)
Kubernetes Fluentd Prometheus Kibana Hyperflex CNI Istio
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 25 Not all workloads created equal!
CPU Intensive Memory GPU Intensive Intensive
• Financial • High paging • 3D Modelling applications Rendering work applications • Apache • In-memory Spark databases • AI / ML • Encoders / Applications decoders with Tensorflow
Kubernetes can manage different types of workloads through tag based node pools
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 Node Pools in Cisco Container Platform
• Machines sizes can be different between pools (high CPU or GPU or high memory) • Individual pool can be separately managed (change size, delete) • Planning to add node pool for Kubernetes masters with multi- master support
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 27 Multi-GPU as a Service
Kubernetes Cluster 1 Kubernetes Cluster N Non-GPU Node Pool GPU Node Pool GPU Node Pool … Worker VM Worker VM Worker VM Worker VM Worker VM
HYPERVISOR HYPERVISOR
UCS Server UCS Server GPU GPU GPU
Cisco Container Platform v4.x • Automate AI/ML workload similar to any K8s workload, CCP provides ”Multi-GPU as a Service” • Multiple GPU per worker node • Optimized GPU selection for UCS Servers • GPU passthrough (native performance), • Auto provisioning of Nvidia CUDA drivers, container runtime, NVIDIA device plugin BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 28 Agenda
• Containers 101 (Docker)
• Kubernetes
• CCP Architecture
• Container Networking
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 29 Cisco Container Platform CNI Options
ACI CNI Contiv Calico
Network Policy • K8s network policy • K8s network policy • K8s network policy • ACI policy (EPGs + Contracts) for K8s network policy Underlay Network Integration • Underlay integration with ACI fabric • Policy extends beyond single K8s cluster across VMs, Bare Metal, Multi- clusters Load Balancer Integration • Hardware L3 Load • Software metalLB L3 Load • Software metalLB L3 Load Balancer integrated with Balancer Balancer ACI CNI to provide optimal data path Istio Integration • Istio integration • Istio integration • Istio Integration
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 30 Network Policy Support (all CNIs)
Ingress Example YAML file From kind: NetworkPolicy • ipBlock apiVersion: networking.k8s.io/v1 • podSelector metadata: • namespaceSelector name: api-allow • Wildcards - {}, [] spec: Egress podSelector: To matchLabels: • ipBlock app: bookstore • Wildcards - {}, [] role: api ingress: - from: Example Ingress Network Policy - podSelector: matchLabels: Default app: bookstore app: app: bookstore bookstore role: api
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 31 Contiv-VPP Architecture
Option for complete container networking solution entirely from user space. - enables rapid upgrades, highly available, no kernel tax Option to replace all eth/kernel interfaces with high performance/scale memif/users space interfaces - high shared memory performance / scale VPP TCP stack for higher performance (bypass kernel host stack and use VPP TCP stack) - envoy sidecar high performance / scale Fully supports legacy apps that use the kernel host stack in the same architecture - enhanced monitoring / debugging for existing apps
K8s Master
High Performance Cloud- Contiv Cloud- High Performance Legacy Netmaster Legacy Apps Apps Native VNFs Native VNFs Apps Apps
PodPod PodPod PodPod PodPod PodPod PodPod Pod Pod Pod Kubelet Kubelet Pod Pod Pod Istio Envoy App App VNF Contiv VNF App App Istio Envoy Etcd
VPP VPP Kernel Host stack CNI K8s policy & state CNI Kernel Host stack TCP CRI CRI TCP memif memif Stack tapv2 distribution tapv2 Stack
VPP Agent Agent VPP Contiv vSwitch … Contiv vSwitch IPv4/IPv6/SRv6 Network BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 32 Agenda
• Containers 101 (Docker)
• Kubernetes
• CCP Architecture
• Container Networking • ACI-CNI
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 33 ACI CNI Solution Overview
Network Policy Kubernetes Technical Description
• Network policies of Kubernetes supported using standard upstream format but enforced through OpFlex / OVS using APIC Host Protection Profiles ACI Policies • Kubernetes app configurations can be moved without modification to/from ACI and non-ACI environments • Embedded fabric and virtual switch load balancing • PBR in fabric for external service load balancing • OVS used for internal service load balancing • VMM Domain for Kubernetes • Stats per namespace, deployment, service, pod OpFlex OVS OpFlex OVS • Physical to container correlation Node Node
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 34 Why Use ACI CNI
Fast, easy, Turnkey solution for Flexible policy: Hardware-accelerated: secure and node and container Native platform Integrated load connectivity policy API and ACI balancing scalable policies networking for your Application Container Platform
Visibility: Live statistics in APIC Enhanced Multitenancy and per container and health metrics unified networking for containers, VMs, bare metal
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 35 ACI allows flexible POD to EPG mapping K8s Network Policy
Cluster Isolation Namespace Isolation Deployment Isolation
Kube-default-EPG namespace-PROD-EPG Frontend-EPG API-Gateway-EPG
POD POD POD POD POD POD POD POD POD Contract POD POD POD POD POD POD POD POD POD
Contract Contract Contract POD POD namespace-QA-EPG Backend-EPG Monitoring-EPG POD POD POD POD POD POD POD POD POD POD POD Contract POD POD POD POD POD
• Default behavior: single EPG for • Each namespace mapped to an • Each deployment mapped to an EPG entire cluster user PODs EPG • Contracts control traffic between • No need for internal contracts • Contracts for inter-namespace microservice tiers traffic are required BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 36 Container to Non-Container Communications
• In production environments certain services like high performance databases will be running as VMs or Bare Metal Servers • This calls for the ability to easily provide communication between Kubernetes PODs and VMs/Bare Metal endpoints • Simply deploy a contract between your EPGs, ACI will do the rest! • This works for any VMM domain and Physical Domains, for example you can have a Container Domain using VXLAN speaking with a Microsoft SCVMM Domain using VLAN.
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 37 Lab Architecture
pod1 pod2
pod3 pod4
Control Plane Tenant Master Tenant Worker (10.139.13.50) Ubuntu Ubuntu Ubuntu
Hyperflex
vmwareESXi
ACI Fabric ACI Fabric Nexus 93180 LEAF Nexus 93180 LEAF
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 38 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 39 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 40 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 41 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 42 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 43 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 44 ~9 minutes
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 45 Demo Cisco Container Platform
Control Plane Cluster Tenant Clusters
Pod Pod
Pod Pod Ops Ops Cluster 1 Cluster Cluster 2 Cluster Cluster 1 Cluster Cluster 2 Cluster Workloads Pod Workloads Pod HX Connect Cluster/ Operations Automation Machine Orchestration Controllers Cluster 1 Kubernetes Cluster 2 Kubernetes
M M M M Control Plane Kubernetes VM VM VM VM VM VM
M VM VM VM VM VM VM VM
Storage (HyperFlex / VMware) Hypervisor Layer (HyperFlex / VMware)
Compute Hardware (UCS)
Networking (Nexus 9K)
Kubernetes Fluentd Prometheus Kibana Hyperflex CNI Istio
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 47 Livewall Architecture (app profile)
livewall-weather
pod
livewall-incidents
pod livewall-frontend livewall-api mysql
pod pod VM frontend api mysql-clemea (EPG) (EPG) (EPG)
livewall-events
pod
collector (EPG)
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 48 Agenda
• Containers 101 (Docker)
• Kubernetes
• CCP Architecture
• Container Networking • ACI-CNI
• Hyperflex-CSI
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 49 HyperFlex Storage – Controller VM
• Virtual machine running on HyperFlex node which assumes control of local disks through PCI pass-through
CONTROLLER CONTROLLER CONTROLLER VM VM VM HYPERVISOR HYPERVISOR HYPERVISOR disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 50 HyperFlex Storage – Controller VM
• Aggregates available storage into cluster-wide datastores configurable by the HyperFlex admin
CONTROLLER CONTROLLER CONTROLLER VM VM VM HYPERVISOR HYPERVISOR HYPERVISOR disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk DATASTORE(s)
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 51 HyperFlex Storage - IOVisor
• Runs as software VIB inside ESXi and works in conjunction with HyperFlex Controller VMs across entire cluster through high-speed data network to provide NFS target for ESXi hosts
CONTROLLER CONTROLLER CONTROLLER VM VM VM IOVISOR IOVISOR HYPERVISOR HYPERVISOR IOVISOR HYPERVISOR disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk disk DATASTORE(s)
High Speed Data Network
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 52 HyperFlex Storage - IOVisor
• HyperFlex provides NFS datastores to vSphere for storing Kubernetes Node VM “vmdk” files
Kubernetes Cluster
Master Node Worker Node Worker Node
CONTROLLER CONTROLLER HYPERVISOR CONTROLLER VM HYPERVISOR HYPERVISOR VM IOVISOR VM IOVISOR IOVISOR
VMDK File VMDK File DATASTORE VMDK File
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 53 HyperFlex Storage for Kubernetes Node VMs
• On the back-end, the “vmdk” files are segmented into blocks and distributed as evenly as possible across all HyperFlex nodes in the cluster
Kubernetes Cluster
Master Node Worker Node Worker Node
CONTROLLER CONTROLLER HYPERVISOR CONTROLLER VM HYPERVISOR HYPERVISOR VM IOVISOR VM IOVISOR IOVISOR
VMDK File VMDK File DATASTORE VMDK File
A C B B A C C B A
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 54 HyperFlex Storage for Kubernetes Node VMs
• The “vmdk” blocks are synchronously replicated within the cluster based on the HyperFlex “Replication Factor”
Kubernetes Cluster
Master Node Worker Node Worker Node
CONTROLLER CONTROLLER HYPERVISOR CONTROLLER VM HYPERVISOR HYPERVISOR VM IOVISOR VM IOVISOR IOVISOR
VMDK File VMDK File DATASTORE VMDK File
A C B B A C C B A B2 A2 A3 A3 C3 B2 A2 C2 C2 C2 B2 C3 C3 B3 A2 B3 A3 B3 Based on cluster-wide Replication Factor RF3 = three copies of data (recommended) BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 55 HyperFlex Storage for Kubernetes Node VMs
• If a node VM were to move via vMotion, it retains access to it’s “vmdk” blocks
Kubernetes Cluster
vMotion MasterMaster VM WorkerWorker 1 VM Worker Node
CONTROLLER CONTROLLER HYPERVISOR CONTROLLER VM HYPERVISOR HYPERVISOR VM IOVISOR VM IOVISOR IOVISOR
VMDK File VMDK File DATASTORE VMDK File
A C B B A C C B A B2 A2 A3 A3 C3 B2 A2 C2 C2 C2 B2 C3 C3 B3 A2 B3 A3 B3
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 56 HyperFlex Storage for Kubernetes Node VMs
• If a host were to fail, node VMs can be restarted and “vmdk” block copies can be leveraged
Kubernetes Cluster
Restart by VMware HA MasterMaster VM WorkerWorker 1 VM Worker Node
CONTROLLER CONTROLLER HYPERVISOR CONTROLLER VM HYPERVISOR HYPERVISOR VM IOVISOR VM IOVISOR IOVISOR
VMDK File VMDK File DATASTORE VMDK File
A C B B A C C B A B2 A2 A3 A3 C3 B2 A2 C2 C2 C2 B2 C3 C3 B3 A2 B3 A3 B3
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 57 HyperFlex CSI Plugin Components
Kubernetes Cluster
Kubernetes Worker Kubernetes Worker Kubernetes Master
csi-nodeplugin-hxcsi(Daemon Set) csi-nodeplugin-hxcsi(Daemon Set) csi-provisioner-hxcsi(Stateful Set) 3 HX HX HX Provisioner Node Driver Registrar Node Driver Registrar Container Container Container Controller Container Container Container 2 2 Pods App App App HyperFlex Pod1 Pod2 Pod3 etcd Pods Pods Storage Class Persistent Persistent Persistent 1• HyperFlexVolume API: Claim automaticallyVolume deployed Claim as part ofVolume HyperFlex Claim cluster API Server 2• HyperFlex NodeKubelet Container: deployed as DaemonSet, instanceKubelet running on each Controller KubernetesSoftware worker iSCSI node Initiator Software iSCSI Initiator 3• HyperFlex Controller Container: deployed as StatefulSet with single replica, one instance per Kubernetes cluster 1 iSCSI iSCSI iSCSI HyperFlex API LUN LUN LUN
HyperFlex Datastore
Cisco HyperFlex
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 58 HyperFlex CSI Plugin Components
Kubernetes Cluster
Kubernetes Worker Kubernetes Worker Kubernetes Master csi-nodeplugin-hxcsi(Daemon Set) csi-nodeplugin-hxcsi(Daemon Set) csi-provisioner-hxcsi(Stateful Set)
HX HX HX Provisioner Node Driver Registrar Node Driver Registrar Container Container Container Controller Container Container Container Pods App App App HyperFlex Pod1 Pod2 Pod3 etcd Pods Pods Storage Class Persistent Persistent Persistent Volume Claim Volume Claim Volume Claim API Server Kubelet Kubelet Controller Software iSCSI Initiator Software iSCSI Initiator
iSCSI iSCSI iSCSI HyperFlex API LUN LUN LUN
HyperFlex Internal Datastore
Cisco HyperFlex
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 59 Cisco Container Platform Kubernetes-as-a-Service
Deploy 1 Tenant Clusters
Control Plane Upstream Cluster Kubernetes Clusters Automatically Deploys & 2 Configures HyperFlex CSI Integration
Cisco HyperFlex
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 60 Message Board Application
message_board:v1
pod
82dae961268c Writeable Container Layer R/W c487928011dc image layer: ADD index.html /usr/share/nginx/html/ 8f8b37538099 image layer: ADD hello.conf /etc/nginx/conf.d/ Image bd8801ea4219 image layer: RUN rm /etc/nginx/conf.d/* 48c8a7c47625 base image: nginx:mainline-alpine
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 61 Message Board Application
message_board:v1 message_board:v2
pod pod
82dae961268c Writeable Container Layer R/W c487928011dc image layer: ADD index.html /usr/share/nginx/html/ 8f8b37538099 image layer: ADD hello.conf /etc/nginx/conf.d/ Image bd8801ea4219 image layer: RUN rm /etc/nginx/conf.d/* 48c8a7c47625 base image: nginx:mainline-alpine
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 62 Message Board Application
message_board:v1 message_board:v2
pod pod
57efu621436f Writeable Container Layer R/W c487928011dc image layer: ADD index.html /usr/share/nginx/html/ 8f8b37538099 image layer: ADD hello.conf /etc/nginx/conf.d/ Image bd8801ea4219 image layer: RUN rm /etc/nginx/conf.d/* 48c8a7c47625 base image: nginx:mainline-alpine
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Agenda
• Containers 101 (Docker)
• Kubernetes
• CCP Architecture
• Container Networking • ACI-CNI
• Hyperflex-CSI
• Cloud
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 65 Cisco Hybrid Solution for Kubernetes on AWS
On-premises Consistent, production-grade environment experience
AppDynamics
CloudCenter Suite Amazon ECR
Cisco Container AWS Identity and Access Amazon Elastic Kubernetes Platform (CCP) Management (IAM) Service (EKS)
Cisco HyperFlex/UCS or other Stealthwatch Cloud Amazon EC2 / Amazon EBS
Cisco Nexus9K/ACI or other Amazon VPC Cisco CSR1000v
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 66 Cisco hybrid architecture for Microsoft Azure
On-premises Consistent, production-grade environment experience
AppDynamics
CloudCenter Suite Azure Marketplace
Cisco Container Identity and Access Azure Kubernetes Platform Management (AD) Service (AKS)
Stealthwatch Cloud HyperFlex | UCS SD-WAN | CSR 1000v Nexus9K | ACI
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 67 Cisco hybrid architecture for Google Cloud Coming soon
On-premises Consistent, production-grade environment experience
AppDynamics BigQuery Cloud SQL Pub/Sub Big Table Cloud Storage CloudCenter Suite Cloud Spanner
Cisco Container Identity and Access Google Kubernetes Platform Management (AD) Engine(GKE)
Stealthwatch Cloud HyperFlex | UCS SD-WAN | CSR 1000v Nexus9K | ACI
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 68 Agenda
• Containers 101 (Docker)
• Kubernetes
• CCP Architecture
• Container Networking • ACI-CNI
• Hyperflex-CSI
• Cloud
• HX-AP
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 69 HyperFlex Application Platform for Containers Turnkey Platform for Application Modernization
Native Kubernetes (100% Upstream) Production grade: hardened, secure, reliable Enterprise ready Full Stack Kubernetes Integrated Management Networking | Storage | Management Monitoring, Telemetry Multi-Tenancy Add-ons Logging | Monitoring | Registry | Service Mesh Lifecycle Operations Physical Infra, VMs, Containers HyperFlex HCI Datacenter & Edge
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 70 HyperFlex Application Platform (HXAP) for Containers
Intersight based “Full-Stack” Management, Intersight SaaS or Monitoring, Telemetry & Multi-Tenancy for On-prem Kubernetes Clusters
Curated, Hardened & Supported Kubernetes with out-of-box add-ons including CSI, CNI, L4/L7 Load Balancer, Service Mesh, Logging/Monitoring, K8s nodes K8s nodes K8s nodes Registry etc. Auth/RBAC integration K8s nodes K8s nodes K8s nodes K8sKubernetes nodes K8sKubernetes nodes K8sKubernetes nodes nodes nodes nodes Enterprise grade compute virtualization with host clustering, resource scheduling, HyperFlex Application HyperFlex Application HyperFlex Application orchestration, HA, live migration etc. Platform Platform Platform Industry leading storage virtualization , distributed persistent storage& data services provided by HX Data Platform
Hyperconvered Infra provided by scalable HyperFlex Cluster HyperFlex Cluster HyperFlex Cluster HX cluster with UCS-FI or Nexus 9K
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 71 HyperFlex Application Platform (HXAP) Architecture
These VMs would run Kubernetes Nodes forming Tenant Kubernetes K8s Tenant cluster 1 K8s Tenant cluster 2 Clusters
Master Worker Worker Worker Master Worker Worker Worker node VM node VM node VM node VM node VM node VM node VM node VM VMs spun up on HXAP cluster with ...... highly available & resilient VM compute & storage resources
HXAP Cluster HXAP compute cluster and HXDP storage cluster together form HXAP HXAP Compute cluster Cluster for HXAP VMs
HXDP Storage cluster HXAP Infra on every node provides highly available compute cluster
HXDP HXDP HXDP HXAP Infra HXAP Infra HXAP Infra Controller Controller Controller HXDP Controller VM on every node provides highly available storage Linux / KVM Linux / KVM Linux / KVM cluster
Each host runs Linux / KVM
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 72 HyperFlex Application Platform (HXAP) Management
Server & Fabric management • Policy driven Multi-site operations K8s Tenant Cluster mgmt. • AI based Recommendation Engine Master/Worker golden Image, • Connected TAC, HCL, Advisories Install/Upgrade, Auto-scaling, Node • SaaS or On-Prem pools, Version match w/ upstream, HXAP Cluster Logging/Monitoring, Interoperability, management Security compliance, RBAC etc. Kubernetes Node VM management K8s-Node (Master/Worker) VM mgmt. K8s Cluster management Tenancy groups for Isolation, Quota & Auth. VM operations including CRUD, Placement, Affinity/Anti-Affinity, HA, K8s Tenant Cluster 1 K8s Tenant Cluster 2 Clones, Monitoring/Telemetry etc.
Master Worker Worker Worker Master Worker Worker Worker ... node VM node VM node VM node VM node VM node VM node VM node VM ... HXAP mgmt including Compute/Storage cluster Install, Upgrade, Expand, Monitor, HXAP Cluster Capacity/Perf reporting etc.
Physical Infra mgmt including BIOS/Boot policies, Firmware, HCL, Networking, security patches etc.
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 73 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 74 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 75 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 76 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 77 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 78 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 80 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 82 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 83 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 84 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 85 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 86 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 88 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 89 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 90 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 91 Cisco Container Platform
Native Kubernetes (100% Upstream) Direct updates and best practices from open source community
Hybrid Cloud Optimized E.g: Google, AWS…
Integrated Networking | Management | Security | Analytics
Turnkey Solution For Production-Grade Container Flexible Deployment Model Environments VM | Bare metal HX, ACI | Public cloud
Easy to acquire, deploy and manage | Open and consistent | Extensible platform | World-class advisory and support
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 92 Deploying production Kubernetes clusters does not have to be hard.
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 93 Complete your online session • Please complete your session survey survey after each session. Your feedback is very important.
• Complete a minimum of 4 session surveys and the Overall Conference survey (starting on Thursday) to receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events Mobile App or by logging in to the Content Catalog on ciscolive.com/emea.
Cisco Live sessions will be available for viewing on demand after the event at ciscolive.com.
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 94 Continue your education
Demos in the Walk-in Cisco campus self-paced labs
Meet the engineer Related sessions 1:1 meetings
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 95 Thank you Call to Action
• Learn more about Kubernetes and how it can help you
• It’s easy to get started…dive right in! • Online FREE Kubernetes “playgrounds” available, for example: • https://www.katacoda.com/courses/kubernetes/playground
• Treasure-trove of information available, the native Kubernetes docs are a pretty good place to start… • https://kubernetes.io/
• Be active in the Kubernetes communities, help shape the future!
• Access Cisco DEVNET • https://developer.cisco.com
• Ivan Kovacevic – Walk In Lab LABCLD-2099 “Kubernetes made easy with CCP”
BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 97