Cisco Container Platform for Infrastructure Teams Justin Barksdale Technical Solutions Architect @3Pings BRKCLD-2005 Cisco Webex Teams Questions? Use Cisco Webex Teams to chat with the speaker after the session How 1 Find this session in the Cisco Events Mobile App 2 Click “Join the Discussion” 3 Install Webex Teams or go directly to the team space 4 Enter messages/questions in the team space BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3 Deploying production Kubernetes clusters does not have to be hard. BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 5 Agenda • Containers 101 (Docker) • Kubernetes • CCP Architecture • Container Networking • ACI-CNI • Hyperflex-CSI • Cloud • HX-AP BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 6 Agenda • Containers 101 (Docker) BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7 Container Images Application Code Container Image Application Dependencies Think of Container Images as “golden” templates from which we deploy container instances BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 8 Dockerfile #Base Image FROM nginx:mainline-alpine # Remove Base conf.d file RUN rm /etc/nginx/conf.d/* # Add new conf.d file ADD hello.conf /etc/nginx/conf.d/ # Add the index ADD index.html /usr/share/nginx/html/ BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9 Docker Image ~ $ docker build -t 3pings/nginx-hello . Sending build context to Docker daemon 214.5kB Step 1/4 : FROM nginx:mainline-alpine mainline-alpine: Pulling from library/nginx 4167d3e14976: Pull complete db94a93dfca0: Pull complete Digest: sha256:9e81b8f9cef5a095f892183688798a5b2c368663276aa0f2be4b1cd283ace53d Status: Downloaded newer image for nginx:mainline-alpine ---> 48c8a7c47625 Step 2/4 : RUN rm /etc/nginx/conf.d/* ---> Running in 88f85486bb0f Removing intermediate container 88f85486bb0f ---> bd8801ea4219 Step 3/4 : ADD hello.conf /etc/nginx/conf.d/ ---> 8f8b37538099 Step 4/4 : ADD index.html /usr/share/nginx/html/ ---> c487928011dc Successfully built c487928011dc Successfully tagged 3pings/nginx-hello:latest BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 Container ~ $ docker history 3pings/nginx-hello IMAGE CREATED CREATED BY SIZE COMMENT c487928011dc 23 seconds ago /bin/sh -c #(nop) ADD file:3ed87b7f3e1501306… 7.19kB 8f8b37538099 23 seconds ago /bin/sh -c #(nop) ADD file:28b8354e8f97bc7e2… 373B bd8801ea4219 24 seconds ago /bin/sh -c rm /etc/nginx/conf.d/* 0B 48c8a7c47625 2 days ago /bin/sh -c #(nop) CMD ["nginx" "-g" "daemon… 0B ~ $ docker run -P -d 3pings/nginx-hello 82dae961268c30c91fa81978bc79c5da436e6e1041843b7b1fd7ce16910ee8d0 ~ $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 82dae961268c 3pings/nginx-hello "nginx -g 'daemon of…" 7 seconds ago Up 6 seconds 0.0.0.0:32770->80/tcp determined_tu 82dae961268c Writeable Container Layer R/W c487928011dc image layer: ADD index.html /usr/share/nginx/html/ 8f8b37538099 image layer: ADD hello.conf /etc/nginx/conf.d/ Image bd8801ea4219 image layer: RUN rm /etc/nginx/conf.d/* 48c8a7c47625 base image: nginx:mainline-alpine BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 11 Agenda • Containers 101 (Docker) • Kubernetes BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12 Comparing Operational Models 10.1.1.5 10.1.1.6 10.1.1.5 10.1.1.6 Nginx Mysql Container: Container: Tomcat Java Nginx Mysql Container: Container: Ubuntu CentOS Tomcat Java VM VM Pod Pod ESXi Host Kubernetes Node VMware vSphere Kubernetes BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13 Single Docker Host Container: Container: Nginx Mysql Container: Container: Tomcat Java Container Runtime Host OS Host Hardware BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 14 The need for Orchestration Container: Container: Container: Container: Container: Container: Nginx Mysql Nginx Mysql Nginx Mysql Container: Container: Container: Container: Container: Container: Tomcat Java Tomcat Java Tomcat Java Container Runtime Container Runtime Container Runtime Host OS Host OS Host OS Host Hardware Host Hardware Host Hardware BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15 Orchestration Wars Circa 2015-2016 BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 16 Docker vs Kubernetes Packaging an Application Running an Application Day 1 Day 2 and Beyond BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 17 What is Kubernetes Kubernetes provides an API for managing application • Service discovery and load balancing • Storage orchestration • Automated rollouts and rollbacks • Automatic bin packing • Self-healing • Secret and configuration management BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 18 Kubernetes does not: • Deploy Source Code • Dictate Logging and Monitoring or alerting solutions • Come with networking • Provide Middleware BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19 Kubernetes Architecture Worker Node Kubelet Kube Proxy Kubectl Container Runtime Master Node Pod Pod Pod K8s API Server Dashboard Worker Node Scheduler Kubelet Kube Proxy Container Runtime REST etcd Pod Pod Pod API Controllers Worker Node Kubelet Kube Proxy Container Runtime • Kubectl: CLI-based remote management of Pod Pod Pod K8s cluster • K8s Dashboard: Native K8s UI BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20 Agenda • Containers 101 (Docker) • Kubernetes • CCP Architecture BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21 Development Orchestrator Container Runtime Host BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 Production Kubernetes is more than just an Orchestrator Development Production CI/CD Repository Service Mesh Addons* Logging Monitoring Load Balancing CCP CNI Orchestrator Container Runtime Orchestrator OS Container Runtime IaaS Host Storage * Addons are available at no additional charge but are not required to be installed. They are supported as part of the platform if they are installed BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 23 Software Layering and CCP Security Scope End-user Application responsibility End-user CCP Application Addons Addons Applications CCP packaging & Kubernetes, Docker, Container Kubernetes, Docker, Container Security infra plugins infra plugins responsibility VMs, Instances, Node OS VMs, Instances, Node OS Physical Infra Hypervisor, Virtualization infra e.g. Hypervisor, Virtualization infra e.g. separate vSphere vSphere setup + responsibility Physical Compute, Network, Physical Compute, Network, Storage Storage Control Plane Cluster Tenant Cluster BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 24 Cisco Container Platform Control Plane Cluster Tenant Clusters Pod Pod Pod Pod Ops Ops Cluster 1 Cluster Cluster 2 Cluster 1 Cluster Cluster 2 Cluster Workloads Pod Workloads Pod HX Connect Cluster/ Operations Automation Machine Orchestration Controllers Cluster 1 Kubernetes Cluster 2 Kubernetes M M M M Control Plane Kubernetes VM VM VM VM VM VM M VM VM VM VM VM VM VM Storage (HyperFlex / VMware) Hypervisor Layer (HyperFlex / VMware) Compute Hardware (UCS) Networking (Nexus 9K) Kubernetes Fluentd Prometheus Kibana Hyperflex CNI Istio BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 25 Not all workloads created equal! CPU Intensive Memory GPU Intensive Intensive • Financial • High paging • 3D Modelling applications Rendering work applications • Apache • In-memory Spark databases • AI / ML • Encoders / Applications decoders with Tensorflow Kubernetes can manage different types of workloads through tag based node pools BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 Node Pools in Cisco Container Platform • Machines sizes can be different between pools (high CPU or GPU or high memory) • Individual pool can be separately managed (change size, delete) • Planning to add node pool for Kubernetes masters with multi- master support BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 27 Multi-GPU as a Service Kubernetes Cluster 1 Kubernetes Cluster N Non-GPU Node Pool GPU Node Pool GPU Node Pool … Worker VM Worker VM Worker VM Worker VM Worker VM HYPERVISOR HYPERVISOR UCS Server UCS Server GPU GPU GPU Cisco Container Platform v4.x • Automate AI/ML workload similar to any K8s workload, CCP provides ”Multi-GPU as a Service” • Multiple GPU per worker node • Optimized GPU selection for UCS Servers • GPU passthrough (native performance), • Auto provisioning of Nvidia CUDA drivers, container runtime, NVIDIA device plugin BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 28 Agenda • Containers 101 (Docker) • Kubernetes • CCP Architecture • Container Networking BRKCLD-2005 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 29 Cisco Container Platform CNI Options ACI CNI Contiv Calico Network Policy • K8s network policy • K8s network policy • K8s network policy • ACI policy (EPGs + Contracts) for K8s network policy Underlay Network Integration • Underlay integration