Containers Infrastructure for Advanced Management Federico Simoncelli – Principal Software Engineer July 2015 Agenda

Running containers on an infrastructure is a “solved problem”. Now you have to run them smoothly.

• Deployment • Metrics Technologies • Analyzing and Correlating Metrics • Smart Analysis – Fleecing • Cross-Provider Relationships • Traceability • Advanced Management Requirements

• Docker automates the deployment of applications inside software containers • manages a cluster of containers as a single system • OpenShift build, deploy, and manage your applications with Docker and Kubernetes • Project Atomic lightweight operating system (rpm-ostree) assembled out of RPM content • complete lifecycle management tool for physical and virtual servers Deployment - Choices

• Deploying Atomic • http://www.projectatomic.io (Fedora and CentOS based images) • Bootable, immutable, versioned filesystem, updates delivered in large sets • Everything that is not in the base system runs as a container • Deploying a Traditional Distribution • More flexible in nodes content • Matrix of components is less tested • Deployment on IaaS (OpenStack, oVirt) • Secure – Improved isolation of tenancy • “Elastic” on demand • Deployment on Bare Metal • Automated with Foreman or Satellite Deployment - Technologies

• Kubernetes project uses Salt • Requires a salt-master and salt-minion running on the nodes • Mostly for GCE (and Vagrant) • OpenShift project uses • https://github.com/openshift/openshift-ansible • Supports AWS GCE libvirt OpenStack Vagrant • OpenStack • Heat - Kubernetes and OpenShift templates can be found on github repositories • Magnum - Container orchestration engines as first class resources in OpenStack • Self-Registration of nodes • Enabled by default - preferred pattern used by most distributions • Can be disabled for strict security policies Metrics Collection

• Containerized solution Kubernetes Master Metrics Analysis • Collecting Metrics • Heapster • Kubelet (cAdvisor)

• Storing Metrics Heapster Metrics DB • InfluxDB • Hawkular • Metrics Analisys • Metrics per Container / Pod / Service / Replica Kubernetes Node 1 Kubernetes Node N • Highly coupled with the metrics database Metrics Cross-Linking

• IaaS metrics collection • OpenStack Ceilometer IaaS Metrics Heapster • oVirt Data Warehouse • Metrics collection and analysis • Nodes, Instances and Hosts Kubernetes Node 1 Kubernetes Node N

Instance load spike Host load spike IaaS Instance 1 IaaS Instance N

IaaS Host X IaaS Host Y Metrics – Hawkular

• Open Source monitoring solution - http://www.hawkular.org • Components can be re-used in applications outside of Hawkular • Capable of receiving monitoring data from various existing sources • Scalable and high performance metric storage based on Cassandra • On-premise or in the cloud (works on OpenShift out of the box) • Other features and work in progress • Alerting • Data Visualization (in progress) • Analytics • Push notification (in progress) Fleecing – Images Inspection

• Case 1 – Strict policy, allow only specific known images • Pre-compute images content • Case 2 – No policy, images are downloaded from any registry • Images discovery • Docker signed images • Extract images (tricky) • Inspect images (e.g. OpenSCAP) • Cross-link with packages and errata database • Store/cache content and analysis (retention after image disappeared) Fleecing – Images Inspection

• Use the same nodes of the cluster infrastructure • Dockerized solution - runs everywhere (Atomic) • Should not execute anything from inside the image (best practice) • docker-fleece – https://github.com/simon3z/docker-fleece • Creates a stopped container from image (fast) • Uses CopyFromContainer to extract files (linear) • Docker Graph Driver independent • docker-mount – https://github.com/rhvgoyal/docker-mount • Creates a stopped container from image (fast) • Mounts externally the Container mountpoint (fast) • Tightly coupled with the Docker Graph Driver (device-mapper, overlayfs, etc.) Fleecing – Docker Fleece Example

• How to extract image content:

$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE docker.io/fedora 22 ded7cd95e059 6 weeks ago 186.5 MB docker.io/fsimonce/docker-fleece latest 847c8293c5b4 7 weeks ago 217.9 MB

$ docker run -ti --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock \ -v /tmp/docker-fleece:/tmp/docker-fleece fsimonce/docker-fleece --image=fedora:22 --path=/tmp/docker-fleece

(3.38 seconds)

$ du -sh /tmp/docker-fleece 194M /tmp/docker-fleece Fleecing – Container Inspection

• Inspect the content of a running container • Analyze the “drift” of a container from its original image • were the packages/binaries tampered with? • Fleecing a running container requires snapshots and merge capabilities • Freeze container processes • Flush cache • Snapshot and clone container • Unfreeze container processes • Analyze cloned container • Merge back container snapshot Kubernetes - Traceability

• Report Docker Container ID in Container Status • Allows to peek in Docker (or other runtimes) low level information • Report Docker Image ID in Container Status • Allows to reliably identify the specific images in use • Add Node System Info • Allows to identify reboots and version of the critical components (Kernel, Docker, etc.) • Add Instance ID and Provider ID to Nodes • Allows to identify the instances of the Kubernetes nodes • Improved Logging • Allows to analyze important events in the cluster and infrastructure ManageIQ

• Open Source Cloud Management Platform - http://manageiq.org • Self-Service Provisioning and Chargeback • Places Virtual Workloads according to your business priorities • Balance your costs, performance, security, and reliability across cloud platforms • Supports Infrastructure (oVirt, VMware) and Cloud (OpenStack) Providers • Smart Analysis of Hosts, Virtual Machines and Instances • Drift of the instances from initial image (detect new applications, new users) • Supports definition of Compliance Policies ManageIQ - Inventory Full Picture

• Inventory collection at all layers for infrastructures, clouds and containers • Containers, Instances, Hosts, Storage and Network

Service Pods Container Image

Cluster Nodes

Region Instance

Host

Storage Network ManageIQ - Inventory Relationship ManageIQ - Inventory Relationship ManageIQ - Inventory Relationship ManageIQ - Capacity and Utilization

• Metrics correlations, and in the future cross-provider correlations ManageIQ - Dashboard and Topology ManageIQ - Dashboard and Topology ManageIQ - Stay in Touch

• Community http://talk.manageiq.org • Code https://github.com/ManageIQ/manageiq #providers/containers • Documentation http://manageiq.org/documentation • Social: • Twitter @manageiq #manageiq

• Contact Me • Federico Simoncelli - [email protected] • Twitter @simon3z