Introduction to Containers and Container Networking

Rohit Agarwalla, Frank Brockners BRKSDN-2115 Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

How 1. Find this session in the Cisco Live Mobile App 2. Click “Join the Discussion” 3. Install Spark or go directly to the space 4. Enter messages/questions in the space

Cisco Spark spaces will be cs.co/ciscolivebot#BRKSDN-2115 available until July 3, 2017.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public April, 26 1956 : SS IdealX leaves New York

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 4 BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 5 Containers Spawned A New Eco-System

• Today, 90+% of all cargo ships in a standard container

• Faster delivery, reduced (dis-)charging times, increased security (less loss/damage of goods)

• Average shipment cost reduced from >25% to <3%: Key enabler to globalization and just-in-time production

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 6 March 18, 1982 : chroot() introduced to BSD

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 7 March 2000: FreeBSD Jails introduced February 2004: Solaris Zones introduced

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 8 August 6, 2008: Initial release of LXC

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 9 March 13, 2013: Initial release of Docker

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 December 2014: Initial release of rkt

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 11 June 22, 2015: Open Container Initiative launched

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12 Open Container Initiative

“The Open Container Initiative is an open governance structure for the express purpose of creating open industry standards around container formats and runtime.”

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 14 Cloud Native Computing Foundation “The Foundation’s mission is to create and drive the adoption of a new computing • xxx paradigm that is optimized for modern distributed systems environments capable of scaling to tens of thousands of self healing multi-tenant nodes.”

Platinum Sponsors

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 15 CNCF Hosted Projects

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 17 Container Technology

Docker

Containers on Cisco devices

Container Orchestration and Networking

Summary

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Containers

• A Linux container lets you run a Linux system within another Linux system. Zones • A container is a group of processes on a Linux machine.

• Those processes form an isolated environment.

• Inside the container, it (almost) looks like a VM.

• Outside the container, it looks like normal processes running on the machine.

• It looks like a VM, but it is more efficient: Containers = Lightweight Virtualization

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 19 Containers and Virtual Machines

App A App A’ App B Containers are isolated but share OS and where Bins/ Bins/ Bins/ VM appropriate bins/libraries Libs Libs Libs

Guest OS Guest OS Guest OS App App App App App App Container A A’ B B’ C’ C’

Hypervisor (Type 2) Bins/Libs Bins/Libs Control Container Host OS Host OS Server Server

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 20 Containers are almost like Virtual Machines

• Containers have their own network interface (and IP address) • Can be bridged, routed... just like with Xen, KVM etc.

• Containers have their own filesystem • For example a Debian host can run Fedora container (and vice-versa)

• Security: Containers are isolated from each other • Two containers can't harm (or even see) each other

• Resource Control: Containers are isolated and can have dedicated resources • Soft & hard quotas for RAM, CPU, I/O... Though…

• Apps in Containers share the kernel of the host OS (i.e. Linux guests only)

• Containers are light-weight, fast to start, allow for >10x density compared to VMs

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 21 Example - Density

• Simulating “The Internet”

• LXC container “the-internet” within which about 250 sub- containers are running, acting as BGP or OSPF routers.

• …on your Laptop!

https://github.com/nsec/the-internet

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 Example – Speed Let’s start 10 simple containers…

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 23 10 simple containers started.. in 4 seconds

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 24 Containers: Classes of Use-Cases

Extend Network Web-Scale Network Function Micro-Services Elements’ Virtualization Functionality

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 25 Web-scale Mirco-Service Deployments Web-Scale Spurred Interest In Container Technology Again… Micro- Services • Micro-Service design principles: 12factor.net

• Continuous Integration/Continuous Delivery: • Go from developer’s laptop, through automated test, to production, and through scaling without modification

• Alternative form of virtualization for multi-tenant services

• Scale-out: • Rapidly scale same application across hundreds or thousands of servers…and scale down as rapidly

• Cross Cloud Deployment • Move the same application across multiple clouds (public, private, or hybrid) without modification or noticeable delay

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 Network Containers for NFV Function Virtualization • Motivation: Containerize Virtual Network Functions • Light-weight approach to NFV • De-compose network functions into a set of containerized “component VNFs”: Compose use-case specific, optimized VNFs • Leverage alternate orchestration & lifecycle-management solutions

• Considerations • Most existing VNFs have specific kernel dependencies/requirements • Component-VNF concept still evolving – optimization vs. isolation vs. orchestration/management tradeoffs • Security/Isolation • Container & lifecycle management solutions mostly focused on micro-service deployments, less so on NFV/networking applications • NFV focused solution might also require a novel approach to current orchestration, incl. OSS/BSS

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 27 Extend Network Motivation: Containers on Network Elements Elements’ Functionality • Network OS Independence • System Modularity • Limit kernel dependencies

• Enable multiple constituents to provide for device enhancements • Cisco Development, Cisco Services, Partners, Customers, …

• Leverage existing tools/tool-chains and eco-systems • Integration of network device with server-centric tool chains

• Isolation / Security • Application-focused deployment, as opposed to base-OS focused deployment (“escape dependency hell”)

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 28 Qualities Of Containers

• Dedicated namespaces for containers Container 1 Container 2 • Hostname, , IPC, FileSystem, Network, User Process Process Process Process Process (subsets are possible)

• Resource Isolation between Process Process Process Process Process container and kernel (cgroups) Container Container File File File System • Shared Kernel System System

• Option: Copy-on-write file- Kernel system to allow for binary reuse between containers

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29 Container Overview

Container 1 Container 2 Container Management P1 P2 Tools P1 P2

Namespace API/ABI Namespace Set 1 Set 2

Kernel

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 30 Namespaces: Isolate System Resources

• Partition essential kernel structures to create virtual environments. E.g., you can have multiple processes with PID 42, in different environments; E.g., you can have multiple accounts called “frank” in different environments

• Different kinds of namespaces • pid (processes) • mnt (mount points, filesystems) • net (network interfaces, ...) • user (UIDs) • ipc (System V IPC) • uts (hostname)

• Namespace creation via the “clone()” system call with extra flags • new processes inherit the parent’s namespace • you can attach to an existing namespace (kernel >= 3.8)

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 31 PID Namespace

• Processes in a PID namespace don't see processes of the whole system

• Each pid namespace has a PID #1

• pid namespaces are actually nested Host

• A given process can have multiple PIDs • One in each namespace it belongs to • So you can easily access processes of children namespace

• Can't see/affect processes in parent/sibling namespace

Container

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 32 PID Namespace PID namespace isolates the Process ID, implemented as a hierarchy

PID namespace1 (Parent) ls /proc (Level 0) 1 2 3 4

pid:1 pid:4

P1 pid:2 pid:3 P4

ls /proc pid:1 pid:1 ls /proc 1 P2 P3 1

PID Namespace2 (Child) PID Namespace3 (Child) (Level 1) (Level 1)

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 33 Mount Namespace Each mount namespace has its own file-system layout

/proc//mounts /proc//mounts /proc//mounts

/ /dev/sda1 / /dev/sda1 / /dev/sda3 /home /dev/sda2 /home /dev/sda2 /boot /dev/sda4

P1 P2 P3

Mount Mount Namespace1 Namespace2

• Mount namespaces are a “deluxe chroot()”

• A mount namespace can have its own rootfs

• Remount special filesystems, e.g. procfs (to see your processes)

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 34 Net Namespace The Net namespace isolates the networking related resources

• Each net namespace has its own… • Network interfaces Net devices: eth0 Net devices: eth1 (and its own lo/127.0.0.1) IP address: 1.1.1.1/24 IP address: 2.2.2.2/24 • IP address(es) Route Route • routing table(s) Firewall rule Firewall rule • iptables rules Sockets Sockets Proc Proc • Communication between containers: sysfs sysfs • UNIX domain sockets … … (=on the filesystem) • Pairs of veth interfaces Net Namespace1 Net Namespace2

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 35 Container Network Models

Shared Dedicated Applications inside the container appear as applications Applications inside the container appear as appliances on a subnet running natively on the host reachable from the host Examples: Nexus 3k, 9k, Cat 3k, 4k, Titanium, NCS xK Examples: ASR 1k, CSR 1kv, ISR4k, ISR 819

Container 1 Container 2 Container 1 Container 2 Network namespace: Host/TPNNS Network namespace: Host/TPNNS Network namespace: N1 Network namespace: N2 Container interfaces Container interfaces Container interfaces Container interfaces

eth0 eth1 eth2 eth0 eth1 eth2 veth0 veth1 veth2 veth0 veth1 veth2

Shared namespace: Linux Bridge Interfaces are directly mapped to container VPG Multiple bridges and virtual topologies possible Host platform Host platform Forwarding Plane Network namespace: Host/TPNNS* Network namespace: Host eth0 eth1 eth2 eth0 eth1 eth2 Physical interfaces Physical interfaces

*Third Party Network Namespace on IOS XR. See also http://www.cisco.com/c/en/us/td/docs/iosxr/AppHosting/AH_Config_Guide/AH_User_Guide_chapter_00.html

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 37 User Namespace

• Kernel-uid (kuid)/Kernel-Group-id (kgid): Original uid/gid, Global

• uid/gid: user id in user namespace, will be translated to kuid/kgid finally

• Only parent User namespace has rights to set map

kernel-uid: kernel-uid: 2000-2004 1000-1009 uid_map uid_map 10 2000 5 0 1000 10 uid: uid: 10-14 0-9

User namespace1 User namespace2

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 38 IPC Namespace IPC namespace isolates the inter-process communication resource (shared memory, semaphore, message queue)

P1 P2 P3 P4 IPC IPC namespace1 namespace2

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 39 UTS Namespace Every UTS namespace has its own UTS related information

ostype: Linux ostype: Linux Unalterable osrelease: 3.8.6 osrelease: 3.8.6 version: … version: …

hostname: cisco1 hostname: cisco2 Alterable domainname: live1 domainname: live2

UTS namespace1 UTS namespace2

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 40 Control Groups (cgroups) Generic process grouping framework (kernel >= 2.6.24)

• Control the resources in a system with many tasks – Linux itself, VMs, containers…

• Subsystems • Group CPU Scheduler • CPU Accounting Controller • Cpuset • Memory • Block IO Controller • Device Whitelist Controller • Freezer • Namespace

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 41 Manage Containers

• Requirement: Userspace container management tool set • Manage containers • Create namespace • Create private filesystem layout for container • Resources controller by cgroup

• Solution options • liblxc • libvirt lxc • libcontainer/Docker

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 42 Container Management – Workflow Example: Docker

Push App A Docker Pull Image Source Bin/Libs Registry Repository Base Libs Update Build Dockerfile App App App A B C

Bin/Libs

Docker Engine Docker Engine

Build Host Target Host

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 43 Containers as a Packaging Mechanism: Build once – Deploy Anywhere

• Containers combine content (the “RPM”) with context (the DependencyDependency Environment DependencyPackagesPackages environment the RPM was Variables Packages built for) Configuration Networking • Containers resolve RPM dependency management Application Process shortcomings Bootstrap Package • How to resolve situations Security where different RPMs require different versions of a Pseudo dependency package? terminals … • RPM built with a different Filesystem toolchain does not guarantee mounts ABI compatibility

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 44 Container World Taxonomy

microservices

• Container Tools • Docker, Rkt, repos/registries applications • micro-OSs – CoreOS, RHEL Atomic, Ubuntu Snappy, ..

• Cluster Control and Services PaaS • Scheduler/Job Monitor – Marathon, Aurora Distributed Scheduler Frameworks • Resource Managers – Mesos, Kubernetes, container / Service Orch/Mgmt Dockerswarms service Cluster Services • Distributed Key/Value/lock managers – management zookeeper, etcd, consul • Service Orchestration/Management Container Tools • Kubernetes, Mesosphere DCOS, Docker IaaS Swarm, HashiCorp Terraform, CoreOS physical & virtual Tectonic cluster nodes

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 45 Container Technology

A glimpse at Docker

Containers on Cisco devices

Container Orchestration and Networking

Summary

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public What is Docker?

• Docker is a container technology similar to Linux Containers (LXC) that… • Provides isolation for application processes from the host processes using Linux namespaces • Provides resource caps for the application using Linux cgroups • Provides industry preferred packaging model using docker images, docker index, and docker registry concepts • Provides the basis for application lifecycle management automation due to good integration with devops automation tools such as Puppet/Chef (not available in 6.1.1)

• A rich repository of certified docker base images are easily available in public as well as private docker registries to cover a variety of application use cases

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 47 When Docker over LXC?

Docker provides the following advantages over LXC containers

• Docker packaging is becoming a packaging model of choice for application packaging and delivery in the industry.

• Standard packaging format provided by docker allows easier integration with devops automation toolchains such as puppet/chef/etc.

• Docker cli provides a git like workflow for developing containerized applications which is tool-friendly

• Docker provides image management capabilities using docker image, docker index and docker registry concepts

• Public and private docker registries provide a repository of starter container images for a variety of application use cases

• Thin packaging/delivery (Docker application download or upgrade is often a thin download due to sharing of common layers among multiple containers)

• Lower foot print for the containers. Docker layers, especially, the base layers can be shared among multiple containers

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 48 Installing Docker

• Docker can be easily installed on a wide variety of platforms (Ubuntu, Mac OS X, RHEL, CentOS and many more).

• Detailed instructions are here: https://docs.docker.com/engine/installation/

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 49 Let’s pull an image from Docker Hub…

Images consist of multiple layers...

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 50 Let’s pull an image from Docker Hub…

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 51 A Docker image is made up of filesystems layered over each other.

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 52 The storage driver is responsible for presenting these layers as a single, unified file system.

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 53 When you start a container, Docker creates an empty, read-write layer on top of the stack – all changes are made in this layer.

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 54 Docker uses “copy-on- write” container layers.

If a file needs to be modified, it is copied into the read- write layer first.

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 55 This means that multiple containers can share a single copy of the image.

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 57 Now that we have an image, what do we do with it?

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 58 Let’s use the image: Start a container and run the bash shell.

/bin/bash

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 59 Interactive Terminal Name of the container Image Command to run

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 60 Let’s start 10 simple containers…

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 62 10 simple containers started.. in 4 seconds

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 63 BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 64 Creating your own images

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 65 Within my simple bash container, we try and run the “traceroute” command, but it isn’t installed:

We could install it inside my container, but as soon as we exit and start another container, traceroute is gone (as we have started a clean container from scratch).

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 66 What if I want to create a new image with the “traceroute” command already installed?

/bin/bash + traceroute

Two options: 1) Commit the container as a new image after making the changes. 2) Use a Dockerfile containing image creation instructions.

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 67 Option 1: Manually compose your container

apt-get update apt-get install inetutils-traceroute

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 68 Option 1: Commit the changes to your container

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 69 Option 1: Make your container available to others: Dockerhub

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 70 Option 1: Once pushed, the new image is available to others

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 71 The second (and recommended) method for building images is to use a Dockerfile.

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 72 Option 2: Let’s create a simple Dockerfile

• A Dockerfile is simply a text file containing instructions on how to build a Docker image – a “Makefile” for container construction.

• Each line in the Dockerfile creates a new image layer

Source Image

Command(s) to execute

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 73 Build the image

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 74 Option 2: Our –v2 Image is now available...

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 75 Option 2: Our –v2 Image is now available...... and we can run it

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 76 Container Technology

Docker

Containers on Cisco devices: IOS-XR

Container Orchestration and Networking

Summary

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Containers in XR

• 64-bit Open Embedded Linux support Classic XR • Container support XR 6.0 • Standard Linux Toolchains • Third-Party Applications Support System System Control Admin • NCS 5500, NCS 5000, NCS1002, Control Admin XRv 9000, ASR 9000 • LXC and Docker XR 6.1.2 32 bit QNX 64 bit Linux • IOS-XR offers application hosting NPU X86 Hardware • natively on the Host OS (WRL 7) • within a Container (managed through libvirt)

• See also IOS-XR Application Hosting Configuration Guide http://www.cisco.com/c/en/us/td/docs/iosxr/AppHosting/AH_Config_Guide.html

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 78 Third Party Network Name Space (TPNNS)

RPMs

• Secure/Isolated environment with Control Plane Admin Apps Apps 3rd-Party read-access to all interfaces Plane NNS outside XR CLI Mgmt Mgmt Internal Gig Gig • Available to processes in the XR IPC Interfaces TenGig TenGig containers or Third Party containers HunGig HunGig • Routing handled by XR TPNNS 64-bit Host OS Routing Processor

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 79 Third Party Network Name Space

RP/0/RP0/CPU0:#run Wed Oct 28 18:45:56.168 IST

[XR-vm_node0_RP0_CPU0:~]$ ip netns exec tpnns bash [XR-vm_node0_RP0_CPU0:~]$ ifconfig Gi0_0_0_0 Link encap:Ethernet HWaddr 52:46:04:87:19:3c inet addr:192.164.168.10 Mask:255.255.255.0 inet6 addr: fe80::5046:4ff:fe87:193c/64 Scope:Link UP RUNNING NOARP MULTICAST MTU:1514 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:3 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:210 (210.0 B)

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 80 Let’s leverage our learnings on Docker in IOS-XR

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 82 IOS XR Docker Architecture – IOS XR 6.1.1

User orchestrates docker containers in XR bash shell

XR LXC Bash Docker Container 1 Docker Container n

Docker Client Binary 1.10 App 1 Binaries … App 2 Binaries

Loopback1

Shared Network Namespace (global-vrf) XR LXC only Storage Volume Docker Storage Pool (upto 1.5 to 2G) libdevmapper/libcontainer

Remote REST API Docker Daemon 1.10

XR

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 84 Linux Build Machine: Compose your App

• Create Dockerfile on a Linux Build Host describing your App

Linux build host BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 85 Linux Build Machine: Compose your App

• Build and image Find existing docker images (initially this will show existing images) • Upload to your repository of choice docker images Using the Dockerfile created earlier build the docker image in the directory (custom or Dockerhub) containing created dockerfile

docker build –t myimage . docker images # should now show myimage

Push docker image to a customer repository

docker push /myimage>

Alternatively, one can just export the docker image as a tarball. This requires creating a docker container.

docker run --net=host --name=mycont --it myimage /bin/bash docker export --output “myimage”

Linux build host BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 86 XR Router: Install your App using the tarball

• Download image tarball Find existing docker images (initially this will show no images) docker images • User downloads docker image in bash Create a docker image using a tarfile (do “docker export --output “myimage” ” shell created using on another machine and scp the file “myimage” to XR bash shell) bash command docker import myimage docker images # should now show myimage • If so required – clean

up  Remove a docker image (must stop and then remove all running containers using that image)

docker rmi myimage docker images “ should show myimage is removed”

XR Router

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 87 XR Router: Install your App using the Docker Registry

• User downloads Find existing docker images (initially this will show no images)

docker image from docker images

registry in bash shell To pull a docker image from a customer’s private registry, download and install ca.crt for the created using bash customer private registry

command mkdir –p /etc/docker/certs.d/ scp /ca.crt /etc/docker/certs.d//ca.crt up  Pull docker images from registry

docker pull /myimage docker images # should now show myimage

Remove a docker image (must stop and then remove all running containers using that image)

docker rmi myimage docker images “ should show myimage is gone”

XR Router

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 88 XR Router: Start your Containers

Find existing docker containers (with their container-id/container-name) • User orchestrates/ manages docker docker ps -a containers in bash Create and start a new docker container (creates, starts and attaches to the container bash)

shell created using docker run --net=host --name=mycont --restart “always” --it myimage bash command /bin/bash Alternatively, create but do not start a new docker container

docker create --net=host --name=mycont --restart “always” --itd myimage /bin/bash

Start existing stopped docker container

docker start -i

Run a new process (e.g., bash shell) in an existing running docker container

docker exec -it|-d /bin/bash >

XR Router

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 89 XR Router: Stop and Remove your Containers

• User orchestrates/ Find existing docker containers (with their container-id/container-name) manages docker docker ps -a containers in bash shell created using bash command Stop existing running docker container docker stop

Remove stopped docker container

docker rm

Check that the docker container is completely removed

docker ps -a

XR Router

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 90 XR Router: Debug your Containers

• User debugs docker Check the state of existing docker containers

containers in bash docker ps -a shell created using

bash command Capture docker logs to capture activity that has already occurred

docker logs --details

Capture docker logs to capture future docker activity

docker logs --follow --details

To debug what is happening in a running container, run another bash shell in the existing docker container

docker exec --it /bin/bash > ps --ef # run this to check processes running inside docker container

XR Router

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 91 Containers on IOS-XR - Resources

• Cisco IOS XR Application Hosting Configuration Guide for NCS 5000 Series Routers: http://www.cisco.com/c/en/us/td/docs/iosxr/ncs5000/app-hosting/b-application- hosting-configuration-guide-ncs5000/b-application-hosting-configuration-guide- ncs5000_chapter_011.html

• Using Puppet with IOS-XR 6.1.1 https://xrdocs.github.io/application-hosting/tutorials/2016-08-22-using-puppet- with-iosxr-6-1-1

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 92 Container Technology

Docker Containers on Cisco devices: Guestshell – IOS-XE/NX-OS Container Orchestration and Networking

Summary

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Technology Spectrum

Native TCL/Python Application hosting Guest Native OS Scripting (“agents/plugins) Shell Access

Fully controlled Uncontrolled Environment Environment

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 94 Guest Shell Application Linux Shell Environment

• Maintain NX-OS/IOS-XE system integrity • Isolated User Space • Fault Isolation Linux • Resource Isolation applications • Integrate into your Linux workflow Guestshell • On-box rapid prototyping • Device-level API Integration Open Application Container • Scripting (Python) API • Linux Commands Network OS

• Integrated with NX-OS/IOS-XE

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 95 Entering the Guest Shell Console

Enter Guest Shell

Exit Guest Shell

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 96 Guest Shell Connectivity

default vrf used

Use chvrf to switch to a specific vrf

chvrf

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 97 Run host OS CLI commands dohost

Return code Command output

Return code and output for each command passed in

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 98 Running Guest Shell commands from Host OS

run guestshell Native guest shell

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 99 Package Management – RPM: Install

yum install

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 102 Front Panel Interfaces in Guestshell

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 105 Playing with Front Panel Interfaces

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 106 Python Code Samples https://github.com/CiscoDevNet/python_code_samples_network

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 107 Integration With DevOps Tools

Puppet Master

REST

Puppet Plugin Puppet Agent Puppet Plugin Puppet Agent

Puppet Agent Plugin Container Switch Compute Switch Compute Node Node onePK API • Puppet agent hosted in a container Network OS Chef plugin works in a similar way

• Leverages configuration API to implement configuration change

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 123 Container Technology

Docker

Containers on Cisco devices

Container Orchestration and Networking

Summary

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 131 What is Docker Swarm mode ?

• Cluster management integrated with Docker Engine

• Multi-host networking Container Docker Engine • Declarative service model Docker Host (s) Swarm Manager Swarm Manager DockerDocker Host Engine (s) Swarm(Leader)(Leader) Manager • Routing Mesh (Leader) Swarm Worker (s) • Multi-Container Deployment with Raft Consensus Group Docker Stacks Gossip Network

docker swarm init docker swarm join :2377

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 132 Container Network Model (CNM)

• Proposed by Docker to provide networking abstractions/API for container networking Docker Container Docker Container • Sandbox contains configuration of a Network Network container's network stack (Linux Sandbox Sandbox network namespace) Endpoint Endpoint • An endpoint is container's interface Blue Green Network into a network (veth pair) Network • A network is collection of endpoints that can communicate with each other (Linux Bridge, VLAN) • A container can belong to multiple endpoints (and therefore multiple networks)

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 133 Docker Networking

• Bridge network driver (--driver=bridge)

• None network driver (--driver=none)

• Host network driver (--driver=host)

• Overlay network driver (--driver=overlay) – Multi-Host using VXLAN

• MACVLAN network driver (--driver=macvlan)

• Remote drivers – compatible with CNM • Contiv, Weave, Calico, Kuryr

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 134 Docker Networking Internals Overlay Networking

eth0 Bridge Networking eth0 iptables : iptables : NAT/port-mapping NAT/port-mapping eth0 docker_gw docker_gw iptables : iptables : Gossip NAT/port-mapping NAT/port-mapping veth pairs veth pairs eth0 eth0 eth0 eth0 C1 C2 C4 Docker0 isolatedbridge C3 eth1 eth1 eth1 eth1 veth pairs veth pairs eth0 eth0 eth0 eth0 ov-net1 ov-net2 ov-net1 ov-net2 C1 C2 C3 C4 VTEP exists in the Docker Host’s network namespace

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 135 Docker Service, Tasks and Containers

S Front End Service Uses IPVS Nginx:latest, replicas = 3

C Front End Container C Front End Container C Front End Container nginx:latest nginx:latest nginx:latest

T Front End Task T Front End Task T Front End Task Nginx.1 Nginx.1 Nginx.1

docker service create --replicas 3 --name frontend --network mynet frontend:latest

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 136 Ingress Networks and Routing Mesh

ToR 8080 8080 eth0 eth0 iptables : NAT/port-mapping iptables : NAT/port-mapping

docker_gw docker_gw Gossip veth pairs veth pairs eth0 eth0 eth0 eth0 C2 eth2 Ingress C1 Ingress eth1 eth1 eth1 eth1

Ingress ov-net1 ov-net1 Ingress Ingress

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 137 Routing Mesh

mynet 8080 8080 8080

frontend frontend

Docker Engine Docker Engine Docker Engine

Swarm Worker (s) Swarm Worker (s) Swarm Worker (s)

docker service create --replicas 2 --name frontend --network mynet -p 8080:80 frontend:latest

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 138 What is Kubernetes ?

• Pod serves as core unit of management

• Cluster Networking Container Pod • Replica Sets Kubernetes Master Pod Node(s) • Services Master(s) Kubelet Kube-Proxy Worker Node(s) • Deployments

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 139 Container Network Interface (CNI)

• Proposed by CoreOS as part of appc specification Container Kubernetes, Rocket…

• Common interface between container Network run time and network plugin namespace Container Network Interface • Gives driver freedom to manipulate network namespace Plugins • Network described by JSON config Driver plumbing • Plugins support two commands: - Add Container to Network - Remove Container from Network

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 140 Kubernetes Networking

• Highly-coupled container-to-container communications

• Pod-to-Pod communications • Cilium, Contiv, Contrail, Flannel, Google Compute Engine, Nuage VCS, OVN, Calico, Romana, Weave, CNI-Genie

• Pod-to-Service communications • Cluster IP

• External-to-Internal communications • Node Port • Load Balancer • External IP

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 141 Kubernetes Networking Internals cbro

Container to Container C C 10.0.2.4 C1 Pod C2 C C --net=container:infra --net=container:infra 10.0.1.3 Node --ipc=container:infra --ipc=container:infra Pod

cbro cbro

Infra C C C C 10.0.1.3 10.0.1.4 10.0.3.3 Pod Pod Pod

Communicate using localhost static ports Node Node Shared namespace – IP address, IPC

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 142 Kubernetes Service, Pods, Replication Controllers, Labels

S Front End Service Manages endpoints app=webapp,role=frontend, version=v1

P Front End v1 Pod P Front End v1 Pod P Front End v2 Pod app=webapp, role=frontend, app=webapp, role=frontend, app=webapp, role=frontend, version=v1 version=v1 version=v2

R Front End v1 Controller R Front End v2 Controller

Desired Count = 2 Desired Count = 1

app=webapp,role=frontend,version=v1 app=webapp,role=frontend,version=v2

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 143 Pod to Service Communication

Service IP gets translated to Pod IP and vice versa

eth0 eth0

iptables iptables

cbr0 cbr0 vethxx vethxx

eth0 eth0 C C C C 10.0.3.3 10.0.4.3 Pod Pod Node Node

Service IP is a virtual IP implemented in IP Tables (connection tracking)

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 144 Performance Suffers Using Overlays Host 2 Host 1 Host 2 Host 1 VM1 VM2 VM1 VM2

C1 Cn C1 Cn C1 Cn C1 Cn

Physical Network Hypervisor Hypervisor Overlay Network Overlay Network Overlay Network Overlay Network - VXLAN - VXLAN - VXLAN - VXLAN Virtual Switching or Overlay Network Overlay Network - VXLAN Overlay Network - VXLAN

Physical Network Hypervisor Hypervisor

Physical Network

Encap over encap over encap – Can not leverage performance and performance suffers security by natively integrating with HW

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 145 What is Contiv Any Infrastructure

ACI integration Application Intent Rich Policy Container,VM,BM LDAP/RBAC Connectvity

DevOps IT Admin

Any Platform Any Networking

The Most Powerful Container Networking Fabric 100% Open Source Rich Policy Model L2, L3, Overlay or ACI

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Introducing Contiv 1.0 What’s New: Commercially Supported Contiv

LDAP+ All New User Kubernetes RBAC Experience 1.4+ Support Cisco Advance and Workflow Services

1 Cisco Technical Docker 1.12+ OpenShift 1.4+ Simple 1-click Services Solutions Support Integration Install Support

Open Source at github.com/contiv © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Cisco ACI + Contiv (ACI mode)

Container Contiv Master Management Contiv APIC Plugin Docker Solution Highlights

OpenShift • Docker containers/OpenShift pods data plane connectivity is provided by ACI fabric

• ACI policies can be extended across physical, virtual machines, and Docker containers/OpenShift Pods

OVS Contiv Plugin

HYPERVISOR HYPERVISOR HYPERVISOR Container/Pod Host

Unified Policy Automation and Enforcement Across Physical, Virtual, and Containers

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 148 Getting More Information / Getting Started

http://contiv.io/ http://blogs.cisco.com/tag/contiv https://contiv.slack.com/

https://hub.docker.com/u/contiv/ https://github.com/contiv http://cs.co/SBxK8sContiv

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 150 Docker+Cisco Partnership

Joint Engineering, Contiv Docker Datacenter Docker Datacenter Sales and Marketing Network Plugin On FlexPod CVD On Cisco UCS

Stronger Together + Open Source community and technology partners to build solutions

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 151 Summary

• Containers – a lightweight tool for virtualization, rapid development and deployment • Web-Scale MicroServices / NFV / Custom Apps on Network-Device • Efficient tool chains for containers drove acceptance

• Containers on routers and switches • Fast/Custom feature delivery: 3rd-party integration tools (Puppet, Chef, …), Guest-Shell, Your stuff…

• Container networking • CNM and CNI models • Contiv – Most powerful container networking and policy fabric

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 152 Complete Your Online Session Evaluation

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card. • Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us.

Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKSDN-2115 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 154 Thank you