Current Events in Container Storage

Keith Hudgins Docker

▪ Cloud Native Computing Foundation (CNCF) ▪ https://www.cncf.io/ ▪ Open Container Initiative (OCI) ▪ https://www.opencontainers.org/ ▪ Both are part of the Linux Foundation ▪ https://www.linuxfoundation.org/

▪ Docker ▪ Default for most container installs, widest user base ▪ 20 million+ Docker Community Engine installs alone ▪ Containerd ▪ Fully graduated CNCF project (as of Feb 2019) ▪ Windows and Linux container runtime ▪ Upstream of Docker (for Linux, anyway) ▪ CoreOS (rkt) ▪ CoreOS acquired by RedHat May 2018 ▪ rkt archived by CNCF as of 8/16/2019 ▪ CRI-O ▪ Default runtime for RH OpenShift as of 4.0, June 2019 ▪ Intended to be Kubernetes-native runtime ▪ Fully open-source ▪ Upstream of these projects* are CNCF efforts. *Docker has non-CNCF upstream 2019 Storage Developer Conference. © Docker, Inc. All Rights Reserved. Components as well 3 Container Storage Lifecycle

▪ Containers use storage in 3 primary contexts: ▪ Raw container image ▪ At rest, either as a file on disk or object storage ▪ At runtime ▪ One of several graph drivers ▪ These are NOT persistent ▪ Persistent storage ▪ Volumes attached to containers for persistent data ▪ Most of this talk will focus on this type of storage

▪ Disk image of container at runtime ▪ Most graph drivers are Copy-On-Write (COW) or UnionFS based ▪ For Docker/Containerd, refer to support matrix for which driver to use ▪ https://docs.docker.com/storage/storagedriver/select-storage-driver/ ▪ Intended for images that don’t write to disk much ▪ Can have performance issues if too many writes ▪ Drivers include Overlay2, AUFS, Btrfs, Zfs, LCFS (Experimental)

▪ https://docs.docker.com/storage/storagedriver/ ▪ https://blog.mobyproject.org/where-are-containerds-graph-drivers-145fc9b7255 ▪ https://integratedcode.us/2016/08/30/storage-drivers-in-docker-a-deep-dive/ ▪ https://github.com/portworx/lcfs ▪ https://docs.docker.com/engine/extend/plugins_graphdriver/

Persistent storage is, continually, one of the hardest problems to solve in the container space.

OS Containers are NOT NEW. chroot was released in 1979

But when Docker introduced immutable containers in 2013, Everything changed for storage.

Each instanced container is meant to be a pluggable component of an application stack.

In production, a container tends to spin up and down, as well as move between nodes rapidly depending on orchestration.

Not every container will need persistence.

But, your data needs to go somewhere!

Prior to pluggable, dynamic volume management:

▪ Data was stored in a bind-mounted volume ▪ Volume was just a directory on disk ▪ Production typically used SAN or NAS directly mounted on the host OS

For persistent storage, there are two primary interfaces:

▪ Docker Volume Driver ▪ Container Storage Interface ▪ For Kubernetes, there’s also: ▪ Native in-tree plugins ▪ FlexVolume ▪ Both are deprecated-ish in favor of CSI

▪ Standard persistence interface for Docker Swarm and Mesosphere ▪ Runs as a container on each host node ▪ Each container implements 2 interfaces: ▪ Orchestrator-facing REST API service ▪ Storage-facing interface ▪ Finally, requires a unique description file (manifest.json) to request privileges and declare requirements for install. ▪ Supports all platforms Docker runs on (Linux, Windows*, i386, ARM) ▪ You will need native drivers for each supported platform ▪ Windows support uses the older, v1 driver API

Interface is very stable ▪ No changes since 2017 ▪ No cross-node knowledge ▪ Can cause problems with scaling ▪ Every volume needs to be exposed to every node in each cluster ▪ Docker EE tracks volumes out-of-band based on Volume ID

Deep Links https://docs.docker.com/engine/extend/plugins_volume/ https://docs.docker.com/registry/spec/manifest-v2-2/ http://mesos.apache.org/documentation/latest/isolators/docker-volume/

Emerging standard, becoming the default

▪ 1.0 spec released in late 2018 ▪ Supported by Kubernetes 1.13+, Mesos, and Cloud Foundry ▪ Initially supports basic CRUD operations on volumes ▪ Expanded feature support coming via microservices-like controllers

CSI drivers need 3 fundamental features: ▪ Node service ▪ Controller service* ▪ Identity service

All of these communicate via gRPC - https://grpc.io/

* See next slide for caveats

Controller service is for: ▪ Volume Operations ▪ Create, Delete, List volumes ▪ “Publish” operations ▪ This designates a volume to a given node ▪ Required for scheduling volumes at runtime ▪ If you don’t do volume operations, you can skip the controller service (See iSCSI, NFS drivers)

Node service is for: ▪ Node operations ▪ Mount/unmount volumes on a node ▪ “Staging” operations ▪ Basically prep ops on a new volume ▪ Create filesystem, etc ▪ Also used in multiple mount scenarios

Identity service is for: ▪ API grease ▪ GetCapability and GetPluginInfo requests ▪ Should be implemented on both Node and Controller

Not necessarily, but probably.

You can absolutely put Node and Controller ops in the same container if you wish

BUT….

Check out what’s new and interesting:

▪ Volume Snapshots ▪ Implemented as a separate service (own container) ▪ Dynamic Volume Resize operations ▪ Likewise separate service ▪ Raw Block Volume Support ▪ Windows CSI Support Deep Links https://github.com/kubernetes/enhancements/blob/master/keps/sig-windows/20190714-windows-csi-support.md https://kubernetes-csi.github.io/docs/features.html

▪ This is a possible topology ▪ Driver is deployed via kube application yaml definition ▪ Or Helm Chart ▪ Or Docker Compose ▪ Or…… ▪ Really recommend standard application yaml - it’s universally supported

▪ https://github.com/kubernetes/community/tree/master/sig-storage ▪ https://medium.com/google-cloud/understanding-the-container-storage-interface-csi-ddbeb966a3b ▪ https://arslan.io/2018/06/21/how-to-write-a-container-storage-interface-csi-plugin/ ▪ https://kubernetes-csi.github.io/docs/

Also, Special Thanks to Deep Debroy and Anusha Ragunathan at Docker for reviewing this talk and keeping me accurate!

Please take a moment to rate this session.

Your feedback matters to us.