CVMFS Build and Release Pipeline Using Docker Microservices Heath Skarlupka ([email protected]) David Schultz ([email protected]) Original Setup NFS Server

SSH CentOS 6 VM NFS SSH CentOS 7 VM NFS SSH Ubuntu 14 VM Stratum0 NFS Local NFS rsync Disk SSH Ubuntu 16 VM NFS Original Setup SlowNFS Server SSH CentOS 6 VM NFS SSH NotCentOS 7 VM NFS SSH Ubuntu 14 VM Stratum0 NFS Local Automated NFS rsync Disk SSH Ubuntu 16 VM NFS

1 Repository Update / Day Could we use Devops and Continuous Delivery principles to build something better? Principles

Infrastructure as code - Our infrastructure deployment model had to be repeatable and scale out as we supported new platforms

Automation - Our build and deployment process needed to be automated and needed to be started with a simple push Tools

Kubernetes - Microservices scheduler across multiple servers

Ceph storage - Shared filesystem and block device deployment in Kubernetes

Docker - Microservices are a key component to infrastructure as code design

Buildbot - Build pipeline software. Pick your poison. Jenkins would work too.

Git - Good way to track versions, bugs, and history New Setup

Ceph Buildbot Block PlatformBuildbot /var/spool PlatformBuildbot WorkerBuildbot WorkerPlatform Ceph Platform Stratum0 Worker Block Buildbot Worker /srv Master Haproxy Server

Ceph Shared FS

Ceph Storage

Kubernetes

Docker Server Docker Server Docker Server Docker Server Buildbot Workflow

CVMFS Transaction

Rsync From Shared FS to Block Volume Git Push Software build on multiple platforms

CVMFS Publish Kubernetes Examples https://github.com/WIPACrepo/cvmfs_buildbot/tree/master/kubernetes

# Shared Storage apiVersion: v1 kind: PersistentVolumeClaim metadata: name: cvmfs-buildbot-worker-pv-claim spec: # Stratum0 Worker Deployment accessModes: https://raw.githubusercontent.com/WIPAC - ReadWriteMany repo/cvmfs_buildbot/master/kubernetes/w persistentVolumeReclaimPolicy: Retain orker-cvmfs-centos7-stratum0.json #StorageClass: cephfs resources: requests: storage: 1Ti factory = util.BuildFactory() factory.addStep(steps.ShellCommand( name='open transaction', command=['cvmfs_server','transaction','icecube.opensciencegrid.org'], Buildbot Examples haltOnFailure=True, )) factory = util.BuildFactory() factory.addStep(steps.ShellCommand( factory.addStep(steps.Git( name='rsync', repourl='git://.com/WIPACrepo/cvmfs.git', command=[ mode='full', 'cvmfs_rsync',

method='clobber', util.Interpolate('/cvmfs-source/icecube.opensciencegrid.org/$(prop:variant)s'), workdir='build',

)) util.Interpolate('/cvmfs/icecube.opensciencegrid.org/$(prop:variant)s'), factory.addStep(steps.ShellCommand( ], name='build cvmfs', haltOnFailure=True, command=[ doStepIf=BuildPassed, 'python', 'builders/build.py', )) '--src', 'icecube.opensciencegrid.org', factory.addStep(steps.ShellCommand( name='publish transaction', '--dest', '/cvmfs/icecube.opensciencegrid.org', command=['cvmfs_server','publish','icecube.opensciencegrid.org'], '--variant', util.Property('variant'), haltOnFailure=True, ], doStepIf=BuildPassed, env={ hideStepIf=lambda results, s: results==SKIPPED, 'CPUS': util.Property('CPUS', default='1'), )) 'MEMORY': util.Property('MEMORY', default='1'), factory.addStep(steps.ShellCommand( }, name='abort transaction', command=['cvmfs_server','abort','-f','icecube.opensciencegrid.org'], workdir='build', haltOnFailure=True, haltOnFailure=True, doStepIf=BuildFailed, locks=[ hideStepIf=lambda results, s: results==SKIPPED, cfg.locks['cvmfs_shared'].access('exclusive') )) ], )) Test Software Improvements

Snapshot Ceph Volume

CVMFS Transaction

Rsync From Shared FS to Block Volume Git Push Software build on multiple platforms

CVMFS Publish