CVMFS Build and Release Pipeline Using Docker Microservices Heath Skarlupka ([email protected]) David Schultz ([email protected]) Original Setup NFS Server SSH CentOS 6 VM NFS SSH CentOS 7 VM NFS SSH Ubuntu 14 VM Stratum0 NFS Local NFS rsync Disk SSH Ubuntu 16 VM NFS Original Setup SlowNFS Server SSH CentOS 6 VM NFS SSH NotCentOS 7 VM NFS SSH Ubuntu 14 VM Stratum0 NFS Local Automated NFS rsync Disk SSH Ubuntu 16 VM NFS 1 Repository Update / Day Could we use Devops and Continuous Delivery principles to build something better? Principles Infrastructure as code - Our infrastructure deployment model had to be repeatable and scale out as we supported new platforms Automation - Our build and deployment process needed to be automated and needed to be started with a simple git push Tools Kubernetes - Microservices scheduler across multiple servers Ceph storage - Shared filesystem and block device deployment in Kubernetes Docker - Microservices are a key component to infrastructure as code design Buildbot - Build pipeline software. Pick your poison. Jenkins would work too. Git - Good way to track versions, bugs, and history New Setup Ceph Buildbot Block PlatformBuildbot /var/spool PlatformBuildbot WorkerBuildbot WorkerPlatform Ceph Platform Stratum0 Worker Block Buildbot Worker /srv Master Haproxy Server Ceph Shared FS Ceph Storage Kubernetes Docker Server Docker Server Docker Server Docker Server Buildbot Workflow CVMFS Transaction Rsync From Shared FS to Block Volume Git Push Software build on multiple platforms CVMFS Publish Kubernetes Examples https://github.com/WIPACrepo/cvmfs_buildbot/tree/master/kubernetes # Shared Storage apiVersion: v1 kind: PersistentVolumeClaim metadata: name: cvmfs-buildbot-worker-pv-claim spec: # Stratum0 Worker Deployment accessModes: https://raw.githubusercontent.com/WIPAC - ReadWriteMany repo/cvmfs_buildbot/master/kubernetes/w persistentVolumeReclaimPolicy: Retain orker-cvmfs-centos7-stratum0.json #StorageClass: cephfs resources: requests: storage: 1Ti factory = util.BuildFactory() factory.addStep(steps.ShellCommand( name='open transaction', command=['cvmfs_server','transaction','icecube.opensciencegrid.org'], Buildbot Examples haltOnFailure=True, )) factory = util.BuildFactory() factory.addStep(steps.ShellCommand( factory.addStep(steps.Git( name='rsync', repourl='git://github.com/WIPACrepo/cvmfs.git', command=[ mode='full', 'cvmfs_rsync', method='clobber', util.Interpolate('/cvmfs-source/icecube.opensciencegrid.org/$(prop:variant)s'), workdir='build', )) util.Interpolate('/cvmfs/icecube.opensciencegrid.org/$(prop:variant)s'), factory.addStep(steps.ShellCommand( ], name='build cvmfs', haltOnFailure=True, command=[ doStepIf=BuildPassed, 'python', 'builders/build.py', )) '--src', 'icecube.opensciencegrid.org', factory.addStep(steps.ShellCommand( name='publish transaction', '--dest', '/cvmfs/icecube.opensciencegrid.org', command=['cvmfs_server','publish','icecube.opensciencegrid.org'], '--variant', util.Property('variant'), haltOnFailure=True, ], doStepIf=BuildPassed, env={ hideStepIf=lambda results, s: results==SKIPPED, 'CPUS': util.Property('CPUS', default='1'), )) 'MEMORY': util.Property('MEMORY', default='1'), factory.addStep(steps.ShellCommand( }, name='abort transaction', command=['cvmfs_server','abort','-f','icecube.opensciencegrid.org'], workdir='build', haltOnFailure=True, haltOnFailure=True, doStepIf=BuildFailed, locks=[ hideStepIf=lambda results, s: results==SKIPPED, cfg.locks['cvmfs_shared'].access('exclusive') )) ], )) Test Software Improvements Snapshot Ceph Volume CVMFS Transaction Rsync From Shared FS to Block Volume Git Push Software build on multiple platforms CVMFS Publish.