DOCSLIB.ORG

CVMFS Build and Release Pipeline Using Docker Microservices

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
CVMFS Build and Release Pipeline Using Docker Microservices CVMFS Build and Release Pipeline Using Docker Microservices Heath Skarlupka ([email protected]) David Schultz ([email protected]) Original Setup NFS Server SSH CentOS 6 VM NFS SSH CentOS 7 VM NFS SSH Ubuntu 14 VM Stratum0 NFS Local NFS rsync Disk SSH Ubuntu 16 VM NFS Original Setup SlowNFS Server SSH CentOS 6 VM NFS SSH NotCentOS 7 VM NFS SSH Ubuntu 14 VM Stratum0 NFS Local Automated NFS rsync Disk SSH Ubuntu 16 VM NFS 1 Repository Update / Day Could we use Devops and Continuous Delivery principles to build something better? Principles Infrastructure as code - Our infrastructure deployment model had to be repeatable and scale out as we supported new platforms Automation - Our build and deployment process needed to be automated and needed to be started with a simple git push Tools Kubernetes - Microservices scheduler across multiple servers Ceph storage - Shared filesystem and block device deployment in Kubernetes Docker - Microservices are a key component to infrastructure as code design Buildbot - Build pipeline software. Pick your poison. Jenkins would work too. Git - Good way to track versions, bugs, and history New Setup Ceph Buildbot Block PlatformBuildbot /var/spool PlatformBuildbot WorkerBuildbot WorkerPlatform Ceph Platform Stratum0 Worker Block Buildbot Worker /srv Master Haproxy Server Ceph Shared FS Ceph Storage Kubernetes Docker Server Docker Server Docker Server Docker Server Buildbot Workflow CVMFS Transaction Rsync From Shared FS to Block Volume Git Push Software build on multiple platforms CVMFS Publish Kubernetes Examples https://github.com/WIPACrepo/cvmfs_buildbot/tree/master/kubernetes # Shared Storage apiVersion: v1 kind: PersistentVolumeClaim metadata: name: cvmfs-buildbot-worker-pv-claim spec: # Stratum0 Worker Deployment accessModes: https://raw.githubusercontent.com/WIPAC - ReadWriteMany repo/cvmfs_buildbot/master/kubernetes/w persistentVolumeReclaimPolicy: Retain orker-cvmfs-centos7-stratum0.json #StorageClass: cephfs resources: requests: storage: 1Ti factory = util.BuildFactory() factory.addStep(steps.ShellCommand( name='open transaction', command=['cvmfs_server','transaction','icecube.opensciencegrid.org'], Buildbot Examples haltOnFailure=True, )) factory = util.BuildFactory() factory.addStep(steps.ShellCommand( factory.addStep(steps.Git( name='rsync', repourl='git://github.com/WIPACrepo/cvmfs.git', command=[ mode='full', 'cvmfs_rsync', method='clobber', util.Interpolate('/cvmfs-source/icecube.opensciencegrid.org/$(prop:variant)s'), workdir='build', )) util.Interpolate('/cvmfs/icecube.opensciencegrid.org/$(prop:variant)s'), factory.addStep(steps.ShellCommand( ], name='build cvmfs', haltOnFailure=True, command=[ doStepIf=BuildPassed, 'python', 'builders/build.py', )) '--src', 'icecube.opensciencegrid.org', factory.addStep(steps.ShellCommand( name='publish transaction', '--dest', '/cvmfs/icecube.opensciencegrid.org', command=['cvmfs_server','publish','icecube.opensciencegrid.org'], '--variant', util.Property('variant'), haltOnFailure=True, ], doStepIf=BuildPassed, env={ hideStepIf=lambda results, s: results==SKIPPED, 'CPUS': util.Property('CPUS', default='1'), )) 'MEMORY': util.Property('MEMORY', default='1'), factory.addStep(steps.ShellCommand( }, name='abort transaction', command=['cvmfs_server','abort','-f','icecube.opensciencegrid.org'], workdir='build', haltOnFailure=True, haltOnFailure=True, doStepIf=BuildFailed, locks=[ hideStepIf=lambda results, s: results==SKIPPED, cfg.locks['cvmfs_shared'].access('exclusive') )) ], )) Test Software Improvements Snapshot Ceph Volume CVMFS Transaction Rsync From Shared FS to Block Volume Git Push Software build on multiple platforms CVMFS Publish.
Load more

Recommended publications
  • Buildbot: a Continuous Integration System
    Buildbot: A continuous integration system Krzysztof Voss January, 2013 Outline • Testing and Continuous Integration • Introduction to Buildbot • BuildMaster • BuildMaster components • BuildSlave • Installation and Usage 1 Testing and continuous integration Tests: • the best specification • safety-net for refactoring • bug identification Tests are the most effective if we: • run them often • run them on different machines/environments • can easily see their results 2 The most straightforward approach would entail: • logging in to different machines • fetching the newest source code • running tests • analyzing their output In case we want to test a few environments, repeating the above steps is tedious. Developers do not focus on the code, instead they run tests. A continuous integration system performs all of these steps for us, so developers can focus on their code. 3 Introduction to Buildbot: Features • run builds on a variety of BuildSlave platforms • arbitrary build process: handles projects using C, Python, . • minimal host requirements: python and Twisted • BuildSlave can be behind a firewall if they can still do checkout • status delivery through web page, email, IRC, other protocols • track builds in progress, provide estimated completion time • flexible configuration by subclassing generic build process classes 4 • debug tools to force a new build, submit fake Changes, query BuildSlave status • released under the GPL source: http://buildbot.net/buildbot/docs/current/manual/introduction.html 5 Introduction to Buildbot: Overview system overview source: http://buildbot.net/buildbot/docs/0.8.1/full.html 6 BuildMaster BuildMaster components source: http://buildbot.net/buildbot/docs/0.8.1/full.html 7 BuildMaster BuildMaster: • holds the configuration of the entire system.
    [Show full text]
  • Analysis of Devops Tools to Predict an Optimized Pipeline by Adding Weightage for Parameters
    International Journal of Computer Applications (0975 – 8887) Volume 181 – No. 33, December 2018 Analysis of DevOps Tools to Predict an Optimized Pipeline by Adding Weightage for Parameters R. Vaasanthi V. Prasanna Kumari, PhD S. Philip Kingston Research Scholar, HOD, MCA Project Manager SCSVMV University Rajalakshmi Engineering Infosys, Mahindra City, Kanchipuram College, Chennai Chennai ABSTRACT cloud. Now-a-days more than ever, DevOps [Development + Operations] has gained a tremendous amount of attention in 2. SCM software industry. Selecting the tools for building the DevOps Source code management (SCM) is a software tool used for pipeline is not a trivial exercise as there are plethora’s of tools development, versioning and enables team working in available in market. It requires thought, planning, and multiple locations to work together more effectively. This preferably enough time to investigate and consult other plays a vital role in increasing team’s productivity. Some of people. Unfortunately, there isn’t enough time in the day to the SCM tools, considered for this study are GIT, SVN, CVS, dig for top-rated DevOps tools and its compatibility with ClearCase, Mercurial, TFS, Monotone, Bitkeeper, Code co- other tools. Each tool has its own pros/cons and compatibility op, Darcs, Endevor, Fossil, Perforce, Rational Synergy, of integrating with other tools. The objective of this paper is Source Safe, and GNU Bazaar. Table1 consists of SCM tools to propose an approach by adding weightage to each with weightage. parameter for the curated list of the DevOps tools. 3. BUILD Keywords Build is a process that enables source code to be automatically DevOps, SCM, dependencies, compatibility and pipeline compiled into binaries including code level unit testing to ensure individual pieces of code behave as expected [4].
    [Show full text]
  • Studying Routing Issues in Vanets by Using NS-3 Bachelor Thesis on Informatics
    Studying Routing Issues in VANETs by Using NS-3 Bachelor Thesis on Informatics by Christos Profentzas Thesis supervisor Dr. Periklis Chatzimisios Alexander Technological Educational Institute of Thessaloniki Department of Informatics A.T.E.I. of Thessaloniki P.O. Box 141 GR -547 00 Thessaloniki, Macedonia, Greece November 2012 i Acknowledgements This research project would not have been possible without the sup- port of many people. The author wishes to express his gratitude to his supervisor, Assistant Professor Periklis Chatzimisios (Alexan- der TEI of Thessaloniki, Greece) and Assistant Professor Gennaro Boggia (Politecnico di Bari, Italy) who was abundantly helpful and offered invaluable assistance, support and guidance. Deepest grati- tude are also due to the members of the supervisory committee, Assis- tant Professor Luigi Alfredo Grieco and Ph.D Student Giuseppe Piro without whose knowledge and assistance this study would not have been successful. Special thanks also to all group members of Telematics Lab at the Electrical & Electronics Engineering Depart- ment of Politecnico di Bari, for sharing the literature, invaluable assis- tance and laboratory facilities. The author would also like to convey thanks to the Office of Erasmus Program and Faculty of Alexander Technological Educational Institution of Thessaloniki for providing the financial means. Abstract A Vehicular Ad-hoc Network (VANET) is a system of nodes (vehi- cles) that are being connected with each other by wireless technolo- gies. Usually the nodes are moving with very high speeds and, thus, the topology is unpredictable and frequently changing. Such networks can be stand alone and making paths along vehicles or may be con- nected by an infrastructure internet.
    [Show full text]
  • This Book Doesn't Tell You How to Write Faster Code, Or How to Write Code with Fewer Memory Leaks, Or Even How to Debug Code at All
    Practical Development Environments By Matthew B. Doar ............................................... Publisher: O'Reilly Pub Date: September 2005 ISBN: 0-596-00796-5 Pages: 328 Table of Contents | Index This book doesn't tell you how to write faster code, or how to write code with fewer memory leaks, or even how to debug code at all. What it does tell you is how to build your product in better ways, how to keep track of the code that you write, and how to track the bugs in your code. Plus some more things you'll wish you had known before starting a project. Practical Development Environments is a guide, a collection of advice about real development environments for small to medium-sized projects and groups. Each of the chapters considers a different kind of tool - tools for tracking versions of files, build tools, testing tools, bug-tracking tools, tools for creating documentation, and tools for creating packaged releases. Each chapter discusses what you should look for in that kind of tool and what to avoid, and also describes some good ideas, bad ideas, and annoying experiences for each area. Specific instances of each type of tool are described in enough detail so that you can decide which ones you want to investigate further. Developers want to write code, not maintain makefiles. Writers want to write content instead of manage templates. IT provides machines, but doesn't have time to maintain all the different tools. Managers want the product to move smoothly from development to release, and are interested in tools to help this happen more often.
    [Show full text]
  • Expert Python Programming Third Edition
    Expert Python Programming Third Edition Become a master in Python by learning coding best practices and advanced programming concepts in Python 3.7 Michał Jaworski Tarek Ziadé BIRMINGHAM - MUMBAI Expert Python Programming Third Edition Copyright © 2019 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Kunal Chaudhari Acquisition Editor: Chaitanya Nair Content Development Editor: Zeeyan Pinheiro Technical Editor: Ketan Kamble Copy Editor: Safis Editing Project Coordinator: Vaidehi Sawant Proofreader: Safis Editing Indexer: Priyanka Dhadke Graphics: Alishon Mendonsa Production Coordinator: Shraddha Falebhai First published: September 2008 Second edition: May 2016 Third edition: April 2019 Production reference: 1270419 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78980-889-6 www.packtpub.com To my beloved wife, Oliwia, for her love, inspiration, and her endless patience.
    [Show full text]
  • Design and Implementation of a Continuous Integration System
    Prakash Acharya DESIGN AND IMPLEMENTATION OF A CONTINUOUS INTEGRATION SYSTEM DESIGN AND IMPLEMENTATION OF A CONTINUOUS INTEGRATION SYSTEM Prakash Acharya Bachelor’s Thesis Spring 2019 Information Technology Oulu University of Applied Sciences ABSTRACT Oulu University of Applied Sciences Information Technology, Internet Services Author: Prakash Acharya Title of the bachelor’s thesis: Design and Implementation of a Continuous Integration System Supervisor: Lasse Haverinen Term and year of completion: Spring 2019 Number of pages: 43 Not having an automated test system in the software development process leaves space for errors to go unnoticed in code review, which can break a codebase if integrated. The aim of the Bachelor’s thesis was to design and implement a Continuous Integration system where tests and build tasks could be automated. The Buildbot framework was used as a Continuous Integration and automation framework. The Ansible playbook was used for automating the deployment of Buildbot configuration. The Buildbot configuration was created in such a way that it allows Buildbot instances to be created easily and in a configurable manner. In addition, automation of the Python static code analysis tool, Prospector, was added to the Continuous Integration system. The result of having a configurable Buildbot system was that the developers could start adding automation tasks and test and verify their change without affecting the production instance. Having an automated static code analysis run on changes brings into attention potential problems and error before they are integrated. Even though the Buildbot system is not taken into use, it was tested in a test environment and it is shown to work.
    [Show full text]
  • Driving and Virtualizing Control Systems: the Open Source Approach Used in Whiterabbit
    Driving and virtualizing control systems: the Open Source approach used in WhiteRabbit Javier Muñoz Mellid & Samuel Iglesias Gonsalvez Academia-Industry Matching event on Technology of Controls for Accelerators and Detectors November 2013 Agenda Knowing Igalia The value of the Open Source approach Open Source approach in WhiteRabbit A walk on technical details and demo Driving and virtualizing control systems: the Open Source approach used in WhiteRabbit Javier Muñoz Mellid & Samuel Iglesias Gonsalvez What is Igalia? Open Source Company 40 engineers and hackers working around the globe Hacking upstream in different technologies and communities kernel (Linux) virtualization (QEMU/KVM) browsers/multimedia (WebKit, Blink, GStreamer...) compilers (V8, JavaScriptCore...) UI (GTK+,...) documents (Evince, LibreOffice...) distros (Debian, Tizen...) automotive/IVI 13 years old now! www.igalia.com Driving and virtualizing control systems: the Open Source approach used in WhiteRabbit Javier Muñoz Mellid & Samuel Iglesias Gonsalvez Partnering Linux Foundation www.linuxfoundation.org/news- media/announcements/2011/04/igalia-joins-linux- foundation W3C www.igalia.com/nc/igalia-247/news/item/igalia-joins- the-world-wide-web-consortium Tizen Association www.igalia.com/nc/igalia-247/news/item/igalia-joins- the-tizen-association-partner-program/ ... Driving and virtualizing control systems: the Open Source approach used in WhiteRabbit Javier Muñoz Mellid & Samuel Iglesias Gonsalvez What services and solutions provides Igalia? We are experts in Open Source
    [Show full text]
  • Splash Documentation Release 3.5
    Splash Documentation Release 3.5 Scrapinghub Jun 16, 2020 Contents 1 Documentation 3 1.1 Installation................................................3 1.2 Splash HTTP API............................................6 1.3 Splash Scripts Tutorial.......................................... 18 1.4 Splash Lua API Overview........................................ 24 1.5 Splash Scripts Reference......................................... 26 1.6 Response Object............................................. 67 1.7 Request Object.............................................. 69 1.8 Element Object.............................................. 71 1.9 Working with Binary Data........................................ 87 1.10 Available Lua Libraries......................................... 88 1.11 Splash and Jupyter............................................ 95 1.12 FAQ.................................................... 97 1.13 Contributing to Splash.......................................... 102 1.14 Implementation Details......................................... 103 1.15 Changes................................................. 105 i ii Splash Documentation, Release 3.5 Splash is a javascript rendering service. It’s a lightweight web browser with an HTTP API, implemented in Python 3 using Twisted and QT5. The (twisted) QT reactor is used to make the service fully asynchronous allowing to take advantage of webkit concurrency via QT main loop. Some of Splash features: • process multiple webpages in parallel; • get HTML results and/or take screenshots; • turn
    [Show full text]
  • Buildbot Documentation Release 0.9.15
    Buildbot Documentation Release 0.9.15 Brian Warner Jan 02, 2018 Contents 1 Buildbot Tutorial 3 1.1 First Run.................................................3 1.2 First Buildbot run with Docker......................................6 1.3 A Quick Tour...............................................9 1.4 Further Reading............................................. 17 2 Buildbot Manual 23 2.1 Introduction............................................... 23 2.2 Installation................................................ 29 2.3 Concepts................................................. 45 2.4 Secret Management........................................... 56 2.5 Configuration............................................... 58 2.6 Transition to “worker” terminology................................... 245 2.7 Customization.............................................. 250 2.8 New-Style Build Steps.......................................... 275 2.9 Command-line Tool........................................... 278 2.10 Resources................................................. 289 2.11 Optimization............................................... 289 2.12 Plugin Infrastructure in Buildbot..................................... 289 2.13 Deployment............................................... 290 3 Buildbot Development 293 3.1 General Documents........................................... 293 3.2 APIs................................................... 377 3.3 Python3 compatibility.......................................... 470 3.4 Classes.................................................
    [Show full text]
  • Automating Build, Test, and Release with Buildbot
    Automating Build, Test, and Release with Buildbot Dustin J. Mitchell Mozilla, Inc. Email: [email protected] Tom Prince Email: [email protected] Abstract—Buildbot is a mature framework for building con- to support a variety of deployments. Schedulers react to new tinuous integration systems which supports parallel execution of data from change sources, external events, or triggers based jobs across multiple platforms, flexible integration with version- on clock time (e.g., for a nightly build), and add new build control systems, extensive status reporting, and more. Beyond the capabilities it shares with similar tools, Buildbot has a number requests to the queue. of unique strengths and capabilities, some of them particu- Each build request comes with a set of source stamps larly geared toward release processes. This paper summarizes identifying the code from each codebase used for a build. A Buildbot’s structure, contrasts the framework with related tools, source stamp represents a particular revision and branch in a describes some advanced configurations, and highlights ongoing source code repository. Build requests also specify a builder improvements to the framework.1 that should perform the task and a set of properties, arbitrary I. INTRODUCTION key-value pairs giving further detail about the requested build. Buildbot is a continuous integration (CI) framework which Each builder defines the steps to take in executing a particular has been in use for well over 10 years[13]. While it began type of build. Once the request is selected for execution, the as a simple build-and-test tool, it has grown into an advanced steps are performed sequentially.
    [Show full text]
  • Mining Software Repositories for Release Engineers - Empirical Studies on Integration and Infrastructure-As-Code
    UNIVERSITÉ DE MONTRÉAL MINING SOFTWARE REPOSITORIES FOR RELEASE ENGINEERS - EMPIRICAL STUDIES ON INTEGRATION AND INFRASTRUCTURE-AS-CODE YUJUAN JIANG DÉPARTEMENT DE GÉNIE INFORMATIQUE ET GÉNIE LOGICIEL ÉCOLE POLYTECHNIQUE DE MONTRÉAL THÈSE PRÉSENTÉE EN VUE DE L’OBTENTION DU DIPLÔME DE PHILOSOPHIÆ DOCTOR (GÉNIE INFORMATIQUE) AOÛT 2016 c Yujuan Jiang, 2016. UNIVERSITÉ DE MONTRÉAL ÉCOLE POLYTECHNIQUE DE MONTRÉAL Cette thèse intitulée: MINING SOFTWARE REPOSITORIES FOR RELEASE ENGINEERS - EMPIRICAL STUDIES ON INTEGRATION AND INFRASTRUCTURE-AS-CODE présentée par: JIANG Yujuan en vue de l’obtention du diplôme de: Philosophiæ Doctor a été dûment acceptée par le jury d’examen constitué de: M. GAGNON Michel, Ph. D., président M. ADAMS Bram, Doctorat, membre et directeur de recherche M. GUÉHÉNEUC Yann-Gaël, Doctorat, membre Mme BAYSAL Olga, Ph. D., membre externe iii DEDICATION To my grandpa Who has become a star in the heaven iv ACKNOWLEDGMENTS Many people I met, many things happened, are hidden as part of this thesis. First of all, I want to thank my parents, for their endless and unconditional love. You respect my every single choice and impractical dream. You always hold the faith in me, even when I myself had lost it. Exclusive thanks are dedicated to my supervisor, Dr.Bram Adams. Humble, enthusiastic and hard-working, for work and for life, you show me what should be like as a researcher and an individual person. You lead me through all the way up here, with enormous patience and kindness. In past four years, what I gained was not only professional skills, but also the will to face new challenges, the courage of learning from failures and the belief in hard working.
    [Show full text]
  • Buildbot This Is the Buildbot Manual
    BuildBot This is the BuildBot manual. Copyright (C) 2005, 2006, 2009, 2010 Brian Warner Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. i Table of Contents 1 Introduction..................................... 1 1.1 History and Philosophy ........................................ 1 1.2 System Architecture............................................ 2 1.2.1 BuildSlave Connections.................................... 3 1.2.2 Buildmaster Architecture .................................. 4 1.2.3 Status Delivery Architecture............................... 6 1.3 Control Flow ................................................... 6 2 Installation ...................................... 8 2.1 Requirements .................................................. 8 2.2 Installing the code ............................................. 8 2.3 Creating a buildmaster ......................................... 9 2.4 Upgrading an Existing Buildmaster............................ 10 2.5 Creating a buildslave.......................................... 10 2.5.1 Buildslave Options ....................................... 12 2.6 Launching the daemons ....................................... 13 2.7 Logfiles ................................................... .... 14 2.8 Shutdown ................................................... 14 2.9 Maintenance .................................................. 15 2.10 Troubleshooting.............................................
    [Show full text]