ECP Continuous Integration Startup Tutorial 2020 ECP Annual Meeting Houston, Texas February 4, 2020 Goals What do we hope you walk away with? 1. An understanding of current and planned GitLab CI capabilities 2. Facility points of contact and resources to get started today 3. Introduction to GitLab CI concepts 2 Agenda ● Introduction to HPC CI and GitLab ● Tutorial: Hello Environment [hands-on] ● GitLab Runner Enhancements ● Break ● HDF5 Group - CI Experience and Feedback ● Current Project Status and Ongoing Efforts ● Tutorial: Heat Equation [hands-on] ● Federation and Zero Trust Networking ● Secure Mirroring ● Recap and Path Forward ● Tutorial: Expanded Heat Equation [hands-on] 3 Introduction to HPC CI and GitLab ECP Project: WBS 2.4.4 Software Deployment at Facilities - Software Integration The Software Integration effort was established to bridge the ECP ST software development effort with the Exascale hardware and software environments deployed at the Facilities. ● Continuous Integration (CI) - Provide the ability to continuously test AD/ST software on facility hardware resources with software environments established at the Facility. Key for software development teams targeting systems being deployed agile feedback loop is key for development 5 Our Team Site GitLab capabilities | CI runners on HPC resources | Account authentication | Test resources / support ● Ryan Adamson ● Noah Ginsburg ● David Nicholaeff ● Anthony Agelastos ● Don Maghrak ● Neil O’Neill ● Paul Bryant ● Mario Melara ● Rebel Powell ● Tiffany Connors ● Thomas Mendoza ● Ryan Prout ● Darel Finkbeiner ● Mark Miller ● Mahesh Rajan ● Jim Galarowicz ● Dave Montoya ● Kyle Shaver ● Todd Gamblin ● Jeff Neel ● Lance Vowell ● Chris West 6 Continuous Integration (CI) Refresher • CI is the process of building / testing code for your project, best done before code is integrated into the repository or released for general usage. – https://about.gitlab.com/product/continuous-integration/ • As developers we most likely expect: – Customizable triggers (e.g. specific changes, schedules, targets, …) – Easy to set up and capable of using existing tests – Feedback (dashboard) identifying pass/fail cases – Access to required test resources – Resources than can handle code build requirements • Many may already use tools such as Travis, Jenkins, or even cron jobs Source: https://about.gitlab.com/product/continuous-integration/ 7 CI Solution for Large-Scale HPC Facilities Need: HPC centers need new security Response: Using GitLab as a basis, extend features within Continuous Integration security, permissions, and auditing features systems to serve thousands of users with while improving intra-Lab and inter-Lab unique hardware and security requirements. accessibility to CI pipelines. 8 Automated testing is an enormous productivity multiplier Projects cannot scale to accept many contributions without automation External contributors Code Review Code Free cloud testing and CI services Automated testing on Accepted unique HPC resources Contributions • Cloud testing is a boon to open source projects – Travis, CircleCI, GitLab, AppVeyor, Azure, etc. provide free cycles! • So-called Continuous Integration (CI) has been hard to implement at HPC centers due to security restrictions – Strict internal rules, lax security in most off-the-shelf products Code Repository • Lack of testing on key platforms will lead to less robust HPC software. 9 GitLab • GitLab is a single platform for the entire DevOps lifecycle – Plan, create, verify, and release • Core functionality (free) – Version control, collaboration, CI/CD, and documentation tools • Open source and capable of self management – MIT License Source: https://docs.gitlab.com/ee/ • Additional tiers of functionality are only available via license – https://about.gitlab.com/pricing/self-manag ed/feature-comparison/ 10 Gitlab - Features and Roadmap GitLab is a continually developing project, with new features being added. However, it is important to note that not everything is available in the “core” (free) tier. 11 GitLab CI/CD • GitLab offers robust support for CI/CD (Continues Delivery) • Primarily controlled via a ‘.gitlab-ci.yml’ file defined in your project repository – Defines the jobs that will be run, their structure, and what will trigger their potential execution – https://docs.gitlab.com/ee/ci/yaml/ • Terms: – Jobs: Core elements of CI, where commands are defined for execution – Stages: Defined order for job execution within a pipeline – Pipelines: Collection of jobs • Pipelines can be triggered through a number of mechanisms, including: commits, manually, api, scheduling, merge requests, upstream projects, and more... 12 GitLab Runner • “GitLab Runner is the open source project that is used to run your CI/CD jobs and send the results back to GitLab” • Using GitLab CI/CD requires management of a runner: – Application supports the execution and reporting of CI jobs on target systems – Installed on target test resources and registered to a GitLab instance – Responsible for the execution of scripts whose creation is dictated by both the user’s CI file as well as administrative configuration 13 Our Vision Delivering a secure, easy-to-use CI solution to support Software Product testing against E6 HPC environments. 14 15 Tutorial: Hello Environment [hands-on] Tutorial Resources • Tutorial resources available for the hands-on sections • Please review and sign the provided agreement – If you don’t have a login or have any questions please let us know • Remember resources may be limited in comparison to the number of attendees, refrain to running jobs that would take a disproportionate amount of compute cycles. 17 Hello Environment - Results 18 Hello Environment - Results 19 Hello Environment - Results 20 Hello Environment - Results 21 Hello Environment - Results 22 Hello Environment - Results 23 Hello Environment - Results 24 GitLab CI Enhancements Security Challenges and Use Cases • All GitLab CI jobs run as the single gitlab-runner user • Achieving any level of isolation between CI jobs requires: – 1) Targeting specific runners such as Docker or VMs – 2) Having each user/team manage their own runner – Neither of these are ideal • Use cases we aim to support – Runner data isolation – Account specific resources – Sensitive data sets – Administrator management 26 Enhanced GitLab Functionality • Developing and supporting enhancements to GitLab applications to ensure targeted HPC testing resources can be available to code teams – Integrating facility feedback to improve technical aspects and support logistical challenges. • Runner capable of setuid, enforcing local permissions – Jobs are executed under a users local account • Batch executor added to interface with Cobalt, LSF, and Slurm – A runner side configuration that dictates how a CI job will be executed. • Federated cross site continuous integration – Previously auth endpoints only used at login, now they have been expanded and integrated into the CI process as well. • Existing GitLab functionality will continue to be supported – For instance continued upstream development efforts and improved to GitLab’s core CI/CD capabilities can be used with HPC focused enhancements 27 Batch Executor • Execute the user defined script (.gitlab-ci.yml) at the job level by submitting it the underlying scheduler – Comparable function to “Shell executor” and leverages setuid functionality – 1 Job = 1 Submission • Able to provide parameters for the job similar to a command line interface – SCHEDULER_PARAMETERS • Simply leverage the underlying Variable used to pass job submission argument scheduler Follows: https://docs.gitlab.com/ee/ci/variables/#priority-of-variables 28 Batch Executor with Setuid Enabled .gitlab-ci.yml Job output (GitLab web UI) 29 Batch Executor - Behavior Cobalt LSF Slurm Script Location where qrun Location where brun Location where srun ok? Element script will run? ok? script will run? ok? script will run? before_script launch/mom node yes launch node yes batch node yes script launch/mom node yes launch node yes batch node yes after_script login node no login node no login node no “Scripts specified in before_script are concatenated with any scripts specified in the main script, and executed together in a single shell.” - GitLab documentation 30 What is the GitLab Runner Doing? • The GitLab runner polls the server to identify available CI jobs – Will only begin executing a job once it has been scheduled by the server • Job local to runner result in generated Bash scripts based upon the contents of the .gitlab-ci.yml – Works similar to upstream GitLab (https://docs.gitlab.com/runner/shells/index.html) – Scripts are executed by piping them into a non-interactive Bash login shell. • Every step in a CI job is executed by a valid local user account – This is a key change with setuid enabled runner • Each user is provided with a clean login shell – Ensures runner environment does not compromise subsequent jobs – Users experience an environment similar to what they would see on a login node – Depending on your environment this may affect the results of your CI 31 Setuid and GitLab Server Requirements • Setuid feature enables GitLab to run bare-metal CI jobs securely on large, multi-tenant systems – All CI job actions are executed under the local user’s account with appropriate privileges observed • Running a setuid enabled runner requires: – Root privileges - The runners require elevated privileges – Trust - GitLab server and the hosts of any setuid enabled runners trust each other – HTTPS