tsi-cc Documentation Release 1.0 David Yuan Mar 23, 2021 Contents 1 Table of contents 3 2 ResOps Training 5 3 Recommended Tools 7 4 EBI Cloud Portal 9 5 Architectural Design Patterns 11 6 About this documentation 13 7 Contact us 15 8 Site map 17 8.1 EMBL-EBI Cloud Portal documentation................................ 17 8.2 ResOps Training & Workshop - 2020.................................. 17 8.3 ResOps Training & Workshop - 2019.................................. 19 8.4 Creating VMs with Terraform on OSK for ResOps........................... 20 8.5 Docker exercises............................................. 23 8.6 Exercise 1: Running images....................................... 23 8.7 Exercise 2: Running a bioinfomatics application............................ 29 8.8 Exercise 3: Creating your own docker image.............................. 32 8.9 Exercise 4: Optimising a docker image................................. 39 8.10 Exercise 5: Using the ‘builder’ pattern to build small images...................... 41 8.11 Exercise 6: Running as root or a non-root user............................. 43 8.12 Exercise 7: Using metadata....................................... 46 8.13 Exercise 8: Removing images from your machine (optional)...................... 48 8.14 Gitlab exercises.............................................. 50 8.15 Exercise 1: Log into gitlab, set up your SSH key............................ 50 8.16 Exercise 2: Create a new project..................................... 56 8.17 Exercise 3: Download and run the docker image from your gitlab project............... 57 8.18 Exercise 4: Use git tags to create a named version of a docker image.................. 59 8.19 Exercise 5: Extend the pipeline by adding further steps......................... 60 8.20 Exercise 6: Change the order of the pipeline steps........................... 63 8.21 Exercise 7: Pass secrets to the build pipeline.............................. 66 8.22 Error-Tracking.............................................. 70 i 8.23 Gitlab to Github pipeline......................................... 71 8.24 Gitlab DevOps.............................................. 72 8.25 Tracing.................................................. 74 8.26 Auto DevOps tutorial........................................... 75 8.27 Additional considerations for research pipelines............................ 77 8.28 Kubernetes on OpenStack........................................ 81 8.29 Kubernetes Practical........................................... 97 8.30 Scaling up Kubernetes for research pipelines.............................. 113 8.31 Accessing GCP node from CLI..................................... 122 8.32 Arvados on Kubernetes......................................... 124 8.33 Cloud Consulting Team toolbox..................................... 127 8.34 DevOps toolchain from GitLab to Docker Hub for Container Build.................. 130 8.35 DevOps toolchain from Gitlab to OpenStack for pipelines on ECP................... 132 8.36 DevOps toolchain from IntelliJ to ReadtheDocs for publishing..................... 137 8.37 HPC with Azure CycleCloud...................................... 139 8.38 HPC with Slurm on GCP......................................... 142 8.39 Installing OpenStack CLI on Mac OS X................................. 143 8.40 Kubeflow for Machine Learning..................................... 146 8.41 Tips and tricks with Docker....................................... 152 8.42 Tips and tricks with Terraform...................................... 154 ii tsi-cc Documentation, Release 1.0 This site is maintained by the Cloud Consultancy Team at EBI Contents 1 tsi-cc Documentation, Release 1.0 2 Contents CHAPTER 1 Table of contents • ResOps Training - ResOps Training • Recommended Tools - a list of recommended tools, and to get the most out of them • EBI Cloud Portal - the EBI multi-cloud way of deploying applications • Architectural Design Patterns - how to build cloud-native applications • About this documentation • Contact us • Site map 3 tsi-cc Documentation, Release 1.0 4 Chapter 1. Table of contents CHAPTER 2 ResOps Training • Notice – Please check the course pre-requisite on the agenda page. • Agenda for ResOps 2020 • Agenda for ResOps 2019 5 tsi-cc Documentation, Release 1.0 6 Chapter 2. ResOps Training CHAPTER 3 Recommended Tools • Ansible: • Cloud Consultancy Team toolbox: – How to install tools • Common Workflow Language – CWL self-contained training course • Docker: – Docker best practice • GitLab: – Overview * How to create docker image * How to deploy ECP instance * How to publish documentation * GitLab best practice – Useful links * GitLab CI with Maven • Google Cloud Platform: – Accessing GCP node from CLI • HPC in the cloud: – HPC with Slurm on GCP – Azure CycleCloud * HPC with CycleCloud on Azure * Useful links: 7 tsi-cc Documentation, Release 1.0 · CycleCloud Overview · Setting up CycleCloud as a container • Kubernetes: – Deployment of Kubernetes cluster on various clouds – Kubeflow for Machine Learning – Useful links: • OneData: – Overview – Oneclient – User Guide – Onedata API – EBI setup details – EBI onedata portal • Terraform: – Best Practice • Supervisor: – Description: Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems. – Reference documentation 8 Chapter 3. Recommended Tools CHAPTER 4 EBI Cloud Portal • EMBL-EBI Cloud Portal documentation – Using the EMBL-EBI Cloud Portal – Packaging Applications for the EMBL-EBI Cloud Portal – Avoid security credentials on git public repository – API Endpoint documentation 9 tsi-cc Documentation, Release 1.0 10 Chapter 4. EBI Cloud Portal CHAPTER 5 Architectural Design Patterns • How to convert monolithic design to microservices 11 tsi-cc Documentation, Release 1.0 12 Chapter 5. Architectural Design Patterns CHAPTER 6 About this documentation This documentation is maintained at in a gitlab repository at EBI. Feel free to clone a local copy. You would need to run a local build to generate a local HTML site from the source files of *.md and *.rst. Install Sphinx with the instructions of Getting started with sphinx. Follow the instructions of Using markdown with sphinx and Local build with Sphinx to generate your local site. The master branch will always be the latest, most up-to-date version. Other versions may be available through the selection drop-down on the bottom-left corner of this page. If you find any mistakes, have suggestions for improvement, have questions, or wish to talk to us about your project, please contact us. Pull requests or issues filed against the git repository are also welcome. 13 tsi-cc Documentation, Release 1.0 14 Chapter 6. About this documentation CHAPTER 7 Contact us You can contact us by email or come find us in pod 14 of the South Building. 15 tsi-cc Documentation, Release 1.0 16 Chapter 7. Contact us CHAPTER 8 Site map 8.1 EMBL-EBI Cloud Portal documentation • EMBL-EBI Cloud Portal documentation – Using the EMBL-EBI Cloud Portal – Packaging Applications for the EMBL-EBI Cloud Portal – Avoid security credentials on git public repository – API Endpoint documentation • Presentations on UKRI 2019 – EMBL-EBI Cloud Portal for Everyone 8.2 ResOps Training & Workshop - 2020 8.2.1 Introduction ResOps training is mainly focus to research community who are willing to develop their cloud native skills. The ResOps course will not be delivered in-person for the foreseeable future, This course can be taken standalone, self- paced and for FREE. 8.2.2 Course objectives This workshop will provide some background to cloud computing and practical experience in building, deploying and running applications in cloud platforms - OpenStack, Google, Amazon and Azure. Using examples drawn from EMBL-EBI’s experiences in the life-sciences, but using tools and technologies that are cross-platform and cross-domain, attendees will come away with a knowledge as to the user-centric hybrid cloud 17 tsi-cc Documentation, Release 1.0 strategy as well as practical aspects of deploying across different clouds and the architectural considerations around porting applications to a cloud. 8.2.3 Prerequisite • An account should be created on Public GitLab CI/CD practical with GitLab. • We are providing sandbox over OpenStack cloud to play exercises. • Workshop participants can use Nano editor by default. Other CLI editors are also available in the sandbox. 8.2.4 Course videos The course videos are all available for download for self-paced study. The slides are linked to the talks in the agenda. 8.2.5 Course guidelines Date & Time (CEST) Duration Topic Monday 05 Oct 14:00 - 14:10 10 min Introduction 14:10 - 14:40 30 min Cloud 101 14:40 - 14:50 10 min EBI Cloud Services (for courses at EBI only) 14:40 - 15:20 40 min Porting apps into cloud [ Part 1| Part 2| Part 3] 15:20 - 16:20 60 min Creating Containers with Docker 16:20 - 16:30 10 min Break 16:30 - 18:00 90 min Docker Practicals Tuesday 06 Oct 14:00 - 15:00 60 min Introduction to Gitlab 15:00 - 16:30 90 min GitLab Practicals 16:30 - 16:45 15 min GitLab Auto DevOps 16:45 - 17:05 20 min Demo - GitLab Auto DevOps 17:05 - 18:00 55 min Q&A, continue with practicals Thursday 08 Oct 14:00 - 15:00 60 min Kubernetes 101 15:00 - 17:00 05 min Overview of Kubernetes (Demo) 115 min Kubernetes (Demo) 17:00 - 17:45 05 min Overview of K8S Practical 40 min Kubernetes Practical 17:45 - 17:50 05 min Overview of Advanced K8S Practical 17:50 - 17:55 05 min Summary Advanced Kubernetes Deployment and Deployment Strategies