DOCSLIB.ORG

Containers in HPC (Summary)

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Containers in HPC (Summary) Presented By: Gregory M. Kurtzer HPC Systems Architect Lawrence Berkeley National Laboratory [email protected] CONTAINERS IN HPC WITH SINGULARITY CONTAINERS IN HPC: SINGULARITY A QUICK REVIEW OF THE LANDSCAPE ▸ Many types of virtualization solutions in industry but we will focus on Hardware and Operating System virtualization (a.k.a. “Containers”) ▸ Virtualization solutions generally focus on network services, isolation and resource efficiency ▸ Hardware virtualization has set the stage for the current understanding and usage models of containers (service virtualization, isolation, usage model, etc.) ▸ People have been discussing and talking about Docker in HPC, but it has yet to be implemented… Why? (and no, NERSC’s Shifter is not Docker) CONTAINERS IN HPC: SINGULARITY CONTAINER ARCHITECTURE Applications which run in a container run with the same “distance” to the host kernel and hardware as natively running applications. Generally the container daemon is a root owned daemon which will separate out all possible namespaces in order to achieve a fully emulated separation from host and other containers. CONTAINERS IN HPC: SINGULARITY COMPARING TO VIRTUAL MACHINES As a comparison to Virtual Machines, applications running inside a VM are not only further away from the physical hardware but the layers they must traverse are redundant. Additionally there is a layer of emulation which contributes to a performance regression. Containers are thus more efficient. CONTAINERS IN HPC: SINGULARITY CONTAINER LIMITATIONS ▸ Architecture compatibility: always limited to CPU architecture (e.g. x86/x86_64 != ARM) and binary format (ELF) - (you could add a level of hardware virtualization) ▸ Portability: containers in general are portable, but there are limitations ▸ Glibc / Kernel compatibility support ▸ Any other kernel / user land API compatibility (e.g. OFED) ▸ Bitrot: containers are subject to stagnation just as any operating system is ▸ Performance: there “may be” a slight theoretical performance penalty with utilizing kernel namespaces ▸ File paths: the file system is different inside the container than outside the container CONTAINERS IN HPC: SINGULARITY WELCOME TO DOCKER ▸ Docker is the most commonly used solution when considering Linux containers today ▸ Docker provides tools, utilities, and wrappers to easily create, maintain and distribute container images ▸ The primary container use case is targeted for network service virtualization (where it has gained popularity very fast) ▸ Computational developers love Docker because they can easily create reproducible and portable images containing their work CONTAINERS IN HPC: SINGULARITY SO WHY NOT JUST IMPLEMENT DOCKER ON HPC RESOURCES? CONTAINERS IN HPC: SINGULARITY DOCKER IN HPC: THE PROBLEM ▸ Docker emulates a virtual machine in many aspects (e.g. users can escalate to root) ▸ Non-authorized users having root access to any of our production networks is considered a security breech ▸ To mitigate this security issue, networks must be isolated for Docker access and thus will preclude access to InfiniBand high performance networks and optimized storage platforms ▸ People then build a virtual cluster within a cluster trying to navigate a solution ▸ Additional complexities arise with existing scheduling architectures and workflows which greatly complicate usage models and/or system architecture (especially with parallel MPI job execution) CONTAINERS IN HPC: SINGULARITY LET’S RE-QUANTIFY THE NEED ▸ The need: Mobility of Compute ▸ Portable source code is so 1990’s, now-a-days it is easier to just distribute a virtual machine or laptop hard drive image ▸ Ease of development: Users and developers want to work in familiar environments that they control and then run that exact code, pipeline or stack anywhere ▸ It must be easily integratable with existing complicated shared infrastructures and scheduling subsystems ▸ It must not create or encourage additional exploit vectors or escalation pathways which could be used to compromise the integrity of the system CONTAINERS IN HPC: SINGULARITY AND NOW FOR AN ENTIRELY DIFFERENT APPROACH. CONTAINERS IN HPC: SINGULARITY SINGULARITY ▸ Container technologies can be utilized to support “Mobility of Compute” and making workflows and applications portable ▸ Container images facilitates modification, distribution, and execution (e.g. OS X packages) ▸ With no direct means to obtain root/superuser within the container, there are no additional security risks thus separation between the container and the host can be blurred (makes efficient use of existing HPC resources, workflows and parallel MPI jobs) ▸ We don’t have to virtualize everything, we can pick and choose on a case by case basis and thus blur the line between container and host ▸ No architectural or workflow changes are necessary to run Singularity containers CONTAINERS IN HPC: SINGULARITY SINGULARITY: ARCHITECTURE The “Singularity Launcher” is run by the user, loads the Singularity container and executes applications as the user. CONTAINERS IN HPC: SINGULARITY SINGULARITY: BOOTSTRAP DEFINITION ▸ The process of building a Singularity container is simple and makes use of standard included distribution tools (e.g. Apt and YUM) ▸ Two steps to build a container from scratch (bootstrapping): ▸ Create the image ▸ Bootstrap the container using a definition file ▸ note: These two steps maybe merged so that the image details can be part of the bootstrap definition CONTAINERS IN HPC: SINGULARITY SINGULARITY: BUILDING THE CONTAINER ▸ Singularity utilizes a definition file to bootstrap a new container ▸ You can include packages, binaries, scripts, libraries, data, and configurations into the definition file ▸ Bootstrap command installs a minimal version of the distribution ▸ Containers can be much smaller then a bootable operating system ▸ The output of the build process is a Singularity container image which can be modified with changes persisted within the image in real time CONTAINERS IN HPC: SINGULARITY SINGULARITY: PERMISSIONS, ACCESS AND PRIVILEGE ▸ User contexts are always maintained when the container is launched (if it is launched by a user, it will always be that user) with no escalation pathway within the container! Thus…. If you want to be root inside of the container, you must first be root outside of the container CONTAINERS IN HPC: SINGULARITY SINGULARITY: LAUNCHING A CONTAINER ▸ When the Singularity container is launched, the namespaces are created and processes will be contained within the image ▸ By default the container image is accessed as read only ▸ Even when the container is mounted as read-write, root still maybe required to make changes (just like on any Linux system) ▸ All expected IO (this includes pipes, program arguments, stdout, stdin, stderr and X11) will all be passed directly through the container ▸ Home directory and other writable directories are automatically shared between the container and the host CONTAINERS IN HPC: SINGULARITY SINGULARITY: WORKFLOWS AND PIPELINES ▸ Using the RunScript bootstrap directive, you can have a container execute a custom pipeline or workflow ▸ You can easily obtain the arguments via the RunScript using standard shell syntax ▸ The runscript exists within the container at ‘/singularity’ ▸ Programs within the container can access data that exists within the container or outside of the container ▸ Singularity containers are a portable and lightweight means to distribute these workflows CONTAINERS IN HPC: SINGULARITY SINGULARITY: IMAGE MANAGEMENT ▸ Singularity does not separate the user Linux namespace ▸ To make changes that require root inside the container requires that you have root access outside the container ▸ You will also need to use the -w/—writable option to make the container writable (without this, not even root can make changes) ▸ Once inside the container, use the system as you would normally CONTAINERS IN HPC: SINGULARITY HPC - SINGULARITY AND OPEN MPI CONTAINERS IN HPC: SINGULARITY SINGULARITY AND OPEN MPI ▸ Open MPI utilizes a parent/children relationship and communication pathway ▸ The parent MPI process (Orted) fork/execs the children processes ▸ Communication from parent to children occurs over the Process Management Interface (PMI) ▸ Singularity becomes the interface to exec the children ▸ Communication between the inside and outside of the container occurs over the PMI interface ▸ This architecture allows for optimal use of the hosts interconnect and storage ▸ So far there has been very little perceived performance penalty (but theoretically there maybe some due to Linux namespace overhead) CONTAINERS IN HPC: SINGULARITY SINGULARITY AND OPEN MPI ‣ Starting with your Bash Shell or resource manager… ‣ MPI run gets executed which forks an orted process ‣ Orted launches Singularity which starts the container process ‣ The MPI application within the container is linked to the Open MPI runtime libraries within the container ‣ The Open MPI runtime libraries then connect and communicate back to the Orted process via a universal PMI CONTAINERS IN HPC: SINGULARITY SINGULARITY: THE ACTUAL OUTPUT Launching `mpirun` on the host runs the Orted process in the background and it then forks and exec’s the given Singularity command. Singularity then exec’s `./ring` which loads and communicates back to the parent Orted process on the host via PMI. CONTAINERS IN HPC: SINGULARITY SHARED MEMORY LATENCY IN OPEN MPI 1800 This is an example shared memory latency graph 1600 comparing the same 1400 application in Singularity 1200 container (orange) and 1000 running directly on the host. 800 You can see both are 600 performing closely and only 400 diverge on very large 200 0 messages. 0 500000 1000000 1500000 2000000 2500000 3000000 3500000 4000000 4500000 Presented By: Gregory M. Kurtzer HPC Systems Architect Lawrence Berkeley National Laboratory [email protected] CONTAINERS IN HPC WITH SINGULARITY.
Load more

Recommended publications
  • Industrial Control Via Application Containers: Migrating from Bare-Metal to IAAS
    Industrial Control via Application Containers: Migrating from Bare-Metal to IAAS Florian Hofer, Student Member, IEEE Martin A. Sehr Antonio Iannopollo, Member, IEEE Faculty of Computer Science Corporate Technology EECS Department Free University of Bolzano-Bozen Siemens Corporation University of California Bolzano, Italy Berkeley, CA 94704, USA Berkeley, CA 94720, USA fl[email protected] [email protected] [email protected] Ines Ugalde Alberto Sangiovanni-Vincentelli, Fellow, IEEE Barbara Russo Corporate Technology EECS Department Faculty of Computer Science Siemens Corporation University of California Free University of Bolzano-Bozen Berkeley, CA 94704, USA Berkeley, CA 94720, USA Bolzano, Italy [email protected] [email protected] [email protected] Abstract—We explore the challenges and opportunities of control design full authority over the environment in which shifting industrial control software from dedicated hardware to its software will run, it is not straightforward to determine bare-metal servers or cloud computing platforms using off the under what conditions the software can be executed on cloud shelf technologies. In particular, we demonstrate that executing time-critical applications on cloud platforms is viable based on computing platforms due to resource virtualization. Yet, we a series of dedicated latency tests targeting relevant real-time believe that the principles of Industry 4.0 present a unique configurations. opportunity to explore complementing traditional automation Index Terms—Industrial Control Systems, Real-Time, IAAS, components with a novel control architecture [3]. Containers, Determinism We believe that modern virtualization techniques such as application containerization [3]–[5] are essential for adequate I. INTRODUCTION utilization of cloud computing resources in industrial con- Emerging technologies such as the Internet of Things and trol systems.
    [Show full text]
  • Portability: Containers, Cloud
    JEDI Portability Across Platforms Containers, Cloud Computing, and HPC Mark Miesch, Rahul Mahajan, Xin Zhang, David Hahn, Francois Vandenberg, Jim Rosinski, Dan Holdaway, Yannick Tremolet, Maryam Abdioskouei, Steve Herbener, Mark Olah, Benjamin Menetrier, Anna Shlyaeva, Clementine Gas Academy website http://academy.jcsda.org/june2019 ‣ Instructions for accessing AWS ‣ Activity instructions ‣ Presentation slides ‣ Doxygen documentation for fv3-bundle We will add further content throughout the week Outline I) JEDI Portability Overview ✦ Unified vision for software development and distribution II) Container Fundamentals ✦ What are they? How do they work? ✦ Docker, Charliecloud, and Singularity III) Using the JEDI Containers ✦ How they are built and deployed ✦ Mac and Windows (Vagrant) IV) HPC and Cloud Computing ✦ Environment modules ✦ Containers in HPC? V) Summary and Outlook JEDI Software Dependencies ‣ Essential ✦ Compilers, MPI ✦ CMake Common versions among users ✦ SZIP, ZLIB and developers minimize ✦ LAPACK / MKL, Eigen 3 stack-related debugging ✦ NetCDF4, HDF5 ✦ udunits ✦ Boost (headers only) ✦ ecbuild, eckit, fckit ‣ Useful ✦ ODB-API, eccodes ✦ PNETCDF ✦ Parallel IO ✦ nccmp, NCO ✦ Python tools (py-ncepbufr, netcdf4, matplotlib…) ✦ NCEP libs ✦ Debuggers & Profilers (ddt/TotalView, kdbg, valgrind, TAU…) The JEDI Portability Vision I want to run JEDI on… Development ‣ My Laptop/Workstation/PC ✦ We provide software containers ✦ Mac & Windows system need to first establish a linux environment (e.g. a Vagrant/VirtualBox virtual machine) Development
    [Show full text]
  • The Component Architecture of Open Mpi: Enabling Third-Party Collective Algorithms∗
    THE COMPONENT ARCHITECTURE OF OPEN MPI: ENABLING THIRD-PARTY COLLECTIVE ALGORITHMS∗ Jeffrey M. Squyres and Andrew Lumsdaine Open Systems Laboratory, Indiana University Bloomington, Indiana, USA [email protected] [email protected] Abstract As large-scale clusters become more distributed and heterogeneous, significant research interest has emerged in optimizing MPI collective operations because of the performance gains that can be realized. However, researchers wishing to develop new algorithms for MPI collective operations are typically faced with sig- nificant design, implementation, and logistical challenges. To address a number of needs in the MPI research community, Open MPI has been developed, a new MPI-2 implementation centered around a lightweight component architecture that provides a set of component frameworks for realizing collective algorithms, point-to-point communication, and other aspects of MPI implementations. In this paper, we focus on the collective algorithm component framework. The “coll” framework provides tools for researchers to easily design, implement, and exper- iment with new collective algorithms in the context of a production-quality MPI. Performance results with basic collective operations demonstrate that the com- ponent architecture of Open MPI does not introduce any performance penalty. Keywords: MPI implementation, Parallel computing, Component architecture, Collective algorithms, High performance 1. Introduction Although the performance of the MPI collective operations [6, 17] can be a large factor in the overall run-time of a parallel application, their optimiza- tion has not necessarily been a focus in some MPI implementations until re- ∗This work was supported by a grant from the Lilly Endowment and by National Science Foundation grant 0116050.
    [Show full text]
  • Introduction to Containers
    Introduction to Containers Martin Čuma Center for High Performance Computing University of Utah [email protected] Overview • Why do we want to use containers? • Containers basics • Run a pre-made container • Build and deploy a container • Containers for complex software 06-Nov-20 http://www.chpc.utah.edu Slide 2 Hands on setup 1. Download the talk slides http://home.chpc.utah.edu/~mcuma/chpc/Containers20s.pdf https://tinyurl.com/yd2xtv5d 2. Using FastX or Putty, ssh to any CHPC Linux machine, e.g. $ ssh [email protected] 3. Load the Singularity and modules $ module load singularity 06-Nov-20 http://www.chpc.utah.edu Slide 3 Hands on setup for building containers 1. Create a GitHub account if you don’t have one https://github.com/join {Remember your username and password!} 2. Go to https://cloud.sylabs.io/home click Remote Builder, then click Sign in to Sylabs and then Sign in with GitHub, using your GitHub account 3. Go to https://cloud.sylabs.io/builder click on your user name (upper right corner), select Access Tokens, write token name, click Create a New Access Token, and copy it 4. In the terminal on frisco, install it to your ~/.singularity/sylabs-token file nano ~/.singularity/sylabs-token, paste, ctrl-x to save 06-Nov-20 http://www.chpc.utah.edu Slide 4 Why to use containers? 06-Nov-20 http://www.chpc.utah.edu Slide 5 Software dependencies • Some programs require complex software environments – OS type and versions – Drivers – Compiler type and versions – Software dependencies • glibc, stdlibc++ versions • Other libraries
    [Show full text]
  • Copy, Rip, Burn : the Politics of Copyleft and Open Source
    Copy, Rip, Burn Berry 00 pre i 5/8/08 12:05:39 Berry 00 pre ii 5/8/08 12:05:39 Copy, Rip, Burn The Politics of Copyleft and Open Source DAVID M. BERRY PLUTO PRESS www.plutobooks.com Berry 00 pre iii 5/8/08 12:05:39 First published 2008 by Pluto Press 345 Archway Road, London N6 5AA www.plutobooks.com Copyright © David M. Berry 2008 The right of David M. Berry to be identifi ed as the author of this work has been asserted by him in accordance with the Copyright, Designs and Patents Act 1988. British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 978 0 7453 2415 9 Hardback ISBN 978 0 7453 2414 2 Paperback Library of Congress Cataloging in Publication Data applied for This book is printed on paper suitable for recycling and made from fully managed and sustained forest sources. Logging, pulping and manufacturing processes are expected to conform to the environmental standards of the country of origin. The paper may contain up to 70% post consumer waste. 10 9 8 7 6 5 4 3 2 1 Designed and produced for Pluto Press by Chase Publishing Services Ltd, Sidmouth, EX10 9QG, England Typeset from disk by Stanford DTP Services, Northampton Printed and bound in the European Union by CPI Antony Rowe, Chippenham and Eastbourne Berry 00 pre iv 5/8/08 12:05:41 CONTENTS Acknowledgements ix Preface x 1. The Canary in the Mine 1 2. The Information Society 41 3.
    [Show full text]
  • Infiniband and 10-Gigabit Ethernet for Dummies
    The MVAPICH2 Project: Latest Developments and Plans Towards Exascale Computing Presentation at OSU Booth (SC ‘19) by Hari Subramoni The Ohio State University E-mail: [email protected] http://www.cse.ohio-state.edu/~subramon Drivers of Modern HPC Cluster Architectures High Performance Accelerators / Coprocessors Interconnects - InfiniBand high compute density, high Multi-core <1usec latency, 200Gbps performance/watt SSD, NVMe-SSD, Processors Bandwidth> >1 TFlop DP on a chip NVRAM • Multi-core/many-core technologies • Remote Direct Memory Access (RDMA)-enabled networking (InfiniBand and RoCE) • Solid State Drives (SSDs), Non-Volatile Random-Access Memory (NVRAM), NVMe-SSD • Accelerators (NVIDIA GPGPUs and Intel Xeon Phi) • Available on HPC Clouds, e.g., Amazon EC2, NSF Chameleon, Microsoft Azure, etc. NetworkSummit Based Computing Sierra Sunway K - Laboratory OSU Booth SC’19TaihuLight Computer 2 Parallel Programming Models Overview P1 P2 P3 P1 P2 P3 P1 P2 P3 Memo Memo Memo Logical shared memory Memor Memor Memor Shared Memory ry ry ry y y y Shared Memory Model Distributed Memory Model Partitioned Global Address Space (PGAS) SHMEM, DSM MPI (Message Passing Interface)Global Arrays, UPC, Chapel, X10, CAF, … • Programming models provide abstract machine models • Models can be mapped on different types of systems – e.g. Distributed Shared Memory (DSM), MPI within a node, etc. • PGAS models and Hybrid MPI+PGAS models are Network Based Computing Laboratorygradually receiving importanceOSU Booth SC’19 3 Designing Communication Libraries for Multi-Petaflop and Exaflop Systems: Challenges Co-DesignCo-Design Application Kernels/Applications OpportuniOpportuni tiesties andand Middleware ChallengeChallenge ss acrossacross Programming Models VariousVarious MPI, PGAS (UPC, Global Arrays, OpenSHMEM), LayersLayers CUDA, OpenMP, OpenACC, Cilk, Hadoop (MapReduce), Spark (RDD, DAG), etc.
    [Show full text]
  • Singularity Container Documentation Release 3.6
    Singularity Container Documentation Release 3.6 User Docs Oct 13, 2020 CONTENTS 1 Getting Started & Background Information3 1.1 Introduction to Singularity........................................3 1.2 Quick Start................................................5 1.3 Security in Singularity.......................................... 15 2 Building Containers 19 2.1 Build a Container............................................. 19 2.2 Definition Files.............................................. 23 2.3 Build Environment............................................ 33 2.4 Support for Docker and OCI....................................... 37 2.5 Fakeroot feature............................................. 78 3 Signing & Encryption 81 3.1 Signing and Verifying Containers.................................... 81 3.2 Key commands.............................................. 86 3.3 Encrypted Containers.......................................... 88 4 Sharing & Online Services 93 4.1 Remote Endpoints............................................ 93 4.2 Cloud Library.............................................. 96 5 Advanced Usage 101 5.1 Bind Paths and Mounts.......................................... 101 5.2 Persistent Overlays............................................ 104 5.3 Running Services............................................. 107 5.4 Environment and Metadata........................................ 118 5.5 OCI Runtime Support.......................................... 128 5.6 Plugins.................................................. 143 5.7
    [Show full text]
  • Cgroups, Containers and Htcondor, Oh My
    Cgroups, containers and HTCondor, oh my Center for High Throughput Computing Outline › Why put contain jobs? › Ersatz HTCondor containment › Docker containers › Singularity containers 2 3 Protections 1) Protect the machine from the job. 2) Protect the job from the machine. 3) Protect one job from another. 3 The ideal container › Allows nesting › Need not require root › Can’t be broken out of › Portable to all OSes › Allows full management: Creation // Destruction Monitoring Limiting 4 Resources a job can (ab)use CPU Memory Disk Network Signals L1-2-3 cache 5 HTCondor’s containment 6 PID namespaces › You can’t kill what you can’t see › Requirements: RHEL 6 or later USE_PID_NAMESPACES = true • (off by default) Must be root 7 PID Namespaces Init (1) Master (pid 15) Startd (pid 26) Starter (pid 73) Starter (pid 39) Condor_init (pid 1) Condor_init (pid 1) Job A (pid 2) Job B (pid 2) 8 MOUNT_UNDER_SCRATCH › Or, “Shared subtrees” › Goal: protect /tmp from shared jobs › Requires Condor 8.0+ RHEL 5 HTCondor must be running as root MOUNT_UNDER_SCRATCH = /tmp,/var/tmp 9 MOUNT_UNDER_SCRATCH MOUNT_UNDER_SCRATCH=/tmp,/var/tmp Each job sees private /tmp, /var/tmp Downsides: No sharing of files in /tmp 10 Control Groups aka “cgroups” › Two basic kernel abstractions: 1) nested groups of processes 2) “controllers” which limit resources 11 Control Cgroup setup › Implemented as filesystem Mounted on /sys/fs/cgroup, Groups are per controller • E.g. /sys/fs/cgroup/memory/my_group • /sys/fs/cgroup/cpu/my_group Interesting contents of virtual groups: • /sys/fs/cgroup/memory/my_group/tasks
    [Show full text]
  • User Manual Revision: C96e096
    Bright Cluster Manager 8.1 User Manual Revision: c96e096 Date: Fri Sep 14 2018 ©2018 Bright Computing, Inc. All Rights Reserved. This manual or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Bright Computing, Inc. Trademarks Linux is a registered trademark of Linus Torvalds. PathScale is a registered trademark of Cray, Inc. Red Hat and all Red Hat-based trademarks are trademarks or registered trademarks of Red Hat, Inc. SUSE is a registered trademark of Novell, Inc. PGI is a registered trademark of NVIDIA Corporation. FLEXlm is a registered trademark of Flexera Software, Inc. PBS Professional, PBS Pro, and Green Provisioning are trademarks of Altair Engineering, Inc. All other trademarks are the property of their respective owners. Rights and Restrictions All statements, specifications, recommendations, and technical information contained herein are current or planned as of the date of publication of this document. They are reliable as of the time of this writing and are presented without warranty of any kind, expressed or implied. Bright Computing, Inc. shall not be liable for technical or editorial errors or omissions which may occur in this document. Bright Computing, Inc. shall not be liable for any damages resulting from the use of this document. Limitation of Liability and Damages Pertaining to Bright Computing, Inc. The Bright Cluster Manager product principally consists of free software that is licensed by the Linux authors free of charge. Bright Computing, Inc. shall have no liability nor will Bright Computing, Inc. provide any warranty for the Bright Cluster Manager to the extent that is permitted by law.
    [Show full text]
  • Easy-Build Singularity Containers
    Easy-Build Singularity Containers Jörg Saßmannshausen Guy's and St Thomas' NHS Foundation Trust and King's College London, UK London, 26.1.2021 Jörg Saßmannshausen Easy-Build Singularity Containers Biography Biography: 1983-1986: apprenticeship as laboratory assistant of Chemistry (Germany) 1993-1994: MSc in Chemistry, University of East Anglia (UK) 1995-1997: PhD in Chemistry, University of Leeds (UK) 1998-2010: various post-doctoral positions in UK, Germany and Austria 2010: venia legendi in Inorganic Chemistry, Technical University of Graz (Austria) 2010: Change from daytime chemist to HPC, gentleman scientist now 2010-2019: Various positions at University College London, Francis Crick Institute (UK) 2019-current: Guy's and St. Thomas' Biomedical Research Centre in London (UK) London, 26.1.2021 Jörg Saßmannshausen Easy-Build Singularity Containers Introduction What is science? In science, we are doing experiments that are reproducible. London, 26.1.2021 Jörg Saßmannshausen Easy-Build Singularity Containers Introduction What is science? In science, we are doing experiments that are reproducible. In the laboratory, we repeat an analysis three times to iron out any random errors. London, 26.1.2021 Jörg Saßmannshausen Easy-Build Singularity Containers Introduction What is science? In science, we are doing experiments that are reproducible. In the laboratory, we repeat an analysis three times to iron out any random errors. On the HPC cluster, we only run one calculation, often for a long time, but we do not repeat this run. Why? London, 26.1.2021 Jörg Saßmannshausen Easy-Build Singularity Containers Introduction What is science? In science, we are doing experiments that are reproducible.
    [Show full text]
  • Using Singularity Containers in Virtual Machines and HPC
    Using Singularity Containers in Virtual Machines and HPC Venkat Mahadevan UBC Advanced Research Computing (ARC) 5/26/2021 Virtualization Containers Containers • Containers provide an additional layer of abstraction over virtualization. Applications and Libraries Container Engine OS Infrastructure Containers • However, containers do not require virtual machines and can happily run on bare-metal. • Lightweight compared to VMs in terms of memory and storage requirements. • Increases portability of applications between datacenters, private clouds, and public clouds. Containers • Can provide a very customized environment specific to research use cases. • Commercial software that requires old, obsolete libraries or operating system versions. • Bottom line: use containers where it makes sense to your workflow. Why Use Containers • Reproducibility. • Portability. • Isolation. • Avoiding complexity aka dependency hell. Docker • The advent of Docker resulted in widespread adoption of container technology. • Uses cgroups, kernel namespaces, OverlayFS (union filesystem) to allow containers to run in a singular Linux instance. • Very mature technology with ~7 million DockerHub users. Docker Singularity • However, Docker is not a perfect fit in the advanced research computing space. • The Docker daemon needs to run in the background with elevated privileges on every node (in the datacenter, private/public Cloud VM, etc.) that hosts containers. • This is a security concern which can be mitigated but limits some features. Singularity • Docker also does not provide close integration with the standard HPC software stack such as MPI as well as schedulers and resource managers such as Slurm & PBS Pro. • Container orchestration technologies such as Kubernetes can replicate some of the aforementioned functionality but the existing HPC stack is well entrenched and robust.
    [Show full text]
  • Setting up a Beowulf Cluster Using Open MPI on Linux
    Setting up a Beowulf Cluster Using Open MPI on Linux https://techtinkering.com/2009/12/02/setting-up-a-beowulf-c... TechTinkering ============= Twitter YouTube Email GitHub RSS Setting up a Beowulf Cluster Using Open MPI on Linux 2 December 2009 Lawrence Woodman #Beowulf Clusters #Distributed Processing #High Performance Computing #Linux #MPI #Parallel Processing I have been doing a lot of work recently on Linear Genetic Programming. This requires a great deal of processing power and to meet this I have been using Open MPI to create a Linux cluster. What follows is a quick guide to getting a cluster running. The basics really are very simple and, depending on the size, you can get a simple cluster running in less than half an hour, assuming you already have the machines networked and running Linux. What is a Beowulf Cluster? A Beowulf Cluster is a collection of privately networked computers which can be used for parallel computations. They consist of commodity hardware running open source software such as Linux or a BSD, often coupled with PVM (Parallel Virtual Machine) or MPI (Message Passing Interface). A standard set up will consist of one master node which will control a number of slave nodes. The slave nodes are typically headless and generally all access the same files from a server. They have been referred to as Beowulf Clusters since before 1998 when the original Beowulf HOWTO was released. What is Open MPI? Open MPI is one of the leading MPI-2 implementations used by many of the TOP 500 Supercomputers. It is open source and is developed and maintained by a consortium of academic, research, and industry partners.
    [Show full text]