DOCSLIB.ORG

Singularity Container Documentation Release 3.0

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Singularity Container Documentation Release 3.0 Singularity Container Documentation Release 3.0 User Docs Mar 25, 2019 CONTENTS 1 Quick Start 1 1.1 Quick Installation Steps.........................................1 1.1.1 Install system dependencies..................................1 1.1.2 Install Go............................................1 1.1.3 Clone the Singularity repository................................2 1.1.4 Install Go dependencies.....................................2 1.1.5 Compile the Singularity binary.................................2 1.2 Overview of the Singularity Interface..................................2 1.3 Download pre-built images........................................4 1.4 Interact with images...........................................5 1.4.1 Shell...............................................6 1.4.2 Executing Commands......................................6 1.4.3 Running a container.......................................7 1.4.4 Working with Files.......................................7 1.5 Build images from scratch........................................8 1.5.1 Sandbox Directories......................................8 1.5.2 Converting images from one format to another.........................8 1.5.3 Singularity Definition Files...................................9 2 Contributing 11 2.1 Join the community........................................... 11 2.1.1 Singularity Google Group................................... 11 2.1.2 Singularity on Slack...................................... 11 2.2 Raise an Issue.............................................. 11 2.3 Write Documentation.......................................... 11 2.4 Contribute to the code.......................................... 12 2.4.1 Step 1. Fork the repo...................................... 12 2.4.2 Step 2. Checkout a new branch................................. 12 2.4.3 Step 3. Make your changes................................... 12 2.4.4 Step 4. Push your branch to your fork............................. 13 2.4.5 Step 5. Submit a Pull Request................................. 13 2.4.6 Step 6. Keep your branch in sync................................ 13 3 Installation 15 3.1 Overview................................................. 15 3.2 Install on Linux.............................................. 15 3.2.1 Before you begin........................................ 15 3.2.1.1 Install Dependencies.................................. 15 3.2.1.2 Install Go....................................... 16 3.2.2 Install from source....................................... 16 i 3.2.2.1 Download Singularity repo (and optionally check out a tag or branch)........ 16 3.2.2.2 Download and install Singularity from a release................... 17 3.2.2.3 Compile Singularity.................................. 17 3.2.2.4 Source bash completion file.............................. 18 3.2.3 Build and install an RPM.................................... 18 3.2.4 Remove an old version..................................... 18 3.2.5 Distribution packages of Singularity.............................. 19 3.2.5.1 Install the Debian/Ubuntu package using apt .................... 19 3.2.5.2 Install the CentOS/RHEL package using yum .................... 20 3.3 Install on Windows or Mac........................................ 20 3.3.1 Setup.............................................. 21 3.3.1.1 Windows........................................ 21 3.3.1.2 Mac.......................................... 21 3.3.2 Singularity Vagrant Box.................................... 21 3.4 Singularity on a shared resource..................................... 22 4 Command Line Interface 25 5 Build a Container 27 5.1 Overview................................................. 27 5.2 Downloading an existing container from the Container Library..................... 27 5.3 Downloading an existing container from Docker Hub.......................... 28 5.4 Creating writable --sandbox directories............................... 28 5.5 Converting containers from one format to another............................ 28 5.6 Building containers from Singularity definition files.......................... 28 5.7 Build options............................................... 29 5.7.1 --builder .......................................... 29 5.7.2 --detached ......................................... 29 5.7.3 --force ............................................ 29 5.7.4 --json ............................................. 29 5.7.5 --library .......................................... 29 5.7.6 --notest ........................................... 30 5.7.7 --remote ........................................... 30 5.7.8 --sandbox .......................................... 30 5.7.9 --section .......................................... 30 5.7.10 --update ........................................... 30 5.8 More Build topics............................................ 30 6 Definition Files 31 6.1 Overview................................................. 31 6.2 Header.................................................. 31 6.3 Sections.................................................. 32 6.3.1 %setup............................................. 33 6.3.2 %files.............................................. 33 6.3.3 %environment.......................................... 34 6.3.4 %post.............................................. 34 6.3.5 %runscript............................................ 35 6.3.6 %startscript........................................... 35 6.3.7 %test.............................................. 36 6.3.8 %labels............................................. 36 6.3.9 %help.............................................. 37 6.4 Apps................................................... 37 6.5 Best Practices for Build Recipes..................................... 39 7 Build Environment 41 ii 7.1 Overview................................................. 41 7.2 Cache Folders.............................................. 41 7.3 Temporary Folders............................................ 41 7.4 Pull Folder................................................ 42 7.5 Environment Variables.......................................... 42 7.5.1 Defaults............................................. 42 7.5.1.1 Docker......................................... 42 7.5.1.2 Library......................................... 42 8 Support for Docker and OCI 43 8.1 Overview................................................. 43 8.2 Running action commands on public images from Docker Hub.................... 43 8.3 Making use of public images from Docker Hub............................. 46 8.4 Making use of private images from Docker Hub............................ 47 8.4.1 Authentication via Interactive Login.............................. 48 8.4.2 Authentication via Environment Variables........................... 48 8.5 Making use of private images from Private Registries.......................... 49 8.6 Building images for Singularity from Docker Registries........................ 50 8.6.1 Working from the Singularity Command Line......................... 50 8.6.1.1 Remotely Hosted Images............................... 50 8.6.1.2 Building Containers Remotely............................ 52 8.6.1.3 Locally Available Images: Cached by Docker.................... 53 8.6.1.4 Locally Available Images: Stored Archives...................... 54 8.6.1.5 Pushing Locally Available Images to a Library.................... 56 8.6.2 Working with Definition Files................................. 58 8.6.2.1 Mandatory Header Keywords: Remotely Boostrapped................ 58 8.6.2.2 Remotely Bootstrapped and Built Containers..................... 59 8.6.2.3 Mandatory Header Keywords: Locally Boostrapped................. 61 8.6.2.4 Optional Header Keywords.............................. 62 8.6.2.5 Private Images and Registries............................. 63 8.6.2.6 Directing Execution.................................. 63 8.6.2.7 Container Metadata.................................. 65 8.7 OCI Image Support............................................ 66 8.7.1 Overview............................................ 66 8.7.1.1 Image Pulls Revisited................................. 66 8.7.1.2 Image Caching in Singularity............................. 67 8.7.1.3 Compliance with the OCI Image Layout Specification................ 67 8.7.1.4 OCI Compliance and the Singularity Cache..................... 68 8.7.2 Building Containers for Singularity from OCI Images..................... 71 8.7.2.1 Working Locally from the Singularity Command Line: oci Boostrap Agent.... 71 8.7.2.2 Working Locally from the Singularity Command Line: oci-archive Boostrap Agent......................................... 72 8.7.2.3 Working from the Singularity Command Line with Remotely Hosted Images.... 74 8.7.2.4 Working with Definition Files: Mandatory Header Keywords............ 75 8.7.2.5 Working with Definition Files: Additonal Considerations.............. 76 8.8 Best Practices............................................... 77 8.9 Troubleshooting............................................. 79 9 Bind Paths and Mounts 81 9.1 Overview................................................. 81 9.2 System-defined bind paths........................................ 81 9.3 User-defined bind paths......................................... 81 9.3.1 Specifying bind paths...................................... 81 9.3.2 A note on using --bind with the --writable flag.................... 82 iii 10 Persistent Overlays 83 10.1 Overview................................................. 83 10.2 Usage..................................................
Load more

Recommended publications
  • Tutorials Point, Simply Easy Learning
    Tutorials Point, Simply Easy Learning UML Tutorial Tutorialspoint.com UNIX is a computer Operating System which is capable of handling activities from multiple users at the same time. Unix was originated around in 1969 at AT&T Bell Labs by Ken Thompson and Dennis Ritchie. This tutorial gives an initial push to start you with UNIX. For more detail kindly check tutorialspoint.com/unix What is Unix ? The UNIX operating system is a set of programs that act as a link between the computer and the user. The computer programs that allocate the system resources and coordinate all the details of the computer's internals is called the operating system or kernel. Users communicate with the kernel through a program known as the shell. The shell is a command line interpreter; it translates commands entered by the user and converts them into a language that is understood by the kernel. Unix was originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Douglas McIlroy, and Joe Ossanna. There are various Unix variants available in the market. Solaris Unix, AIX, UP Unix and BSD are few examples. Linux is also a flavour of Unix which is freely available. Several people can use a UNIX computer at the same time; hence UNIX is called a multiuser system. A user can also run multiple programs at the same time; hence UNIX is called multitasking. Unix Architecture: Here is a basic block diagram of a UNIX system: 1 | P a g e Tutorials Point, Simply Easy Learning The main concept that unites all versions of UNIX is the following four basics: Kernel: The kernel is the heart of the operating system.
    [Show full text]
  • Linux at 25 PETERHISTORY H
    Linux at 25 PETERHISTORY H. SALUS Peter H. Salus is the author of A n June 1991, at the USENIX conference in Nashville, BSD NET-2 was Quarter Century of UNIX (1994), announced. Two months later, on August 25, Linus Torvalds announced Casting the Net (1995), and The his new operating system on comp.os.minix. Today, Android, Google’s Daemon, the Gnu and the Penguin I (2008). [email protected] version of Linux, is used on over two billion smartphones and other appli- ances. In this article, I provide some history about the early years of Linux. Linus was born into the Swedish minority of Finland (about 5% of the five million Finns). He was a “math guy” throughout his schooling. Early on, he “inherited” a Commodore VIC- 20 (released in June 1980) from his grandfather; in 1987 he spent his savings on a Sinclair QL (released in January 1984, the “Quantum Leap,” with a Motorola 68008 running at 7.5 MHz and 128 kB of RAM, was intended for small businesses and the serious hobbyist). It ran Q-DOS, and it was what got Linus involved: One of the things I hated about the QL was that it had a read-only operating system. You couldn’t change things ... I bought a new assembler ... and an editor.... Both ... worked fine, but they were on the microdrives and couldn’t be put on the EEPROM. So I wrote my own editor and assembler and used them for all my programming. Both were written in assembly language, which is incredibly stupid by today’s standards.
    [Show full text]
  • Process Manager Ext2fs Proc Devfs Process Management Memory Manager Scheduler Minix Nfs Msdos Memory Management Signaling
    Embedded Systems Prof. Myung-Eui Lee (A-405) [email protected] Embedded Systems 1-1 KUT Linux Structure l Linux Structure proc1 proc2 proc3 proc4 proc5 procn User Space System Call Interface Filesystem Manager Process Manager Ext2fs proc devfs Process Management Memory Manager Scheduler minix nfs msdos Memory Management Signaling Buffer Cache Kernel Space Device Manager Network Manager char block Ipv6 ethernet Console KBD SCSI CD-ROM PCI network ….. Device Interface dev1 dev2 dev3 dev4 devn Embedded Systems 1-2 KUT Linux Structure l Linux Structure Applications: Graphics UI, Compilers, Media player UI Text UI = Shell: sh, csh, ksh, bash, tcsh, zsh Compiler libraries (libc.a) API System Shared Libraries (System Call Interface) Memory File Process management management management Kernel Device Drives BIOS Computer Hardware Embedded Systems 1-3 KUT System Call Interface Standard C Lib. System Call C program invoking printf() library call, API – System call – OS Relationship which calls write() system call ar –t /usr/lib/libc.a | grep printf /usr/src/linux-2.4/arch/i386/kernel/entry.S Embedded Systems 1-4 KUT System Call l System calls define the programmer interface to Linux system » Interface between user-level processes and hardware devices u CPU, memory, disks etc. » Make programming easier u Let kernel take care of hardware-specific issues » Increase system security u Let kernel check requested service via system call » Provide portability u Maintain interface but change functional implementation l Roughly five categories of system calls
    [Show full text]
  • Processes in Linux/Unix
    Processes in Linux/Unix A program/command when executed, a special instance is provided by the system to the process. This instance consists of all the services/resources that may be utilized by the process under execution. • Whenever a command is issued in unix/linux, it creates/starts a new process. For example, pwd when issued which is used to list the current directory location the user is in, a process starts. • Through a 5 digit ID number unix/linux keeps account of the processes, this number is call process id or pid. Each process in the system has a unique pid. • Used up pid’s can be used in again for a newer process since all the possible combinations are used. • At any point of time, no two processes with the same pid exist in the system because it is the pid that Unix uses to track each process. Initializing a process A process can be run in two ways: 1. Foreground Process : Every process when started runs in foreground by default, receives input from the keyboard and sends output to the screen. When issuing pwd command $ ls pwd Output: $ /home/geeksforgeeks/root When a command/process is running in the foreground and is taking a lot of time, no other processes can be run or started because the prompt would not be available until the program finishes processing and comes out. 2. Backround Process : It runs in the background without keyboard input and waits till keyboard input is required. Thus, other processes can be done in parallel with the process running in background since they do not have to wait for the previous process to be completed.
    [Show full text]
  • CENG251: Assignment #5
    Out: March 30, 2021 Due: Wed Apr 14, 2021 There are 30 marks. Please keep and submit a time log to account for what you’ve done. CENG251: Assignment #5 Part I: C Language Features (8 marks) In class we demonstrated re-entrant code by comparing the use of ctime vs ctime_r. We showed that ctime always returned the same memory location and that ctime_r avoided this problem by passing the memory to be filled which was then returned with a different result; Write a single small C program that 1. Demonstrates that getpwuid and getpwnam are not reentrant by calling each of these twice with different valid arguments and displaying the return address and the content of the passwd structure returned. Prepare your output string using fmemopen and then writing the output to a string using a file pointer before writing the string to the terminal. An example of using fmemopen can be found in the example program stringIODemo.c (3) 2. Continue the program and call getpwuid_r and getpwnam_r with correct arguments. Prepare your output string using sprintf and then display the string to the terminal. Examples of long structured format strings can be seen in the example program stringIODemo.c (3) 3. In ctimeDemo.c we demonstrated the use of strdup to preserve the result of ctime from one function call to the next. Use malloc and memcpy to do the same for getpwuid and show that this worked displaying the values from the copy. (2) Part II: Signals (8 marks) The following exercises should be done as a single program.
    [Show full text]
  • Unit-Iv Process,Signals and File Locking
    UNIT-IV PROCESS, SIGNALS AND FILE LOCKING III-I R14 - 14BT50502 PROCESS: When you execute a program on your UNIX system, the system creates a special environment for that program. This environment contains everything needed for the system to run the program as if no other program were running on the system. Whenever you issue a command in UNIX, it creates, or starts, a new process. When you tried out the ls command to list directory contents, you started a process. A process, in simple terms, is an instance of a running program. When you start a process (run a command), there are two ways you can run it − Foreground Processes Background Processes FOREGROUND PROCESSES: By default, every process that you start runs in the foreground. It gets its input from the keyboard and sends its output to the screen. Example: $ls ch*.doc When a program is running in foreground and taking much time, we cannot run any other commands (start any other processes) because prompt would not be available until program finishes its processing and comes out. BACKGROUND PROCESSES: A background process runs without being connected to your keyboard. If the background process requires any keyboard input, it waits. The advantage of running a process in the background is that you can run other commands; you do not have to wait until it completes to start another! The simplest way to start a background process is to add an ampersand (&) at the end of the command. Example: $ls ch*.doc & PROCESS IDENTIFIERS: The process identifier – normally referred to as the process ID or just PID, is a number used by most operating system kernels such as that of UNIX, Mac OS X or Microsoft Windows — to uniquely identify an active process.
    [Show full text]
  • IBM Delivers Red Hat Enterprise Linux for IBM Z and Linuxone to Run Linux on IBM Z and IBM Linuxone Servers
    IBM United States Software Announcement 220-135, dated May 5, 2020 IBM delivers Red Hat Enterprise Linux for IBM Z and LinuxONE to run Linux on IBM Z and IBM LinuxONE servers Table of contents 1 Overview 4 Technical information 2 Key requirements 5 Ordering information 2 Planned availability date 6 Terms and conditions 2 Description 7 Prices 3 Program number 7 Order now 4 Publications At a glance Red Hat Enterprise Linux (RHEL) for IBM Z and LinuxONE can now be ordered through the Entitled Software (ESW) configurator using Shopz to run Linux on IBM Z and LinuxONE. The Linux operating system benefits from the mainframe's capabilities and strengths, including: • Exceptional data security • Availability • Performance • Cost savings and efficiency RHEL is an open source operating system (OS), subject to the terms set forth by Red Hat, that provides a foundation to scale existing applications and deploy emerging technologies across the following environments: • Bare metal • Virtual • Container • Cloud RHEL has been tested and certified by IBM on IBM Z and IBM LinuxONE servers. Overview IBM Z and Red Hat have a long history of partnering to deliver enterprise-grade Linux computing. One of the first mainframe ports of Linux, Think Blue Linux, was based on Red Hat Linux 6.1. Over the past 20 years, this partnership has grown and has resulted in RHEL being an attractive operating system on IBM Z and LinuxONE. The combination of IBM Z and RHEL brings Linux applications to the mainframe and supports efficient management across Linux workloads. Many workloads that run on IBM Linux on Z and LinuxONE today can be deployed to drive high levels of consolidation.
    [Show full text]
  • Processes – CSC252
    Processes – CSC252 PROF. EARL Some slides adopted from Dr. John Carelli and Dr. Lisa Frye User • Programs that the user interacts with User vs or launches themselves Kernel Kernel • Provides services to user space. Space Manages hardware and resources for multiple users. User processes access this space through system calls Hardware Processes https://goo.gl/images/381BdN Processes A single process is the execution of command by the Linux Kernel Every process (besides init) has a Process ID (PID) and Parent Process ID (PPID) Processes are “spawned” from another System Calls Allow a program in user space to interact with the kernel fork() – Allows for a process to create a new sub-process (A child process) The PPID of the new process, would be the PID of the process that called fork() exec() – Execute a new command replace the process with the new one. Sessions The lifetime a user login process When a user logins into a linux machine, a new shell process is created job – A command pipeline ; - Separate Commands & - Run the command in the background Basic Process Commands history – log of past commands jobs – List of currently running jobs kill – Terminal a process fg,bg – Move a process to the foreground/background ps – List processes top – View top running processes uptime – View how long system has been running free – View available memory (RAM) pstree – Show process tree nohup – run command immune to hangups Review the student presentations as posted on D2L (except for ps) The Process Table Data structure that exists in Kernel Space Contains the following: Process ID Process Owner Process Priority Environment variables for each process The Parent Process Location of its code, data, stack, and user area Pending Signals The ps command UNIX command that allows you to view information from the process table.
    [Show full text]
  • Linux - Friheden Til at Vælge Installation
    Linux - Friheden til at vælge installation Version 5.8.20040526 - 2020-12-31 Peter Toft og mange andre Linux - Friheden til at vælge installationVersion 5.8.20040526 - 2020-12-31 af Peter Toft og og mange andre Ophavsret © 1998-2005 Forfatterne har ophavsret til bogen, men udgiver den under "Åben dokumentlicens (ÅDL) - version 1.0". Skrevet af mange Linux-brugere til nye brugere som vil hurtigt igang med Linux og tilhørende programmer. Indholdsfortegnelse Forord........................................................................................................................................................ix 1. Forord............................................................................................................................................ix 2. Linux-bøgerne...............................................................................................................................ix 3. Ophavsret.......................................................................................................................................x 4. Om forfatterne og bogens historie.................................................................................................xi 5. Vi siger tak for hjælpen............................................................................................................... xii 6. Typografi.....................................................................................................................................xiv 1. Hvad er Linux?......................................................................................................................................1
    [Show full text]
  • Tutorial 2 : Process Management
    Tutorial 2 : Process Management August 20, 2019 Objective : • This assignment is intended to learn how to create, work with and ma- nipulate processes in Linux. You are expected to refer to the text book and references mentioned in the course website befor you start the lab. Some sample codes for process creation using fork system call have been provided for your reference. Instructions • You are expected to run all the sample codes provided in the Helpful Re- sources section. It will help you understand how OS virtualizes CPU and memory. Some of the codes are related to process creation and execution using Unix based system calls such as fork,exit,wait and exec Tutorials 1. Tut1 : Memory Virtualization : Physical memory model presents Memory as an array of bytes. The data stored at an address of memory can be read by providing the address of the location wehere the data resides. Similarly, while writing data to any address of a memory, one needs to specify the address for writing as well as the data to be written to the given address. Memory is accessed all the time when a program is running. A program keeps all of its data structures in memory, and accesses them through various instruction. In- structions themeselves reside in the memory which have to be fetched. Instructions : Execute the code mem.c from the Helpful Resources sec- tion. Make sure to include the common.h file in the same folder as the mem.c. To execute use the following commands : gcc mem.c ./a.out ox200000 The program does a couple of things.
    [Show full text]
  • Exec(“/Bin/Ls”) : Overwrites the Memory Code Image with Binary from /Bin/Ls and Execute
    Assignment 1 Υπεύθυνοι βοηθοί: Ειρήνη Δέγκλερη ( [email protected] ) Ηλίας Παπαδόπουλος ( [email protected] ) System Calls ■ If a process is running a user program in user mode and needs a system service, such as reading data from a file, it has to execute a trap instruction to transfer control to the operating system. Application – USER SPACE System Call KERNEL SPACE System Calls A system call is a request for service that a program makes of the kernel. The service is generally something that only the kernel has the privilege to do, such as doing I/O. Fork() ■ Fork creates a new process (child process). – It creates an exact duplicate of the original process, including all the file descriptors, registers—everything. ■ The fork is called once, but returns twice! – After the fork, the original process and the copy (the parent and child) go their separate ways. – The fork call returns a value, which is zero in the child and equal to the child's process identifier or PID in the parent. ■ Now consider how fork is used by the shell. When a command is typed, the shell forks off a new process. This child process must execute the user command. Fork() – PID (Process IDentity) ■ pid < 0 → the creation of a child process was unsuccessful. ■ pid == 0 → the newly created child. ■ pid > 0 → the process ID of the child process passes to the parent. Consider a piece of program: #include <unistd.h> P1 fork() C1 pid_t pid = fork(); PID:28 PID:34 printf(“PID: %d\n”, pid); …. The parent will print: PID: 34 And the child will always print: PID: 0 Fork() #define TRUE 1 while (TRUE) { /* repeat forever */ type_prompt(); /* display prompt on the screen */ read_command(command, parameters); /* read input from terminal */ if (fork() != 0) { /* fork off child process */ /* Parent code.
    [Show full text]
  • Assignment 1
    Assignment 1 Teaching Assistant: Michalis Pachilakis ( [email protected]) System Calls If a process is running a user program in user mode and needs a system service, such as reading data from a file, it has to execute a trap instruction to transfer control the operating system. Application –USER SPACE System call KERNEL SPACE System Calls A system call is a request for service that a program makes of the kernel. The service is generally something that only the kernel has the privilege to do, such as doing I/O. SYSTEM CALLS PROCESS CONTROL fork(), wait(), exec(),exit() FILE open(), close(), MANIPULATION read(), write() DIRECTORIES mkdir(),rmdir(), MANAGEMENT mount(),link() OTHER chdir(),chmod(), kill(),time() Fork() Fork creates a new process(child process). It creates an exact duplicate of the original process, including all the file descriptors, registers etc. The fork is called once, but returns twice! After the fork, the original process and the copy(the parent and the child) go their separate ways. The fork call returns a value, which is zero in the child and equal to the child’s process identifier (PID) in the parent. Now consider how fork is used by the shell. When a command is typed, the shell forks off a new process. This child process must execute the user command. Fork() – PID (Process IDentity) pid < 0 the creation of a child process was unsuccessful. pid == 0 the newly created child. pid > 0 the process ID of the child process passes to the parent Consider the program: #include <unistd.h> P1 fork() C1 PID:28 PID:34 pid_t pid = fork(); printf(“PID:%d\n”,pid); … The parent will print: PID:34 The child will always print: PID:0 Fork() #define TRUE 1 while (TRUE) { /* repeat forever */ type_prompt(); /* display prompt on the screen */ read_command(command, parameters); /* read input from terminal */ if (fork() != 0) { /* fork off child process */ /* Parent code.
    [Show full text]