Status of 휇CernVM

D Berzano, J Blomer, P Buncic, I Charalampidis, G Ganis, G Lestaris [email protected]

CERN PH-SFT

1 / 12 Introduction

Production CernVM: • 2.X series, 2.7.0 ready to be released • Scientific 5 based • Uses Conary • Versatile image: runs on practically all hypervisors runs all LHC experiment software frameworks can take different roles (desktop, batch, head, BOINC)

Next generation CernVM (휇CernVM): • Scientific Linux 6 based • Based on RPM • Drop remaining dependencies on rPath tools • Simplify the image building and distribution process Virtual appliance without a hard disk image

2 / 12 CernVM Building Blocks

Classic CernVM

XMPP CernVM Online CernVM Co-Pilot HTTP (Amazon EC2) cloud-init (in development)

Contextualization HEPIX CernVM-FS System Libraries & rAA Tools HTTP Cache Hierarchy

Fuse Kernel

• Uniform and portable environment for physics data processing • Minimal operating system derived from application dependencies • Easy to maintain and to distribute

3 / 12 CernVM Building Blocks

휇CernVM

XMPP CernVM Online CernVM Co-Pilot HTTP (Amazon EC2) cloud-init (in development)

OS + Extras

AUFS R/W Overlay HD Scratch

initrd: CernVM-FS + 휇Contextualization

AUFS Fuse Kernel ISO Image

Prototyped by Ioannis Charalampidis Idea: Operating system on CernVM-FS Instead of 400 MB hard disk image: 10 MB ISO image + 100 MB cache.

⇒ From “just enough operating system” to “operating system on demand”

3 / 12 휇CernVM Root File System Stack

• AUFS well-maintained kernel Read/Write Scratch Area module CernVM-FS Read-Only AUFS • < 5 % performance loss (untar) (Union File System)

Read/Write • Some use cases faster due to Interface CernVM-FS meta-data handling

• Root file system stack created by init script on ramdisk • Tweaks required to CernVM-FS: • Closing all read-write file descriptors in order to unraval file system stack on shutdown • Redirect syslog messages to a plain file • In-flight change of DNS server (tbd.) ⇒ This file system stack requires special support froma read-only Fuse branch in order to work.

4 / 12 휇CernVM Boot Process

CernVM Kernel: 3.7.10 (tbd: next long-term kernel) Features: KSM, zRam, THP, cgroups, X32-ABI Extra modules: AUFS, VMware drivers, VBox drivers, OpenAFS

“Virtualization-friendly”, minimal set of device drivers: 100 modules / 8 MB as compared to 2 000 modules / 120 MB in SL6

1 SYSLINUX boot loader 2 Decompress and load Linux kernel 3 Decompress init ramdisk, execute /init a) Start networking b) Contextualize (tbd.) c) [Partition and] [format and] mount scratch space d) Mount CernVM-FS e) Mount AUFS root file system stack

5 / 12 Booting 휇CernVM

6 / 12 Scientific Linux on CernVM-FS

Naive approach: install packages with into CernVM-FS chroot Problem: package managers are not designed to preserve an environemnt

• Scientific Linux and EPEL repositories contain newest packages The same command at a different point in time leads to a different environment • Bootstrapping an entire system is time-consuming, incremental updates depend on the order of packages • Little support for special rules Such as: use a specific package version, prefer packages from a specific repository • Dependency resolution is heuristic (not optimal)

Idea: fully versioned meta RPM “closure” (dependencies resolved upfront), mirror of used packages from upstream repository

7 / 12 Build Process: Scientific Linux on CernVM-FS

Maintenance of the repository must not become a Linux distributor’s job But: should be reproducible and well-documented Idea: Automatically generate a fully versioned, closed package list from an unversioned “shopping list” of packages

Scientific Linux EPEL CernVM Extras (≈ 40)

··· yum install on CernVM-FS Closure

!""# Dependency $%&"'# (#

Formulate dependencies as Integer Linear Program (ILP) Mirror

8 / 12 Build Process: Package Dependency ILP

Normalized (Integer) Linear Program: ⎛x1⎞ ⎛a11 ··· a1n ⎞ ⎛x1⎞ ⎛b1 ⎞ (c ··· c ) · ⎜ . ⎟ ⎜ . . . ⎟ · ⎜ . ⎟ ≤ ⎜ . ⎟ Minimize 1 n ⎝ . ⎠ subject to ⎝ . .. . ⎠ ⎝ . ⎠ ⎝ . ⎠ xn am1 ··· amn xn bm

Here: every available (package, version) is mapped to a xi ∈ {0, 1}. Cost vector: newer versions are cheaper than older versions. (Obviously: less packages cheaper than more packages.) Dependencies: Package xa requires xb or xc : xb + xc − xa ≥ 0. Packages xa and xb conflict: xa + xb ≤ 1. (. . . )

Figures ≈17 000 available packages (n = 17000), 500 packages on “shopping list” ≈160 000 inequalities (m = 160000), solving time <10 s (glpk) Meta RPM: ≈1 000 fully versioned packages, dependency closure

Idea: Mancinelli, Boender, di Cosmo, Vouillon, Durak (2006) 9 / 12 CernVM: VM Life Cycle

11. Retire 10. Feedback

2. Prepare 1. Plan 9. Terminate 8. Monitor Repositories Development Deployment Cycle Cycle

3. Build 4. Test 6. Instantiate 7. Contextualize

5. Endorse

CernVM Infrastructure User Infrastructure

Figure 1: Visual representation of the two sub-cycles that form the Virtual Machine Lifecycle.

10 / 12

2. The Virtual Machine Lifecycle A virtual machine passes through various different stages throughout it’s life. These stages are just a logical separation of the fundamental procedures that are common for the maintenance of every virtual machine (VM). They are usually independent and are associated with a specific set of tools. For instance, the life of the VM begins when the specifications of the build process are prepared and stored in a reference database, and it is terminated after it has completed the job it was instantiated for. In order to find an optimal solution it is important to identify those stages, the tools associated with them and their dependencies. This way the appropriate tools can be grouped with the stages and form a stand-alone and independently-maintained component. In the CernVM Project we pass through all stages of the every time we release a new version. In our case, after a VM instance completes it’s cycle, user feedback is processed and a new development cycle begins. Because of this cycling pattern, we decided to use the term lifecycle to refer to the life of CernVM. This lifecycle can be split into two logical sub-cycles: the development cycle and the deployment cycle (Figure 1). The development cycle begins with the definition of the specifications and finishes with the production of the distributable VM media. This cycle is performed entirely inside the CernVM infrastructure. The deployment cycle begins with the instantiation of the released image and finishes with the termination of the instance. This cycle is performed outside the CernVM infrastructure, such as a public or private cloud infrastructure (e.g. Amazon or OpenNebula ) or an individual computer (e.g. desktop hypervisors or a small computer farm). In all these cases, the OS needs to contact the CernVM infrastructure in order to obtain contextualization information and software packages from our repository. The two cycles are connected via two intermediate stages: The release of the produced image to the public and the feedback that is collected from the users and triggers a new development cycle. The two stages are in the borders that split the private infrastructure and the public. As was mentioned before, each stage is independent and is typically supported by a number of specialized tools.

Plan: This is a stage on which the desired functionality of the VM is planned. The resulting CernVM: VM Life Cycle

11. Retire 10. Feedback

2. Prepare 1. Plan 9. Terminate 8. Monitor Repositories Development Deployment Cycle Cycle

3. Build 4. Test 6. Instantiate 7. Contextualize

5. Endorse

CernVM Infrastructure User Infrastructure

Avoids: Image Building Solves: Image Distribution Figure 1: Visual representation of the two sub-cycles that form the Virtual Machine Lifecycle. Options for updating: stay, diverge, rebase

2. The Virtual Machine Lifecycle 10 / 12 A virtual machine passes through various different stages throughout it’s life. These stages are just a logical separation of the fundamental procedures that are common for the maintenance of every virtual machine (VM). They are usually independent and are associated with a specific set of tools. For instance, the life of the VM begins when the specifications of the build process are prepared and stored in a reference database, and it is terminated after it has completed the job it was instantiated for. In order to find an optimal solution it is important to identify those stages, the tools associated with them and their dependencies. This way the appropriate tools can be grouped with the stages and form a stand-alone and independently-maintained component. In the CernVM Project we pass through all stages of the every time we release a new version. In our case, after a VM instance completes it’s cycle, user feedback is processed and a new development cycle begins. Because of this cycling pattern, we decided to use the term lifecycle to refer to the life of CernVM. This lifecycle can be split into two logical sub-cycles: the development cycle and the deployment cycle (Figure 1). The development cycle begins with the definition of the specifications and finishes with the production of the distributable VM media. This cycle is performed entirely inside the CernVM infrastructure. The deployment cycle begins with the instantiation of the released image and finishes with the termination of the instance. This cycle is performed outside the CernVM infrastructure, such as a public or private cloud infrastructure (e.g. Amazon or OpenNebula ) or an individual computer (e.g. desktop hypervisors or a small computer farm). In all these cases, the OS needs to contact the CernVM infrastructure in order to obtain contextualization information and software packages from our repository. The two cycles are connected via two intermediate stages: The release of the produced image to the public and the feedback that is collected from the users and triggers a new development cycle. The two stages are in the borders that split the private infrastructure and the public. As was mentioned before, each stage is independent and is typically supported by a number of specialized tools.

Plan: This is a stage on which the desired functionality of the VM is planned. The resulting Hypervisor Support Status

Hypervisor / Cloud Controller Status VirtualBox X VMware X KVM X Microsoft HyperV ? Amazon EC2 (Xen based) X1 CERN ibex E2 Google Compute Engine ?3

1 Only tested with ephemeral storage, not with EBS backed instances 2 Until early user space, block device configuration needs adjustments 3 No support for custom kernels, no aufs support

11 / 12 Outlook and Open Issues

휇CernVM technology preview has been released: http://cernvm.cern.ch/portal/ucernvm

So far we managed to • Compile ROOT • Run LHCb reconstruction jobs

Open Issues: • Many small things (see tbd. items in this presentation) + graphics fine-tuning, copy & paste between host and guest, . • Replacement for the rPath appliance agent (summer student project) • Appliance “re-base utility” in order to upgrade the appliance from one version to another (summer student project) • Contextualization step during early boot (summer student project)

12 / 12