DOCSLIB.ORG

Checkpoint and Restore of Singularity Containers

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Checkpoint and Restore of Singularity Containers Universitat politecnica` de catalunya (UPC) - BarcelonaTech Facultat d'informatica` de Barcelona (FIB) Checkpoint and restore of Singularity containers Grado en ingenier´ıa informatica´ Tecnolog´ıas de la Informacion´ Memoria 25/04/2019 Director: Autor: Jordi Guitart Fernandez Enrique Serrano G´omez Departament: Arquitectura de Computadors 1 Abstract Singularity es una tecnolog´ıade contenedores software creada seg´unlas necesidades de cient´ıficos para ser utilizada en entornos de computaci´onde altas prestaciones. Hace ya 2 a~nosdesde que los usuarios empezaron a pedir una integraci´onde la fun- cionalidad de Checkpoint/Restore, con CRIU, en contenedores Singularity. Esta inte- graci´onayudar´ıaen gran medida a mejorar la gesti´onde los recursos computacionales de las m´aquinas. Permite a los usuarios guardar el estado de una aplicaci´on(ejecut´andose en un contenedor Singularity) para poder restaurarla en cualquier momento, sin perder el trabajo realizado anteriormente. Por lo que la posible interrupci´onde una aplicaci´on, debido a un fallo o voluntariamente, no es una p´erdidade tiempo de computaci´on. Este proyecto muestra como es posible realizar esa integraci´on. Singularity ´esuna tecnologia de contenidors software creada segons les necessitats de cient´ıfics,per ser utilitzada a entorns de computaci´od'altes prestacions. Fa 2 anys desde que els usuaris van comen¸cara demanar una integraci´ode la funcional- itat de Checkpoint/Restore, amb CRIU, a contenidors Singularity. Aquesta integraci´o ajudaria molt a millorar la gesti´odels recursos computacionals de les m`aquines.Permet als usuaris guardar l'estat d'una aplicaci´o(executant-se a un contenidor Singularity) per poder restaurar-la en qualsevol moment, sense perdre el treball realitzat anteriorment. Per la qual la possible interrupci´od'una aplicaci´o,a causa de una fallada o volunt`ariament, no ´esuna p`erduade temps de computaci´o. Aquest projecte mostra com ´espossible realitzar la integraci´o. Singularity is a software container technology created thinking in the needs of the scientists, so it can be used on High Performance Computing environments. It has been 2 years since the users started demanding an integration of the Check- point/Restore functionality, using CRIU on Singularity containers. This integration would help to a large degree to improve the management of the machine's computational re- sources. It allows the users to save the state of an application (being executed on a Singularity container) to be able of restoring it anytime, without losing the work already done. So that the prospect of interrupting an application, due to an error or voluntarily, is not a computation time lost. This project shows how the integration can be done. 2 ´Indice de contenidos 1 Contexto 8 1.1 Uso de Contenedores . .9 1.2 Singularity . 10 1.3 Actores implicados . 10 2 Estado del arte 11 2.1 Checkpoint ....................................... 11 2.2 Sistemas de Checkpoint ................................ 12 2.3 Container Checkpoint ................................. 12 2.4 Checkpoint/Restore in Userspace, CRIU ...................... 13 2.5 Conclusiones . 13 3 Formulaci´ondel problema 14 4 Planteamiento inicial 15 4.1 Alcance . 15 4.1.1 Posibles obst´aculos . 15 4.2 Metodolog´ıay rigor . 16 4.2.1 Herramientas . 16 4.2.2 M´etodo de validaci´on . 17 4.3 Planificaci´on. 18 4.3.1 Descripci´onde las fases y tareas . 18 4.3.1.1 Planificaci´oninicial . 18 4.3.1.2 An´alisisProyecto . 18 4.3.1.3 Estructuraci´onSingularity . 18 4.3.1.4 Integraci´onCRIU-Singularity . 18 4.3.1.5 Finalizaci´onproyecto . 19 4.4 Recursos . 20 4.5 Valoraci´onde alternativas y plan de acci´on. 21 4.5.1 Planificaci´onincorrecta . 21 4.5.2 Inviabilidad de integraci´onde CRIU . 21 4.6 Identificaci´ony estimaci´onde los costes . 22 4.6.1 Costes Directos . 22 4.6.1.1 Recursos Humanos . 22 4.6.1.2 Amortizaciones . 23 4.6.1.3 Costes indirectos . 23 4.6.2 Imprevistos . 23 4.6.3 Contingencias . 24 4.6.4 Resumen . 24 4.7 Control de gesti´on . 25 5 Planteamiento final 26 5.1 Alcance y Objetivos . 26 5.1.1 Obst´aculosencontrados . 26 5.2 Metodolog´ıay rigor . 27 3 5.2.1 M´etodo de validaci´on . 27 5.3 Planificaci´on. 28 5.3.1 Revisi´on:Estructuraci´onde Singularity . 28 5.3.2 Documentaci´oninicial . 28 5.3.3 An´alisisde las tecnolog´ıas . 28 5.3.4 Integraci´onCRIU-Singularity . 28 5.3.5 Finalizaci´ondel proyecto . 29 5.4 Recursos . 30 5.5 Costes del proyecto . 31 5.5.1 Costes Directos . 31 5.5.1.1 Recursos Humanos . 31 5.5.1.2 Amortizaciones . 32 5.5.2 Costes indirectos . 32 5.5.3 Imprevistos y Contingencias . 32 5.5.4 Resumen . 32 5.6 Revisi´on:Valoraci´onde alternativas, plan de acci´ony control de gesti´on . 33 5.6.1 Planificaci´onIncorrecta . 33 5.6.2 Inviabilidad de integraci´onde CRIU . 33 6 Sostenibilidad del Proyecto 34 6.1 Ambiental . 34 6.1.1 Proyecto puesto en producci´on(PPP) . 34 6.1.2 Vida ´util. 34 6.1.3 Riesgos . 34 6.2 Econ´omico. 35 6.2.1 Proyecto puesto en producci´on(PPP) . 35 6.2.2 Vida ´util. 35 6.2.3 Riesgos . 35 6.3 Social . 36 6.3.1 Proyecto puesto en producci´on(PPP) . 36 6.3.2 Vida ´util. 36 6.3.3 Riesgos . 36 7 Leyes y regulaciones relativas al proyecto 37 8 An´alisisde las tecnologias 38 8.1 Singularity . 38 8.1.1 Imagen de Contenedor Singularity . 38 8.1.2 Generaci´ondel SIF . 39 8.1.3 Singularity definition files .......................... 40 8.1.4 Namespaces de Linux . 41 8.1.4.1 Mount namespaces ......................... 41 8.1.4.2 PID namespaces .......................... 42 8.1.5 Creaci´onde contenedores . 43 8.1.6 Caracter´ısticasde una imagen en ejecuci´on. 44 8.1.7 Comunicaci´oncon el contenedor . 47 8.2 CRIU, Checkpoint/Restore in User Space ...................... 49 8.2.1 Checkpoint .................................. 49 4 8.2.2 Restore .................................... 52 8.2.3 Uso de CRIU . 54 9 Integraci´onCRIU-Singularity 55 9.1 External Checkpoint/Restore ............................ 56 9.1.1 Checkpoint .................................. 56 9.1.2 Restore .................................... 58 9.2 Internal Checkpoint/Restore ............................. 60 9.2.1 Soluci´on . 61 9.2.2 Pruebas de checkpoint/restore con el manager . 62 9.2.2.1 Contador . 63 9.2.2.2 Contadores . 64 9.2.2.3 Escritor . 65 9.2.2.4 Lector . 66 9.2.2.5 Cliente/Servidor TCP Contador . 67 9.2.2.6 Manager y contenedores en instancias . 68 10 Conclusiones 69 11 Glosario 70 12 Anexo A: Singularity Definition Files 71 13 Anexo B: Scripts 72 14 Anexo C: Debug de Singularity 78 15 Referencias 81 ´Indice de tablas 1 Horas por tarea y sus dependencias. 19 2 Costes directos por actividad. 22 3 Precio/hora por rol. 23 4 Amortizaciones Hardware. 23 5 Costes indirectos. 23 6 Resumen presupuesto. 24 7 Horas por tarea dedicadas. 29 8 Costes directos de las actividades realizadas. 31 9 Precio/hora por rol. 31 10 Amortizaciones Hardware. 32 11 Costes indirectos 7 meses. 32 12 Resumen presupuesto, . 32 5 ´Indice de figuras 1 Arquitectura Contenedor Software ..........................8 2 Arquitectura M´aquinavirtual ............................9 3 Escenario t´ıpico de uso de Checkpoint/Restore ................... 11 4 M´etodo iterativo. ................................... 16 5 Diagrama de flujo de trabajo. ..
Load more

Recommended publications
  • Checkpoint and Restoration of Micro-Service in Docker Containers
    3rd International Conference on Mechatronics and Industrial Informatics (ICMII 2015) Checkpoint and Restoration of Micro-service in Docker Containers Chen Yang School of Information Security Engineering, Shanghai Jiao Tong University, China 200240 [email protected] Keywords: Lightweight Virtualization, Checkpoint/restore, Docker. Abstract. In the present days of rapid adoption of micro-service, it is imperative to build a system to support and ensure the high performance and high availability for micro-services. Lightweight virtualization, which we also called container, has the ability to run multiple isolated sets of processes under a single kernel instance. Because of the possibility of obtaining a low overhead comparable to the near-native performance of a bare server, the container techniques, such as openvz, lxc, docker, they are widely used for micro-service [1]. In this paper, we present the high availability of micro-service in containers. We investigate capabilities provided by container (docker, openvz) to model and build the Micro-service infrastructure and compare different checkpoint and restore technologies for high availability. Finally, we present preliminary performance results of the infrastructure tuned to the micro-service. Introduction Lightweight virtualization, named the operating system level virtualization technology, partitions the physical machines resource, creating multiple isolated user-space instances. Each container acts exactly like a stand-alone server. A container can be rebooted independently and have root access, users, IP address, memory, processes, files, etc. Unlike traditional virtualization with the hypervisor layer, containerization takes place at the kernel level. Most modern operating system kernels now support the primitives necessary for containerization, including Linux with openvz, vserver and more recently lxc, Solaris with zones, and FreeBSD with Jails [2].
    [Show full text]
  • Evaluating and Improving LXC Container Migration Between
    Evaluating and Improving LXC Container Migration between Cloudlets Using Multipath TCP By Yuqing Qiu A thesis submitted to the Faculty of Graduate and Postdoctoral Affairs in partial fulfillment of the requirements for the degree of Master of Applied Science in Electrical and Computer Engineering Carleton University Ottawa, Ontario © 2016, Yuqing Qiu Abstract The advent of the Cloudlet concept—a “small data center” close to users at the edge is to improve the Quality of Experience (QoE) of end users by providing resources within a one-hop distance. Many researchers have proposed using virtual machines (VMs) as such service-provisioning servers. However, seeing the potentiality of containers, this thesis adopts Linux Containers (LXC) as Cloudlet platforms. To facilitate container migration between Cloudlets, Checkpoint and Restore in Userspace (CRIU) has been chosen as the migration tool. Since the migration process goes through the Wide Area Network (WAN), which may experience network failures, the Multipath TCP (MPTCP) protocol is adopted to address the challenge. The multiple subflows established within a MPTCP connection can improve the resilience of the migration process and reduce migration time. Experimental results show that LXC containers are suitable candidates for the problem and MPTCP protocol is effective in enhancing the migration process. i Acknowledgement I would like to express my sincerest gratitude to my principal supervisor Dr. Chung-Horng Lung who has provided me with valuable guidance throughout the entire research experience. His professionalism, patience, understanding and encouragement have always been my beacons of light whenever I go through difficulties. My gratitude also goes to my co-supervisor Dr.
    [Show full text]
  • The Aurora Operating System
    The Aurora Operating System Revisiting the Single Level Store Emil Tsalapatis Ryan Hancock Tavian Barnes RCS Lab, University of Waterloo RCS Lab, University of Waterloo RCS Lab, University of Waterloo [email protected] [email protected] [email protected] Ali José Mashtizadeh RCS Lab, University of Waterloo [email protected] ABSTRACT KEYWORDS Applications on modern operating systems manage their single level stores, transparent persistence, snapshots, check- ephemeral state in memory, and persistent state on disk. En- point/restore suring consistency between them is a source of significant developer effort, yet still a source of significant bugs inma- ACM Reference Format: ture applications. We present the Aurora single level store Emil Tsalapatis, Ryan Hancock, Tavian Barnes, and Ali José Mash- (SLS), an OS that simplifies persistence by automatically per- tizadeh. 2021. The Aurora Operating System: Revisiting the Single sisting all traditionally ephemeral application state. With Level Store. In Workshop on Hot Topics in Operating Systems (HotOS recent storage hardware like NVMe SSDs and NVDIMMs, ’21), June 1-June 3, 2021, Ann Arbor, MI, USA. ACM, New York, NY, Aurora is able to continuously checkpoint entire applications USA, 8 pages. https://doi.org/10.1145/3458336.3465285 with millisecond granularity. Aurora is the first full POSIX single level store to han- dle complex applications ranging from databases to web 1 INTRODUCTION browsers. Moreover, by providing new ways to interact with Single level storage (SLS) systems provide persistence of and manipulate application state, it enables applications to applications as an operating system service. Their advantage provide features that would otherwise be prohibitively dif- lies in removing the semantic gap between the in-memory ficult to implement.
    [Show full text]
  • Instant OS Updates Via Userspace Checkpoint-And
    Instant OS Updates via Userspace Checkpoint-and-Restart Sanidhya Kashyap, Changwoo Min, Byoungyoung Lee, and Taesoo Kim, Georgia Institute of Technology; Pavel Emelyanov, CRIU and Odin, Inc. https://www.usenix.org/conference/atc16/technical-sessions/presentation/kashyap This paper is included in the Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC ’16). June 22–24, 2016 • Denver, CO, USA 978-1-931971-30-0 Open access to the Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC ’16) is sponsored by USENIX. Instant OS Updates via Userspace Checkpoint-and-Restart Sanidhya Kashyap Changwoo Min Byoungyoung Lee Taesoo Kim Pavel Emelyanov† Georgia Institute of Technology †CRIU & Odin, Inc. # errors # lines Abstract 50 1000K 40 100K In recent years, operating systems have become increas- 10K 30 1K 20 ingly complex and thus more prone to security and per- 100 formance issues. Accordingly, system updates to address 10 10 these issues have become more frequently available and 0 1 increasingly important. To complete such updates, users 3.13.0-x 3.16.0-x 3.19.0-x May 2014 must reboot their systems, resulting in unavoidable down- build/diff errors #layout errors Jun 2015 time and further loss of the states of running applications. #static local errors #num lines++ We present KUP, a practical OS update mechanism that Figure 1: Limitation of dynamic kernel hot-patching using employs a userspace checkpoint-and-restart mechanism, kpatch. Only two successful updates (3.13.0.32 34 and → which uses an optimized data structure for checkpoint- 3.19.0.20 21) out of 23 Ubuntu kernel package releases.
    [Show full text]
  • Task Migration at Scale Using CRIU Linux Plumbers Conference 2018
    Task Migration at Scale Using CRIU Linux Plumbers Conference 2018 Victor Marmol Andy Tucker [email protected] [email protected] 2018-11-15 Confidential + Proprietary Who we are Outside of Google, we’ve worked on open source cluster management and containers lmctfy Confidential + Proprietary 2 Who we are Inside Google: we’re part of the Borg team ● Manages all compute jobs ● Runs on every server ConfidentialImages + Proprietary by Connie Zhou3 What is Borg? Google’s cluster management system ● Borgmaster: Cluster control and main API entrypoint ● Borglet: On-machine management daemon ● Suite of tools and UIs for managing jobs ● Many purpose-built platforms created on top of Borg ● Everything runs on Borg and everything runs in containers Confidential + Proprietary 4 Borg basics Allocated Ports Base compute primitive: Task ● A priority signals how quickly a task should Processes Packages schedule Processes Packages Processes Packages ● It’s appclass describes a task as either Processes Packages serving (latency sensitive) or batch ● Static content/binaries provided by Container packages Task ● A container isolates a task’s resources ● Native Linux processes Borg Machine ● Share an IP with the machine, ports are allocated for each task Confidential + Proprietary 5 Borg basics: evictions When a task is forcefully terminated by Borg ● Typically receive a notification: 1-5min ● Our SLO allows for quite a few evictions ● Applications must handle them Reasons for evictions ● Preemption: a higher priority task needs the resources ● Software upgrades (e.g.: kernel, firmware) ● Re-balancing for availability or performance Confidential + Proprietary 6 Evictions are impactful and hard to handle Technical Complexity ● Handling evictions requires state management ○ How and what state to serialize and where to store it ● Application-specific and not very reusable Lost Compute ● Batch jobs run at lower priorities and get preempted often ● Even platforms that handle them for users, don’t do a great job Task 0 Evicted!Shard 1 Task 2 Task 3 Task 4 Compute is lost..
    [Show full text]
  • Open Source Software Packages
    Hitachi Ops Center 10.5.1 Open Source Software Packages Contact Information: Hitachi Ops Center Project Manager Hitachi Vantara LLC 2535 Augustine Drive Santa Clara, California 95054 Name of package Version License agreement @agoric/babel-parser 7.12.7 The MIT License @angular-builders/custom-webpack 8.0.0-RC.0 The MIT License @angular-devkit/build-angular 0.800.0-rc.2 The MIT License @angular-devkit/build-angular 0.901.12 The MIT License @angular-devkit/core 7.3.8 The MIT License @angular-devkit/schematics 7.3.8 The MIT License @angular/animations 9.1.11 The MIT License @angular/animations 9.1.12 The MIT License @angular/cdk 9.2.4 The MIT License @angular/cdk-experimental 9.2.4 The MIT License @angular/cli 8.0.0 The MIT License @angular/cli 9.1.11 The MIT License @angular/common 9.1.11 The MIT License @angular/common 9.1.12 The MIT License @angular/compiler 9.1.11 The MIT License @angular/compiler 9.1.12 The MIT License @angular/compiler-cli 9.1.12 The MIT License @angular/core 7.2.15 The MIT License @angular/core 9.1.11 The MIT License @angular/core 9.1.12 The MIT License @angular/forms 7.2.15 The MIT License @angular/forms 9.1.0-next.3 The MIT License @angular/forms 9.1.11 The MIT License @angular/forms 9.1.12 The MIT License @angular/language-service 9.1.12 The MIT License @angular/platform-browser 7.2.15 The MIT License @angular/platform-browser 9.1.11 The MIT License @angular/platform-browser 9.1.12 The MIT License @angular/platform-browser-dynamic 7.2.15 The MIT License @angular/platform-browser-dynamic 9.1.11 The MIT License @angular/platform-browser-dynamic
    [Show full text]
  • Red Hat Enterprise Linux 7 7.8 Release Notes
    Red Hat Enterprise Linux 7 7.8 Release Notes Release Notes for Red Hat Enterprise Linux 7.8 Last Updated: 2021-03-02 Red Hat Enterprise Linux 7 7.8 Release Notes Release Notes for Red Hat Enterprise Linux 7.8 Legal Notice Copyright © 2021 Red Hat, Inc. The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/ . In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux ® is the registered trademark of Linus Torvalds in the United States and other countries. Java ® is a registered trademark of Oracle and/or its affiliates. XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and other countries. Node.js ® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
    [Show full text]
  • Seccomp Update
    seccomp update https://outflux.net/slides/2015/lss/seccomp.pdf Linux Security Summit, Seattle 2015 Kees Cook <[email protected]> (pronounced “Case”) What is seccomp? ● Programmatic kernel attack surface reduction ● Used by: – Chrome – vsftpd – OpenSSH – Systemd (“SystemCallFilter=...”) – LXC (blacklisting) – … and you too! (easiest via libseccomp) seccomp update 2/8 Linux Security Summit 2015 Seattle, Aug 21 Architecture support ● x86: v3.5 ● s390: v3.6 ● arm: v3.8 ● mips: v3.15 ● arm64: v3.19, AKASHI Takahiro ● powerpc: linux-next (v4.3), Michael Ellerman seccomp update 3/8 Linux Security Summit 2015 Seattle, Aug 21 split-phase internals ● v3.19, Andy Lutomirski ● Splits per-architecture calls to seccomp into 2 phases ● Speeds up simple (no tracing) callers ● Only used on x86 so far seccomp update 4/8 Linux Security Summit 2015 Seattle, Aug 21 Regression tests ● v4.2: moved the 48 tests from github into the kernel: tools/testing/selftests/seccomp/ ● Shows some interesting glitches with restart_syscall on arm (hidden) and arm64 (hidden, unless compat, then exposed) ● Gained big-endian support during powerpc port ● Added s390 seccomp support today seccomp update 5/8 Linux Security Summit 2015 Seattle, Aug 21 Minor changes ● v4.0: SECCOMP_RET_ERRNO capped at MAX_ERRNO – Avoid confusing userspace ● v4.1: asm-generic for seccomp.h – Easier architecture porting seccomp update 6/8 Linux Security Summit 2015 Seattle, Aug 21 Future ● Argument inspection ● CRIU (checkpoint/restore) – PTRACE_O_SUSPEND_SECCOMP with CAP_SYS_ADMIN: linux-next (v4.3), Tycho Andersen – Serialize dump/restore of filters. ● eBPF – Use maps or tail calls instead of balanced if/else trees for checking syscall numbers. seccomp update 7/8 Linux Security Summit 2015 Seattle, Aug 21 Questions? https://outflux.net/slides/2015/lss/seccomp.pdf @kees_cook [email protected] [email protected] [email protected] seccomp update 8/8 Linux Security Summit 2015 Seattle, Aug 21.
    [Show full text]
  • Mini-Ckpts: Surviving OS Failures in Persistent Memory
    Mini-Ckpts: Surviving OS Failures in Persistent Memory David Fiala, Frank Mueller Kurt Ferreira Christian Engelmann North Carolina State University∗ Sandia National Laboratories† Oak Ridge National Laboratory‡§ davidfi[email protected], [email protected] [email protected] [email protected] ABSTRACT minate due to tight communication in HPC. Therefore, Concern is growing in the high-performance computing the OS itself must be capable of tolerating failures in a (HPC) community on the reliability of future extreme- robust system. In this work, we introduce mini-ckpts, scale systems. Current efforts have focused on appli- a framework which enables application survival despite cation fault-tolerance rather than the operating system the occurrence of a fatal OS failure or crash. mini- (OS), despite the fact that recent studies have suggested ckpts achieves this tolerance by ensuring that the crit- that failures in OS memory may be more likely. The OS ical data describing a process is preserved in persistent is critical to a system’s correct and efficient operation of memory prior to the failure. Following the failure, the the node and processes it governs — and the parallel na- OS is rejuvenated via a warm reboot and the applica- ture of HPC applications means any single node failure tion continues execution effectively making the failure generally forces all processes of this application to ter- and restart transparent. The mini-ckpts rejuvenation and recovery process is measured to take between three ∗This work was supported in part by a subcontract from to six seconds and has a failure-free overhead of between Sandia National Laboratories and NSF grants CNS- 3-5% for a number of key HPC workloads.
    [Show full text]
  • Netlink Sockets
    CHAPTER 2 Netlink Sockets Chapter 1 discusses the roles of the Linux kernel networking subsystem and the three layers in which it operates. The netlink socket interface appeared first in the 2.2 Linux kernel as AF_NETLINK socket. It was created as a more flexible alternative to the awkward IOCTL communication method between userspace processes and the kernel. The IOCTL handlers cannot send asynchronous messages to userspace from the kernel, whereas netlink sockets can. In order to use IOCTL, there is another level of complexity: you need to define IOCTL numbers. The operation model of netlink is quite simple: you open and register a netlink socket in userspace using the socket API, and this netlink socket handles bidirectional communication with a kernel netlink socket, usually sending messages to configure various system settings and getting responses back from the kernel. This chapter describes the netlink protocol implementation and API and discusses its advantages and drawbacks. I also talk about the new generic netlink protocol, discuss its implementation and its advantages, and give some illustrative examples using the libnl library. I conclude with a discussion of the socket monitoring interface. The Netlink Family The netlink protocol is a socket-based Inter Process Communication (IPC) mechanism, based on RFC 3549, “Linux Netlink as an IP Services Protocol.” It provides a bidirectional communication channel between userspace and the kernel or among some parts of the kernel itself. Netlink is an extension of the standard socket implementation. The netlink protocol implementation resides mostly under net/netlink, where you will find the following four files: •฀ af_netlink.c •฀ af_netlink.h •฀ genetlink.c •฀ diag.c Apart from them, there are a few header files.
    [Show full text]
  • Eyeglass Search OSS Licenses and Packages V9
    ECA 15.1 opensuse https://en.opensuse.org/openSUSE:License Package Licence Name Version Type Key Licence Name opensuse 15.1 OS annogen:annogen 0.1.0 JAR Not Found antlr:antlr 2.7.7 JAR BSD Berkeley Software Distribution (BSD) aopalliance:aopalliance 1 JAR Public DomainPublic Domain asm:asm 3.1 JAR Not Found axis:axis 1.4 JAR Apache-2.0 The Apache Software License, Version 2.0 axis:axis-wsdl4j 1.5.1 JAR Not Found backport-util-concurrent:backport-util-concurrent3.1 JAR Public DomainPublic Domain com.amazonaws:aws-java-sdk 1.1.7.1 JAR Apache-2.0 The Apache Software License, Version 2.0 com.beust:jcommander 1.72 JAR Apache-2.0 The Apache Software License, Version 2.0 com.carrotsearch:hppc 0.6.0 JAR Apache-2.0 The Apache Software License, Version 2.0 com.clearspring.analytics:stream 2.7.0 JAR Apache-2.0 The Apache Software License, Version 2.0 com.drewnoakes:metadata-extractor 2.4.0-beta-1JAR Public DomainPublic Domain com.esotericsoftware:kryo-shaded 3.0.3 JAR BSD Berkeley Software Distribution (BSD) com.esotericsoftware:minlog 1.3.0 JAR BSD Berkeley Software Distribution (BSD) com.fasterxml.jackson.core:jackson-annotations2.6.1 JAR Apache-2.0 The Apache Software License, Version 2.0 com.fasterxml.jackson.core:jackson-annotations2.5.0 JAR Apache-2.0 The Apache Software License, Version 2.0 com.fasterxml.jackson.core:jackson-annotations2.2.0 JAR Apache-2.0 The Apache Software License, Version 2.0 com.fasterxml.jackson.core:jackson-annotations2.9.0 JAR Apache-2.0 The Apache Software License, Version 2.0 com.fasterxml.jackson.core:jackson-annotations2.6.0
    [Show full text]
  • Restoring Uniqueness in Microvm Snapshots
    Restoring Uniqueness in MicroVM Snapshots Marc Brooker Adrian Costin Catangiu Mike Danilov Amazon Web Services Amazon Web Services Amazon Web Services Alexander Graf Colm MacCarthaigh Andrei Sandu Amazon Web Services Amazon Web Services Amazon Web Services Abstract MicroVM boot ~200ms Code initialization—the step of loading code, executing static Download 20ms to code, filling caches, and forming re-used connections—tends 60s to dominate cold-start time in serverless compute systems Initialize “cold start” such as AWS Lambda. Post-initialization memory snapshots, cloned and restored on start, have emerged as a viable solution Invoke <10ms to this problem, with incremental snapshot and fast restore “Warm start” Invoke support in VMMs like Firecracker. Saving memory introduces the challenge of managing … high-value memory contents, such as cryptographic secrets. Cloning introduces the challenge of restoring the uniqueness Death of the VMs, to allow them to do unique things like generate UUIDs, secrets, and nonces. This paper examines solutions Figure 1: Time line of the life cycle of a microVM in AWS to these problems in the every microsecond counts context of Lambda serverless cold-start, and discusses the state-of-the-art of avail- able solutions. We present two new interfaces aimed at solving this problem—MADV_WIPEONSUSPEND and VmGenId—and control, leaving little opportunity to optimize or improve it on compare them to alternative solutions. behalf of the customer (beyond simply allocating additional resources to its execution). 1 Introduction Figure1 shows a typical time line of the life cycle of a microVM in Lambda. The microVM boot, booting a minimal When AWS Lambda receives a request to invoke a server- Linux kernel in Firecracker, with a minimal set of userspace less function, the system attempts to match the request to an daemons, and network setup, takes approximately 200ms.
    [Show full text]