DOCSLIB.ORG

BIG Data, BIG Responsibility (Data Lineage Management with Template for Reproducible Scientific Papers)

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
BIG Data, BIG Responsibility (Data Lineage Management with Template for Reproducible Scientific Papers) BIG Data, BIG responsibility (Data lineage management with template for reproducible scientific papers) Mohammad Akhlaghi Instituto de Astrof´ısicade Canarias (IAC), Tenerife, Spain 10th Iberian Grid Conference (Ibergrid2019), Santiago de Compostela (Spain), September 23rd, 2019 Slides available at http://akhlaghi.org/pdf/reproducible-paper.pdf Reproducibility is critically important in the sciences (example from astronomy) Example: Detecting outer regions of M51 in a single exposure SDSS image, using NoiseChisel, with default and optimized parameters. I When optimized, outer wing detected to S=N =1=4, or 28:3 mag/arcsec2. I Complete tutorial in manual fully describes how to derive/reproduce optimized result: I Run-time options/configuration. I Steps before/after NoiseChisel. Input image Default NoiseChisel I Deep/orange image from Watkins+2015 (arXiv:1501.04599). I Therefore: I Default settings not enough. I Final number not just from NoiseChisel (more software involved). Simply reporting in your paper that \we used NoiseChisel" is not enough to reproduce, un- Optimized NoiseChisel Much deeper image derstand, or verify your result. Perspectives on Reproducibility and Sustainability of Open-Source Scientific Software \It is our interest that NASA adopt an open-code policy because without it, reproducibility in computational science is needlessly hampered". From Oishi+2018, (arXiv:1801.08200). Schroedinger's code: source code availability and link persistence in astrophysics \We were unable to find source code online ... for 40:4% of the codes used in the research we looked at". From Allen+2018, (arXiv:1801.02094). Reproducibility crisis in the sciences/astronomy Snakes on a Spaceship { An Overview of Python in Heliophysics \...inadequate analysis descriptions and loss of scientific data have made scientific studies difficult or impossible to replicate". From Burrell+2018, (arXiv:1901.00143). Schroedinger's code: source code availability and link persistence in astrophysics \We were unable to find source code online ... for 40:4% of the codes used in the research we looked at". From Allen+2018, (arXiv:1801.02094). Reproducibility crisis in the sciences/astronomy Snakes on a Spaceship { An Overview of Python in Heliophysics \...inadequate analysis descriptions and loss of scientific data have made scientific studies difficult or impossible to replicate". From Burrell+2018, (arXiv:1901.00143). Perspectives on Reproducibility and Sustainability of Open-Source Scientific Software \It is our interest that NASA adopt an open-code policy because without it, reproducibility in computational science is needlessly hampered". From Oishi+2018, (arXiv:1801.08200). Reproducibility crisis in the sciences/astronomy Snakes on a Spaceship { An Overview of Python in Heliophysics \...inadequate analysis descriptions and loss of scientific data have made scientific studies difficult or impossible to replicate". From Burrell+2018, (arXiv:1901.00143). Perspectives on Reproducibility and Sustainability of Open-Source Scientific Software \It is our interest that NASA adopt an open-code policy because without it, reproducibility in computational science is needlessly hampered". From Oishi+2018, (arXiv:1801.08200). Schroedinger's code: source code availability and link persistence in astrophysics \We were unable to find source code online ... for 40:4% of the codes used in the research we looked at". From Allen+2018, (arXiv:1801.02094). Original image from https://www.redbubble.com General outline of a project Software Build Run software on data Paper Hardware/data General outline of a project Software Build Run software on data Paper Hardware/data https://heywhatwhatdidyousay.wordpress.com General outline of a project What version? Software Build Run software on data Paper Hardware/data https://heywhatwhatdidyousay.wordpress.com Different package managers have different versions of software (repology.org, 2019/08/19) GNU Astronomy Utilities (Gnuastro) GGnuastronuastro packagingpackaging statusstatus DDebianebian OldstableOldstable 00.2.33.2.33 DDebianebian StableStable 00.8.8 DDebianebian TestingTesting 00.10.10 Astropy DDebianebian UnstableUnstable 00.10.10 DDeepineepin 00.5.5 PPackagingackaging statusstatus DDevuanevuan 2.02.0 (ASCII)(ASCII) 00.2.33.2.33 Debian Stable 3.1.2 Debian Stable 3.1.2 DDevuanevuan 3.03.0 (Beowulf)(Beowulf) 00.8.8 Debian Testing 3.1.2 Debian Testing 3.1.2 DDevuanevuan UnstableUnstable 00.10.10 DDebianebian UnstableUnstable 33.2.1.2.1 DDPortsPorts 00.9.9 DDeepineepin 33.0.2.0.2 FFreeBSDreeBSD PortsPorts 00.10.10 DDevuanevuan 3.03.0 (Beowulf)(Beowulf) 33.1.2.1.2 FFuntoountoo 1.31.3 00.3.3 DDevuanevuan UnstableUnstable 33.2.1.2.1 GGentooentoo 00.3.3 KKaliali LinuxLinux RollingRolling 33.1.2.1.2 KKaliali LinuxLinux RollingRolling 00.10.10 PParrotarrot 33.1.2.1.2 oopenSUSEpenSUSE LeapLeap 15.115.1 00.8.8 PPureOSureOS greengreen 33.1.2.1.2 oopenSUSEpenSUSE TumbleweedTumbleweed 00.8.8 PPureOSureOS landinglanding 33.1.2.1.2 oopenSUSEpenSUSE ScienceScience TumbleweedTumbleweed 00.8.8 RRaspbianaspbian StableStable 33.1.2.1.2 PPardusardus 00.2.33.2.33 RRaspbianaspbian TestingTesting 33.1.2.1.2 PParrotarrot 00.10.10 UUbuntubuntu 18.0418.04 33.0.0 PPLDLD LLinuxinux 00.8.8 UUbuntubuntu 18.1018.10 33.0.4.0.4 PPureOSureOS greengreen 00.8.8 UUbuntubuntu 19.0419.04 33.1.1.1.1 PPureOSureOS landinglanding 00.8.8 UUbuntubuntu 19.1019.10 33.1.2.1.2 RRaspbianaspbian OldstableOldstable 00.2.33.2.33 Raspbian Stable 0.8 UUbuntubuntu 19.1019.10 ProposedProposed 33.2.1.2.1 Raspbian Stable 0.8 RRaspbianaspbian TestingTesting 00.10.10 UUbuntubuntu 18.0418.04 00.5.5 UUbuntubuntu 18.1018.10 00.7.7 UUbuntubuntu 19.0419.04 00.8.8 UUbuntubuntu 19.1019.10 00.8.8 UUbuntubuntu 19.1019.10 ProposedProposed 00.10.10 General outline of a project Repository? What version? Software Build Run software on data Paper Hardware/data https://heywhatwhatdidyousay.wordpress.com General outline of a project Repository? What version? Dependencies? Software Build Run software on data Paper Hardware/data https://heywhatwhatdidyousay.wordpress.com General outline of a project Repository? Dep. versions? What version? Dependencies? Software Build Run software on data Paper Hardware/data https://heywhatwhatdidyousay.wordpress.com General outline of a project Config options? Repository? Dep. versions? What version? Dependencies? Software Build Run software on data Paper Hardware/data https://heywhatwhatdidyousay.wordpress.com General outline of a project Config environment? Config options? Repository? Dep. versions? What version? Dependencies? Software Build Run software on data Paper Hardware/data https://heywhatwhatdidyousay.wordpress.com Example: Matplotlib (a Python visualization library) build dependencies From \Attributing and Referencing (Research) Software: Best Practices and Outlook from Inria" (Alliez et al. 2019, hal-02135891) Impact of \Dependency hell" on native building in various hardware (CPU architectures) Astropy depends on Matplotlib GNU Astronomy Utilities doesn't. General outline of a project Existing solutions: Config environment? Virtual machines Containers (e.g., Docker) OSs (e.g., Nix, GNU Guix) Config options? Repository? Dep. versions? What version? Dependencies? Software Build Run software on data Paper Hardware/data https://heywhatwhatdidyousay.wordpress.com General outline of a project Existing solutions: Config environment? Virtual machines Containers (e.g., Docker) OSs (e.g., Nix, GNU Guix) Config options? Repository? Dep. versions? What version? Dependencies? Software Build Run software on data Paper Hardware/data Data base, or PID? https://heywhatwhatdidyousay.wordpress.com General outline of a project Existing solutions: Config environment? Virtual machines Containers (e.g., Docker) OSs (e.g., Nix, GNU Guix) Config options? Repository? Dep. versions? What version? Dependencies? Software Build Run software on data Paper Hardware/data Data base, or PID? Calibration/version? https://heywhatwhatdidyousay.wordpress.com General outline of a project Existing solutions: Config environment? Virtual machines Containers (e.g., Docker) OSs (e.g., Nix, GNU Guix) Config options? Repository? Dep. versions? What version? Dependencies? Software Build Run software on data Paper Hardware/data Data base, or PID? Calibration/version? Integrity? https://heywhatwhatdidyousay.wordpress.com General outline of a project Existing solutions: Config environment? Virtual machines Containers (e.g., Docker) OSs (e.g., Nix, GNU Guix) Config options? Repository? Dep. versions? What version? Dependencies? Software Build What order? Run software on data Paper Hardware/data Data base, or PID? Calibration/version? Integrity? https://heywhatwhatdidyousay.wordpress.com General outline of a project Existing solutions: Config environment? Virtual machines Containers (e.g., Docker) OSs (e.g., Nix, GNU Guix) Config options? Repository? Dep. versions? What version? Dependencies? Runtime options? Software Build What order? Run software on data Paper Hardware/data Data base, or PID? Calibration/version? Integrity? https://heywhatwhatdidyousay.wordpress.com General outline of a project Existing solutions: Config environment? Virtual machines Containers (e.g., Docker) OSs (e.g., Nix, GNU Guix) Config options? Repository? Dep. versions? Human error? What version? Dependencies? Runtime options? Software Build What order? Run software on data Paper Hardware/data Data base, or PID? Calibration/version? Integrity? https://heywhatwhatdidyousay.wordpress.com General outline of a project Existing solutions: Config environment? Virtual machines
Load more

Recommended publications
  • Using System Call Interposition to Automatically Create Portable Software Packages
    CDE: Using System Call Interposition to Automatically Create Portable Software Packages Philip J. Guo and Dawson Engler Stanford University April 5, 2011 (This technical report is an extended version of our 2011 USENIX ATC paper) Abstract as dependency hell. Or the user might lack permissions to install software packages in the first place, a common It can be painfully difficult to take software that runs on occurrence on corporate machines and compute clusters one person’s machine and get it to run on another ma- that are administered by IT departments. Finally, the chine. Online forums and mailing lists are filled with user (recalling bitter past experiences) may be reluctant discussions of users’ troubles with compiling, installing, to perturb a working environment by upgrading or down- and configuring software and their myriad of dependen- grading library versions just to try out new software. cies. To eliminate this dependency problem, we created As a testament to the ubiquity of software deployment a system called CDE that uses system call interposition to problems, consider the prevalence of online forums and monitor the execution of x86-Linux programs and pack- mailing list discussions dedicated to troubleshooting in- age up the Code, Data, and Environment required to run stallation and configuration issues. For example, almost them on other x86-Linux machines. The main benefits half of the messages sent to the mailing lists of two ma- of CDE are that creating a package is completely auto- ture research tools, the graph-tool mathematical graph matic, and that running programs within a package re- analysis library [10] and the PADS system for processing quires no installation, configuration, or root permissions.
    [Show full text]
  • GNU Guix Cookbook Tutorials and Examples for Using the GNU Guix Functional Package Manager
    GNU Guix Cookbook Tutorials and examples for using the GNU Guix Functional Package Manager The GNU Guix Developers Copyright c 2019 Ricardo Wurmus Copyright c 2019 Efraim Flashner Copyright c 2019 Pierre Neidhardt Copyright c 2020 Oleg Pykhalov Copyright c 2020 Matthew Brooks Copyright c 2020 Marcin Karpezo Copyright c 2020 Brice Waegeneire Copyright c 2020 Andr´eBatista Copyright c 2020 Christine Lemmer-Webber Copyright c 2021 Joshua Branson Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled \GNU Free Documentation License". i Table of Contents GNU Guix Cookbook ::::::::::::::::::::::::::::::: 1 1 Scheme tutorials ::::::::::::::::::::::::::::::::: 2 1.1 A Scheme Crash Course :::::::::::::::::::::::::::::::::::::::: 2 2 Packaging :::::::::::::::::::::::::::::::::::::::: 5 2.1 Packaging Tutorial:::::::::::::::::::::::::::::::::::::::::::::: 5 2.1.1 A \Hello World" package :::::::::::::::::::::::::::::::::: 5 2.1.2 Setup:::::::::::::::::::::::::::::::::::::::::::::::::::::: 8 2.1.2.1 Local file ::::::::::::::::::::::::::::::::::::::::::::: 8 2.1.2.2 `GUIX_PACKAGE_PATH' ::::::::::::::::::::::::::::::::: 9 2.1.2.3 Guix channels ::::::::::::::::::::::::::::::::::::::: 10 2.1.2.4 Direct checkout hacking:::::::::::::::::::::::::::::: 10 2.1.3 Extended example ::::::::::::::::::::::::::::::::::::::::
    [Show full text]
  • Introduction to the Nix Package Manager
    Introduction Nix concepts Usage examples Conclusion Introduction to the Nix Package Manager Millian Poquet 2021-05-12 — Datamove (Inria) seminar 1 / 16 Introduction Nix concepts Usage examples Conclusion Why Nix? Control your software environment! Programs/libraries/scripts/configurations + versions Why is it important for us? Use/develop/test/distribute software Manually install many dependencies? No, just type nix-shell Shared env for whole team (tunable) and test machines Bug only on my machine? Means this is hardware or OS related Reproducible research Repeat experiment in exact same environment Introduce or test variation 2 / 16 Introduction Nix concepts Usage examples Conclusion What is Nix? Nix: package manager Download and install packages Shell into well-defined environment (like virtualenv) Transactional (rollback works) Cross-platform: Linux, macOS, Windows (WSL) Nix: programming language Define packages Define environments (set of packages) Functional, DSL NixOS: Linux distribution Declarative system configuration Uses the Nix language Transactional (rollback still works) 3 / 16 Introduction Nix concepts Usage examples Conclusion Nix in numbers Started in 2003 Nix 1: 10k commits, 28k C++ LOC Nixpkgs 2: 285k commits, 55k packages 3 1. https://github.com/NixOS/nix 2. https://github.com/NixOS/nixpkgs 3. https://repology.org/repositories/statistics 4 / 16 Introduction Nix concepts Usage examples Conclusion Presentation summary 2 Nix concepts 3 Usage examples 4 Conclusion 5 / 16 Introduction Nix concepts Usage examples Conclusion Traditional
    [Show full text]
  • Functional Package and Configuration Management with GNU Guix
    Functional Package and Configuration Management with GNU Guix David Thompson Wednesday, January 20th, 2016 About me GNU project volunteer GNU Guile user and contributor since 2012 GNU Guix contributor since 2013 Day job: Ruby + JavaScript web development / “DevOps” 2 Overview • Problems with application packaging and deployment • Intro to functional package and configuration management • Towards the future • How you can help 3 User autonomy and control It is becoming increasingly difficult to have control over your own computing: • GNU/Linux package managers not meeting user needs • Self-hosting web applications requires too much time and effort • Growing number of projects recommend installation via curl | sudo bash 1 or otherwise avoid using system package managers • Users unable to verify that a given binary corresponds to the source code 1http://curlpipesh.tumblr.com/ 4 User autonomy and control “Debian and other distributions are going to be that thing you run Docker on, little more.” 2 2“ownCloud and distribution packaging” http://lwn.net/Articles/670566/ 5 User autonomy and control This is very bad for desktop users and system administrators alike. We must regain control! 6 What’s wrong with Apt/Yum/Pacman/etc.? Global state (/usr) that prevents multiple versions of a package from coexisting. Non-atomic installation, removal, upgrade of software. No way to roll back. Nondeterminstic package builds and maintainer-uploaded binaries. (though this is changing!) Reliance on pre-built binaries provided by a single point of trust. Requires superuser privileges. 7 The problem is bigger Proliferation of language-specific package managers and binary bundles that complicate secure system maintenance.
    [Show full text]
  • Chapter 7 Package Management
    Chapter 7 Package Management CERTIFICATION OBJECTIVES 7.01 The Red Hat Package Manager ✓ Two-Minute Drill 7.02 More RPM Commands Q&A Self Test 7.03 Dependencies and the yum Command 7.04 More Package Management Tools fter installation is complete, systems are secured, filesystems are configured, and other initial setup tasks are completed, you still have work to do. Almost certainly before your system is in the state you desire, you will be required to install or remove packages. To make sure the right updates are installed, you need to know how to get a system working with ARed Hat Subscription Management (RHSM) or the repository associated with a rebuild distribution. To accomplish these tasks, you need to understand how to use the rpm and yum commands in detail. Although these are “just” two commands, they are rich in detail. Entire books have been dedicated to the rpm command, such as the Red Hat RPM Guide by Eric 344 Chapter 7 Package Management Foster-Johnson. For many, that degree of in-depth knowledge of the rpm command is no longer necessary, given the capabilities of the yum command and the additional package management tools provided in RHEL 7. CERTIFICATION OBJECTIVE 7.01 The Red Hat Package Manager One of the major duties of a system administrator is software management. New applications are installed. Services are updated. Kernels are patched. Without the right tools, it can be difficult to figure out what software is on a system, what is the latest update, and what applications depend on other software.
    [Show full text]
  • Bulletin Issue 25
    Issue 25 Bulletin November 2014 Contents Free software needs your vote Free software needs your 1 by John Sullivan vote Executive Director What would a free 3 t the Free Software Foundation, software world look like? Awe want to empower all computer GNU Guix and GNU’s 4 users everywhere to do everything they 31st Birthday might need or want to do on any com- Appropriate legal 5 puter, using only free software, with- notices out having to ask permission. Free tools for the FSF 6 By definition, proprietary software Common misconceptions 7 does not empower users in this way. in licensing It places limits on what they can do, Volunteer opportunities 9 such as preventing sharing of the soft- at the FSF ware, or looking at its code to see how See you at LibrePlanet 10 it works. 2015! Proprietary software enables users Around the world in (a 11 to pursue everything they might need hundred and) eighty or want to do, only as long as the soft- days ware distributor approves. The four freedoms that define free software — to run the program (0), to study and modify it (1), to share it (2), and to share modifications (3) — are meant for everyone, in their inter- actions with any program. Free soft- ware is a means to protect the individ- ual freedom of computer users. But why would someone who has Register for LibrePlanet at u.fsf.org/14w. no intention of ever reading the source code of programs running on their computer, much less in modifying it, care about Freedom 1, or Freedom 3? Why do they need or want the freedom to do things they might never need or want to do? 1 One reason is that any computer general, the right to vote can be a pow- user can ask someone else to do those erful check on government behavior.
    [Show full text]
  • GNU/Linux Operating System
    A Bibliography of Publications about the GNU/Linux Operating System Nelson H. F. Beebe University of Utah Department of Mathematics, 110 LCB 155 S 1400 E RM 233 Salt Lake City, UT 84112-0090 USA Tel: +1 801 581 5254 FAX: +1 801 581 4148 E-mail: [email protected], [email protected], [email protected] (Internet) WWW URL: http://www.math.utah.edu/~beebe/ 07 April 2021 Version 2.135 Title word cross-reference [Tho05]. 0-13-167984-8 [Sta07b]. 0-596-00482-6 [Sch04]. 0-7821-4428-4 [Koh06]. '03 [ACM03b]. 046 [Sav11]. '05 [ACM05b, MS05]. + [Ste01e]. $100 [CS95]. $39.95 [Sch04]. $44.99 [Sta07b]. $49.95 [Jen05]. $49.99 1 [FOP06, Jen05, She03]. 1-59327-036-4 [Hid04, Tho05]. $59.99 [Koh06]. $99 [Jen05]. 1-GHz [Ano03b]. 1.0 [Coc01]. 1.2 [Kro00]. = [Ste01e]. × [Hun99]. [Gar98]. 1.x [KGG00]. 10 [DWV06]. 10-Gigabit [cFJH+03]. 10th [USE96a]. * [TYKZ07]. */ [TYKZ07]. *BSD [Den99a]. 12-step [Mil01]. 12th [MS05]. 1394 *icomment [TYKZ07]. [Ale00, HKP09]. 14-16 [ACM06]. 18th [KD96]. 1999 [Den99b, Tim99]. 19th -dienste [WF03]. [ACM03b, SS05b]. 1Z0 [Sav11]. 1Z0-046 [Sav11]. /*icomment [TYKZ07]. /GNOME [Wri00, Pen99]. 2 [Ano94c, Com00, Com03, Gab07, MK04]. 2.0 [B¨ol01, Car98, McN99, PF97, Swe01]. 0 [Hid04, Koh06, Sch04, Sta07b, Tho05]. 2.0.1 [ISO05]. 2.1 [BR95, CV00]. 2.2 0-13-101415-3 [Hid04]. 0-13-144853-6 1 2 [Ano00b, BB99b, Bra04]. 2.4 [Cal00]. 2.6 [Mon00b, GR09]. Action [NR03]. ActiveX [BS05, PTS+14, TCM07]. 2000 [Kro99]. activity [MB08]. Acumen [Kro99]. [Bru02, Kro00, MYH00, War01]. 2003 Ada [SB99]. Ada95 [Gar09].
    [Show full text]
  • CDE: Run Any Linux Application On-Demand Without Installation
    CDE: Run Any Linux Application On-Demand Without Installation Philip J. Guo Stanford University [email protected] Abstract with compiling, installing, and configuring software and their myriad of dependencies. For example, the official There is a huge ecosystem of free software for Linux, but Google Chrome help forum for “install/uninstall issues” since each Linux distribution (distro) contains a differ- has over 5800 threads. ent set of pre-installed shared libraries, filesystem layout In addition, a study of US labor statistics predicts that conventions, and other environmental state, it is difficult by 2012, 13 million American workers will do program- to create and distribute software that works without has- ming in their jobs, but amongst those, only 3 million will sle across all distros. Online forums and mailing lists be professional software developers [24]. Thus, there are are filled with discussions of users’ troubles with com- potentially millions of people who still need to get their piling, installing, and configuring Linux software and software to run on other machines but who are unlikely their myriad of dependencies. To address this ubiqui- to invest the effort to create one-click installers or wres- tous problem, we have created an open-source tool called tle with package managers, since their primary job is not CDE that automatically packages up the Code, Data, and to release production-quality software. For example: Environment required to run a set of x86-Linux pro- grams on other x86-Linux machines. Creating a CDE • System administrators often hack together ad- package is as simple as running the target application un- hoc utilities comprised of shell scripts and custom- der CDE’s monitoring, and executing a CDE package re- compiled versions of open-source software, in or- quires no installation, configuration, or root permissions.
    [Show full text]
  • Functional Package Management with Guix
    Functional Package Management with Guix Ludovic Courtès Bordeaux, France [email protected] ABSTRACT 1. INTRODUCTION We describe the design and implementation of GNU Guix, a GNU Guix1 is a purely functional package manager for the purely functional package manager designed to support a com- GNU system [20], and in particular GNU/Linux. Pack- plete GNU/Linux distribution. Guix supports transactional age management consists in all the activities that relate upgrades and roll-backs, unprivileged package management, to building packages from source, honoring the build-time per-user profiles, and garbage collection. It builds upon the and run-time dependencies on packages, installing, removing, low-level build and deployment layer of the Nix package man- and upgrading packages in user environments. In addition ager. Guix uses Scheme as its programming interface. In to these standard features, Guix supports transactional up- particular, we devise an embedded domain-specific language grades and roll-backs, unprivileged package management, (EDSL) to describe and compose packages. We demonstrate per-user profiles, and garbage collection. Guix comes with a how it allows us to benefit from the host general-purpose distribution of user-land free software packages. programming language while not compromising on expres- siveness. Second, we show the use of Scheme to write build Guix seeks to empower users in several ways: by offering the programs, leading to a \two-tier" programming system. uncommon features listed above, by providing the tools that allow users to formally correlate a binary package and the Categories and Subject Descriptors \recipes" and source code that led to it|furthering the spirit D.4.5 [Operating Systems]: Reliability; D.4.5 [Operating of the GNU General Public License|, by allowing them to Systems]: System Programs and Utilities; D.1.1 [Software]: customize the distribution, and by lowering the barrier to Applicative (Functional) Programming entry in distribution development.
    [Show full text]
  • Reproducible Builds Summit II
    Reproducible Builds Summit II December 13-15, 2016. Berlin, Germany Aspiration, 2973 16th Street, Suite 300, San Francisco, CA 94103 Phone: (415) 839-6456 • [email protected] • aspirationtech.org Table of Contents Introduction....................................................................................................................................5 Summary.......................................................................................................................................6 State of the field............................................................................................................................7 Notable outcomes following the first Reproducible Builds Summit..........................................7 Additional progress by the reproducible builds community......................................................7 Current work in progress.........................................................................................................10 Upcoming efforts, now in planning stage................................................................................10 Event overview............................................................................................................................12 Goals.......................................................................................................................................12 Event program........................................................................................................................12 Projects participating
    [Show full text]
  • Hurd's PCI Arbiter
    Hurd’s PCI arbiter Samuel Thibault & Joan Lledo 2018 February 3rd 1 The Hurd is all about freedom #0 “The freedom to run the program, for any purpose” I.e.: ● Freedom from sysadmin! ● WTH is fdisk/mke2fs/... hidden in /sbin? ● I should be able to just work with my disk/network access ● Freedom to innovate ● Experimental filesystem, personal work-flow, new kind of process combination,… ● Give a PCI card function to a process ● Freedom from misbehaving programs and drivers 2 Micro-kernel layering ext2fs sh auth cp pfnet proc root user Kernel Tasks, memory, IPC 15 Micro-kernel layering ext2fs sh auth cp pfnet proc root user Kernel Tasks, memory, IPC 16 Micro-kernel layering ● Server crash? Not a problem ● “Computer bought the farm” is just an error, not something-of-the-death ● Easier to debug/tune ● Just run gdb, gprof, … ● Can dare crazy things ● The Hurd console has dynamic font support – See chinese support in pseudo-graphical mode (actually pure VGA textmode!) of Debian installer. – And Emojis! ● Kernel only handles Tasks, memory, IPC 17 Hurd possibilities ext2fs ftpfs sh auth cp pfnet proc isofs root user Kernel 18 Hurd possibilities € settrans -c ~/ftp: /hurd/hostmux /hurd/ftpfs / (just once for good) € settrans -a ~/mnt /hurd/iso9660fs ~/ftp://ftp.gnu.org/old-gnu/gnu-f2/hurd-F2-main.iso € ls ~/mnt "#ADME-or-FA'( … ● Only downloads what is needed. ● Can be permanently stored in ext2fs € settrans ~/.signature /hurd/run /usr/games/fortune 19 But also € remap /)in/sh *+,&#/)in/sh € remap /)in *+,&#/union)in … ● Check out Stow/Nix/Guix! 21 How does it work? ext2fs sh ftpfs libc auth cp pfnet libc proc isofs root user Kernel 22 Rationale - Everything is an (interposable) RPC - Translators exposed in the FS ● The user gets to decide what/how to interpose ● Without need for costly ptrace or fragile libc symbols interposition.
    [Show full text]
  • Conflict Resolution Via Containerless Filesystem Virtualization
    Dependency Heaven: Conflict Resolution via Containerless Filesystem Virtualization Anonymous Author(s) Abstract previous installation, effectively preventing concurrent ver- Resolving dependency versioning conflicts in applications sions of that library from coexisting. The same is true for is a long-standing problem in software development and packages whose executable names does not change across deployment. Containers have become a popular way to ad- releases; unless the user renames the existing executable dress this problem, allowing programs to be distributed in a files prior to the installation of a new version it is notpos- portable fashion and to run them under strict security con- sible to keep both installations around. The problem with straints. Due to the popularity of this approach, its use has that approach is that it breaks package managers, as the re- started to extend beyond its original aim, with users often named files will not be featured in the package manager’s creating containers bundling entire Linux distributions to database and, consequently, will not be tracked anymore. run mundane executables, incurring hidden performance Further, unless executables depending on the renamed files and maintenance costs. This paper presents an alternative are modified to reflect their new path, users need todefine approach to the problem of versioning resolution applied to which executable to activate at a given time, usually through locally-installed applications, through a virtualization tool tricky management of symbolic
    [Show full text]