arXiv:2104.00048v1 [cs.SE] 31 Mar 2021 ihu ytmr-ul n re-flash. and re-build system a without ecm,kok,adohrpout fe udea bundle often products other and kiosks, webcams, blt:sfwr yial antb ntle rupgrade or installed us- be in cannot disadvantages typically significant on ability: with reproducible comes based but guarantees firmware behavior, This read-only configurations. into declarative system boxes, such set-top GNU/ systems TV , Linux industrial Embedded , OS. IoT reproducible as the for from allow behavior models configuration Declarative drivers. malfunctioning broken or as ssh, such inaccessible behavior boot, in system result in can configura- and changes state, system unpredictable initial the known the to from diverges modification tion Each uninstall, with [2]. install, filesystem ) (i.e. mutable commands a management on imperative operate managers Package versioning architectures. and processor availability across vulnerable vary Package can potentially [1]. and software out-of-date system in con resulting version creating flicts often dependencies, co system with dependencies exist Application maintainers. managed distribution files, system by to and applications managers upgrade package and binary install use portability. lack distributions Linux typically Most BSPs result firmware a and as root , operate, kernel, to mutable of combination requires a unique platform a Each manager. containing package a with (BSPs) filesystem "board packages" distribute vendors support (SBC) Board Single Computer Board Single Terms Index (OTA). host and Over-the-Air upgraded separates The backed-up independently applications. are cleanly containers the and approach from system support This hardware device. any the to or configurations which platform system tool system re-targets cross-compiler layering embedded automatically configuration the a distributions for and containerized applications, hosting and for sys- GNU/Linux optimized cross-compiled tem require- minimal stored these a with addresses firmware ments SkiffOS memory. immutable read-only com- through in OS, host achieved the from monly behavior ro- reproducible require and applications bust and These robotics (IoT). as Things such of tasks real-time for used Abstract Ebde iu rcsosaeincreasingly are processors Linux —Embedded kffS iia rs-opldLnxfor Linux Cross-compiled Minimal SkiffOS: Cnanr meddLnx Robotics; Linux; Embedded —Container; .Introduction I. meddContainers Embedded [email protected] prueRbtc LLC. Robotics Aperture hita Stewart Christian d - - h kffSGtrpstr a eebde nexten- in embedded be can repository SkiffOS The emu- It experience. user the in variance minimal with sKbree n okrSamcnb sdt remotely [6]. to workloads used monitor be and can deploy and Swarm Docker be portability and Kubernetes such can enhance as platforms Workloads management to Container parallel. [5]. images reproducibility in container as run defined be containerized and can distributions applications Multiple from system. used independently then host software manage the can and and managers install to imported Package used images be be container SkiffOS can as given directly userspaces a system that Existing output. ensure identical to produce always used will tree. commit are Git builds SkiffOS offline the reproducible is and within pinning, version Buildroot package sub-module Checksumming, out-of- overrides. a with as and extended referenced packages and configuration sub-module a tree as projects sion in compiled "workspaces." be with boot can parallel to configurations system system the Multiple are installing media. targets and Makefile formatting computer. for system target available a the producing for virtual- time, tuned build image and at enabled op- systems are performance timizations target Hardware-specific numerous environments. ization for Configurations packages. supplied named are into installa and scripts hooks, tion/utility build files, Buildroot system packages, files, extension configuration bundle layers Configuration storage. envi- persistent package-manager to containerized of attached with ronments utility distributions the GNU/Linux be- based providing reproducible while produces which havior, immutable image an system with operating approaches platforms, firmware compute traditional lates of set diverse host- a for across system containers cross-compiled ing minimal a a builds across SkiffOS [4]. development configurations rapid hardware configura- of for set and diverse allowing structure language understand and utilities, to tion system pack- easy major software an all 2500 provides for over support includes providing It ages [3]. userspa drivers, and kernels, KConfig- , and GNU/Linux including of systems, Makefile cross-compilation a automated for providing tool project based a is Buildroot I kfO:Frwr o otn Containers Hosting for SkiffOS: II. ces, 1 - 2

III. Implementation A. Configuration Layers The SkiffOS base configuration, builds a minimal host SkiffOS is available under the MIT license, and references with hardware support, OpenSSH server, Buildroot as a sub-module under the GPLv2 license, with and system management tools. Enabling additional config- a series providing additional features and bug fixes. uration layers adds use-case specific functionality. Configu- Changes are frequently submitted upstream to the Build- rations are organized into logical units called "layers" with root mailing list. the following structure:

As of Release 2020.11.7 the target support table is: • cflags: additional target compiler flags • buildroot: buildroot configuration fragments • buildroot_ext: buildroot extensions System Config Package Kernel • buildroot_patches: buildroot package patches Apple Macbook () apple/macbook 5.11.2 • extensions: utility commands • hooks: scripts hooking pre/post build steps BananaPi M1 bananapi/m1 5.11.2 • kernel: kernel configuration fragments BananaPi M1+/Pro bananapi/m1plus 5.11.2 • kernel_patches: kernel .patch files • root_overlay: root overlay files BananaPi M2+ bananapi/m2plus 5.11.2 • metadata: metadata files BananaPi M3 bananapi/m3 5.11.2 – commands: targets in "extensions" makefile Docker Container virt/docker N/A – dependencies: comma-separated layers – description: single-line description Intelx86/64 intel/x64 5.11.2 – unlisted: if exists, hidden from "help" Msft Windows (WSL) virt/wsl N/A • resources: support files Nano jetson/nano 4.9.140 • scripts: used by extensions and/or hooks • uboot: u-boot configuration fragments NVIDIA Jetson TX2 jetson/tx2 4.9.140 • uboot_patches: u-boot .patch files Odroid C2 odroid/c2 tb-5.9.16 Configuration layers can override options set by previous Odroid C4 odroid/c4 tb-5.9.16 layers, making it simple to re-target configurations to Odroid HC1/2, XU3/4 odroid/xu tb-5.9.16 various compute platforms by merging with the desired Odroid U odroid/u tb-5.9.16 hardware configuration layer. The set of desired configura- tion layers is defined as an ordered comma-separated list, OrangePi Lite orangepi/lite 5.11.2 for example: OrangePiZero orangepi/zero 5.11.2 SKIFF_CONFIG=pi/4,core/gentoo PcDuino 3 pcduino/3 5.11.2 SkiffOS can be extended [7] by adding it as a sub-module PcEngines APU2 pcengines/apu2 5.11.2 of a project repo. Project configurations can then be Pi0 pi/0 5.4.72 specified in additional configuration layers. Pi1 pi/1 5.4.72 B. Persistent data partition Pi3(and1/2) pi/3 5.4.72 The persist partition contains a skiff directory with: Pi4 pi/4 5.4.72 H64 pine64/h64 5.8.0 • connections: network-manager connections • core: skiff-core configuration and state PinePhone pine64/phone megi-5.9.11 • docker: docker state and storage PinebookPro pine64/book 5.11.2 • etc: configuration tree overlay • hostname: system hostname Qemu (VM) virt/qemu 5.11.2 • journal: -journald system logs Virtualbox (VM) virt/qemu 5.11.2 • keys: public keys for access to "root" user • Rockpro64 pine64/rockpro64 5.9.0 ssh: ssh server keys and persistent configuration System startup scripts mount the persist partition and Legal information and licenses for all dependencies can create the file structure at boot time. Most configurations be bundled together with the legal-info command. will create a memory swapfile to avoid failure due to an out- Some board support packages include proprietary binary of-memory condition. OpenSSH is configured for public- blobs (typically firmware) and are denoted as "Proprietary" key access with the persist "keys" directory or in the OS in the produced licensing information bundle. image. 3

The persist partition is automatically resized to fit the directory contains patches for Buildroot packages, i.e. fixes remainder of the available storage space on first boot. for platform errata. For convenience, kernel and uboot Bootloaders such as u-boot are sometimes copied to the patches can also be specified in the kernel_patches and beginning of the storage media. Some systems use more uboot_patches directories. complex partition layouts mandated by the hardware ven- dor. Skiff and Buildroot’s flexible configuration language F. SkiffOS extensions supports this variance between target systems. The extensions configuration layer directory contains a Makefile which implements custom commands made . Over-the-air (OTA) available to the user in the help screen: The SkiffOS system typically consists of fewer than five cmd/pi/common/format: Format a SD card. files, including the root filesystem /initramfs, ker- cmd/pi/common/install: Install to a SD card. nel image, and kernel modules squashfs. The kernel and immutable boot system can be atomically upgraded by re- Commands are prefixed by their layer name and placing these files at run-time. The push_image.sh script can be executed as Makefile targets, for example, is provided, using rsync and ssh to upload the updated make cmd/pi/common/install, and are declared in the files to a running system. In some cases, the firmware metadata/commands text file: and/or bootloader is also upgraded by the script. format Format a SD card and install bootloader. ./scripts/push_image.sh root@my-device-ip install Installs to a formatted SD card.

D. Kconfig configuration fragments G. Root filesystem overlay Buildroot organizes software components into "packages." The root_overlay trees are copied to the target filesystem Packages can be enabled and configured using the Kconfig image at the end of the build in the order that the layers language. The available options can be explored using the were specified. This is used to add additional configuration make or make xconfig configuration menus. files or scripts to the root SkiffOS system image. For Configuration options are specified in "fragments" that are example, the pi/common layer adds configuration under merged together into a single ".config" file and provided to the etc directory, and firmware configuration under the the Buildroot build system. The Buildroot build system usr directory. manages merging together kconfig and u-boot configura- tion fragments and applying patches. H. Temporary local overrides Example buildroot configuration fragment: Configuration overrides can be specified in the overrides BR2_LINUX_KERNEL_DEFCONFIG="versatile" directory tree, which contains additional configuration lay- BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="5.11.2" ers implicitly added to the build. For example, temporary local buildroot overrides for all workspaces can be declared Buildroot and the Linux kernel use the Kconfig configura- in overrides/buildroot, and kernel configuration frag- tion language. The available options can be explored with ments affecting the pi4 workspace alone can be declared make br/menuconfig and make br/kernel-menuconfig in overrides/workspaces/pi4/kernel. configuration menus. Configuration fragments are merged together in SKIFF_CONFIG order followed by lexicographic IV. Skiff Core: Containerized Environments filename order. Skiff Core includes the Docker containerization runtime This example kernel configuration fragment enables the and a Go program for automating the creation and initial ext3/ext4 filesystem: setup of containerized environments. It can be enabled with the skiff/core configuration layer. User sessions CONFIG_EXT3_FS=m are routed to the container assigned to their account. CONFIG_EXT3_FS_SECURITY=y Multiple OS distributions can be installed simultaneously CONFIG_EXT3_FS_XATTR=y on a single machine. Existing userspaces including vendor- CONFIG_EXT3_POSIX_ACL=y provided software images can be imported as container CONFIG_EXT4_FS=y images. CONFIG_EXT4_FS_SECURITY=y CONFIG_EXT4_POSIX_ACL=y The skiff-core binary is configured as the user shell, and intercepts incoming SSH sessions to redirect them to the The m option denotes a feature built as a kernel module corresponding container. The container setup process is instead of "built-in." displayed if the container(s) are not yet ready. The usual userspace init daemon, typically systemd, is run as PID E. Buildroot extensions 1, and standard approaches for defining systemd services Configuration layers can extend Buildroot with packages and the systemctl CLI tool function identically to when in the buildroot_ext directory. The buildroot_patches running without containerization. 4

The "core" system can be updated or rolled-back inde- currently based on . To pull, for example, the pendently from the "root" operating system. Mountpoints gentoo-lto image, which includes the Gentoo core image are used to mount user home directories and other tem- with gentooLTO optimizations, the Docker CLI command porary paths so that "docker export" and "docker save" is: include system files only. Containers are portable between docker pull skiffos/skiff-core-gentoo-lto:latest machines of similar architecture without target-specific configuration. When using "Skiff Core," the Docker container engine is used, which leverages Linux namespaces and not vir- The container system is configured with a YAML file: tualization or emulation, implementing "OS-level virtual- containers: ization as opposed to hardware " [10]. The core: PID namespace is used to allow running the usual system image: skiffos/skiff-core-gentoo:latest initialization and management daemons as the privileged mounts: PID 1 within the container. - /dev:/dev The results of [11] show "an almost negligible impact - /etc/resolv.conf:/etc/resolv.conf:ro of the [Docker container] layer in terms of performance, - /mnt/persist/data:/home if compared to native execution." While throughput is [ ] ... not significantly affected, the results of [6] show that : users namespacing can cause a measurable signal processing : core delay. To mitigate potential impacts of processing latency, : container core most container isolation is disabled. containerUser: core :{ : } auth copyRootKeys true V. Comparison with Existing Approaches images: skiffos/skiff-core-gentoo:latest: This section discusses several of the current approaches pull: in use today, and their disadvantages when applied to policy: ifnotexists embedded Linux and/or robotics development: registry: quay.io 1) Binary Package Distributions: Traditional board- build: support packages use binary package distribution systems source: /opt/skiff/coreenv/base such as ’s Advanced Package Tool (APT). Several key disadvantages of this approach are lack of hardware- Several OS distribution configurations are available as specific optimizations, reliance on third-party infrastruc- configuration layers. If a pre-built image is unavailable, or ture for builds and maintenance, inability to reproduce a the pull section of skiff-core.yaml is empty, the system system from , and difficulties with portability. will instead build the included Dockerfile. The rolling nature of package upgrades introduces unpre- OS Distribution Config Package Website dictable behavior when upgrading a system, particularly when a long time has passed since the previous upgrade Gentoo core/gentoo gentoo.org [1]. core/manjaro manjaro.org Current widely-used package management tools install NixOS core/ nixos.org system files, firmware, and user applications together into a single mutable "root" filesystem. Imperative install, re- Ubuntu skiff/core ubuntu.org move, and upgrade commands instruct the package man- ager to perform operations on the system. Interrupted Additional images optimized for specific use cases are operations might leave some files in a available: partially written state. If the affected (now corrupted) files Image Description are critical to system boot and/or reachability, the only recourse to fix the system may be to remove the boot gentoo-lto-exwm LTO, Apps, Emacs X11 Workflow media from the machine, connect it to a different device, and fix the issue manually. gentoo- KDE Desktop w/ apps gentoo-lto O3, Graphite, Link-time optimize SkiffOS addresses the concern of always having repro- ducible boot-up and reachability behavior in embedded nasa-cfs Flight systems framework systems by to an immutable root filesystem image, nasa-fprime Flight systems framework mounting persistent storage, and running user applications and operating systems inside lightweight Linux containers. -neon KDE Neon for PinePhone This mitigates the risks of mid-upgrade power brownout: -manjaro-kde Manjaro KDE for PinePhone the immutable portion of the system is always reachable, allowing the user can connect to the container and fix the The NASA Fprime[8] and NASA cFS[9] images are problem without physical intervention. 5

Optimization of compute performance and reduction of upgrades in containers. Software not compiled by Nix can energy usage are important considerations for resource- still be run in a concurrent container. constrained embedded devices (the "power budget"). For 3) Buildroot: The Buildroot [3] project provides a auto- most robotics applications, the "power budget" is a sig- mated system cross-compiler. Users typically download nificant factor in determining maximum range/endurance. a Buildroot release, create a configuration by hand, and Compilation of the operating system and applications from store this along with their other project files. It focuses source allows fine-tuning of build output to the hardware on providing a toolset for Embedded Linux developers to make maximum use of energy saving optimizations. to produce system images for use with a single target platform and/or product. Backing up the system by creating a full bit-for-bit copy is a time-consuming process which often leads to infrequent SkiffOS extends Buildroot to simplify this process with backups and, as a result, occasional data loss due to an easy to understand configuration layering architecture, storage card failure. SkiffOS provides an alternate ap- which merges together the platform support configs with proach, in which existing OS distributions can be imported the selected components to configure the build automat- as containers. Containers are easy to back up, restore, ically. The configuration layers can be stored externally roll-back, and are portable between machines of similar to the SkiffOS tree. Existing Buildroot projects can be architecture. Any vendor-provided BSP can be imported adapted as configuration layers and ported to any of the as a container and used without sacrifices to workflow or available target configurations. compatibility. 4) : The Yocto Project [13] is closely com- 2) NixOS: NixOS [2] argues that imperative package parable to Buildroot, and is primarily focused on the managers destructively update the state of the system, python-based "" tool, which is derived from Gen- leading to "inability to roll back changes easily, to run too’s "," with a focus on cross-compilation of com- multiple versions of a package side-by-side, to reproduce a plex embedded Linux systems. This is contrasted with configuration deterministically on another machine, or to Buildroot’s focus on simplicity, using the Makefile and reliably upgrade a system." NixOS includes a declarative Kconfig architecture [14]. Yocto system configurations are package manager which performs modifications according fragmented into "overlay" repositories from various sources, to a specification of target system state. Nix can perform while Buildroot focuses on a consolidated and curated atomic upgrades with the ability to roll-back. package tree with strict and opinionated code style. Yocto has many more packages than Buildroot. SkiffOS is designed around the same principles of im- mutability and declarative configuration, but provides Buildroot [3] forms a linear commit history with release these guarantees through compilation of the system in checkpoints, and Skiff releases pin the version of the advance, loading an immutable and ephemeral root system Buildroot sub-module along with the versions of the device at boot-up. It also addresses the other issues described firmware, kernels, and board support files. A given Git by the NixOS , including running multiple systems checkout is reproducible forever [4]. Yocto setups may side-by-side in Linux containers, and reproducing a system suffer from loss of availability of overlay repositories, or configuration deterministically at a later time. de-synchronization between changes to the overlay reposi- tories and changes to the core system configurations. NixOS breaks the Filesystem Hierarchy Standard (FHS) [12] declared by the and followed by VI. Conclusion most distributions, instead using a flat symbolic link-based SkiffOS offers reproducible system behavior, offline com- structure. This causes incompatibility with software not pilation, immutable host system for containers attached compiled by Nix. On the other hand, Buildroot [3] (and to persistent storage, board-specific build re-targeting via therefore SkiffOS) follow the FHS for compatibility with config layers, and Over-The-Air upgrade/downgrade. The existing glibc-based binaries. host system and/or containers can be easily customized for SkiffOS focuses on providing a minimal "shim" to ab- specific use cases with cross-platform configuration layers. stract away the differences between hardware (particu- The "Skiff Core" workflow provides an easy way to import larly Single-Board ) such that the containers any existing Linux userspace such that users will likely not are portable to new platforms. It does not distribute realize they are now working in a container. Background complex packages like web browsers, graphical interfaces, workloads can run in parallel Linux containers, indepen- and other user applications. This responsibility is left to dently isolated from the user workspace(s) including re- the distributions running in Linux containers attached source management and quality-of-service (QoS). to persistent storage and/or customization of the cross- SkiffOS is available under the MIT license at: compiled system. https://github.com/skiffos/skiffos NixOS has been integrated with SkiffOS as a "core" container configuration. This approach uses Buildroot to Buildroot is available under the GPLv2 license at: manage the hardware specifics, and NixOS to handle https://buildroot.org declarative system configuration and rolling application 6

References

[1] R. D. Cosmo, S. Zacchiroli, and P. Trezentos, “Package up- grades in FOSS distributions: Details and challenges,” CoRR, vol. abs/0902.1610, 2009. arXiv: 0902.1610. [Online]. Available: http://arxiv.org/abs/0902.1610. [2] E. Dolstra and A. Löh, “Nixos: A purely functional ,” in Proceedings of the 13th ACM SIG- PLAN International Conference on Functional Programming, ser. ICFP ’08, Victoria, BC, Canada: Association for Com- puting Machinery, 2008, pp. 367–378, isbn: 9781595939197. doi: 10 . 1145 / 1411204 . 1411255. [Online]. Available: https://doi.org/10.1145/1411204.1411255. [3] Buildroot. [Online]. Available: https://buildroot.org/ (visited on 03/17/2021). [4] J. Diamond and K. Martin, “Managing a real-time embedded linux platform with buildroot,” Fermi National Accelerator Lab, 2015. [Online]. Available: https://www.osti.gov/biblio/1250794. [5] P. Z. Vaillancourt, J. E. Coulter, R. Knepper, and B. Barker, Self-scaling clusters and reproducible containers to enable sci- entific computing, 2020. arXiv: 2006.14784 [cs.DC]. [6] Y. Wang and Q. Bao, “Adapting a container infrastructure for autonomous vehicle development,” in 2020 10th Annual Computing and Communication Workshop and Conference (CCWC), 2020, pp. 0182–0187. doi: 10.1109/CCWC47524.2020.9031129. [7] Skiffos extension template. [Online]. Available: https : / / . com / skiffos / skiff - ext - example (visited on 03/17/2021). [8] R. Bocchino, T. Canham, G. Watney, L. Reder, and J. Levison, “F prime: An open-source framework for small-scale flight software systems,” 2018. [9] D. McComas, “Nasa/gsfc’s flight software core flight system,” in Flight Software Workshop, vol. 11, 2012. [10] J. M. Mbongue, D. T. Kwadjo, and C. Bobda, Performance exploration of virtualization systems, 2021. arXiv: 2103.07092 [cs.DC]. [11] R. Morabito, “A performance evaluation of container on devices,” in 2016 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), 2016, pp. 999–1000. doi: 10.1109/INFCOMW.2016.7562228. [12] Filesystem hierarchy standard. [Online]. Available: https://refspecs.linuxfoundation.org/FHS_3.0/fhs- 3.0.pdf (visited on 03/17/2021). [13] Yocto project. [Online]. Available: https://www.yoctoproject.org/ (visited on 03/17/2021). [14] T. P. Alexandre Belloni, “Buildroot vs. /yocto project: A four hands discussion,” in Embedded Linux Conference, 2016. [Online]. Available: https://events.static.linuxfound.org/sites/events/files/slides/belloni-petazzoni-buildroot-oe_0.pdf (visited on 03/17/2021).