Building applications at once for Flatpak, Snapd, Docker
Valentin David @valentindavid
Codethink Ltd.
All Systems Go! 2019 September 21st
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 1 / 22 Table of Contents
1 Motivation
2 Building for multiple backends
3 BuildStream
4 Current state
5 Appendix
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 2 / 22 Porting an application to Linux
A developer has made and application for Android, iOS, MacOS X and Windows and decides to port it to “the Linux platform”.
They decide to download “the Linux SDK”...
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 3 / 22 What should I use to publish my application?
Flatpak Snapd Docker, Podman, Buildah Live image for virtual machine Source tarball only AppImage Binary tarballs with manual installation Self-extracting script .deb package, Debian? Ubuntu? Which versions? RPM package, Fedora? OpenSUSE? Which versions? pip, gem, cpan, npm
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 4 / 22 Fragmentation
Software source
.json manifest Freedesktop SDK snap.yml Ubuntu Core Dockerfile Many SDKs
FlatpakBuilder snapcraft build Buildah, Docker
Flatpak ostree ref .snap file OCI/Docker image
flat-manager-client snapcraft push docker/podman push
flat-manager repository (e.g. Flathub) Snapcraft store Registry (e.g. Docker Hub)
Flatpak (flatpak install) Snapd (snap install) docker/podman pull Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 5 / 22 Table of Contents
1 Motivation
2 Building for multiple backends
3 BuildStream
4 Current state
5 Appendix
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 6 / 22 What is Freedesktop SDK?
Small userland runtime for desktop applications Not a distro: no package manager, configuration tools, or installation. Base for GNOME SDK and KDE SDK for Flatpak Targetting first Flatpak Built with BuildStream
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 7 / 22 What is BuildStream?
Integration of software stacks Backend agnostic, it creates artifacts Extensible with plugins
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 8 / 22 Factorizing the build process
Software source Freedesktop SDK
BuildStream project
BuildStream
Flatpak repo .snap file Docker/OCI image
flat-manager-client snapcraft push docker/podman push
flat-manager (e.g. Flathub) Snapcraft store Registry (e.g. Docker Hub)
Flatpak (flatpak install) Snapd (snap install) Snapd (docker/podman pull)
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 9 / 22 Table of Contents
1 Motivation
2 Building for multiple backends
3 BuildStream
4 Current state
5 Appendix
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 10 / 22 Development with BuildStream
A BuildStream project contains multiple elements. An element is a .bst file. An element typically builds one project (e.g. gcc.bst builds GCC). Elements have dependencies (build and/or runtime) to other elements. It is a graph. Each elements is built in an environment only containing build dependencies. Built artifacts are cached, locally and eventually remotely. To integrate elements together, they can have integration commands. Like a post-install script. Element can depend on elements from external BuildStream projects. We call it a “junction”.
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 11 / 22 Example of element
kind: autotools depends: - filename: bootstrap-import.bst junction: freedesktop-sdk.bst - filename: public-stacks/buildsystem-autotools.bst junction: freedesktop-sdk.bst type: build - filename: components/curl.bst junction: freedesktop-sdk.bst - filename: libevent.bst - filename: components/gtk3.bst junction: freedesktop-sdk.bst - filename: intltool.bst type: build - filename: libappindicator.bst
variables: conf - local: >- --with - gtk --enable-cli --enable-daemon sources: - kind: git_tag url: https://github.com/transmission/transmission.git track: master track-extra: -’2.9x’ track - tags: true ref: 2.94-0-gd8e60ee44f4295935bd98bf741f85ed19f5a7dfb
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 12 / 22 Plugins
Flatpak flatpak image prepares the image for flatpak build-export. flatpak repo builds the Flatpak repository with multiple flatpak image elements.
Snap snap image prepares the image and metadata. We use the generic script to call mksquashfs.
Docker/OCI oci plugin creates a layer chain or image per element. Either Docker 1.2 or OCI.
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 13 / 22 OCI image layering
OCI images are stacks of layers, BuildStream uses graphs Each Docker/OCI layer has an identifier. But implementation typically use chain identifier. So layers in mis-ordered or partial chains are not reused. Some filesystem implementation might not handle well 500 layers. We let the user explictly decide of the layering to improve reuse. Because downstream projects import the Freedestkop SDK graph, it imports also the OCI layers.
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 14 / 22 OCI image layering example
debug-oci.bst
sdk.bst sdk-oci.bst
Dependency graph platform.bst platform-oci.bst
bootstrap.bst bootstrap-oci.bst
junction junction junction
Dependency graph transmission-daemon.bst transmission-oci.bst
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 15 / 22 Table of Contents
1 Motivation
2 Building for multiple backends
3 BuildStream
4 Current state
5 Appendix
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 16 / 22 Examples
Firefox For Flatpak and Snap https://gitlab.com/valentindavid/firefox-buildstream
Transmission GTK UI for Flatpak and Snap Server for Docker/OCI https://gitlab.com/valentindavid/transmission-buildstream
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 17 / 22 Current issues
Strict confinement not yet working with Snapd. Snapd and Flatpak have a different way to mount /etc. Snapd mounts /etc from host and replaces some files from runtime Flatpak mounts /etc from runtime and bind some files from host. Nvidia drivers not yet tested in Snap. We need an equivalent to Flatpak extensions in OCI and Snapd. Difficult to deal with users for OCI images.
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 18 / 22 Questions?
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 19 / 22 Table of Contents
1 Motivation
2 Building for multiple backends
3 BuildStream
4 Current state
5 Appendix
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 20 / 22 Firefox dependencies
flatpak/repo.bst snap/image.bst
flatpak/org.mozilla.Firefox.Debug.bst flatpak/org.mozilla.Firefox.bst snap/firefox.bst
flatpak/flatpak-debug-image.bst flatpak/flatpak-image.bst flatpak/export.bst snap/firefox-composed.bst
firefox.bst snap/desktop-file.bst snap/firefox-wrapper.bst
libevent.bst cbindgen.bst dbus-glib.bst gtk2.bst icu.bst nodejs.bst startup-notification.bst yasm.bst autoconf-2.13.bst
python2.bst nghttp2.bst c-ares.bst libuv.bst app-lib.bst
freedesktop-sdk
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 21 / 22 Transmission dependencies
snap/image.bst
flatpak/repo.bst snap/transmission.bst
oci/transmission-docker.bst oci/transmission-oci.bst flatpak/com.transmissionbt.Transmission.Debug.bst flatpak/com.transmissionbt.Transmission.bst snap/transmission-composed.bst
oci/layers/transmission.bst flatpak/flatpak-debug-image.bst flatpak/export.bst flatpak/flatpak-image.bst snap/desktop-file.bst snap/transmission-wrapper.bst
transmission-daemon.bst transmission-gtk.bst
transmission.bst
libevent.bst libappindicator.bst
libindicator.bst libdbusmenu.bst
intltool.bst dbus-glib.bst
app-lib.bst freedesktop-sdk
Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 22 / 22