Building applications at once for , Snapd, Docker

Valentin David @valentindavid

Codethink Ltd.

All Systems Go! 2019 September 21st

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 1 / 22 Table of Contents

1 Motivation

2 Building for multiple backends

3 BuildStream

4 Current state

5 Appendix

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 2 / 22 Porting an application to

A developer has made and application for Android, iOS, MacOS X and Windows and decides to port it to “the Linux platform”.

They decide to download “the Linux SDK”...

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 3 / 22 What should I use to publish my application?

Flatpak Snapd Docker, Podman, Buildah Live image for virtual machine Source tarball only AppImage Binary tarballs with manual installation Self-extracting script . package, ? ? Which versions? RPM package, Fedora? OpenSUSE? Which versions? , gem, cpan, npm

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 4 / 22 Fragmentation

Software source

.json manifest Freedesktop SDK .yml Ubuntu Core Dockerfile Many SDKs

FlatpakBuilder snapcraft build Buildah, Docker

Flatpak ostree ref .snap file OCI/Docker image

flat-manager-client snapcraft push docker/podman push

flat-manager repository (e.g. Flathub) Snapcraft store Registry (e.g. Docker Hub)

Flatpak (flatpak install) Snapd (snap install) docker/podman pull Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 5 / 22 Table of Contents

1 Motivation

2 Building for multiple backends

3 BuildStream

4 Current state

5 Appendix

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 6 / 22 What is Freedesktop SDK?

Small userland runtime for desktop applications Not a distro: no , configuration tools, or installation. Base for GNOME SDK and KDE SDK for Flatpak Targetting first Flatpak Built with BuildStream

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 7 / 22 What is BuildStream?

Integration of stacks Backend agnostic, it creates artifacts Extensible with plugins

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 8 / 22 Factorizing the build process

Software source Freedesktop SDK

BuildStream project

BuildStream

Flatpak repo .snap file Docker/OCI image

flat-manager-client snapcraft push docker/podman push

flat-manager (e.g. Flathub) Snapcraft store Registry (e.g. Docker Hub)

Flatpak (flatpak install) Snapd (snap install) Snapd (docker/podman pull)

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 9 / 22 Table of Contents

1 Motivation

2 Building for multiple backends

3 BuildStream

4 Current state

5 Appendix

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 10 / 22 Development with BuildStream

A BuildStream project contains multiple elements. An element is a .bst file. An element typically builds one project (e.g. gcc.bst builds GCC). Elements have dependencies (build and/or runtime) to other elements. It is a graph. Each elements is built in an environment only containing build dependencies. Built artifacts are cached, locally and eventually remotely. To integrate elements together, they can have integration commands. Like a post-install script. Element can depend on elements from external BuildStream projects. We call it a “junction”.

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 11 / 22 Example of element

kind: autotools depends: - filename: bootstrap-import.bst junction: freedesktop-sdk.bst - filename: public-stacks/buildsystem-autotools.bst junction: freedesktop-sdk.bst type: build - filename: components/curl.bst junction: freedesktop-sdk.bst - filename: libevent.bst - filename: components/gtk3.bst junction: freedesktop-sdk.bst - filename: intltool.bst type: build - filename: libappindicator.bst

variables: conf - local: >- --with - gtk --enable-cli --enable-daemon sources: - kind: git_tag url: https://github.com/transmission/transmission.git track: master track-extra: -’2.9x’ track - tags: true ref: 2.94-0-gd8e60ee44f4295935bd98bf741f85ed19f5a7dfb

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 12 / 22 Plugins

Flatpak flatpak image prepares the image for flatpak build-export. flatpak repo builds the Flatpak repository with multiple flatpak image elements.

Snap snap image prepares the image and metadata. We use the generic script to call mksquashfs.

Docker/OCI oci plugin creates a layer chain or image per element. Either Docker 1.2 or OCI.

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 13 / 22 OCI image layering

OCI images are stacks of layers, BuildStream uses graphs Each Docker/OCI layer has an identifier. But implementation typically use chain identifier. So layers in mis-ordered or partial chains are not reused. Some filesystem implementation might not handle well 500 layers. We let the user explictly decide of the layering to improve reuse. Because downstream projects import the Freedestkop SDK graph, it imports also the OCI layers.

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 14 / 22 OCI image layering example

debug-oci.bst

sdk.bst sdk-oci.bst

Dependency graph platform.bst platform-oci.bst

bootstrap.bst bootstrap-oci.bst

junction junction junction

Dependency graph transmission-daemon.bst transmission-oci.bst

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 15 / 22 Table of Contents

1 Motivation

2 Building for multiple backends

3 BuildStream

4 Current state

5 Appendix

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 16 / 22 Examples

Firefox For Flatpak and Snap https://gitlab.com/valentindavid/firefox-buildstream

Transmission GTK UI for Flatpak and Snap Server for Docker/OCI https://gitlab.com/valentindavid/transmission-buildstream

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 17 / 22 Current issues

Strict confinement not yet working with Snapd. Snapd and Flatpak have a different way to mount /etc. Snapd mounts /etc from host and replaces some files from runtime Flatpak mounts /etc from runtime and bind some files from host. Nvidia drivers not yet tested in Snap. We need an equivalent to Flatpak extensions in OCI and Snapd. Difficult to deal with users for OCI images.

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 18 / 22 Questions?

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 19 / 22 Table of Contents

1 Motivation

2 Building for multiple backends

3 BuildStream

4 Current state

5 Appendix

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 20 / 22 Firefox dependencies

flatpak/repo.bst snap/image.bst

flatpak/org..Firefox.Debug.bst flatpak/org.mozilla.Firefox.bst snap/firefox.bst

flatpak/flatpak-debug-image.bst flatpak/flatpak-image.bst flatpak/export.bst snap/firefox-composed.bst

firefox.bst snap/desktop-file.bst snap/firefox-wrapper.bst

libevent.bst cbindgen.bst dbus-glib.bst gtk2.bst icu.bst nodejs.bst startup-notification.bst yasm.bst autoconf-2.13.bst

python2.bst nghttp2.bst -ares.bst libuv.bst app-lib.bst

freedesktop-sdk

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 21 / 22 Transmission dependencies

snap/image.bst

flatpak/repo.bst snap/transmission.bst

oci/transmission-docker.bst oci/transmission-oci.bst flatpak/com.transmissionbt.Transmission.Debug.bst flatpak/com.transmissionbt.Transmission.bst snap/transmission-composed.bst

oci/layers/transmission.bst flatpak/flatpak-debug-image.bst flatpak/export.bst flatpak/flatpak-image.bst snap/desktop-file.bst snap/transmission-wrapper.bst

transmission-daemon.bst transmission-gtk.bst

transmission.bst

libevent.bst libappindicator.bst

libindicator.bst libdbusmenu.bst

intltool.bst dbus-glib.bst

app-lib.bst freedesktop-sdk

Valentin David (Codethink) Building for Flatpak, Snapd & Docker All Systems Go! 2019 22 / 22