Building Products with Debian and Isar Jan Kiszka Baurzhan Ismagulov Open Source Summit Japan 2019 About us Jan Kiszka <[email protected]> Baurzhan Ismagulov <[email protected]> ● Member of embedded Linux team at Siemens ● Work for ilbers GmbH Corporate Technology ● Provide software development services ● (In-house) consultant, architect, developer for ● Focus on embedded, real-time, networking, OSS virtualization ● Focus on kernel, real-time, virtualization, ● embedded build systems Contribute to Linux and other FOSS projects ● Upstream contributor / maintainer Unrestricted © 2019 Siemens AG, ilbers GmbH Page 2 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Agenda • Motivation & Approach 4 • Isar Image Build System 8 • How Isar Works 10 • Product Building Overview 12 • Isar Users 16 • Usage Patterns 19 • Summary 26 Unrestricted © 2019 Siemens AG, ilbers GmbH Page 3 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Motivation ● Build Linux-based products ● Linux base system ● Customizations ● Product-specific applications ● We want a product build system ● Tools are distribution-specific Unrestricted © 2019 Siemens AG, ilbers GmbH Page 4 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Which Distribution? Source distributions, e.g.: Binary distributions, e.g.: Yocto, OE, buildroot Debian, Ubuntu, SUSE, Fedora ● Toolchain bootstrap, cross-build => long ● Prebuilt binary packages => keep build times upstream QA ● Highly customizable, smaller footprint ● Larger images, slower boot by default ● High maintenance costs ● Maintained releases Unrestricted © 2019 Siemens AG, ilbers GmbH Page 5 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Debian ● Mature, high-quality, mainstream Linux distribution ● Support for many new and old hardware architectures ● Suitable for small and big installations ● Security updates ● License information Unrestricted © 2019 Siemens AG, ilbers GmbH Page 6 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Embedded Debian: Wishlist ● Build all sources in the right order with one command ● Bootable image generator ● Easy package customization ● Reproducible images ● Efficient variant management Unrestricted © 2019 Siemens AG, ilbers GmbH Page 7 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Isar ● Integration System for Automated Root filesystem generation https://github.com/ilbers/isar ● Goals ● Build systems in a Debian way ● Developer-centric workflow: One-command building ● Make customizations easy and repeatable ● Efficient building ● The best of both worlds ● Debian: Tested binary packages, tools, security updates ● Yocto: bitbake, recipes, layers ● Reuse Yocto knowledge of developers Unrestricted © 2019 Siemens AG, ilbers GmbH Page 8 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Hello, Image $ git clone https://github.com/ilbers/isar # Install packages and configure sudo as described in user_manual.md $ . isar-init-build-env ../build $ bitbake multiconfig:rpi-stretch:isar-image-base $ ls tmp/deploy/images/rpi isar-image-base.rpi-sdimg Unrestricted © 2019 Siemens AG, ilbers GmbH Page 9 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Image Generation Sequence of Isar 1 debootstrap Debian for target, also for host if cross-building debootstrap 2 Create buildchroots (target and host) 3 Build custom Debian packages buildchroot • pre-debianized packages • ad-hoc debianized packages package A package B (customizations/overlays, u-boot, …) 4 Assemble rootfs package C • debootstrap output • external packages • self-built packages target rootfs 5 Run imager (typically wic) • Filesystem image generation boot imaging • Partitioning • Bootloader installation and configuration Unrestricted © 2019 Siemens AG, ilbers GmbH Page 10 2019-07-18 Jan Kiszka, Baurzhan Ismagulov BitBake dpkg.bbclass ● A tool for task description and execution do_apt_fetch() { ... } ● Recipes: – Describe tasks (python, shell) do_install_builddeps() { ... } – May depend on other recipes do_runbuild() { ... } ● Tasks: e.g., fetch, patch, build ● Dependencies define: hello.bb – Which tasks to execute – Execution order DESCRIPTION = "Hello app" DEPENDS += "libhello" ● Classes: Repetitive stuff SRC_URI = "\ ● Efficient building git://gh.com/ilbers/hello.git \ file://0001.patch" – Parallel building SRCREV = "86cc719..." – Multiple targets in one run (multiconfig) inherit dpkg Unrestricted © 2019 Siemens AG, ilbers GmbH Page 11 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Building a Product Requirements • Directly bootable images for product boards • Product-specific base system customizations • Product-specific software • Reproducible by developers Tasks • Generate rootfs • Perform customizations • Build custom software: boot loaders, kernels, https://en.wikipedia.org/wiki/Newbear_77-68 drivers, libs, apps • Generate flash images for physical boards Unrestricted © 2019 Siemens AG, ilbers GmbH Page 12 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Layers product-x ● Group by product / department / feature... ├── conf │ └── layer.conf ● Config files, recipes, classes / include files, ... └── recipes-core ● ├── customizations E.g.: │ ├── customizations.bb ● meta: Isar core │ └── files │ └── postinst ● meta-company: Company common stuff └── images └── x-image.bb ● meta-prodline: Product line common stuff ● meta-bsp: Product X COTS BSP product-x product-y ● product-x: Product X-specific stuff meta-bsp meta-codecs ● meta-codecs: Product Y COTS libs meta-prodline ● product-y: Product Y-specific stuff meta-company meta Unrestricted © 2019 Siemens AG, ilbers GmbH Page 13 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Product Lifecycle ● Deployment: Copy image to flash ● Updates: Product-specific ● Install packages via apt ● Overwrite complete images ● Security updates: Packages and alerts from Debian ● Long-term support for chosen packages by Debian and Civil Infrastructure Project ● License compliance: Product-specific ● Rely on Debian license information ● FOSSology + manual checking Unrestricted © 2019 Siemens AG, ilbers GmbH Page 14 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Is Isar Right for Me? ● Build with one command ● No infrastructure needed upfront ● Use mainstream build tools ● Reuse binary packages ● Reuse code in multiple products ● Collaborate with multiple vendors Unrestricted © 2019 Siemens AG, ilbers GmbH Page 15 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Who Uses Isar? • Jailhouse images https://github.com/siemens/jailhouse-images • Xenomai images https://gitlab.denx.de/Xenomai/xenomai-images • Civil Infrastructure Project core https://gitlab.com/cip-project/cip-core/isar-cip-core • Used at Siemens for an increasing number of embedded products https://en.wikipedia.org/wiki/Isar • Core of Mentor Embedded Linux Omni OS https://www.mentor.com/embedded-software/linux/ mel-omni-os Unrestricted © 2019 Siemens AG, ilbers GmbH Page 16 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Example: jailhouse-images jailhouse-images Isar Debian packages Unrestricted © 2019 Siemens AG, ilbers GmbH Page 17 2019-07-18 Jan Kiszka, Baurzhan Ismagulov jailhouse-images in a Nutshell • Started early 2018 with x86 QEMU/KVM setup - ./build-images.sh • By now 8 targets: #)ailable images demo images: 1: /01U/341 Intel-x87 virtual target • 2 virtual 2: /01U #$174 )irtual target • 6 physical 3: Orange Pi Zero >8?6 1B editionA 4: Intel B2% (B2%7%#C, 6 G@ R#1A • 3 architectures 5: Marvell 0F<$0FFObin >1 EB editionA • Prerequisite for building: docker (privileged) 6: Marvell 1#%%&5#G;bin 7: Le1aker &iKey >Kirin 78+ So%D 2 EB editi • git clone https://github.com/ 8: #)net UltraJ7 siemens/jailhouse-images 0: all (may ta*e hours...) • ./build-images.sh Felect images to build (space-separated • ./start-qemu.sh <ARCH> index listA: . 2 9 • dd if=target.img of=/dev/mmcblk0 … Unrestricted © 2019 Siemens AG, ilbers GmbH Page 18 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Distro or Custom Kernel? • Set 30$B0IKB#10 to select target linux-${KERNEL_NAME} • Distro kernel is Isar default (and automatically provided) • linux-my-lovely-kernel.bb • require recipes-*ernel/linux/linux- custom.inc • Usage straightforward, just needs F$%K2$5 and 30$B0IKO0P%;BP5E • meta-isar/recipes-kernel/linux contains two custom kernel examples (CIP and mainline) Unrestricted © 2019 Siemens AG, ilbers GmbH Page 19 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Adding Out-of-Tree Kernel Modules • require recipes-kernel/linux-module/module.inc • Package name = recipe name + kernel name e.g. example-module-${KERNEL_NAME} • Module Makefile according to kbuild rules, e.g. obj-m := example-module.o • #2G;I;#O: list of modules to be loaded during boot Unrestricted © 2019 Siemens AG, ilbers GmbH Page 20 2019-07-18 Jan Kiszka, Baurzhan Ismagulov U-Boot • Distro U-Boot supported, though mostly available on maker boards (Raspberry Pi & Co.) • require recipes-bsp/u-boot/u-boot- custom.inc • Trival case just needs setting 2K@;;GK%;BP5E and 2K@;;GK@5B • Complex case may require overloading debian/rules file • Examples • meta-isar: mainline 2018.09 • jailhouse-images: MACCHIATObin, Ultra96 Unrestricted © 2019 Siemens AG, ilbers GmbH Page 21 2019-07-18 Jan Kiszka, Baurzhan Ismagulov Image Partitioning and Generation • Default case: wic tool • .wks files control partition layout and content • Same core as OE, but additional source plugins • bootimg-efi-isar (EFI-based boot partition) • bootimg-pcbios-isar (legacy boot partition) • rootfs-u-boot (used with u-boot-script) • wic references • Yocto manual: https://www.yoctoproject.org/docs/latest/dev-manu al/dev-manual.html#creating-partitioned-images-u sing-wic • Example wks files • Additional image formats (more planned) • ext4.img • fit • ubifs