U-Boot – Bootloader for IoT Platform? Alexey Brodkin Embedded Linux Conference Europe 2018, Edinburgh, Scotland, UK © 2018 Synopsys, Inc. 1 Agenda U-Boot for IoT device Shrinking memory footprint Execution from ROM Run-time issues © 2018 Synopsys, Inc. 2 Alexey Brodkin Engineering manager at Synopsys • Maintainer of ARC architecture in U-Boot • Contributor to: – Linux kernel – Buildroot – OpenEmbedded – OpenWrt – uClibc etc © 2018 Synopsys, Inc. 3 Previous ARC boards with U-Boot Single-board computers with Linux AXS103 HSDK • Dual-core ARC HS38 @ 100 MHz • Quad-core ARC HS38 @ 1 GHz in FPGA in silicon • BootROM • BootROM • 2 GiB of DDR • 4 GiB of DDR © 2018 Synopsys, Inc. 4 New board – new fun Meet IoT development kit • ARC EM9D @ 150 MHz • Memories: – eFlash 256 KiB @ 0x0000_0000 – ICCM 256 KiB @ 0x2000_0000 – SRAM 128 KiB @ 0x3000_0000 – DCCM 128 KiB @ 0x8000_0000 – SPI flash 2 MiB • Peripherals:* – SD-card (DW MobileStorage) – USB OTG (DW USB OTG) – Serial port (DW APB UART) * There’re much more peripherals on the board but those make no sense for a bootloader © 2018 Synopsys, Inc. 5 Why U-Boot Add support of a new board in a blink of an eye • Mature and well-known bootloader $ git show --stat --pretty=oneline 5396e8b • Supports: 5396e8b arc: Add support for IoT development kit arch/arc/Kconfig | 5 ++++ – 12 CPU architectures arch/arc/dts/Makefile | 1 + arch/arc/dts/iot_devkit.dts | 45 ++++++++++++++++++++++ – Lots of peripherals: UART, SPI, I2C, Ethernet, board/synopsys/iot_devkit/Kconfig | 12 ++++++++ SD, USB … board/synopsys/iot_devkit/MAINTAINERS | 5 ++++ – File-systems: Fat, Ext, Yaffs2, Btrfs, ubifs … board/synopsys/iot_devkit/Makefile | 7 +++++ board/synopsys/iot_devkit/config.mk | 2 ++ – Networking protocols: TFTP, NFS, DHCP board/synopsys/iot_devkit/iot_devkit.c | 168 ++++++++++++++++++++++ • “Device-tree” used for drivers initialization board/synopsys/iot_devkit/u-boot.lds | 77 ++++++++++++++++++++++ configs/iot_devkit_defconfig | 38 ++++++++++++++++++++++ • Allows for flexible scripting in “hush” shell include/configs/iot_devkit.h | 84 ++++++++++++++++++++++ 11 files changed, 444 insertions(+) • Allows re-use of its stdio and C run-time libs by user applications © 2018 Synopsys, Inc. 6 Starting point: 422 KiB total CONFIG_ARC=y CONFIG_ISA_ARCV2=y • DW USB OTG & DW MMC drivers CONFIG_CPU_ARCEM6=y • Read/write FAT file-system CONFIG_TARGET_IOT_DEVKIT=y CONFIG_CMD_MMC=y • Built-in .dtb CONFIG_CMD_USB=y CONFIG_CMD_FAT=y CONFIG_OF_CONTROL=y CONFIG_OF_EMBED=y CONFIG_ENV_IS_IN_FAT=y CONFIG_ENV_FAT_INTERFACE="mmc" CONFIG_ENV_FAT_DEVICE_AND_PART="0:1" CONFIG_DM=y CONFIG_MMC=y CONFIG_MMC_DW=y CONFIG_DM_SERIAL=y CONFIG_SYS_NS16550=y CONFIG_USB=y CONFIG_DM_USB=y $ size u-boot CONFIG_USB_DWC2=y text data bss dec hex filename 257129 9372 155928 422429 6721d u-boot CONFIG_USB_STORAGE=y © 2018 Synopsys, Inc. 7 Shrinking memory footprint We only have 256 KiB of ROM and 128 KiB of RAM © 2018 Synopsys, Inc. 8 Disable useless options: 366 KiB total As simple as deselecting items in menuconfig • No plans to load OS # CONFIG_CMD_BOOTD is not set • No plans to use flash (as of now) # CONFIG_CMD_BOOTM is not set # CONFIG_CMD_ELF is not set • There’s no Ethernet controller # CONFIG_CMD_XIMG is not set • No memory to load application from Elf # CONFIG_CMD_FLASH is not set # CONFIG_CMD_LOADB is not set – Load Elf # CONFIG_CMD_LOADS is not set – Copy sections from Elf to RAM # CONFIG_NET is not set • No need to load via serial port $ size u-boot text data bss dec hex filename 216009 7544 142468 366021 595c5 u-boot © 2018 Synopsys, Inc. 9 Get rid of dead code: 311 KiB total Might require changes in sources • Put all functions, global & static variables in CPPFLAGS += -ffunction-sections -fdata-sections their own sections and strip unused on final LDFLAGS += --gc-sections linkage. • Should be enabled by default for all architectures and boards in U-Boot except toolchain doesn’t support it. • Was not the case for ARC – fixed now fac4790491f6 (“arc: Eliminate unused code and data with GCC's garbage collector”) $ size u-boot text data bss dec hex filename 163532 6948 140928 311408 4c070 u-boot © 2018 Synopsys, Inc. 10 Shrink statically allocated buffers: 188 KiB total • tmpbuf_cluster & $ nm --size-sort --reverse-sort u-boot | head -n 5 get_contents_vfatname_block of 00010000 b tmpbuf_cluster 65 KiB each! 00010000 B get_contents_vfatname_block • CONFIG_FS_FAT_MAX_CLUSTSIZE=4096 00001414 b hist_lines 00000a96 t do_fdt • Save 120 KiB of memory! 00000812 t set_contents #define MAX_CLUSTSIZE CONFIG_FS_FAT_MAX_CLUSTSIZE static __u8 tmpbuf_cluster[MAX_CLUSTSIZE] __aligned(ARCH_DMA_MINALIGN); __u8 get_contents_vfatname_block[MAX_CLUSTSIZE] $ size u-boot text data bss dec hex filename 163532 6948 18048 188528 2e070 u-boot © 2018 Synopsys, Inc. 11 Compile-time optimizations summary: /2 size Memory foot-print reduced from 422 to 188 KiB (saved 234 KiB) • Analyze – Main contributors were huge statically allocated buffers – Primary tools: – bloat-o-meter (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/bloat-o-meter) – size – nm • Be practical – Unused options might add to memory usage significantly (56 KiB in our case) • Use advanced features of the toolchain – 5% size reduction due to dead code elimination – Link Time Optimization (LTO) might help a bit more © 2018 Synopsys, Inc. 12 Execution from ROM Relocation & memory partitioning © 2018 Synopsys, Inc. 13 [Self-] relocation Fundamental feature of U-Boot • Why relocation? – RAM is much larger – DDR might require initialization before use – We’ll need RAM anyways so why not? Before After relocation • 2 major stages: – Pre-relocation (common/board_f.c) Stack, heap, ENV etc Execute code from ROM/flash with limited RAM options: u-boot.bin Payload u-boot.bin – On-chip SRAM ROM RAM – Locked D$ lines (x86) – DDR (sometimes) – After-relocation (common/board_r.c) Executing from RAM (usually DDR) © 2018 Synopsys, Inc. 14 Don’t relocate* What if RAM size < ROM size = u-boot.bin • Only supported for ARC as of today • Add support for your architecture: 264d298fda39 (“arc: Introduce a possibility to not relocate U-boot”) • In platform/board code signal your intention • Keep executing code from ROM/flash u-boot.bin • Use RAM only for data ROM RAM – Heap – Stack – .data – Environment – Payload © 2018 Synopsys, Inc. 15 Memory partitioning Standard U-Boot CONFIG_xxx constants ROM .ivt, .text, .rodata .data CONFIG_SYS_TEXT_BASE CONFIG_SYS_SDRAM_SIZE RAM Stack .data .bss Heap ENV CONFIG_SYS_SDRAM_BASE CONFIG_SYS_MALLOC_LEN CONFIG_ENV_SIZE CONFIG_SYS_INIT_SP_ADDR © 2018 Synopsys, Inc. 16 Definition of derived constants configs/iot_devkit.h board/synopsys/iot_devkit/u-boot.lds #define CONFIG_SYS_MONITOR_BASE CONFIG_SYS_TEXT_BASE MEMORY { ROM : ORIGIN = ROM_BASE, LENGTH = ROM_SIZE #define SRAM_BASE 0x30000000 RAM : ORIGIN = RAM_DATA_BASE, LENGTH = RAM_DATA_SIZE #define SRAM_SIZE SZ_128K } #define DCCM_BASE 0x80000000 SECTIONS #define DCCM_SIZE SZ_128K { . = CONFIG_SYS_MONITOR_BASE; #define CONFIG_SYS_SDRAM_BASE DCCM_BASE .ivt : { *(.ivt); } > ROM #define CONFIG_SYS_SDRAM_SIZE DCCM_SIZE .text : { *(.text*); } > ROM #define CONFIG_SYS_INIT_SP_ADDR (CONFIG_SYS_SDRAM_BASE + SZ_32K) .rodata : { *(.rodata*); } > ROM #define CONFIG_SYS_MALLOC_LEN SZ_64K __rom_end = .; #define CONFIG_SYS_BOOTM_LEN SZ_128K .data : { #define CONFIG_SYS_LOAD_ADDR SRAM_BASE __ram_start = .; #define ROM_BASE CONFIG_SYS_MONITOR_BASE *(.data*) #define ROM_SIZE SZ_256K __ram_end = .; #define RAM_DATA_BASE CONFIG_SYS_INIT_SP_ADDR } > RAM AT > ROM #define RAM_DATA_SIZE CONFIG_SYS_SDRAM_SIZE - \ .bss : { (CONFIG_SYS_INIT_SP_ADDR - \ __bss_start = .; CONFIG_SYS_SDRAM_BASE) - \ *(.bss*) CONFIG_SYS_MALLOC_LEN - \ __bss_end = .; CONFIG_ENV_SIZE } > RAM https://sourceware.org/binutils/docs/ld/Output-Section-LMA.html#Output} -Section-LMA © 2018 Synopsys, Inc. 17 Memory partitioning Derived constants ROM_SIZE ROM .ivt, .text, .rodata .data ROM_BASE __rom_end RAM_DATA_SIZE RAM Stack .data .bss Heap ENV RAM_DATA_BASE __bss_end __ram_start __bss_start __ram_end © 2018 Synopsys, Inc. 18 Required quirks They are not too many • Signal intention to skip relocation /* 1. Don't relocate U-Boot */ – Set GD_FLG_SKIP_RELOC flag gd->flags |= GD_FLG_SKIP_RELOC; • Copy .data section from ROM to RAM /* 2. Copy data from ROM to RAM */ u8 *src = __rom_end; • Zero .bss as usual in clear_bss() u8 *dst = __ram_start; while (dst < __ram_end) *dst++ = *src++; /* 3. Zero .bss as usual in clear_bss() */ size_t len = (size_t)&__bss_end - (size_t)&__bss_start; memset((void *) &__bss_start , 0x00, len); © 2018 Synopsys, Inc. 19 Run-time issues © 2018 Synopsys, Inc. 20 -ENOMEM Even though we boot to command prompt “usb start” fails • Problem starting USB... – Driver attempts to allocate 64 KiB buffer USB0: probe failed, error -12 USB error: all controllers failed lowlevel init • Fix [14] malloc(bytes = 66328) = dlmalloc.c!1241 – 42637fdae833 (“usb: dwc2: Allow selection of data [13] memalign()+0x78 = dlmalloc.c!1922 buffer size”) [12] alloc_priv()+0x1a = device.c!269 – Set CONFIG_USB_DWC2_BUFFER_SIZE = 16 [11] device_probe()+0x9c = device.c!325+0x6 (instead of default 64) [10] usb_init()+0xa2 = usb-uclass.c!276 [ 9] do_usb_start()+0xc = usb.c!586+0x4 • Hint [ 8] do_usb()+0x4e = usb.c!657 [ 7] cmd_call()+0xc = command.c!499+0xc – Check malloc() return value early! [ 6] cli_simple_run_command()+0x94 = cli_simple.c!249 [ 5] cli_simple_loop()+0x36 = cli_simple.c!299+0xa [ 4] main_loop()+0x30 = main.c!66 [ 3] run_main_loop()+0x6 = board_r.c!645+0x4 [ 2] initcall_run_list()+0x2a