U-root: A Go-based, Firmware Embeddable Root File System with On-demand Compilation Ronald G. Minnich, Google; Andrey Mirtchovski, Cisco https://www.usenix.org/conference/atc15/technical-session/presentation/minnich This paper is included in the Proceedings of the 2015 USENIX Annual Technical Conference (USENIC ATC ’15). July 8–10, 2015 • Santa Clara, CA, USA ISBN 978-1-931971-225 Open access to the Proceedings of the 2015 USENIX Annual Technical Conference (USENIX ATC ’15) is sponsored by USENIX. U-root: A Go-based, firmware embeddable root file system with on-demand compilation Ronald G. Minnich Andrey Mirtchovski Google Cisco Abstract booted long enough to boot another kernel; in others, the kernel that is booted and the file system it contains con- U-root is an embeddable root file system intended to be stitute the operational environment of the device[15]. placed in a FLASH device as part of the firmware im- These so-called “embedded root file systems” also age, along with a Linux kernel. The program source contain a set of standard Unix-style programs used for code is installed in the root file system contained in the both normal operation and maintenance. Space on the firmware FLASH part and compiled on demand. All the device is at a premium, so these programs are usually u-root utilities, roughly corresponding to standard Unix written in C using, e.g., the BusyBox toolkit[26]; or in an utilities, are written in Go, a modern, type-safe language interpretive languages, such as Perl[24] or Forth. Busy- with garbage collection and language-level support for Box in particular has found wide usage in embedded ap- concurrency and inter-process communication. pliance environments, as the entire root file system can Unlike most embedded root file systems, which con- be contained in under one MiB. sist largely of binaries, U-root has only five: an init pro- Embedded systems, which were once standalone, gram and 4 Go compiler binaries. When a program is are now almost always network connected. Network- first run, it and any not-yet-built packages it uses are connected systems face a far more challenging security compiled to a RAM-based file system. The first invo- environment than even a few years ago. In response to cation of a program takes a fraction of a second, as it is the many successful attacks against shell interpreters[11] compiled. Packages are only compiled once, so the slow- and C programs[8], we have started to look at using a est build is always the first one, on boot, which takes more secure, modern language in embedded root file sys- about 3 seconds. Subsequent invocations are very fast, tems, namely, Go[21][16]. usually a millisecond or so. Go is a new systems programming language created U-root blurs the line between script-based distros by Google. Go has strong typing; language level support such as Perl Linux[24] and binary-based distros such as for concurrency; inter-process communication via chan- BusyBox[26]; it has the flexibility of Perl Linux and the nels, a la Occam[13], Limbo[17], and Alef[27]; runtime performance of BusyBox. Scripts and builtins are writ- type safety and other protective measures; dynamic al- ten in Go, not a shell scripting language. U-root is a new location and garbage collection; closures; and a package way to package and distribute file systems for embedded syntax, similar to Java, that makes it easy to determine systems, and the use of Go promises a dramatic improve- what packages a given program needs. ment in their security. The modern language constructs make Go a much safer language than C. This safety is critical for network- Introduction attached embedded systems, which usually have network utilities written in C, including web servers, network Embedding kernels and root file systems in BIOS servers including sshd, and programs that provide ac- FLASH is a common technique for gaining boot time cess to a command interpreter, itself written in C. All are performance and platform customization[25][14][23]. proving to be vulnerable to the attack-rich environment Almost all new firmware includes a multiprocess oper- that the Internet has become. Buffer overflow attacks ating system with a full complement of file systems, net- affecting C-based firmware code (among other things) work drivers, and protocol stacks, contained in an em- in 2015 include GHOST and the so-called FSVariable.c bedded file system. In some cases, the kernel is only bug in Intel’s UEFI firmware. Buffer overflows in Intel’s USENIX Association 2015 USENIX Annual Technical Conference 577 UEFI and Active Management Technology (AMT)have if the build succeeds, the command is executed. This also been discovered in several versions in recent years. first invocation takes a fraction of a second, depending Both UEFI[12] and AMT[4] are embedded operating on program complexity; after that, the RAM-based, stat- systems, loaded from FLASH, that run network-facing ically linked binaries run in about a millisecond. software; attacks against UEFI have been extensively U-root blurs the boundary between script-based root studied[9]. Most printers are network-attached and are file systems such as Perl Linux[24] and binary-based root a very popular exploitation target[6]. file systems such as BusyBox[26]; it has the flexibility of Firmware is not visible to most users and is up- Perl Linux and the performance of BusyBox. Scripts are dated much less frequently (if at all) than programs. written in Go, not a shell scripting language, with two It is the first software to run, at power on reset. Ex- benefits: the shell can be simple, with fewer corner cases; ploits in firmware are extremely difficult to detect, be- and the scripting environment is substantially improved, cause firmware is designed to be as invisible as possible. since Go is more powerful than most shell scripting lan- Firmware is extremely complex; UEFI is roughly equiv- guages, but also less fragile and less prone to parsing alent in size and capability to a Unix kernel. Firmware bugs. is usually closed and proprietary, with nowhere near the level of testing of kernels. These properties make firmware an ideal place for so-called advanced persistent The U-root design threats[10][18][5]. Once an exploit is installed, it is al- The u-root boot image is a build toolchain and a set of most impossible to remove, since the exploit can inhibit programs in source form. When first used, a program its removal by corrupting the firmware update process. and any needed but not-yet-built packages are built and The only sure way to mitigate a firmware exploit is to installed, typically in a fraction of a second. On second destroy the hardware. and later uses, the binary is executed. The root file sys- Even the most skilled programmers make simple mis- tem is almost entirely unformed on boot; /init sets up the takes that in C can be fatal, especially on network- key directories and mounts, including common ones such connected systems; nowadays, even the lowest-level as /etc and /proc. firmware in our PCs, printers, and thermostats is Since the init program itself is only 132 lines of code network-connected. These mistakes are either impossi- and is easy to change, the structure is very flexible and ble to make in Go or, if made, are detected at runtime and allows for many use cases. result in the program exiting. Perhaps surprisingly, the case for using a high-level, safe language like Go in very Additional binaries: if the 3 seconds it takes to get • low level embedded firmware might be stronger than for to a shell is too long (some applications such as au- user programs, because exploits at the firmware level are tomotive computing require 800 ms startup time), nearly impossible to detect and mitigate. and there is room in FLASH, some programs can The challenge to using Go in a storage-constrained en- be precompiled into /bin. vironment such as firmware is that advanced language features lead to big binaries. Even a date program is Build it all on boot: if on-demand compilation is not • about 2 MiB. One Go binary, implementing one func- desired, a background thread in the init process can tion, is twice as large as a BusyBox binary implement- build all the programs on boot. ing many functions. As of this writing, a typical BIOS FLASH part is 16 MiB. Fitting many Go binaries into a Selectively remove binaries after use: if RAM space • single BIOS flash part is not practical. is at a premium, once booted, a script can remove The Go compiler is very fast and its sheer speed points everything in /bin; those things that are used will be to a solution: to compile programs only when they are rebuilt on demand. used. We can build a root file system which has almost no binaries except the Go compiler itself. The compiled Always build on demand: it is possible to run in a • programs and packages can be saved to a RAM-based mode in which programs are never written to /bin file system. and always rebuilt on demand; this mode is surpris- U-root is our proof of concept of this idea. U-root con- ingly comfortable to use, given that program com- tains only 5 binaries, 4 of them from the Go toolchain, pilation is so fast1. and the 5th an init binary. The rest of the programs Lockdown: if desired, the system can be locked are contained in BIOS FLASH in source form, includ- • ing packages.