WEPGF096 Proceedings of ICALEPCS2015, Melbourne, Australia MANAGING A REAL-TIME EMBEDDED LINUX PLATFORM WITH BUILDROOT J. Diamond#, K. Martin, FNAL*, Batavia, IL 60510, U.S.A. Abstract Developers of real-time embedded software often need LINUX “FROM SCRATCH” to build the operating system, kernel, tools and supporting The Linux from Scratch website [5] was chosen as the applications from source to work with the differences in model for the attempt to build a custom Linux platform their hardware configuration. The first attempts to from source. Linux from Scratch is an on-line resource introduce Linux-based real-time embedded systems into that explains in step-by-step fashion how to build an the Fermilab accelerator controls system used this entire Linux operating system from source. First the user approach but it was found to be time-consuming, difficult partitions the target’s disk on the build system. Then the to maintain and difficult to adapt to different hardware necessary source packages (about three dozen) and configurations. Buildroot is an open source build system patches (another dozen) are downloaded onto the build with a menu-driven configuration tool (similar to the system and unpacked into a special “chroot environment”. Linux kernel build system) that automates this process. A Then the toolchain packages are built and installed – not a customized Buildroot [1] system has been developed for trivial task in its own right. The toolchain is then used to use in the Fermilab accelerator controls system that build a complete Linux system from inside of the chroot includes several hardware configuration profiles environment - kernel, boot-loader, shell, etc. Once the (including Intel, ARM and PowerPC) and packages for system is in place the boot-scripts are installed and Fermilab support software. A bootable image file is configured. Finally, the boot-loader is installed on the produced containing the Linux kernel, shell and target CF disk and an attempt is made to boot the target supporting software suite that varies from 3 to 20 (things rarely work on the first try). megabytes large – ideal for network booting. The result As illustrated above, the process is exhausting and can is a platform that is easier to maintain and deploy in take several days to complete. It should be noted that the diverse hardware configurations. target platform was x86 so a cross-compile toolchain was not attempted (ARM targets came later). Once developed INTRODUCTION the root filesystem could be used to boot any target with a Scientific Linux [2] is a Linux distribution produced by similar hardware architecture. a collaborate effort between Fermilab, CERN and other The root filesystems produced with this method were in members of the global Linux community. Scientific the range of two to three hundred megabytes. One Linux has been used successfully as a target platform for significant drawback is that a boot image of this size is data acquisition in the Fermilab accelerator control still too large to network boot. The decision was made to system but only on commodity PC hardware and not with store the kernel, boot-scripts and network configuration real-time extensions [3]. When attempting to adopt on a Compact Flash (CF) disk installed with each target Scientific Linux to a real-time embedded application node and the root filesystem on a common network some road-blocks were encountered. The first was that filesystem that could be mounted by each target at boot Linux is not designed to be a hard real-time operating time. The network filesystem also contains the system. There are however, several options for making application programs, libraries and data areas. Linux real-time and all of these options require a custom One issue that was noticed early on was the failure rate kernel be built. At Fermilab, one route chosen was to use of the CF disks. CF disks are susceptible to failure after the Real-Time Application Interface (RTAI) [4]. The next many write-cycles. As noted above, application data is road-block was the minimum foot-print of the installed written to a network filesystem instead of the CF disk. To system. The minimum Scientific Linux install profile reduce the CF write cycles the /tmp and /log partitions requires 1.5GB of storage. This is a concern because it is were moved to a RAM disk and the system log was desirable to network boot the targets. The requirement for redirected to a syslog server. These efforts were able to a network boot capability drastically reduces the mitigate the failure rate of the CF disks. acceptable footprint size as the entire boot image needs to Another significant challenge with this platform was be transferred over the network. Finally, Scientific Linux maintainability. Installing new software packages in the is a binary distribution targeted for x86 architecture, thus root filesystem was easy but deploying to targets already it would not be possible to use as a platform for ARM and in the field meant physically replacing the CF disks. PowerPC targets. These concerns drove the decision to Furthermore, the root filesystem installed on the network use a custom Linux platform built from the ground-up. filesystem would creep away from its original state over ____________________________________________ * Operated by Fermi Research Alliance, LLC under Contract time without version control system in place. No. De-AC02-07CH11359 with the United States Department of Energy # [email protected] ISBN 978-3-95450-148-9 Copyright ©926 2015 CC-BY-3.0 and by the respective authors Software Technology Evolution Proceedings of ICALEPCS2015, Melbourne, Australia WEPGF096 CONTROL SYSTEM INTEGRATION interface remained the same and the bridge software was moved into the Erlang framework. An alternate method At first the support libraries available for integrating was developed that replaced the PEP protocol with RTAI with the Fermilab accelerator controls system (ACNET) message queues and shared memory. This method allows [6] [7] only supported VxWorks. Package Exchange large amounts of data to be shared between the ACNET Protocol (PEP), a light weight UDP protocol was process and the data acquisition process without developed to bridge communication between Linux transferring through a UDP socket. targets and VxWorks targets (Fig. 1). Support software that ran on a VxWorks target was deployed to bridge communication between ACNET and Linux targets deployed in the field. This method was successful and supported many, but not all of the features expected from an ACNET front end node. Figure 3. Replacing PEP with RTAI IPC methods. LOOKING TO IMPROVE Out of the experience of building a real-time embedded Linux platform from scratch came three desired improvements: Figure 1. Method for using PEP to bridge ACNET request 1. A smaller footprint operating system that could be to Linux Targets. packaged into a network-deliverable boot-image 2. An automated system for building kernel and root filesystem that could support multiple target Later, a suite of software was developed for interfacing architectures and hardware configurations Linux front end nodes with ACNET [8]. This ACNET 3. An integrated system for building and deploying support software is written in Erlang and runs inside of application software the Erlang virtual machine as its own process (Fig. 2). To address these improvements an effort was begun to Using this software the user can write data acquisition look to the open source community for a “best-practice” drivers in Erlang or using a C++ API. The challenge in approach to deploying an embedded Linux platform. this model is how to attach the data acquisition driver, running in its own process to the real-time application BUILDROOT running in another process. The decision was made to again utilize the PEP protocol to forward ACNET Buildroot is an open source build system with a menu- requests to the real-time application process. In this driven configuration tool (similar to the Linux kernel architecture the communication was done with local build system) that completely automates this process sockets instead of over Ethernet. (Fig. 4). It supports ucLibc [9], a low-footprint alternative to the GNU standard C library and Busybox [10], which combines many of the standard UNIX utilities and a shell into a single low-footprint executable. Figure 2. ACNET process utilizing PEP for inter-process communication. Using this architecture the existing applications using the bridge method were adapted to the new Linux ACNET platform without much effort (Fig. 3). The Figure 4. Buildroot configuration menu. ISBN 978-3-95450-148-9 Software Technology Evolution 927 Copyright © 2015 CC-BY-3.0 and by the respective authors WEPGF096 Proceedings of ICALEPCS2015, Melbourne, Australia In addition to providing a kernel, shell and basic GRUB must be built with the support for the network utilities, hundreds of software packages are supported. adapter on the target. The network configuration is stored The user simply selects which software package to in GRUB menu.lst file, which resides on the flash disk’s include in the boot-image and the build system boot partition. downloads, unpacks, configures, compiles and installs it. The Buildroot configuration system includes an option to APPLICATION BUILD ENVIRONMENT combine the kernel and root filesystem into a single All software that runs on the target must be built with bzImage at the end of the build process. A boot-loader the toolchain that Buildroot builds as the first step in the and this bzImage file are the only components needed to build process. A set of Makefiles was developed that boot an embedded Linux target. could be included in a developer’s C/C++ project that The first build usually takes a couple of hours, builds using this toolchain. Deploying to remote targets depending on the processing power and network is supported using scp and ssh or by mounting a remote bandwidth of the build machine.