SYSADMIN Upstart James Thew

Quick booting with Upstart, a replacement for the legacy Sys V JUMP START

The slow boot has troubled users for years. Now the Upstart userspace processes and initializing the machine. The process and its helper project offers a fresh approach to the problem of booting Linux. scripts load kernel modules, check and mount the filesystem, set up the net- BY NICO DIETRICH, DIRK VON SUCHODOLETZ work, launch servers, and call the graph- ical login manager. Init has to launch the he history of Linux includes verages many developments in modern services in a meaningful order. For ex- many attempts to address the Linux systems. Upstart, which has Unix ample, it doesn’t make sense to set the Tlong Linux boot process. This roots, cleverly steers clear of unneces- system time by polling a time server on comes as no surprise, as a long boot sary waits, runs start scripts simultane- the network until the machine becomes marathon will annoy all but the most pa- ously, and reduces the boot time to a available on the network. To do this, init tient of users. The legacy Unix System V minimum. The long-term plan is to re- has to first initialize the network hard- boot design was once revolutionary, but place generic background system ser- ware, and set up at least one network in- has turned out to be a millstone around vices such as the at daemon, cron, and terface for external access. the necks of many distributions. others with Upstart. 6.10 (Edgy The number of services and back- Although several tricks for speeding Eft) demonstrates the first effects of this ground agents has grown over the years, up the process have appeared through promising software. and this has made the init process the years, most boot reforms have clumsy. In contrast, desktop usage, turned out to be unworkable in practice, It All Starts with Init which is typical for Ubuntu, demands a and many of the turbo loaders employed Most Unix-style systems share the init dynamic system configuration. Mobile by gurus are inaccessible to regular concept. The process calls the kernel devices also make life really difficult for users. A new tool, Upstart, takes a fresh and assigns the kernel process ID 1. This Sys V Init. Roving devices require an ad- approach to speeding up Linux boot. sequence is hard-coded into the kernel – hoc approach to setting up network con- The high-flying Upstart project [1] in- in /usr/src/linux/init/main.c on Linux. nections, as well as a means for dynami- troduces a generic init daemon that le- Init has the task of launching all other cally configuring hardware.

62 ISSUE 76 MARCH 2007 WWW.LINUX - MAGAZINE.COM Upstart SYSADMIN

To cope with this, Linux programmers When they started to think about the pleted,” “Apache running,” or (in the fu- have developed a number of tools: acpid new design, they had to choose between ture) “USB stick plugged.” and apmd for power management, the a target or result-oriented option for the HAL Device Manager for dynamic system launch. Target-oriented would Event Horizon mounting of drives, and the Resource mean defining the services that should Events are basically just simple strings. Manager for dynamic device privilege be running at the end of the start se- The Upstart developers divide events assignments to the user working with quence (KDM, SSH daemon, etc.). into three classes: the GUI desktop. Each of these sytems In this case, it would be necessary to • Simple edge events, such as “the sys- implements its own configuration logic, investigate each service and determine tem is booting,” or “the user has just and admins have to be familiar with this which other services it relied on. Based pressed a button.” logic to be able to run a required task at on these dependencies, the init system • Level events have an additional pa- the right point in time. would need to derive a meaningful rameter, such as the network interface Not all processes and services are tied startup sequence. This is exactly the ap- status. Services and tasks run either to starting or stopping a machine. For ex- proach that Gentoo adopted with its de- for any Level Event, or only when a ample, there are some special services, pend system (see the “Gentoo” box); parameter has reached a specific like cron and the at daemon, that launch Suse also follows this approach with a value. other processes at a certain time. They modified version of the Sys V Init (see • Temporal events occur after a specific are not linked to the system in the “Suse Linux” box). interval, or at a specific point in time. any way, although they have a similar In the other corner of the ring, there The developers kept to the open source underlying logic. This is another thing were events. Instead of formulating de- codex of release early, release often. that Upstart looks to change [3]. pendencies, which a script would proba- Thus, the code was released at a very bly need to handle at system start time, early stage, and the self-confident devel- Design Issues an event-based system would not run a opers presented a running system, in the Before the Ubuntu programmers decided script until a specific set of preconditions guise of Edgy Eft, to demonstrate how to develop a new system, they first in- was fulfilled. For example, it would not far they have gotten. Their aim is to col- vestigated contemporary alternatives to make sense to call an NFS client until lect as much feedback as possible from the familiar Sys V system [2]. None of the authoritative NFS became available. developers working on other Linux dis- the designs they looked at fulfilled their The system that Ubuntu opted for also tributions. expectations or was available under an understands more complex conditions, However, this also means that the acceptable license. such as “network configuration com- specifications might change in the

Sys V Init The early Unix versions used a simple This lets administrators update the start any system. Three have clearly shell script to configure the machine and scripts without endangering the local defined tasks: Halt (0), Single User Mode, launch services. The design behind the configuration. and Reboot (6). BSD family’s /etc/rc, for example, was Sys V Init uses a far more flexible, but The /etc/inittab file specifies which run- convincingly simple, but it did have one also more complex, approach than BSD, levels exist and defines the runlevel the major drawback. Integrating third-party introducing runlevels that define specific system enters on booting (Figure 1). software, or custom-built extensions, machine states, based on the processes The Sys V design assumes that the sys- meant modifying the shell script. Unfor- that run in them. A total of eight runlev- tem will use a small number of defined tunately, modifying this code is quite els is possible, but not mandatory, for states, such as without network, with dangerous – a single mis- network, with X11, and so take could lead to an un- on. Administrators can bootable system. change the runlevel by In many cases, it takes giving the init Runlevel more than just a simple command. command to launch a Another advantage of this service, with the details approach is the separate varying depending on the scripts for each service or current environment. For configuration task. For ex- example, the ISC DHCP ample, calling /etc/init.d/ server can be set up to lis- dhcpd restart lets you re- ten on specific Ethernet launch the DHCP server interfaces, rather than on without affecting any all ports. To remove the other services. The idea of need for administrators using a collection of sym- to modify the start script links to determine the to do this, daemons often scope and order of the come with configuration scripts used in each run- files that parse the script. Figure 1: This typical inittab defines runlevels 0 through 6. level is also a good one.

WWW.LINUX - MAGAZINE.COM ISSUE 76 MARCH 2007 63 SYSADMIN Upstart

course of the next few months. The ex- splash (by deleting the splash token down, or runlevel-3 through runlevel-5 amples described here refer to version from the kernel command line in grub), occur (lines 7 through 10). In the future, 0.3 from early December. you won’t see too many messages. If you more complex semantics will support This version replaces the existing init want more, just remove the quiet entry. conditions with logical operators and process, however, don't assume that all On first inspection, the changes under pass in parameters to event scripts, if the start scripts have been modified to the hood are also hidden. If you give the needed. These files play the same role use the event mechanism. Upstart does man init or man telinit command, you as the entries in the legacy /etc/inittab. not support temporal events right now, are informed that the runlevel system This is why Edgy Eft has both rc2 and and there are plans to use other pro- has a new engine. No /etc/inittab file is the files shown at the top of Figure 2. grams, such as and the ACPI or another telltale clue. /etc/init.d and the APM daemons, as event sources. accompanying start scripts still exist at Upstart-Compliant Jobs present as Ubuntu currently runs Upstart There are two ways of defining your own State of the Art in compatibility mode. The mid-term jobs. The simple method uses the exec The current version of Ubuntu boots plan is to allow /etc/event.d to handle /path/program -O --optional parameter quite quickly, although you won’t see job definitions, which boil down to sim- approach. This works just like in the much of what’s going on if you use the ple, non-executable files like the one shell. Upstart actually uses a shell to default setup. The boot splash screen shown in Listing 1. The example takes handle quotes, "" or $. If the job defini- with its progress indicator, and the OS the easy way out, and simply calls the tion contains more than a simple com- logo hide any useful information. Al- old start scripts for runlevel 2 (line 20). mand line, the shell script can reside be- though normal users may not object to As you can see from line 5, we want tween the script and end script tokens this, it takes some getting used to for ad- the script to run whenever the runlevel-2 (Listing 1, lines 12 through 21). mins. Even if you remove the colorful event occurs. It ends if the events shut- There are two variations on this script- ing theme, start script and stop script. Gentoo The start script does what the service re- As one of the more recent additions to match. While the start, stop, and restart quires, like creating directories or check- the Linux family, Gentoo solved the sections keep to the traditional design, problem of organizing runlevel scripts in more interesting things happen in de- Listing 1: Job Definitions its own special way. To do so, it does not pend. A service depends on other ser- use simple bash scripts as runlevel vices or preparatory settings on the one 01 # /etc/event.d/rc2 scripts, but instead launches a separate hand; but on the other hand, it can pro- 02 # Runlevel 2 compatibility interpreter: /sbin/runscript. An example vide specific functions that other ser- script for Upstart of a typical structure follows: vices require: 03 # This job runs the old • need service: Depends on the service. #!/sbin/runscript Sys V scripts. • use service: Uses the service. opts="depend start stop restart" 04 • provide functionality: Provides a spe- depend() { 05 start on runlevel-2 cific functionality. # dependencies and conditions • before service: Should start before the 06 } other service. 07 stop on shutdown start() { • after service: Should start after the 08 stop on runlevel-3 specified service. # commands for starting 09 stop on runlevel-4 services Gentoo also supports virtual services, 10 stop on runlevel-5 such as net, as there are various kinds of # including preparatory tasks network (Ethernet, Modem, WLAN). This 11 } also applies to mail servers (mta). The 12 script stop() { start script can even determine depen- 13 set $(runlevel --set 2 dencies dynamically, as the /etc/init.d/ # commands for stopping syslog-ng example shows: || true) services case $(sed 's/#.*//'U 14 if [ "$1" != "unknown" # plus clean-up actions ]; then /etc/syslog-ng/syslog-ng.conf) in } 15 PREVLEVEL=$1 *source*tcp*|*source*udp*|U restart() { 16 RUNLEVEL=$2 *destination*tcp*|*destination*udp*) # Restarting as service 17 export PREVLEVEL need net ;; } RUNLEVEL esac The string that follows opts lists all the 18 fi functions provided by the runlevel script. As long as the change does not conflict 19 If you need to add your own functions, with existing dependencies, admins can you just add them to the list, and script a alter the order in which services are 20 exec /etc/init.d/rc 2 function block of the same name to started, using before or after. 21 end script

64 ISSUE 76 MARCH 2007 WWW.LINUX - MAGAZINE.COM Upstart SYSADMIN

Figure 2: Edgy Eft stores files that define jobs for the typical events from the legacy inittab in /etc/ event.d. ing access privileges. The stop script exec /usr/ cleans up after the service terminates. local/bin/U simpleserver.U Self-Executing sh Using a simple server as an example, Figure 3: The initctl list command provides you with a system status look at what creating your own Upstart To start and stop overview. scripts involves. The server doesn’t have services, including to do anything but keep on running. The the one just defined, we still rely on the sages on screen at boot time, you can following section is based on a two-liner start and stop commands, plus there is a check them later, of course. in /usr/local/bin/simpleserver.sh: new status call. Generally speaking, the Upstart script A simple start simple-server brings the output is passed to the logd included #/bin/sh service to life. To see whether the com- with the package, and the daemon while true ; do sleep 1U mand worked, use initctl list or status hands them on to /var/log/boot (Listing ; done simple-server. stop simple-server termi- 3). The initctl list call provides another nates the service (Listing 2). source of debugging output (Figure 3). Let’s call the event script for this service If everything is working, most users /etc/event.d/simple-server. If we only do not want to see system messages. Upstarting want to support manual launching of the However, log information can be useful, As Ubuntu is based on Debian, the service, all we need is a single line in the especially if you just modified the sys- chances of accelerating Debian, thanks event script to start the server: tem. If you prefer not to output mes- Listing 2: Service Control Suse Linux 01 root@EdgyEft:~# start While the traditional Sys V Init folllows a # Should-Start: $syslog simple-server strictly linear approach, more recent # Required-Stop: 02 simple-server (start) running, Suse Linux versions (10.0 or newer) sup- port parallelization of boot script calls. # Default-Start: 3 5 process 6507 active Administrators can enable this feature in # Default-Stop: 0 1 2 6 03 root@EdgyEft:~# stop the /etc/sysconfig/boot file by setting the # Description: U simple-server RUN_PARALLEL variable to yes. This Start Network Blockdevice Daemon 04 simple-server (stop) running, changes the legacy sequence defined by process 6507 killed S00script1 through S99script25. ### END INIT INFO Instead, the .depend.boot, .depend.start, This hides much of the complexity from 05 root@EdgyEft:~# status and .depend.stop dependencies are ap- the user, however, the approach does simple-server plied. If an administrator adds a simple not provide much in the line of speed 06 simple-server (stop) waiting script, say, S12nbd-server, to rc3.d, by benefits. When we tested the Suse-style 07 root@EdgyEft:~# start creating a link in the traditional manner, boot on the X41 mentioned earlier on – the system will just ignore the change. the machine admittedly has a fairly lame simple-server The insserv command handles this task hard disk – the parallel boot took just 08 simple-server (start) running, by evaluating the file header to ensure over a minute, which is fairly close to the process 6517 active correct resolution of dependencies: 70-second value for the legacy approach. 09 root@EdgyEft:~# status You can actually see some evidence of ### BEGIN INIT INFO simple-server parallel service launching; the screen # Provides: nbd-server output is mixed up. 10 simple-server (start) running, # Required-Start: $network process 6517 active

WWW.LINUX - MAGAZINE.COM ISSUE 76 MARCH 2007 65 SYSADMIN Upstart

Listing 3: Upstart Boot can simply give the -get install In this scenario, you will need to modify sysvinit command to achieve your goal. the scripts you dropped into /etc/event.d. 01 [...] To do this, just add the following line 02 Dec 3 18:44:59 rc2: Self-Administration after the script line in rc-default and * Starting deferred execution If you decide to build Upstart yourself, rcS-sulogin: scheduler atd [ ok ] you should manually remove the sysvinit U 03 Dec 3 18:44:59 rc2: package. If you fail to do so, make install export PATH=/opt/upstart/ * Starting periodic command will overwrite the central binaries, and sbin:$PATH scheduler... [ ok ] the package manager will either ignore the changes, or worse, decide that your Because the Upstart directory is at the 04 Dec 3 18:44:59 rc2: * Enabling additional system is corrupted. This said, it is easy beginning of your search path, scripts executable to build and install Upstart from the will use the new telinit command. binary formats ...[ ok ] source code [1]: The system will still boot Sys V Init by default, but on booting, you can tell the 05 Dec 3 18:44:59 rc2: ./configure --prefix=/usr U kernel to use an alternative to the legacy * Checking battery state... U init. The following kernel command line [ ok ] --exec-prefix=/ --sysconfdir=/etc parameters will do the trick: init=/opt 06 Dec 3 18:44:59 rc2: make /upstart/sbin/init. * Running local boot scripts make install For simple tests, you might prefer to (/etc/rc.local) [ ok ] enter the parameters at the bootloader After completing these steps, the system prompt, but you could add a bootloader to the changes in Ubuntu, are pretty will ask you for your init scripts. configuration menu entry if you prefer high. If you are prepared to take the risk, To get started, download the exam- (Listing 3a, line 5). you can opt to either replace the existing ple-jobs-2.tar.gz tarball from the /down- Sys V Init completely, or to use Upstart load directory [1] and unpack in /etc/ Analysis in parallel. event.d. Bootchart [4] gives admins an excellent The steps for implementing plan A method for comparing the legacy and (using Upstart to completely replace the Parallel Worlds new boot processes. The tool logs the legacy system) in Debian Unstable are If you want to avoid an unsuccessful Up- CPU load and hard disk I/ O performance fairly simple – the developers of the dis- start installation wrecking your working at boot time and later converts the re- tribution have already completed the System V Init system, you can install Up- sults to a neat graph. To allow this to preparatory work by separating the sys- start side by side with Sys V. To do so, happen, you need to install the Boot- vinit-utils from the sysvinit package. follow the same steps as for the Upstart- chart package and add an entry to the This means that you can easily replace only install, but keep the sysvinit pack- kernel command line. bootchartd runs as sysvinit with Upstart, and just keep the age, and make sure the new init ends up an initial process and launches the init old scripts. in /opt/upstart: process proper. There is an Upstart package in the Listing 3b shows you the entry for Debian Experimental repository [2]. To ./configure -prefix=/opt/U Grub. If you are running Upstart in side- use the package, add the following entry upstart --sysconfdir=/etcU by-side mode with the legacy Init, let to your /etc/apt/sources.list: --enable-compat Bootchart know by adding the following to the kernel command line: http://ftp.de.debian.org/U Listing 3a: Grub debian/experimental main Configuration bootchart_init=/opt/U upstart/sbin/init 01 # /boot/grub/menu.lst After doing this, give the command to remove the legacy sysvinit package: 02 [...] After doing so, Bootchart will log any 03 title Ubuntu, kernel interesting process data every 0.2 sec- apt-get install upstartU 2.6.17-10-generic onds and store the information in /var upstart-compat-sysv 04 root (hd0,0) 05 kernel /boot/vmlinuz- Listing 3b: Bootchart Because sysvinit is tagged as required, 2.6.17-10-generic root=/dev/ the package manager will wait until you hdb1 ro quiet splash init=/ 01 # /boot/grub/menu.lst type Yes, do as I say!. The next pitfall oc- opt/ 02 [...] curs when you update. apt-get dist-up- upstart/sbin/init 03 kernel /boot/vmlinuz grade will remove the Upstart packages 06 initrd /boot/initrd.img -2.6.17-10-generic you just installed, and reinstate the sys- -2.6.17-10-generic root=/dev/hdb1 ro quiet vinit package. 07 boot splash init=/sbin/bootchartd If this is your intention – that is, if re- 08 [...] 04 [...] ally do want to reinstate Sys V Init – you

66 ISSUE 76 MARCH 2007 WWW.LINUX - MAGAZINE.COM advertisement SYSADMIN Upstart

major boot speed gains until Edgy is re- placed by Feisty Fawn. Conclusion The days when a computer booted in a couple of seconds are gone, and the kind of surgery that many distributions use to patch up boot performance are not likely to correct the problem. The future looks good for new designs like Upstart. Even if Upstart does not offer the same kind of “switch on and go” feeling you get from a gaming console, it requires less patience than the legacy system. Of course, a fair level of administrative skill is required to migrate a working Linux machine without having to reinstall. Our crystal ball reveals that Upstart is Figure 4a: This Bootchart analysis shows Debian GNU/ Linux booting with the legacy init pro- aiming to do more than simply rejuve- cedure. nate the boot process: the developers en- visage a central service daemon that will /log/bootchart.tgz once the boot process manager is launched. The fact that Up- take over chores that are handled by a has completed. The bootchart -f png gen- start launches jobs in parallel means that potpourri of tools right now. This in- erates a PNG graphic from the data, with this step simply begins earlier, before all cludes executing specific events at spe- svg and eps as your other options. the other critical boot processes have cific times, in other words, replacing If you compare the Sys V boot graph completed. cron and at. In the meantime, Upstart in Figure 4a with the Upstart graph in It would be unfair to the Upstart proj- still has to demonstrate its ability to or- 4b, the results are misleading. On our ect to ignore the promise of Upstart on ganize the boot process in a meaningful lab machine, Bootchart reports that the the basis of the current test results. Re- way, and as of now, it’s well on the way legacy Sys V Init takes a hefty 33 sec- member that the init successor will be to doing just that. ■ onds, whereas Upstart is all done in 23 running in compatibility mode until seconds flat. more progress has been made. INFO Checking these results with a stop- You can expect major speed gains as [1] Ubuntu Upstart: watch revealed that the gain is actually soon as the individual start scripts have http:// upstart. ubuntu. com a mere two seconds. Bootchart stops the been adapted to support the new system. [2] Upstart in the Debian Experimental clock as soon as KDM or another login Thus, Ubuntu does not expect to see branch: http:// packages. debian. org/ experimental/ admin/ upstart [3] Scott’s blog entry on Upstart: http:// www. netsplit. com/ blog/ articles/ 2006/ 08/ 26/ upstart-in-universe [4] “Boot Camp” by Charly Kühnast, Linux Magazine, March 2005, pg. 61

Nico Dietrich studies computer sci- ence at the Technical University in Berlin, Germany, and is interested in free software, direct democracy, and developing countries. He will be looking to combine these fields after graduating. Dirk von Suchodoletz works as an assistant at the Department of Com- munication Systems at the Univer- sity of Freiburg, Germany. His devel-

THEAUTHORS opment work focuses on diskless Linux projects, which explains his interest in system boot variants in general, and in accelerating the boot Figure 4b: Replaying Sys V Init with Upstart does not change the results greatly. process in particular.

68 ISSUE 76 MARCH 2007 WWW.LINUX - MAGAZINE.COM