Improving Linux Startup Time Using Software Resume (and other techniques) Hiroki Kaminaga Sony Corporation [email protected] Abstract 1 Software suspend This paper presents a new resume operation as Snapshot boot is based on the current software well as other startup time improvement tech- suspend technology in the Linux kernel. Soft- niques which are aimed at achieving fast startup ware suspend is independent of APM or ACPI, time for embedded Linux systems. A new which makes it more applicable to embedded fast boot method called snapshot boot is intro- systems, where APM or ACPI is not present in duced. Snapshot boot is essentially a resume- many cases. Before describing snapshot boot, from-disk operation, which is a system resume the standard procedure of software suspend is from a semi-permanent snapshot image stored described to assist in understanding the proce- on disk or flash memory, that restores the ma- dure of snapshot boot. chine to a known running state. As opposed to a standard resume operation, a snapshot im- age is made only once, stored on disk or flash memory, and same image is used repeatedly, 1.1 Suspend states in Linux kernel every time the system is powered on. Other fast boot techniques that are discussed are: use of prelinking, a scheme to reduce the startup There are three suspend states in the Linux ker- cost of symbol relocation overhead for links to nel [1]. They are: dynamic libraries, execute in place (XIP) to re- duce or avoid OS and application loading de- lays, toolchain modifications to collect global • Standby state constructors in one place to accomplish a lo- cality benefit, and making the program .data • Suspend-to-RAM state section demand-paged from flash to avoid fully loading its pages on startup. • Suspend-to-disk state Unless otherwise stated, the startup time re- ferred to in this paper is the time from the sys- tem power on to the time user can manipulate Unless otherwise stated, the term suspend is the device. This includes userland application referred to as Suspend-to-disk. This is also startup as well as kernel startup time. known as hibernation. 18 • Improving Linux Startup Time Using Software Resume (and other techniques) 1.2 Suspend procedure 6. Save processor state This is done by calling the save_ processor_state() procedure. It will The suspend procedure is shown below: save registers other than generic ones, such as segment registers, co-processor registers, and so on. 1. Trigger Suspend procedure is triggered from writ- 7. Save processor registers ing disk to /sys/power/state opera- This is done by calling the swsusp_ tion. The call stack to the entrance of the arch_suspend() procedure. It will save suspend procedure is shown in Figure 1. general registers. This is written in assem- bly language, since the stack may not be sys_write() used. +-vfs_write() +-sysfs_write_file() 8. Allocate memory for snapshot image +-flush_write_buffer() This is done by calling the swsusp_ +-subsys_attr_store() +-state_store() alloc() procedure. Page directories +-enter_state() get allocated by calling __get_free_ +-pm_suspend_disk() pages(), and pages for the image itself gets allocated by get_zeroes_page() Figure 1: Call graph to entrance of software for each page directory entry. suspend 9. Copy memory contents to allocated area This is done by calling the copy_data_ 2. Freeze processes pages() procedure. It calls memcpy() This is done by calling the freeze_ for each page to copy. processes() procedure. It freezes user processes, and then freezes kernel tasks. 10. Restore processor state This is done by calling the 3. Free unnecessary memory restore_processor_state() pro- This is done by calling the free_some_ cedure in swsusp_suspend(). This memory() procedure. It calls shrink_ is where the software suspend resume all_memory() inside. procedure comes back. It restores the previously saved processor state. 4. Suspend devices This is done by calling the device_ 11. Power up devices suspend() procedure. It calls suspend_ This is done by calling the device_ device() and then the suspend() power_up() procedure. It resumes sys- method for all listed active devices. tem devices and all listed power off de- vice. 5. Power down devices This is done by calling the device_ 12. Resume devices power_down() procedure. It calls This is done by calling the device_ suspend_device() to for all listed resume() procedure. It resumes devices power off devices. in power off device list. 2006 Linux Symposium, Volume Two • 19 13. Write page, pagedir, header image to swap 5. Read page directory into allocated mem- Now that the devices are active, write ory to swap could be performed. This is This is done by calling the read_ done by calling the write_suspend_ pagedir() procedure. It allocates page image() procedure. It writes image data, directory memory space by using __get_ page directories, and then image header free_pages() and reads page directory into swap. information with bio_read_page(). 14. Power down devices 6. Relocate page directory (if necessary) This is done by calling the device_ shutdown() procedure. It calls the shut- 7. Read swap image into allocated memory down() method for each device. Then, the This is done by calling the data_read() system device is shutdown. procedure. The page directory area gets relocated if it collides with the snapshot 15. Halt machine image. Then the snapshot image is read This is done by calling the machine_ from swap with bio_read_page(). power_off() procedure. It calls pm_ power_off() and the machine halts. 8. Prepare resume 9. Freeze process 1.3 Resume procedure 10. Free unnecessary memory 11. Suspend devices The resume procedure is shown below: These steps are taken to accomplish con- sistency between suspend and resume, and in case resume fails. These steps are same Start resume 1. as Steps 2 to 4 of the suspend procedure. Resume starts by calling the software_resume() procedure in 12. Power down devices do_initcalls(), at late_initcall This step is taken to accomplish consis- timing. tency between suspend and resume. This step is same as Step 5 of the suspend pro- 2. Check kernel parameter cedure. In software_resume(), it checks for the kernel command line for the resume 13. Save processor state swap device. These steps are taken in case resume fails. These steps are same as Step 6 of the sus- 3. Check signature and header of snapshot pend procedure. image This is done by calling the check_sig() 14. Copy snapshot image in allocated memory and check_header() procedures. It to its original address checks swap image signature for snapshot image, and that the header for the kernel 15. Restore processor registers used for suspend and resume is the same. This is done by calling the swsusp_ arch_resume() procedure. It copies all 4. Allocate memory space for snapshot im- image pages from the allocated memory age address to its original memory address. 20 • Improving Linux Startup Time Using Software Resume (and other techniques) It also restores general purpose registers. Kit (OSK5912) reference board was used [4]. Since registers are restored, the return ad- Since this board does not have a disk, NOR dress used by this function would be the flash is used to store the snapshot image. same as the one in effect for the swsusp_ arch_suspend() procedure call at sus- To enable software suspend, the CONFIG_PM pend time. and CONFIG_SOFTWARE_SUSPEND configura- tion options must be set when building the linux 16. Restore processor state kernel. After the target is booted with the new kernel, the following commands were issued to 17. Power up devices enter suspend. This is exactly the same as Steps 10 to 11 of the suspend procedure. # mkswap /dev/mtdblock3 18. Free memory allocated for image # swapon /dev/mtdblock3 This is done by calling the free_image_ # mount -t sysfs none /sys pages() procedure. It does free_ # echo disk >/sys/power/state page() for all pages in image. Page di- rectories are also freed. A kernel message is printed to console, and the 19. Resume devices system would halt. At the next system power resume=/dev/ This is done by calling the device_ on, passing the argument: mtdblock3 resume() procedure. The process is the (in the above case) will trigger same as Step 12 of the suspend procedure. software resume. 20. Thaw processes This is done by calling the thaw_ 2 Snapshot boot processes() procedure. This wakes up every thread by calling the wake_up_ process() procedure. 2.1 Preserving snapshot image 1.4 Software suspend support for ARM ar- For normal use of software suspend and re- chitecture sume, a snapshot image is created and de- stroyed on every suspend/resume cycle. Since the aim of using software suspend in an embed- Software suspend does not support the ARM ded product in this paper is to improve startup architecture in a vanilla kernel. To port soft- time, a snapshot image is created only once, ware suspend for other architectures, a porting stored on disk or flash memory, and the same note [2] was followed which shows how to port image is used repeatedly, every time the system for the ARM architecture. The patch for soft- is powered on. This is accomplished by not in- ware suspend ARM support is posted to a pub- validating the snapshot image at resume time. lic mailing list [3]. 2.2 Principle of software resume for im- 1.5 Execution of software suspend proving startup time To evaluate software suspend for an embedded The time to a certain point in running system system, an ARM-based OMAP 5912 Starter state could be roughly expressed as follows: 2006 Linux Symposium, Volume Two • 21 startup time = ∑Storage to RAM + • The snapshot image is copied twice. ∑setup I/O state + ∑setup RAM state • There is redundancy in device state transi- tions during booting.