Keeping the Linux Kernel Honest Testing Kernel.org kernels Kamalesh Babulal Balbir Singh IBM IBM [email protected] [email protected] Abstract 10000000 9500000 TM 9000000 The Linux Kernel release cycle has been short v2.6.24 v2.6.23 with various intermediate releases and with the lack 8500000 v2.6.22 of a separate kernel development tree. There have v2.6.21 8000000 v2.6.20 v2.6.19 been many challenges with this rapid development v2.6.18 Lines of Code 7500000 v2.6.17 v2.6.16 such as early bug reporting, regression tracking, func- v2.6.15 v2.6.14 tional/performance testing, and test coverage by dif- 7000000 v2.6.13 v2.6.12 v2.6.11 ferent individuals and projects. Many kernel develop- 6500000 ers/testers have been working to keep the quality of the 6000000 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 kernel high, by testing as many possible subsystems as Days of Development they can. Figure 1: Size of the kernel with every release In this paper, we present our kernel testing methodology, infrastructure, and results used for the v2.6 kernels. We summarize the bug reporting statistics based on the dif- • Testing on wide range of architectures and plat- ferent kernel subsystems, trends, and observations. We forms, will also present code coverage analysis by subsystem • Regressions carried from previous release, for different test suites. • Different development trees, and 1 Introduction • Various configurations and boot options to be tested. The Linux kernel has been growing with every release as a result of the sheer number of new features being In Section 2 we briefly explain the challenges. Section 3 merged. These changes are being released at a very high outlines our methodology to meet the challenges. Sec- rate, with each release containing a very large number tion 4 presents the results of code coverage analysis, in- of changes and new features. The time latency at which cluding the fault injection coverage results for the stan- these changes are made is very short between every re- dard test cases and we discuss our future plans in Sec- lease cycle. All of this is occurring across many dis- tion 5. parate hardware architectures. This presents unique problems for a tester who must 2 Challenges take all of the following into account: 2.1 Different Trees • Lines of code added, • Time interval between each release, In the past, Linux kernel development had separate de- velopment and stable trees. The stable trees had an even • Number of intermediate releases, number and the development trees had an odd number • 19 • 20 • Keeping the Linux Kernel Honest as their second release number. As an example 2.5 was a regular basis would allow the development and stable development release, while 2.4 was stable release. With releases to be based on the -next tree in the future (re- the 2.6 kernel development, there are no separate devel- fer to [5] for more information on kernel trees). opment and stable trees. All the development is based upon the current stable tree and once the current devel- opment tree is marked as stable, it is released as the next mainline release. maintainers tree -mm -stable There are intermediate development releases between major releases of a 2.6 kernels, each having its own -rc1 distributions significance. Figure 2 shows the different development trees and the flow of patches from development trees to the mainline kernel. -gits The -stable tree contains critical fixes for security -rcN release -next users problems or significant regressions identified for the mainline tree it is based upon. Once the mainline tree is released, the developers start contributing to the new mainline release features to be included in the next release. The new fea- tures are accepted for the inclusion during the first two weeks of development following the mainline release. Bug Fixes Features Rebase Releases After two weeks the merge window closes and the ker- nel is in feature freeze; no further features will be ac- cepted into this release. This is released as -rc1. After Figure 2: Linux Kernel Development Cycle -rc1 the features are tested well and stabilized. During this period roughly weekly release candidates, -rc, are produced, as well as intermediate snapshots of Linus’s 2.2 Release Frequency git tree. Testing is essential to deliver high quality software and The -rc releases are a set of patches for the bug fixes testing every phase of development is important if we and other important security fixes, based on the previous are to catch bugs early. This is especially true for -rc release. For example, 2.6.25-rc3 will have the fixes projects like Linux where the source is growing by for the bugs identified in 2.6.25-rc2. 2.32% [1] with every release, as represented in Figure 1. In addition to the mainline releases, we have a number These numbers are pretty high for any project. Starting of testing trees for less stable features, as well as sub- from 2.6 kernel development series, there is no sepa- system specific trees. The -mm tree has experimental rate stable and development tree, which means that more patches and critical fixes that are planned to be pushed code is being added and it needs to be tested thoroughly. to the mainline kernel. For a new feature it is recom- Bugs caught and fixed early in the cycle helps to main- mended that it be tested in -mm, since that tree under- tain the kernel quality by: goes rigorous testing, which in-turn helps in stabilizing the feature. -mm is rebased often to development re- leases to test the patch set against the development tree. • Providing a pleasant user experience, The -next release was introduced with the 2.6.24 • Avoiding building on top of buggy code, and development series. The -next tree has the patch • Easier debugging, since the code is fresh in the au- changesets from different maintainers, intended to be thors mind. merged into the next release. Changesets are rebased to the current development tree, which helps to resolve the merge conflicts and bugs before they get introduced The stable kernel releases are made approximately ev- in the next release. Resolving the conflicts and bugs on ery 2-3 months, during which time an average of 3.18 2008 Linux Symposium, Volume One • 21 v2.6 kernels • Regression Testing, versions .20 .21 .22 .23 .24 Total stable 22 8 20 18 5 73 • Stress Testing, stable-git 16 18 18 26 22 100 • Performance Testing, and stable-mm 2 2 0 1 1 4 stable-mm • Functional Testing. +hotfixe(s) 0 0 0 3 6 9 rc 7 7 11 8 8 41 These efforts are not being captured completely because rc-git 59 52 57 70 47 285 most of these efforts are not visible. We will not be able rc-mm 12 6 10 7 4 41 to account for any testing done unless it is being pub- rc-mm lished or shared. Many features are accepted into the +hotfixe(s) 0 10 18 14 3 45 mainline after stringent code reviews and ample testing next 0 0 0 0 35 35 in the development releases. But not many developers Total 118 103 134 147 131 633 provide the test cases or guidelines to test their func- tionality even though it is in the interest of developers to Table 1: Summary of releases between kernel versions keep the quality of their code high. Over the years there have been many test projects and changes are being accepted every hour. This has been suites with a primary focus to improve the quality of the trend for the past 1 1 years. Linux. They are constantly undergoing lots of changes 2 between every release. They have been adding new test Table 1 shows the data of various intermediate releases cases to test the new features getting merged into the 1 kernel, but the updating is not fast enough to catch those made for past 1 2 years. These incremental changes to the kernel source ensure that any code changes made early bugs and for some features we do not have test to the kernel are well tested before they accepted into cases available. mainline. Any person who is interested in testing the kernel has no With the average of 126 kernels (refer to Table 1) be- single, nor even a small number, of test projects which ing released between two sequential major releases, we can cover most of the kernel subsystems. We do have have at least one kernel per day being released. Devel- a large number of test projects, but each is independent opers end up limiting their testing to unit testing for the requiring installation and configuration; the test setup is changes they make and start concentrating on new de- a huge effort for a tester. Not all the testers have the velopment. It is not practical for most of the developers harness infrastructure in place for testing due to the lim- to test their changes across all architectures supported itations of the hardware they own. by Linux. A significant amount of testing is being done Most of the test projects act at best as regression, stress, by others from the community, which includes testers, performance test suites, or combinations thereof, but we developers, and vendors (including distribution compa- do not have tests which can be used for functional test- nies). ing of the new features being merged into the kernel.