Keeping the Linux Kernel Honest Testing Kernel.org kernels
Kamalesh Babulal Balbir Singh IBM IBM [email protected] [email protected]
Abstract 10000000 9500000
TM 9000000 The Linux Kernel release cycle has been short v2.6.24
v2.6.23 with various intermediate releases and with the lack 8500000 v2.6.22 of a separate kernel development tree. There have v2.6.21 8000000 v2.6.20 v2.6.19 been many challenges with this rapid development v2.6.18 Lines of Code 7500000 v2.6.17 v2.6.16 such as early bug reporting, regression tracking, func- v2.6.15 v2.6.14 tional/performance testing, and test coverage by dif- 7000000 v2.6.13 v2.6.12 v2.6.11 ferent individuals and projects. Many kernel develop- 6500000 ers/testers have been working to keep the quality of the 6000000 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 kernel high, by testing as many possible subsystems as Days of Development they can. Figure 1: Size of the kernel with every release In this paper, we present our kernel testing methodology, infrastructure, and results used for the v2.6 kernels. We summarize the bug reporting statistics based on the dif- • Testing on wide range of architectures and plat- ferent kernel subsystems, trends, and observations. We forms, will also present code coverage analysis by subsystem • Regressions carried from previous release, for different test suites. • Different development trees, and
1 Introduction • Various configurations and boot options to be tested. The Linux kernel has been growing with every release as a result of the sheer number of new features being In Section 2 we briefly explain the challenges. Section 3 merged. These changes are being released at a very high outlines our methodology to meet the challenges. Sec- rate, with each release containing a very large number tion 4 presents the results of code coverage analysis, in- of changes and new features. The time latency at which cluding the fault injection coverage results for the stan- these changes are made is very short between every re- dard test cases and we discuss our future plans in Sec- lease cycle. All of this is occurring across many dis- tion 5. parate hardware architectures.
This presents unique problems for a tester who must 2 Challenges take all of the following into account: 2.1 Different Trees • Lines of code added,
• Time interval between each release, In the past, Linux kernel development had separate de- velopment and stable trees. The stable trees had an even • Number of intermediate releases, number and the development trees had an odd number
• 19 • 20 • Keeping the Linux Kernel Honest as their second release number. As an example 2.5 was a regular basis would allow the development and stable development release, while 2.4 was stable release. With releases to be based on the -next tree in the future (re- the 2.6 kernel development, there are no separate devel- fer to [5] for more information on kernel trees). opment and stable trees. All the development is based upon the current stable tree and once the current devel- opment tree is marked as stable, it is released as the next mainline release. maintainers tree -mm -stable There are intermediate development releases between major releases of a 2.6 kernels, each having its own -rc1 distributions significance. Figure 2 shows the different development trees and the flow of patches from development trees to the mainline kernel. -gits
The -stable tree contains critical fixes for security -rcN release -next users problems or significant regressions identified for the mainline tree it is based upon. Once the mainline tree is released, the developers start contributing to the new mainline release features to be included in the next release. The new fea- tures are accepted for the inclusion during the first two weeks of development following the mainline release. Bug Fixes Features Rebase Releases After two weeks the merge window closes and the ker- nel is in feature freeze; no further features will be ac- cepted into this release. This is released as -rc1. After Figure 2: Linux Kernel Development Cycle -rc1 the features are tested well and stabilized. During this period roughly weekly release candidates, -rc, are produced, as well as intermediate snapshots of Linus’s 2.2 Release Frequency git tree. Testing is essential to deliver high quality software and The -rc releases are a set of patches for the bug fixes testing every phase of development is important if we and other important security fixes, based on the previous are to catch bugs early. This is especially true for -rc release. For example, 2.6.25-rc3 will have the fixes projects like Linux where the source is growing by for the bugs identified in 2.6.25-rc2. 2.32% [1] with every release, as represented in Figure 1. In addition to the mainline releases, we have a number These numbers are pretty high for any project. Starting of testing trees for less stable features, as well as sub- from 2.6 kernel development series, there is no sepa- system specific trees. The -mm tree has experimental rate stable and development tree, which means that more patches and critical fixes that are planned to be pushed code is being added and it needs to be tested thoroughly. to the mainline kernel. For a new feature it is recom- Bugs caught and fixed early in the cycle helps to main- mended that it be tested in -mm, since that tree under- tain the kernel quality by: goes rigorous testing, which in-turn helps in stabilizing the feature. -mm is rebased often to development re- leases to test the patch set against the development tree. • Providing a pleasant user experience,
The -next release was introduced with the 2.6.24 • Avoiding building on top of buggy code, and development series. The -next tree has the patch • Easier debugging, since the code is fresh in the au- changesets from different maintainers, intended to be thors mind. merged into the next release. Changesets are rebased to the current development tree, which helps to resolve the merge conflicts and bugs before they get introduced The stable kernel releases are made approximately ev- in the next release. Resolving the conflicts and bugs on ery 2-3 months, during which time an average of 3.18 2008 Linux Symposium, Volume One • 21
v2.6 kernels • Regression Testing, versions .20 .21 .22 .23 .24 Total stable 22 8 20 18 5 73 • Stress Testing, stable-git 16 18 18 26 22 100 • Performance Testing, and stable-mm 2 2 0 1 1 4 stable-mm • Functional Testing. +hotfixe(s) 0 0 0 3 6 9 rc 7 7 11 8 8 41 These efforts are not being captured completely because rc-git 59 52 57 70 47 285 most of these efforts are not visible. We will not be able rc-mm 12 6 10 7 4 41 to account for any testing done unless it is being pub- rc-mm lished or shared. Many features are accepted into the +hotfixe(s) 0 10 18 14 3 45 mainline after stringent code reviews and ample testing next 0 0 0 0 35 35 in the development releases. But not many developers Total 118 103 134 147 131 633 provide the test cases or guidelines to test their func- tionality even though it is in the interest of developers to Table 1: Summary of releases between kernel versions keep the quality of their code high.
Over the years there have been many test projects and changes are being accepted every hour. This has been suites with a primary focus to improve the quality of the trend for the past 1 1 years. Linux. They are constantly undergoing lots of changes 2 between every release. They have been adding new test Table 1 shows the data of various intermediate releases cases to test the new features getting merged into the 1 kernel, but the updating is not fast enough to catch those made for past 1 2 years. These incremental changes to the kernel source ensure that any code changes made early bugs and for some features we do not have test to the kernel are well tested before they accepted into cases available. mainline. Any person who is interested in testing the kernel has no With the average of 126 kernels (refer to Table 1) be- single, nor even a small number, of test projects which ing released between two sequential major releases, we can cover most of the kernel subsystems. We do have have at least one kernel per day being released. Devel- a large number of test projects, but each is independent opers end up limiting their testing to unit testing for the requiring installation and configuration; the test setup is changes they make and start concentrating on new de- a huge effort for a tester. Not all the testers have the velopment. It is not practical for most of the developers harness infrastructure in place for testing due to the lim- to test their changes across all architectures supported itations of the hardware they own. by Linux. A significant amount of testing is being done Most of the test projects act at best as regression, stress, by others from the community, which includes testers, performance test suites, or combinations thereof, but we developers, and vendors (including distribution compa- do not have tests which can be used for functional test- nies). ing of the new features being merged into the kernel. It is very critical for new code to be tested thoroughly, 2.3 Test Case Availability even before it gets merged in to the mainline.
In the existing scenario there are many valuable test There is a significant testing effort in the community by cases/scripts available from individuals, which could various individual testers, developers, distribution com- expose bugs on other environments untested by them. panies, and Independent Software Vendors. They con- Sharing these test cases and scripts with the community tribute to the effort with combinations of testing meth- through one of the test projects will help in improving ods such as: the testing much more by:
• Compile and Boot test (including the cross com- • Enabling the code to be tested on a variety of hard- piler tests), ware, 22 • Keeping the Linux Kernel Honest
• Improving the test coverage on executing all possi- 2.5 Existing Test Projects ble code paths, • Avoiding duplication of test case development, and There are many existing test projects used by the indi- vidual test contributors. We summarize some of them • Making reproduction of bugs easier. along with their key features:
2.4 Kernel options LTP (Linux Test Project)2 is a Regression/Functional test suite. It contains 3000+ test cases to test the ba- The Linux kernel is highly configurable, which allows sic functionality of the kernel. It is capable of test- specific features to be enabled as required. This means ing/stressing filesystem, memory, scheduler, disk I/O, that to test the kernel fully we would have to test with network, and system calls. It also provides some ad- each combination of these options. Testing kernels com- ditional test suites such as pounder, kdump, open-hpi, piled only with the best known1 configurations cannot open-posix, code coverage, and others. expose any bugs hidden under the untried combinations. As an example the kernel can be configured to use any It is ideal for running the basic functionality verification, one of the following different memory models: with sufficient stress generated from the test cases. LTP does not support the kernel build test. LTP results can CONFIG_FLATMEM be formatted as HTML pages. It lacks the support for CONFIG_DISCONTIGMEM machine parseable logs. The test case results are either CONFIG_SPARSEMEM PASS or FAIL, which makes it complex for a tester to CONFIG_SPARSEMEM_EXTREME understand the reason behind test failure. CONFIG_SPARSEMEM_VMEMMAP IBM autobench is a client harness project which sup- ports setting up the test execution environment, execu- They are mutually exclusive, which means the kernel tion of the test suites, and capturing the logs with envi- can be compiled with only one of these options. Testing ronmental statics. It supports a kernel build test along all combinations of memory models would require five with support for profiling. It is capable of executing test different build and test cycles. Some of the combina- cases in parallel. The job control support is basic, allow- tions should be tested to improve the testing coverage of ing user to have minimal control over the way the tests the kernel. are executed. The tool is written using bash/perl scripts. Many of the new features getting merged into the kernel 3 are not tested by all individuals because their existence Autotest is an open source based client/server harness is not known to the tester. The example above shows capable of running as a standalone client or is easily how quickly permutations and combinations can grow. plugged into an existing server harness. Test cases in- Usually what gets tested is the defaults on each architec- cluded are capable of regression, functionality, stress, ture and the defaults that depend on the machine (testers performance, kernel build tests, and they support vari- do not deviate from a configuration that works for them). ous profilers. Autotest is written in python which al- lows the user to take more control of job execution by It is important to test the responsiveness of the kernel including python syntax in the job control file. Its object with different boot options (refer to [10] for more avail- oriented and has a cleaner design. able kernel parameters). For example, booting with less memory by passing mem=
Results Analysis: tests produce large amount of logs
New Release Patch and other information. Analyzing the test information collected is time consuming. Relevant information is extracted and displayed as status using colour combi- Jobs Trigger Manual Job nations each representing the percentage of test cases completed successfully. Results can be viewed in differ- 9 All Kernels Selective Kernels ent layouts. Performance data is analysed on selective benchmarks to provide historical performance graphs.
+ Build + Regression Additional Testcases Results Publication: after automated analysis, the re- sults are made available on the TKO website. Human
Jobs Queues monitoring is needed to take action on test failures or performance regressions. The problems are reported to the community (mostly via email) with the links to the Client Harness test results.
Found Problem: when a test failure or performance re- Results Collation Found Problem gression is noticed, it is reported back to the community (mostly via email) by the person monitoring the results. Results Analysis Depending on the kernel release, another round of jobs are queued with the additional patches received for the
Results Publication problem reported. Currently only IBM engineers can manually submit the jobs on ABAT though the results and performance graphs are published as is done for reg- ular jobs. Figure 3: Infrastructure of kernel.org testing 4 Results Analysis more than just a simple queuing system. ABAT test framework is not open source, but the results are The test procedure attempts to execute as many as pos- published to the community through http://test. sible code paths in the Linux kernel using different test kernel.org8 projects. When combined together, the various tests tend to cover most of the kernel, but there has always Client harness: both IBM autobench and autotest are been a gap between tested and untested code. Code cov- supported as the client harness tools though their control erage helps us to quantify this gap. file syntax is different. The client starts the test execu- tion reading the job control file that is passed over when In this section we compare the code coverage re- the job gets scheduled. It is responsible for building the sults of 2.6.20, 2.6.21, 2.6.22, 2.6.23 TM appropriate kernel, running the tests, capturing the logs and 2.6.24 for x86 and Power architecture. We and other information, and making the results available. also look at the results of executing some of these tests Section 2.5 explains more about these clients. using the fault injection framework.
Results Collation: results are gathered asynchronously 4.1 Code Coverage Setup as the jobs complete and are pushed to test. kernel.org. They are grouped relevantly by TKO The gcov kernel patch10 and lcov package11 from LTP before publishing them. Kernel binaries and other sys- were used for the code coverage. Table 2 shows tem information dumps are stripped off the results. Re- 9user selects the row and column heads. Condition based views sults published at TKO are of a standard set of test cases are available. that are used for testing. 10http://ltp.sourceforge.net/coverage/gcov. php 8In collaboration between development and test team at IBM 11http://ltp.sourceforge.net/coverage/lcov. Linux Technology Center php 2008 Linux Symposium, Volume One • 25
the benchmarks used. Coverage was run on x86 and TM Benchmarks Description Power architectures, with following configurations: runltp A collection of tools for testing 8P Intel R XEONTM , 10GB Memory the Linux kernel and related fea- 2P IBM R POWER5+TM , 7GB Memory tures. 500000 Lines Instrumented dbench Filesystem benchmark that gen- Lines Executed erates good filesystem load. 450000 aio-stress Filesystem benchmark that gen- 400000 erates asynchronous I/O stress load. 350000 aio-cp Testing tool that copies files by 300000
using async I/O state machine. Lines of code 250000 hackbench A benchmark for measuring the performance, overhead, and scal- 200000
ability of the Linux scheduler. 150000 vmmstress Performs general stress with 100000 memory race condtions between 2.6.20 2.6.21 2.6.22 2.6.23 2.6.24 simultaneous read fault write Kernel versions fault, copy on write (COW) fault. Figure 4: Code coverage on x86 architecture kernbench A CPU throughput benchmark. It is designed to compare kernels on Kernels Lines Instrumented Lines Executed the same machine, or to compare 2.6.20 380,333 133,718 hardware. 2.6.21 383,800 134,800 ltp-stress Stresses the system using the LTP 2.6.22 386,868 136,653 test suite. 2.6.23 395,520 137,472 hugepage-tests Perform basic functional and 2.6.24 401,802 139,052 stress tests for large pages ramsnake Allocate 1/8 of the system RAM Table 3: Lines Instrumented Vs Executed on x86 archi- and kick off threads to use them tecture for 3600 seconds. random_syscall Pounds on syscall interface and Figure 4 and Table 3 show number of lines instrumented does random syscalls and the code covered of the various kernels mentioned reaim A multiuser benchmark that tests for the x86 architecture. The figure shows the following and measures the performance of trends: open system multiuser comput- ers. sdet Workload created by parallel ex- • The number of lines instrumented shows a modest ecution of common UNIX com- increase of 5.64%. When compared to the rate of mands. change of the kernel, it does not seem significant. libhugetlbfs Interacts with the Linux hugetlbfs The code coverage data above fails to show that to make large pages available to changed lines are also covered as the kernel version applications in a transparent man- changes. ner. • The code coverage shows a modest increase of Others Tested NFS, CIFS and the autofs 3.98%. It is very encouraging to see code cover- filesystem. age increase as the number of lines in the kernel Table 2: Benchmarks used for Code Coverage increase. • Versions 2.6.23 and 2.6.24 show a trend of de- cline in code coverage percentage. The coverage 26 • Keeping the Linux Kernel Honest
500000 Lines Instrumented Lines Executed 450000 120000 400000 Instrumented Lines Executed Lines
350000 100000
300000 80000 Lines of code 250000 60000 200000 Lines of code 40000 150000
100000 20000 2.6.20 2.6.21 2.6.22 2.6.23 2.6.24 Kernel versions 0 Figure 5: Code coverage on PowerTM architecture mm net kernel lib arch driver include
Kernels Lines Instrumented Lines Executed Figure 6: Component-wise code coverage on 2.6.24 ker- 2.6.20 388,019 116,736 nel with x86 architecture 2.6.21 382,552 115,242 2.6.22 385,853 116,010 2.6.23 391,993 117,168 2.6.24 407,194 122,431 Directories Lines Instrumented Lines Executed Table 4: Lines Instrumented Vs Executed on PowerTM mm 1,3147 8,597 architecture net 63,802 20,855 kernel 23,633 10,962 lib 5,152 2,785 of 2.6.20 kernel was 35.15%, where as for 2.6.24 it arch 14,836 4,810 is 34.6%. driver 116,623 30,031 include 7,751 4,707
Figure 5 and Table 4 show the coverage of the various Table 5: Lines Instrumented Vs Executed on 2.6.24 ker- kernels mentioned for the PowerTM architecture. The fig- nel with x86 architecture ure shows the following trends:
• The number of lines of instrumented code show a trend similar to the ones for the x86 architecture.
• The code coverage has increased with increasing Directories Lines Instrumented Lines Executed kernel versions. mm 14,585 9,133 net 65,441 20,749 • The code coverage percentage however is less than kernel 23,407 10,916 that of the x86 architecture, close to 30%. lib 5,062 2,799 arch 24,954 5,700 • There is no decline in the code coverage percentage driver 88,195 8,930 as seen on the x86 architecture, indicating that x86 include 7,638 4,434 is growing rapidly.12 Table 6: Lines Instrumented Vs Executed on 2.6.24 ker- TM Figures 6,7 and Tables 5,6 show the component-wise nel with Power architecture break up for the code coverage obtained for the 2.6.24
12which might be true, due to the x86 and x86-64 merge. 2008 Linux Symposium, Volume One • 27
90000 Instrumented Lines Directories Lines Instrumented Lines Executed Executed Lines 80000 Disabled Forced
70000 mm 13,147 8,597 8,688 net 63,802 20,855 21,306 60000 kernel 23,633 10,962 11,431 50000 lib 5,152 2,785 2,853 40000 arch 14,836 4,810 4,918 Lines of code
30000 driver 116,623 30,031 30,552
20000 include 7,751 4,707 4,777
10000 Table 7: Fault Injection code coverage on 2.6.24 kernel
0 with x86 architecture mm net kernel lib arch driver include
Figure 7: Component-wise code coverage on 2.6.24 ker- 13 nel with PowerTM architecture We used the Fault Injection framework to get more coverage of the error handling path. The kernel was con-
120000 figured with: Instrumented Lines Executed Lines Executed Lines With Fault Injection 100000 N > /debug/fail_page_alloc/task-filter
80000 20 > /debug/fail_page_alloc/probability 2000 > /debug/fail_page_alloc/interval -1 > /debug/fail_page_alloc/times 60000 0 > /debug/fail_page_alloc/space
Lines of code 1 > /debug/fail_page_alloc/verbose 40000 N > /debug/fail_page_alloc/ignore-gfp-wait
20000 N > /debug/fail_make_request/task-filter 20 > /debug/fail_make_request/probability
0 2000 > /debug/fail_make_request/interval mm net kernel lib arch driver include -1 > /debug/fail_make_request/times 0 > /debug/fail_make_request/space Figure 8: Fault injection code coverage on 2.6.24 kernel 1 > /debug/fail_make_request/verbose N > /debug/fail_make_request/ignore-gfp-wait with x86 architecture
TM kernel on the x86 and Power architectures respec- (refer to [9] for more configuration details) The results tively. The component-wise break up shows some in- of the coverage are shown in Figure 8. Getting cover- teresting trends as well: age results with fault injection enabled turned out to be very challenging, since most applications are not ready to deal with failures. The test applications we saw would • The mm, kernel, lib, and include subdirec- fail and abort their operation on error. We had to man- tories are among those that have the highest code ually make changes to get coverage data with fault in- coverage. jection and we were forced to keep the failure rate very low. Due to these factors, we did not see a significant • The subsystem with highest coverage is mm, with improvement in code coverage with the fault injection close to 65% coverage framework enabled, it was just 0.6% improvement. The test cases and infrastructure need to be enhanced to deal with the failures forced from the fault injection frame- • drivers and arch subdirectories are among work. those that have the least code coverage, The main focus of our testing is not to test architecture or 13injects errors at various kernel layers, helping to test error han- platform-specific code. dling 28 • Keeping the Linux Kernel Honest
5 Future Plans 8 Legal Statement
c International Business Machines Corporation 2008. Per- We plan to extend our testing by adopting and updat- mission to redistribute in accordance with Linux Symposium ing test cases from various test projects and test scripts submission guidelines is granted; all other rights reserved. published on the mailing list to improve the coverage of untested code. We intend to test the practically possible This work represents the view of the authors and does not permutations of kernel configurations and boot options necessarily represent the view of IBM. with selective releases. We intend to build tests with the IBM, IBM logo and ibm.com are trademarks of International a cross-compiler setup will help us in finding the build Business Machines Corporation in the United States, other errors over various platforms. Testing the kernel error countries, or both. path is very critical to avoiding surprises under certain situations. We could perform error handling path testing Linux is a registered trademark of Linus Torvalds in the on selective kernels using the fault injection framework United States, other countries, or both. available in the kernel. Other company, product, and service names may be trade- marks or service marks of others.
6 Conclusion References in this publication to IBM products or services do not imply that IBM intends to make them available in all countries in which IBM operates. Testing kernel releases earlier and often, helps in fix- ing the bugs earlier in the cycle. The earlier the prob- INTERNATIONAL BUSINESSMACHINES CORPORA- lem gets fixed, the lower the costs involved in fix- TION PROVIDES THIS PUBLICATION "AS IS" WITH- ing it. Our infrastructure tests various kernels across OUT WARRANTY OF ANY KIND, EITHER EX- different hardware, using many benchmarks and test PRESS OR IMPLIED, INCLUDING, BUT NOT LIM- suites performing build, regression, stress, functional, ITED TO, THE IMPLIED WARRANTIES OF NON- and performance testing. Benchmarks are run selec- INFRINGEMENT, MERCHANTABILITY OR FITNESS tively depending upon the kernel release. Testing re- FOR A PARTICULAR PURPOSE. Some states do not al- sults are contributed back to the community through low disclaimer of express or implied warranties in certain test.kernel.org. We discussed some of the har- transactions, therefore, this statement may not apply to you. nesses commonly used by the projects. This information could include technical inaccuracies or ty- pographical errors. Changes are periodically made to the in- Code coverage results explain the gap between the lines formation herein; these changes will be incorporated in new of code being added and tested. For better and more editions of the publication. IBM may make improvements complete testing of the kernel, we need test cases that and/or changes in the product(s) and/or the program(s) de- can help us better test existing and new features. For scribed in this publication at any time without notice. these we require developers to share their tests and test- ing methodology. The fault injection framework helps References testing the error handling part of the kernel, so improve- ments made to the framework could result in better ker- [1] Greg Kroah-Hartman, Jonathan Corbet, and nel coverage. Amanda McPherson, How Fast it is Going, Who is Doing It, What They are Doing, and Who is Sponsoring It, http://www. 7 Acknowledgments linux-foundation.org/publications/ linuxkerneldevelopment.php, April 2008.
We would like to thank Andy Whitcroft for his input to [2] Fully Automated Testing of the Linux Kernel, and review of drafts of this paper. Martin Bligh and Andy P. Whitcroft. In Proceedings of the Linux Symposium 2006. We also owe lot of thanks to Sudarshan Rao, Premalatha Nair, and our teammates for their active support and en- [3] Linux Test Project, thusiasm. http://ltp.sourceforge.net/ 2008 Linux Symposium, Volume One • 29
[4] Autotest, http://test.kernel.org/autotest
[5] Kernelsource/Documentation/HOWTO.
[6] Linux Kernel Mailing List. [email protected], http://lkml.org/
[7] Linux Test Project - Test Tools Matrix. http:// ltp.sourceforge.net/tooltable.php
[8] The Cathedral and the Bazaar, Eric Steven Raymond.
[9] kernelsource/Documentation/ fault-injection/fault-injection. txt
[10] kernelsource/Documentation/ kernel-parameters.txt 30 • Keeping the Linux Kernel Honest Proceedings of the Linux Symposium
Volume One
July 23rd–26th, 2008 Ottawa, Ontario Canada Conference Organizers
Andrew J. Hutton, Steamballoon, Inc., Linux Symposium, Thin Lines Mountaineering C. Craig Ross, Linux Symposium
Review Committee
Andrew J. Hutton, Steamballoon, Inc., Linux Symposium, Thin Lines Mountaineering Dirk Hohndel, Intel Gerrit Huizenga, IBM Dave Jones, Red Hat, Inc. Matthew Wilson, rPath C. Craig Ross, Linux Symposium
Proceedings Formatting Team
John W. Lockhart, Red Hat, Inc. Gurhan Ozen, Red Hat, Inc. Eugene Teo, Red Hat, Inc. Kyle McMartin, Red Hat, Inc. Jake Edge, LWN.net Robyn Bergeron Dave Boutcher, IBM Mats Wichmann, Intel
Authors retain copyright to all submitted papers, but have granted unlimited redistribution rights to all as a condition of submission.