Autotest Documentation Release 0.16.3-44-G0d527f
Total Page:16
File Type:pdf, Size:1020Kb
autotest Documentation Release 0.16.3-44-g0d527f Autotest Team May 31, 2018 Contents 1 Autotest Documentation 3 1.1 General Information...........................................3 1.2 Local (Former Client).......................................... 20 1.3 Remote (Former Server)......................................... 41 1.4 Frontend................................................. 70 1.5 System Administration.......................................... 125 1.6 Scheduler................................................. 146 1.7 Developer................................................. 154 2 client Package 195 2.1 autotest_local Module...................................... 195 2.2 base_sysinfo Module........................................ 195 2.3 base_utils Module.......................................... 196 2.4 bkr_proxy Module.......................................... 200 2.5 bkr_xml Module............................................ 202 2.6 client_logging_config Module................................. 203 2.7 cmdparser Module.......................................... 203 2.8 common Module............................................. 204 2.9 config Module............................................. 204 2.10 cpuset Module............................................. 204 2.11 fsdev_disks Module......................................... 206 2.12 fsdev_mgr Module.......................................... 208 2.13 fsinfo Module............................................. 208 2.14 harness Module............................................ 209 2.15 harness_autoserv Module..................................... 210 2.16 harness_beaker Module...................................... 210 2.17 harness_simple Module...................................... 212 2.18 harness_standalone Module................................... 212 2.19 job Module............................................... 212 2.20 kernel Module............................................. 216 2.21 kernel_config Module....................................... 218 2.22 kernel_versions Module...................................... 219 2.23 kernelexpand Module........................................ 219 2.24 kvm_control Module......................................... 220 2.25 local_host Module.......................................... 220 2.26 lv_utils Module........................................... 221 i 2.27 optparser Module.......................................... 222 2.28 os_dep Module............................................. 222 2.29 parallel Module........................................... 225 2.30 partition Module.......................................... 226 2.31 profiler Module........................................... 230 2.32 setup Module.............................................. 230 2.33 setup_job Module.......................................... 230 2.34 setup_modules Module....................................... 231 2.35 sysinfo Module............................................ 232 2.36 test Module.............................................. 232 2.37 test_config Module......................................... 232 2.38 utils Module.............................................. 233 2.39 xen Module............................................... 233 2.40 Subpackages............................................... 234 3 frontend Package 369 3.1 Subpackages............................................... 369 4 Indices and tables 371 Python Module Index 373 ii autotest Documentation, Release 0.16.3-44-g0d527f Autotest is a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms. It’s an open-source project under the GPL and is used and developed by a number of organizations, including Google, IBM, Red Hat, and many others. Please check Avocado, a next generation test automation framework being developed by several of the original Autotest team members: http://avocado-framework.github.io/ Contents 1 autotest Documentation, Release 0.16.3-44-g0d527f 2 Contents CHAPTER 1 Autotest Documentation 1.1 General Information 1.1.1 Contact information • Autotest mailing list • Autotest IRC channel: irc.oftc.net #autotest 1.1.2 Who uses autotest? An active community of users is fundamental to sustain an open source project. Currently, there are quite a few projects that use autotest as their main test automation platform: • Fedora’s AutoQA • Chrome OS • KVM kernel based virtual machine • Goobuntu (internal customized version of ubuntu for google) Besides these projects, other people from several different companies/affiliations use autotest for their test automation needs. 1.1.3 Autotest structure overview This document intends to be a high level overview of the autotest project structure. We try to be brief and show the high level diagrams. Simplified block diagram For the sake of clarity, some things are simplified here, but it gives you a good idea of the overall layout. 3 autotest Documentation, Release 0.16.3-44-g0d527f Web interface and command line interface The web interface and the command line interface are complementary ways to interact with autotest and create jobs. Both were designed to have the same functionality, to add to the user’s convenience. The interfaces allows you to: • Manage jobs? - create, monitor, abort, etc. • Manage client hosts • Look at results. The frontends will inject jobs into the server by creating records in a mysql database. Server The server consists of three main parts: • A mysql database that holds information on all jobs, clients (test machines), users and tests. • The dispatcher (monitor_db) - chooses the jobs from the database to run. It’s input is the database, pretty much all it does is start autoserv processes to service requests. – There is normally one dispatcher process per machine – Client side jobs are run asyncronously (as client machines become available) – Server side jobs are run syncronously (ie we wait for all clients before commencing) • Autoserv: the server manages clients via autoserv processes - there will be one autoserv process per running job?. Each autoserv process: – controls and monitors one or more clients – verifies clients are working properly, and if it fails verification, attempts to repair it – manages the execution of a job? – updates the autotest software on each client before commencing work. 4 Chapter 1. Autotest Documentation autotest Documentation, Release 0.16.3-44-g0d527f The mysql database can live on a different machine than the dispatcher. There can be multiple dispatchers to spread the workload, though each can service a few thousand clients, so this is not normally necessary. Client The client does most of the work of running a job?; this can be invoked: • manually - from client/autotest-local <control_file_name> • via the server A typical job workflow is as follows: Results repository A directory tree of all the results. Each job has a well formatted directory structure Results MySQL DB A simple mysql database containing the jobs, test results, and performance metrics for each test 1.1. General Information 5 autotest Documentation, Release 0.16.3-44-g0d527f Overall structure With all the parts of the code briefly commented, it’s easier to understand the overall structure diagram: 1.1.4 Autotest White Paper Abstract This paper describes the motivation for, and design goals of, the autotest and test.kernel.org projects. Autotest is a framework for fully automated testing, that is designed primarily to test the Linux kernel, though is useful for many other functions too. Test.kernel.org is a framework for communicating, sharing, and analysing test results. In a traditional corporate systems software development environment, there is normally a large test team responsible for assuring the quality of the final product. Open source projects do not have that luxury, and we need to find another way to run testing. We feel that the only realistic way to achieve the goal is to fully automate the test process, and drastically reduce the need for human staffing. It turns out that this also solves several other critical problems with testing. • Consistency - it’s much easier to guarantee the tests are run the same way as last time. • Knowledge capture - the knowledge of how to run testing is not held in one person, but within a system. • Sharing - you can easily share tests with vendors, partners, and across a wide community. • Reproducibility - they say 90% of fixing a bug is to get an easily reproducible test case. Testing is not about running tests . testing is about finding and fixing bugs. We have to: • Run the tests • Find a bug • Classify the bug 6 Chapter 1. Autotest Documentation autotest Documentation, Release 0.16.3-44-g0d527f • Hand the bugs off to a developer • Developer investigates bug (cyclical) • Developer tests some proposed fix (cyclical) • Fix checked in • New release issued to test team. So many test systems I see are oriented only around the first two (or even one!) steps. This is massively inefficient - so often I see developers writing a simple testcase to reproduce what happens in a more complex test, or proprietary application, and then these are thrown away. If we started with open tests that we could freely and easily share, much effort and time would be saved. This is not just about the cost of the people’s time, salaries and machine resources. It’s about the opportunity cost of stalling