6 UNIT TESTING Many find unit testing to be arduous and time-consuming, and some people and projects have no testing policy. This chapter assumes that you see the wisdom of unit testing! Writing code that is not tested is fun- damentally useless, as there’s no way to conclusively prove that it works. If you need convincing, I suggest you start by reading about the benefits of test-driven development. In this chapter you’ll learn about the Python tools you can use to con- struct a comprehensive suite of tests that will make testing simpler and more automated. We’ll talk about how you can use tools to make your software rock solid and regression-free. We’ll cover creating reusable test objects, running tests in parallel, revealing untested code, and using virtual environments to make sure your tests are clean, as well as some other good- practice methods and ideas. The Basics of Testing Writing and running unit tests is uncomplicated in Python. The process is not intrusive or disruptive, and unit testing will greatly help you and other developers in maintaining your software. Here I’ll discuss some of the abso- lute basics of testing that will make things easier for you. Some Simple Tests First, you should store tests inside a tests submodule of the application or library they apply to. Doing so will allow you to ship the tests as part of your module so that they can be run or reused by anyone—even after your soft- ware is installed—without necessarily using the source package. Making the tests a submodule of your main module also prevents them from being installed by mistake in a top-level tests module. Using a hierarchy in your test tree that mimics the hierarchy of your module tree will make the tests more manageable. This means that the tests covering the code of mylib/foobar.py should be stored inside mylib/ tests/test_foobar.py. Consistent nomenclature makes things simpler when you’re looking for the tests related to a particular file. Listing 6-1 shows the simplest unit test you can write. def test_true(): assert True Listing 6-1: A really simple test in test_true.py This will simply assert that the behavior of the program is what you expect. To run this test, you need to load the test_true.py file and run the test_true() function defined within. However, writing and running an individual test for each of your test files and functions would be a pain. For small projects with simple usage, the pytest package comes to the rescue—once installed via pip, pytest pro- vides the pytest command, which loads every file whose name starts with test_ and then executes all functions within that start with test_. With just the test_true.py file in our source tree, runningpytest gives us the following output: $ pytest -v test_true.py ========================== test session starts =========================== platform darwin -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/local/opt/python/bin/python3.6 cachedir: .cache rootdir: examples, inifile: collected 1 item test_true.py::test_true PASSED [100%] ======================== 1 passed in 0.01 seconds ======================== 76 Chapter 6 The -v option tells pytest to be verbose and print the name of each test run on a separate line. If a test fails, the output changes to indicate the failure, accompanied by the whole traceback. Let’s add a failing test this time, as shown in Listing 6-2. def test_false(): assert False Listing 6-2: A failing test in test_true.py If we run the test file again, here’s what happens: $ pytest -v test_true.py ========================== test session starts =========================== platform darwin -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/ local/opt/python/bin/python3.6 cachedir: .cache rootdir: examples, inifile: collected 2 items test_true.py::test_true PASSED [ 50%] test_true.py::test_false FAILED [100%] ================================ FAILURES ================================ _______________________________ test_false _______________________________ def test_false(): > assert False E assert False test_true.py:5: AssertionError =================== 1 failed, 1 passed in 0.07 seconds =================== A test fails as soon as an AssertionError exception is raised; our assert test will raise an AssertionError when its argument is evaluated to something false (False, None, 0, etc.). If any other exception is raised, the test also errors out. Simple, isn’t it? While simplistic, a lot of small projects use this approach and it works very well. Those projects require no tools or libraries other than pytest and thus can rely on simple assert tests. As you start to write more sophisticated tests, pytest will help you under- stand what’s wrong in your failing tests. Imagine the following test: def test_key(): a = ['a', 'b'] b = ['b'] assert a == b When pytest is run, it gives the following output: $ pytest test_true.py ========================== test session starts =========================== Unit Testing 77 platform darwin -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 rootdir: /Users/jd/Source/python-book/examples, inifile: plugins: celery-4.1.0 collected 1 item test_true.py F [100%] ================================ FAILURES ================================ ________________________________ test_key ________________________________ def test_key(): a = ['a', 'b'] b = ['b'] > assert a == b E AssertionError: assert ['a', 'b'] == ['b'] E At index 0 diff: 'a' != 'b' E Left contains more items, first extra item: 'b' E Use -v to get the full diff test_true.py:10: AssertionError ======================== 1 failed in 0.07 seconds ======================== This tells us that a and b are different and that this test does not pass. It also tells us exactly how they are different, making it easy to fix the test or code. Skipping Tests If a test cannot be run, you will probably want to skip that test—for example, you may wish to run a test conditionally based on the presence or absence of a particular library. To that end, you can use the pytest.skip() func- tion, which will mark the test as skipped and move on to the next one. The pytest.mark.skip decorator skips the decorated test function unconditionally, so you’ll use it when a test always needs to be skipped. Listing 6-3 shows how to skip a test using these methods. import pytest try: import mylib except ImportError: mylib = None @pytest.mark.skip("Do not run this") def test_fail(): assert False @pytest.mark.skipif(mylib is None, reason="mylib is not available") def test_mylib(): assert mylib.foobar() == 42 78 Chapter 6 def test_skip_at_runtime(): if True: pytest.skip("Finally I don't want to run it") Listing 6-3: Skipping tests When executed, this test file will output the following: $ pytest -v examples/test_skip.py ========================== test session starts =========================== platform darwin -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/ local/opt/python/bin/python3.6 cachedir: .cache rootdir: examples, inifile: collected 3 items examples/test_skip.py::test_fail SKIPPED [ 33%] examples/test_skip.py::test_mylib SKIPPED [ 66%] examples/test_skip.py::test_skip_at_runtime SKIPPED [100%] ================= 3 skipped in 0.01 seconds ================= The output of the test run in Listing 6-3 indicates that, in this case, all the tests have been skipped. This information allows you to ensure you didn’t accidentally skip a test you expected to run. Running Particular Tests When using pytest, you often want to run only a particular subset of your tests. You can select which tests you want to run by passing their directory or files as an argument to the pytest command line. For example, calling pytest test_one.py will only run the test_one.py test. Pytest also accepts a directory as argument, and in that case, it will recursively scan the directory and run any file that matches thetest_*.py pattern. You can also add a filter with the -k argument on the command line in order to execute only the test matching a name, as shown in Listing 6-4. $ pytest -v examples/test_skip.py -k test_fail ========================== test session starts =========================== platform darwin -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/ local/opt/python/bin/python3.6 cachedir: .cache rootdir: examples, inifile: collected 3 items examples/test_skip.py::test_fail SKIPPED [100%] Unit Testing 79 === 2 tests deselected === === 1 skipped, 2 deselected in 0.04 seconds === Listing 6-4: Filtering tests run by name Names are not always the best way to filter which tests will run. Commonly, a developer would group tests by functionalities or types instead. Pytest provides a dynamic marking system that allows you to mark tests with a key- word that can be used as a filter. To mark tests in this way, use the -m option. If we set up a couple of tests like this: import pytest @pytest.mark.dicttest def test_something(): a = ['a', 'b'] assert a == a def test_something_else(): assert False we can use the -m argument with pytest to run only one of those tests: $ pytest -v test_mark.py -m dicttest === test session starts === platform darwin -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/ local/opt/python/bin/python3.6 cachedir: .cache rootdir: examples, inifile: collected 2 items test_mark.py::test_something PASSED [100%] === 1 tests deselected === === 1 passed, 1 deselected in 0.01 seconds === The -m marker accepts more complex queries, so we can also run all tests that are not marked: $ pytest test_mark.py -m 'not dicttest' === test session starts === platform darwin -- Python 3.6.4, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 rootdir: examples, inifile: collected 2 items test_mark.py F [100%] === FAILURES === test_something_else 80 Chapter 6 def test_something_else(): > assert False E assert False test_mark.py:10: AssertionError === 1 tests deselected === === 1 failed, 1 deselected in 0.07 seconds === Here pytest executed every test that was not marked as dicttest—in this case, the test_something_else test, which failed.