Code Coverage: Covered in Depth Sebastian Bergmann | May 21st 2015 Sebastian Bergmann Helps teams to successfully craft the right software. sharing experience "The beauty of testing is found not in the effort but in the efficiency. Knowing what should be tested is beautiful, and knowing what is being tested is beautiful." — Murali Nandigama Code Coverage
"In computer science, code coverage is a measure used to describe the degree to which the source code of a program is tested by a particular test suite. A program with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a program with low code coverage."
— Wikipedia
2003 Xdebug 1.2.0
2007 Xdebug 2.0.1 2009 PHP_CodeCoverage
PHP_CodeCoverage 2.0
Drivers Software Metrics Reporters » Xdebug » Line Coverage » HTML » HHVM » Function / Method Coverage » Clover XML » Class / Trait Coverage » Crap4J XML » Change Risk Anti-Patterns (CRAP) » PHPUnit XML » Patch Coverage » Plain Text » PHP Line Coverage
The Line Coverage software metric measures whether each executable line was executed.
$ phpunit -c build --coverage-text PHPUnit 4.6.6 by Sebastian Bergmann and contributors.
Configuration read from /usr/local/src/money/build/phpunit.xml
......
Time: 988 ms, Memory: 19.00Mb
OK (33 tests, 52 assertions)
Generating code coverage report in Clover XML format ... done
Generating Crap4J report XML file ... done
Generating code coverage report in HTML format ... done
Code Coverage Report: 2014-06-01 17:18:58
Summary: Classes: 1.60% (3/187) Methods: 13.81% (29/210) Lines: 0.77% (87/11233)
\SebastianBergmann\Money::Currency Methods: 100.00% ( 7/ 7) Lines: 100.00% ( 11/ 11) \SebastianBergmann\Money::IntlFormatter Methods: 100.00% ( 2/ 2) Lines: 100.00% ( 8/ 8) \SebastianBergmann\Money::Money Methods: 100.00% (20/20) Lines: 100.00% ( 68/ 68)
// ... return $coverage; $ wget https://phar.phpunit.de/phpcov.phar $ chmod +x phpcov.phar $ mv phpcov.phar /usr/local/bin/phpcov
$ parallel --gnu ::: \ 'phpunit --coverage-php /tmp/coverage/FooTest.cov tests/FooTest' \ 'phpunit --coverage-php /tmp/coverage/BarTest.cov tests/BarTest'
$ phpcov merge --clover /tmp/clover.xml /tmp/coverage phpcov 2.0.1 by Sebastian Bergmann.
Generating code coverage report in Clover XML format ... done Function / Method Coverage
The Function / Method Coverage software metric measures whether each function or method has been invoked.
PHP_CodeCoverage only considers a function or method as covered when all of its executable lines are covered. Class / Trait Coverage
The Class / Trait Coverage software metric measures whether each method of a class or trait is covered.
PHP_CodeCoverage only considers a class or trait as covered when all of its methods are covered.
Change Risk Anti-Patterns (CRAP) Index
The Change Risk Anti-Patterns (CRAP) Index "predict[s] the amount of effort, pain, and time required to maintain an existing body of code". It is calculated based on its cyclomatic complexity and code coverage.
Code that is not too complex and has an adequate test coverage will have a low CRAP index. The CRAP index can be lowered by writing tests and by refactoring the code to lower its complexity.
Patch Coverage
The Patch Coverage software metric measures whether each executable line that is changed by a patch is covered by a test. $ wget https://phar.phpunit.de/phpcov.phar $ chmod +x phpcov.phar $ mv phpcov.phar /usr/local/bin/phpcov
$ git diff HEAD^1 > /tmp/patch.txt
$ phpunit --coverage-php /tmp/coverage.cov
$ phpcov patch-coverage --patch /tmp/patch.txt \ --path-prefix /path/to/project \ /tmp/coverage.cov phpcov 2.0.1 by Sebastian Bergmann.
1 / 2 changed executable lines covered (50.00%)
Changed executable lines that are not covered:
Example.php:11 Including and Excluding Files
» By default, all source files that ontainc at least one line of code that has been executed (and only these files) are included in the odec coverage report. » By default, a blacklist is used to exclude files from the odec coverage report. This blacklist is pre-filled with the source files fo PHPUnit and its dependencies. » It is best practice to use a whitelist instead of the blacklist. » Optionally, all whitelisted files can be added ot the code coverage report. This allows the inclusion of files that are not estt ed yet at all. $ phpunit -c build PHPUnit 4.7-g86f0995 by Sebastian Bergmann and contributors.
Runtime: PHP 5.6.9 with Xdebug 2.3.2 Configuration: /usr/local/src/money/build/phpunit.xml Warning: No whitelist configured for code coverage
......
Time: 1.15 seconds, Memory: 14.75Mb
OK (44 tests, 70 assertions)
Generating code coverage report in Clover XML format ... done
Generating Crap4J report XML file ... done
Generating code coverage report in HTML format ... done
Generating code coverage report in PHPUnit XML format ... done Annotations
// ... } $ phpunit --strict-coverage
PHPUnit 4.6.6 by Sebastian Bergmann and contributors.
Configuration read from /usr/local/src/money/build/phpunit.xml
...... R......
Time: 388 ms, Memory: 19.25Mb
There was 1 risky test:
1) SebastianBergmann\Money\MoneyTest::testExceptionIsRaisedForInvalidConstructorArguments This test executed code that is not listed as code to be covered or used: - /usr/local/src/money/src/Currency.php:1169 - /usr/local/src/money/src/Currency.php:1175 - /usr/local/src/money/src/Currency.php:1176
OK, but incomplete, skipped, or risky tests! Tests: 33, Assertions: 52, Risky: 1.
Generating code coverage report in HTML format ... done
// ... } @covers @uses Test Unit Dependency
$ wget https://phar.phpunit.de/test-mapper.phar $ chmod +x test-mapper.phar $ mv test-mapper.phar /usr/local/bin/test-mapper
$ test-mapper --graphviz /tmp/testmap.dot /path/to/tests test-mapper 1.0.0 by Sebastian Bergmann.
Wrote test map in GraphViz/DOT format to /tmp/testmap.dot
$ dot -Tpdf -o /tmp/testmap.pdf /tmp/testmap.dot PHP_CodeCoverage 2.1
» Part of the PHPUnit 4.7 release cycle » Introduces "Three Shades of Green" feature based on PHPUnit's @small, @medium, and @large test size annotations
PHP_CodeCoverage 3.0
Drivers Software Metrics Reporters » Xdebug » Line Coverage » HTML » HHVM » Branch Coverage » Clover XML » Path Coverage » Crap4J XML » Function / Method Coverage » PHPUnit XML » Class / Trait Coverage » Plain Text » PHP » Change Risk Anti-Patterns (CRAP) » Patch Coverage Branch Coverage
Measures whether the boolean expression of each control structure evaluated to both true and false while running the test suite. Path Coverage
Measures whether each of the possible execution paths in a function or method has been followed.
An execution path is a unique sequence of branches from the entry of the function or method to its exit.
true false
Condition 2 Condition 2
true false true false
Condition 3 Condition 3 Condition 3 Condition 3
true false true false true false true false
...... Condition 1
true false
Condition 2 Condition 2
true false true false
Condition 3 Condition 3 Condition 3 Condition 3
true false true false true false true false
...... Condition 1
true false
Condition 2 Condition 2
true false true false
Condition 3 Condition 3 Condition 3 Condition 3
true false true false true false true false
...... Branch and Path Coverage: Status
» Xdebug 2.3 has support for branch and path coverage » Support for branch and path coverage in PHP_CodeCoverage is under development talks.thePHP.cc [email protected] @s_bergmann sharing experience