Squish Coco 5.1.0 Copyright © 2003–2011 Sébastien Fricker, 2011–2021 froglogic GmbH CONTENTS Contents I Overview2 1 Squish Coco – A code coverage tool for Tcl, QML, C# and C/C++3 1.1 CoverageScanner – Instrumentation as part of the build process...................4 1.2 CoverageBrowser – Viewing, analyzing and managing code coverage results.............5 II Setup and Tutorials7 2 Installation and Setup 8 2.1 Choosing a license...........................................8 2.2 Installation...............................................9 2.2.1 Installation of a license server.................................9 2.2.2 Running the license server................................... 10 2.3 Updates................................................. 10 3 Instrumentation of a simple project 11 3.1 UNIX and macOS setup........................................ 11 3.1.1 Setup.............................................. 11 3.1.2 Structure of the parser directories.............................. 12 3.1.3 Compiling and testing..................................... 12 3.1.4 Instrumentation......................................... 13 3.1.5 How the project is instrumented................................ 13 3.1.6 Additional changes....................................... 14 3.2 Microsoft Windows setup........................................ 15 3.2.1 Setup.............................................. 15 3.2.2 Structure of the parser directories.............................. 15 3.2.3 Compiling and testing..................................... 15 3.2.4 Instrumentation......................................... 16 Squish Coco 5.1.0 - i - froglogic GmbH CONTENTS 3.2.5 How the project is instrumented................................ 17 3.2.6 Additional changes....................................... 18 3.3 Beyond the minimal instrumentation.................................. 18 3.3.1 Excluding code from instrumentation............................. 18 3.3.2 Making the test names visible................................. 19 3.3.3 Patch file analysis........................................ 19 3.3.4 The patch analysis report.................................... 20 3.3.5 Reverse patches......................................... 21 3.3.6 Bug location.......................................... 21 3.3.7 The function profiler...................................... 23 3.4 Using the function profiler for performance improvents........................ 26 3.4.1 Application Building...................................... 26 3.4.2 Analyzing the Performance Issues............................... 27 4 Code coverage for a Qt application 31 4.1 Compiling the example application................................... 31 4.1.1 The first code coverage results................................. 33 4.1.2 Interactive testing........................................ 34 4.1.3 Writing unit tests........................................ 36 4.2 Working with code coverage data.................................... 41 4.2.1 Post mortem analysis...................................... 42 4.2.2 Evaluating the impact of a hot fix............................... 43 4.2.3 Black-box testing/distributed testing.............................. 44 4.2.4 Verifying if a bug fix is correctly tested............................ 45 4.3 Conclusion............................................... 46 III About Code Coverage 48 5 Squish Coco’s code coverage analysis 49 5.1 Coverage metrics supported by Squish Coco .............................. 49 5.1.1 Coverage metrics and safety standards............................. 50 5.2 Description of the coverage metrics.................................. 51 5.2.1 Statement Block Coverage................................... 51 5.2.2 Decision Coverage....................................... 52 5.2.3 Condition Coverage...................................... 53 Squish Coco 5.1.0 - ii - froglogic GmbH CONTENTS 5.2.4 Multiple Condition Coverage.................................. 54 5.2.5 MC/DC............................................. 55 5.3 Display of the results.......................................... 57 5.4 Performance............................................... 58 5.5 Statistics................................................. 58 5.6 Problems with line coverage...................................... 61 6 Testing Methodologies 63 6.1 Hit vs Count............................................... 63 6.2 Strategies................................................ 63 6.2.1 Manual tests.......................................... 64 6.2.2 Black box tests......................................... 64 6.2.3 Unit Tests............................................ 64 6.2.4 Automatic Tests........................................ 65 7 Code metrics 66 7.1 eLOC.................................................. 66 7.2 McCabe................................................. 66 7.2.1 Definition............................................ 66 7.2.2 Properties............................................ 67 7.2.3 eLOC vs McCabe – Which metric to choose?......................... 67 7.2.4 Variants of McCabe metrics.................................. 67 7.2.4.1 McCabe with cases grouped............................. 68 7.2.4.2 McCabe with condensed switches.......................... 68 IV CoverageScanner Reference 69 8 Introduction 70 8.1 Invoking CoverageScanner ....................................... 70 9 Command Line Arguments 72 9.1 Environment variables......................................... 72 9.2 List of options.............................................. 73 9.2.1 Coverage methods....................................... 73 9.2.2 Fine-tuning of the instrumentation............................... 74 9.2.3 File and path inclusion..................................... 76 Squish Coco 5.1.0 - iii - froglogic GmbH CONTENTS 9.2.4 Function inclusion....................................... 78 9.2.5 Debugging........................................... 79 9.2.6 Execution report........................................ 79 9.2.7 Command line modification.................................. 81 9.2.8 Generated code......................................... 81 9.2.9 Miscellaneous......................................... 83 9.3 Instrumenting using preprocessor symbols............................... 84 9.4 Regular Expressions.......................................... 84 10 Library calls 86 10.1 C and C++ Library........................................... 86 10.1.1 __coveragescanner_install() ............................... 86 10.1.2 __coveragescanner_testname() .............................. 87 10.1.3 __coveragescanner_teststate() ............................. 87 10.1.4 __coveragescanner_add_html_comment() ......................... 88 10.1.5 __coveragescanner_clear_html_comment() ....................... 88 10.1.6 __coveragescanner_save() ................................ 88 10.1.7 __coveragescanner_clear() ................................ 88 10.1.8 __coveragescanner_filename() .............................. 89 10.1.9 __coveragescanner_get_filename() ........................... 89 10.1.10 __coveragescanner_register_library() ......................... 89 10.1.11 __coveragescanner_unregister_library() ....................... 90 10.1.12 __coveragescanner_register_squish() ......................... 90 10.1.13 __coveragescanner_set_custom_io() ........................... 91 10.2 C# Library................................................ 92 10.2.1 CoverageScanner.__coveragescanner_init() ...................... 92 10.2.2 CoverageScanner.__coveragescanner_testname() ................... 92 10.2.3 CoverageScanner.__coveragescanner_teststate() .................. 92 10.2.4 CoverageScanner.__coveragescanner_memory_pool_stat() .............. 93 10.2.5 CoverageScanner.__coveragescanner_add_html_comment() .............. 93 10.2.6 CoverageScanner.__coveragescanner_clear_html_comment() ............ 93 10.2.7 CoverageScanner.__coveragescanner_save() ...................... 93 10.2.8 CoverageScanner.__coveragescanner_clear() ..................... 94 10.2.9 CoverageScanner.__coveragescanner_filename() ................... 94 Squish Coco 5.1.0 - iv - froglogic GmbH CONTENTS 10.2.10 CoverageScanner.__coveragescanner_set_custom_io() ................ 94 11 Controlling the instrumentation from the source code 96 11.1 Annotations............................................... 96 11.1.1 Comments........................................... 96 11.1.2 Manual Validation....................................... 97 11.2 C and C++ Pragmas........................................... 98 11.3 C# regions................................................ 99 V QML Coverage 101 12 QML coverage 102 12.1 Setup.................................................. 102 12.1.1 Compilation of the plugin................................... 102 12.2 Use of CocoQML............................................ 103 12.2.1 Instrumentation......................................... 103 12.2.2 Measuring the coverage.................................... 103 12.3 Reference................................................ 104 12.3.1 cocoqmlscanner ........................................ 104 12.3.2 The tracker plugin....................................... 104 13 QML coverage with a library 106 13.1 Overview................................................ 106 13.2 Installation............................................... 106 13.2.1 General remark......................................... 107 13.2.2 Finding the right version of the archive............................ 107 13.2.3 Linux™ installation......................................