Squish Coco 5.1.0
Copyright © 2003–2011 Sébastien Fricker, 2011–2021 froglogic GmbH CONTENTS
Contents
I Overview2
1 Squish Coco – A code coverage tool for Tcl, QML, C# and C/C++3 1.1 CoverageScanner – Instrumentation as part of the build process...... 4 1.2 CoverageBrowser – Viewing, analyzing and managing code coverage results...... 5
II Setup and Tutorials7
2 Installation and Setup 8 2.1 Choosing a license...... 8 2.2 Installation...... 9 2.2.1 Installation of a license server...... 9 2.2.2 Running the license server...... 10 2.3 Updates...... 10
3 Instrumentation of a simple project 11 3.1 UNIX and macOS setup...... 11 3.1.1 Setup...... 11
3.1.2 Structure of the parser directories...... 12 3.1.3 Compiling and testing...... 12 3.1.4 Instrumentation...... 13 3.1.5 How the project is instrumented...... 13 3.1.6 Additional changes...... 14 3.2 Microsoft Windows setup...... 15 3.2.1 Setup...... 15
3.2.2 Structure of the parser directories...... 15 3.2.3 Compiling and testing...... 15 3.2.4 Instrumentation...... 16
Squish Coco 5.1.0 - i - froglogic GmbH CONTENTS
3.2.5 How the project is instrumented...... 17 3.2.6 Additional changes...... 18 3.3 Beyond the minimal instrumentation...... 18 3.3.1 Excluding code from instrumentation...... 18 3.3.2 Making the test names visible...... 19 3.3.3 Patch file analysis...... 19 3.3.4 The patch analysis report...... 20 3.3.5 Reverse patches...... 21 3.3.6 Bug location...... 21 3.3.7 The function profiler...... 23 3.4 Using the function profiler for performance improvents...... 26 3.4.1 Application Building...... 26 3.4.2 Analyzing the Performance Issues...... 27
4 Code coverage for a Qt application 31 4.1 Compiling the example application...... 31 4.1.1 The first code coverage results...... 33 4.1.2 Interactive testing...... 34 4.1.3 Writing unit tests...... 36 4.2 Working with code coverage data...... 41 4.2.1 Post mortem analysis...... 42 4.2.2 Evaluating the impact of a hot fix...... 43 4.2.3 Black-box testing/distributed testing...... 44 4.2.4 Verifying if a bug fix is correctly tested...... 45 4.3 Conclusion...... 46
III About Code Coverage 48
5 Squish Coco’s code coverage analysis 49 5.1 Coverage metrics supported by Squish Coco ...... 49 5.1.1 Coverage metrics and safety standards...... 50 5.2 Description of the coverage metrics...... 51 5.2.1 Statement Block Coverage...... 51 5.2.2 Decision Coverage...... 52 5.2.3 Condition Coverage...... 53
Squish Coco 5.1.0 - ii - froglogic GmbH CONTENTS
5.2.4 Multiple Condition Coverage...... 54 5.2.5 MC/DC...... 55 5.3 Display of the results...... 57 5.4 Performance...... 58 5.5 Statistics...... 58 5.6 Problems with line coverage...... 61
6 Testing Methodologies 63 6.1 Hit vs Count...... 63 6.2 Strategies...... 63 6.2.1 Manual tests...... 64 6.2.2 Black box tests...... 64 6.2.3 Unit Tests...... 64 6.2.4 Automatic Tests...... 65
7 Code metrics 66 7.1 eLOC...... 66 7.2 McCabe...... 66 7.2.1 Definition...... 66 7.2.2 Properties...... 67 7.2.3 eLOC vs McCabe – Which metric to choose?...... 67 7.2.4 Variants of McCabe metrics...... 67 7.2.4.1 McCabe with cases grouped...... 68 7.2.4.2 McCabe with condensed switches...... 68
IV CoverageScanner Reference 69
8 Introduction 70 8.1 Invoking CoverageScanner ...... 70
9 Command Line Arguments 72 9.1 Environment variables...... 72 9.2 List of options...... 73 9.2.1 Coverage methods...... 73 9.2.2 Fine-tuning of the instrumentation...... 74 9.2.3 File and path inclusion...... 76
Squish Coco 5.1.0 - iii - froglogic GmbH CONTENTS
9.2.4 Function inclusion...... 78 9.2.5 Debugging...... 79 9.2.6 Execution report...... 79 9.2.7 Command line modification...... 81 9.2.8 Generated code...... 81 9.2.9 Miscellaneous...... 83 9.3 Instrumenting using preprocessor symbols...... 84 9.4 Regular Expressions...... 84
10 Library calls 86 10.1 C and C++ Library...... 86
10.1.1 __coveragescanner_install() ...... 86
10.1.2 __coveragescanner_testname() ...... 87
10.1.3 __coveragescanner_teststate() ...... 87
10.1.4 __coveragescanner_add_html_comment() ...... 88
10.1.5 __coveragescanner_clear_html_comment() ...... 88
10.1.6 __coveragescanner_save() ...... 88
10.1.7 __coveragescanner_clear() ...... 88
10.1.8 __coveragescanner_filename() ...... 89
10.1.9 __coveragescanner_get_filename() ...... 89
10.1.10 __coveragescanner_register_library() ...... 89
10.1.11 __coveragescanner_unregister_library() ...... 90
10.1.12 __coveragescanner_register_squish() ...... 90
10.1.13 __coveragescanner_set_custom_io() ...... 91 10.2 C# Library...... 92
10.2.1 CoverageScanner.__coveragescanner_init() ...... 92
10.2.2 CoverageScanner.__coveragescanner_testname() ...... 92
10.2.3 CoverageScanner.__coveragescanner_teststate() ...... 92
10.2.4 CoverageScanner.__coveragescanner_memory_pool_stat() ...... 93
10.2.5 CoverageScanner.__coveragescanner_add_html_comment() ...... 93
10.2.6 CoverageScanner.__coveragescanner_clear_html_comment() ...... 93
10.2.7 CoverageScanner.__coveragescanner_save() ...... 93
10.2.8 CoverageScanner.__coveragescanner_clear() ...... 94
10.2.9 CoverageScanner.__coveragescanner_filename() ...... 94
Squish Coco 5.1.0 - iv - froglogic GmbH CONTENTS
10.2.10 CoverageScanner.__coveragescanner_set_custom_io() ...... 94
11 Controlling the instrumentation from the source code 96 11.1 Annotations...... 96 11.1.1 Comments...... 96 11.1.2 Manual Validation...... 97 11.2 C and C++ Pragmas...... 98 11.3 C# regions...... 99
V QML Coverage 101
12 QML coverage 102 12.1 Setup...... 102 12.1.1 Compilation of the plugin...... 102 12.2 Use of CocoQML...... 103 12.2.1 Instrumentation...... 103 12.2.2 Measuring the coverage...... 103 12.3 Reference...... 104
12.3.1 cocoqmlscanner ...... 104 12.3.2 The tracker plugin...... 104
13 QML coverage with a library 106 13.1 Overview...... 106 13.2 Installation...... 106 13.2.1 General remark...... 107 13.2.2 Finding the right version of the archive...... 107 13.2.3 Linux™ installation...... 107 13.2.4 macOS installation...... 108 13.2.5 Microsoft® Windows installation...... 108 13.3 CocoQML Tutorial...... 108
13.3.1 Compile a Qt application...... 109 13.3.2 Enable code coverage...... 109 13.3.3 Excluding source files from coverage...... 110 13.4 Resolving instrumentation problems...... 110 13.5 Environment Variables...... 111
Squish Coco 5.1.0 - v - froglogic GmbH CONTENTS
13.6 The CocoQML scanner...... 112 13.6.1 Syntax...... 112
VI Tcl Coverage 114
14 Command Line Arguments 115
15 Getting started 117 15.1 Using more than one Tcl version on one system...... 118
16 Coco TCL Tutorial 119 16.1 Analysis of Tcl script coverage...... 119 16.1.1 Demo application build...... 119 16.1.2 Ad-hoc Tcl coverage run...... 119 16.1.3 Result analysis...... 120 16.1.4 Filtered Tcl coverage run...... 120 16.1.5 Report Generation...... 120 16.2 Analysis of C/C++ code coverage...... 121 16.2.1 Instrumentation...... 121 16.2.2 Execution...... 121 16.2.3 Result analysis...... 121 16.2.4 Report Generation...... 122 16.2.5 Analysis of mixed Tcl and C/C++ code coverage...... 122 16.2.6 Combined coverage run...... 122 16.2.7 Merging measurement data...... 122 16.2.8 Report Generation...... 123
VII CoverageBrowser Reference 124
17 Introduction 125 17.1 Command Line Arguments...... 125
18 Black box and white box testing 127
19 The windows of CoverageBrowser 129 19.1 The Executions Window...... 129
Squish Coco 5.1.0 - vi - froglogic GmbH CONTENTS
19.1.1 Principles...... 129 19.1.2 Loading an Execution Report...... 131 19.1.3 The Execution Comparison Analysis Mode...... 133 19.2 The Source Browser Window...... 134 19.3 The Function Browser Window...... 135 19.4 The Source Viewer Window...... 136 19.4.1 Source Display...... 136 19.4.2 Color Convention...... 137 19.4.3 Comments...... 138 19.4.3.1 Editing Comments...... 138 19.4.3.2 Removing Comments...... 139 19.5 The Explanation Window...... 139 19.6 The Statistics Window...... 140
20 Working with CoverageBrowser 141 20.1 Filtering...... 141 20.1.1 Wildcard Expressions...... 141 20.1.2 Regular Expression...... 141 20.1.2.1 Pattern matching...... 142 20.1.2.2 String substitution...... 143 20.2 Code Coverage Level...... 143 20.3 Code Coverage Algorithm...... 143 20.4 Optimized execution order...... 144 20.5 Bug Location...... 144 20.5.1 Theory...... 144 20.5.2 Usage...... 148 20.6 Patch Analysis...... 148 20.7 Comparing Releases...... 149 20.7.1 Reference Database...... 150 20.7.2 Coverage analysis of modified/unmodified source code...... 150 20.8 Changing the Instrumentation Database...... 150 20.8.1 Merging Instrumentations...... 150 20.8.2 Importing Unit Tests...... 150 20.8.3 Importing Reviewer Comments...... 151
Squish Coco 5.1.0 - vii - froglogic GmbH CONTENTS
20.9 Function Profiler...... 151 20.9.1 Comparing Executions Together...... 151 20.9.2 Comparing Two Software Versions Together...... 152
21 Generation of Reports 153 21.1 HTML/CSV Report...... 153
21.2 EMMA-XML Report...... 153
21.3 Cobertura XML Report...... 154 21.4 JUnit Report...... 154 21.5 Text Report...... 154
22 Preferences 155 22.1 Save/Load Project...... 155 22.2 Comments...... 155 22.3 Thresholds...... 155 22.4 Cache...... 156
VIII Other Coco Tools 157
23 cmedit – Edit the paths in a csmes file 158
24 cmmerge – Merging Utility 160
25 cmcsexeimport – Command Line Import Utility 162
26 cmreport – Code coverage report generation 164 26.1 File selection options...... 165 26.2 Execution selection options...... 165 26.3 Release comparison...... 166 26.4 Options for HTML or CSV reports...... 167 26.4.1 Options only for HTML...... 168 26.4.2 Options only for CSV...... 169 26.5 Options for text reports...... 169
26.6 Options for EMMA-XML reports...... 170 26.7 Options for JUnit reports...... 170
26.8 Options for Cobertura reports...... 170
Squish Coco 5.1.0 - viii - froglogic GmbH CONTENTS
27 cocolic – License activation 172
28 cocolicwizard – GUI for license activation 174
29 cocolicserver – License Server 175
30 The Visual Studio® Coco Wizard 177 30.1 How it works...... 177 30.2 The dialog steps of the Coco Wizard...... 177
IX Howtos 179
31 Creating an instrumented project 180 31.1 C++ on Microsoft Visual Studio using the Microsoft Visual Studio Add-in...... 180 31.2 C# on Microsoft Visual Studio...... 180 31.3 Command Line Tools...... 181
32 Generating instrumentations without modifying projects 183 32.1 GNU Make...... 183 32.2 Microsoft NMake...... 183 32.3 Microsoft Visual Studio...... 184 32.4 Microsoft MSBuild...... 184 32.5 Mono C# XBuild...... 184
33 Code Coverage of Libraries 185 33.1 Code Coverage of Static/Shared Libraries and DLL...... 185 33.2 Code Coverage of Plugins/Manually Loaded Shared Libraries...... 185 33.2.1 Generating Code Coverage Information directly from the Main Application...... 185 33.2.2 Generating Code Coverage Information directly from the Plugin...... 186 33.2.2.1 Code Coverage of Plugins Generated with Microsoft Visual Studio...... 187 33.2.2.2 Code Coverage of Plugins Generated with GNU gcc...... 187
34 Test suites and Squish Coco 189 34.1 Execution comment, name and status...... 189 34.2 Unit testing...... 190
35 Special compiler versions for cross-compiling 191
Squish Coco 5.1.0 - ix - froglogic GmbH CONTENTS
36 Multi-Platform Application 192 36.1 Principle...... 192 36.2 Restrictions...... 193 36.2.1 Code Generators...... 193 36.2.2 Platform depending macros...... 193
37 Control of execution report generation 194 37.1 Execution report generation with UNIX signals...... 194 37.2 Execution report generation with Microsoft Windows events...... 194 37.2.1 A program to send events to applications...... 195 37.2.2 Global events...... 195 37.2.3 Write your own program...... 196
38 Customizing I/O of CoverageScanner library 198 38.1 Custom I/O using C file access...... 198 38.2 Custom I/O using SFTP protocol...... 200
39 Using the CoverageScanner library with a memory pool 207 39.1 Estimation of the size of the memory pool...... 207 39.2 Monitoring of the size of the memory pool...... 207
40 Generation of diff files 209 40.1 Comparing directories...... 209 40.2 Version control systems...... 209
X Coco Integration Handbook 211
41 Support for IDEs and toolkits 212 41.1 GNU Makefiles...... 212 41.2 CygWin...... 212 41.3 Scratchbox...... 213 41.4 CMake...... 213 41.4.1 Adding new build type for instrumented compilation...... 213 41.4.2 Microsoft Visual Studio...... 214 41.4.3 Microsoft NMake...... 214 41.4.4 Compilation with GNU GCC...... 215
Squish Coco 5.1.0 - x - froglogic GmbH CONTENTS
41.5 Qt...... 216 41.5.1 Qt Creator...... 216
41.5.2 qmake ...... 217 41.5.3 moc...... 218 41.5.4 qbs...... 219
41.6 SCons ...... 219 41.7 ARM Keil...... 220 41.8 ARM-DS...... 222 41.9 Green Hills...... 226 41.9.1 Installation...... 226 41.9.2 Command Line Tools...... 227 41.9.3 CoverageScanner Library...... 227 41.10Visual DSP...... 228 41.11Microsoft Visual Studio...... 230 41.11.1 Microsoft Visual Studio .NET C# Compiler...... 230 41.11.2 Microsoft Visual Studio .NET C++ Compiler...... 230 41.11.3 Microsoft Visual C++ Express...... 233 41.11.4 Microsoft Visual Studio 6.0...... 234 41.12eMbedded Visual C++...... 235 41.13Eclipse...... 236 41.14Xcode...... 238 41.14.1 VxWorks™ support on Linux™ ...... 239 41.15IAR Embedded Workbench...... 241 41.16TASKING Support...... 243
42 Support for specific test frameworks 248 42.1 CppUnit...... 248 42.2 QTestLib...... 250 42.3 GoogleTest...... 252 42.4 CxxTest...... 254 42.5 boost::test...... 258 42.6 xUnit...... 262 42.7 NUnit...... 267 42.8 Catch2...... 267
Squish Coco 5.1.0 - xi - froglogic GmbH CONTENTS
42.9 Squish...... 269 42.9.1 General Approach...... 269 42.9.2 Simplified for Reuse...... 270
43 Installing Coco in Docker 274 43.1 Prerequisites...... 274 43.2 Installation of Docker...... 274 43.2.1 Installation from the Debian repository...... 274 43.2.2 Installation from the Docker repository...... 275 43.2.3 Verification of the installation...... 275 43.3 Installation of Coco in a Docker container...... 276
44 Atlassian Bamboo integration 278 44.1 Introduction...... 278 44.2 Agent configuration...... 279 44.3 Importing an execution report...... 279 44.4 Merging the instrumentation databases...... 279 44.5 Generating the coverage report...... 280 44.6 Examples...... 280 44.6.1 A coverage report from a single execution...... 280 44.6.2 Coverage report for unit tests...... 281
45 Jenkins CI integration 284 45.1 Prerequisites...... 284 45.1.1 Getting a license...... 284 45.1.2 Getting an EMMA plugin...... 285 45.2 Adaption of the project to Jenkins...... 285
XI Coco Internals 287
46 File system and registry 288 46.1 Location of the installed files...... 288 46.1.1 Microsoft Windows...... 288 46.1.2 Linux...... 289 46.1.3 macOS...... 289 46.2 Location of the license...... 289
Squish Coco 5.1.0 - xii - froglogic GmbH CONTENTS
46.2.1 Node-locked licenses...... 289 46.2.2 Address of the license server...... 290 46.3 Location of the temporary files...... 290 46.4 Location of the program settings...... 291 46.5 Location of the installation log file...... 291
47 Supported compilers 292 47.1 C# compilers...... 292 47.2 Microsoft Visual C++...... 292 47.3 Intel C++ Compiler...... 293 47.4 GNU gcc...... 293
48 Code insertion 294
49 Code Coverage Benchmarks 298 49.1 Test Algorithm...... 298 49.2 Benchmarks...... 299
50 CoverageScanner Adaptation to a Tool Suite 301 50.1 Profile Parameters for Architecture Settings...... 302 50.2 Profile Parameters for Instrumentation Settings...... 302 50.3 Profile Parameters for Preprocessor Settings...... 302 50.4 Profile Parameters for Linker Settings...... 303 50.5 Profile Parameters for Compiler Settings...... 306 50.6 Profile Parameters for Precompiled Headers Support...... 310 50.7 Profile Parameters for Execution Time Measurement...... 310 50.8 Profile Parameters for Profiling...... 311 50.9 Profile Parameters for Custom IO...... 311 50.10Miscellaneous Profile Parameters...... 312 50.11Instrumentation Parameters...... 315
A Release Notes 316 A.1 Squish Coco v5.1.0...... 317 A.2 Squish Coco v5.0.3...... 318 A.3 Squish Coco v5.0.2...... 318 A.4 Squish Coco v5.0.1...... 318
Squish Coco 5.1.0 - xiii - froglogic GmbH CONTENTS
A.5 Squish Coco v5.0.0...... 319 A.6 Squish Coco v4.3.3...... 319 A.7 Squish Coco v4.3.2...... 320 A.8 Squish Coco v4.3.1...... 320 A.9 Squish Coco v4.3.0...... 321 A.10 Squish Coco v4.2.2...... 321 A.11 Squish Coco v4.2.1...... 321 A.12 Squish Coco v4.2.0...... 322 A.13 Squish Coco v4.1.2...... 322 A.14 Squish Coco v4.1.1...... 322 A.15 Squish Coco v4.1.0...... 322 A.16 Squish Coco v4.0.4...... 323 A.17 Squish Coco v4.0.3...... 323 A.18 Squish Coco v4.0.2...... 323 A.19 Squish Coco v4.0.1...... 324 A.20 Squish Coco v4.0.0...... 324 A.21 Squish Coco v3.4.1...... 324 A.22 Squish Coco v3.4.0...... 325 A.23 Squish Coco v3.3.3...... 325 A.24 Squish Coco v3.3.2...... 326 A.25 Squish Coco v3.3.1...... 326 A.26 Squish Coco v3.3.0...... 327 A.27 Squish Coco v3.2.3...... 327 A.28 Squish Coco v3.2.2...... 328 A.29 Squish Coco v3.2.1...... 328 A.30 Squish Coco v3.2.0...... 328 A.31 Squish Coco v3.1.0...... 329 A.32 Squish Coco v3.0.2...... 329 A.33 Squish Coco v3.0.1...... 330 A.34 Squish Coco v3.0.0...... 331 A.35 Squish Coco v3.0.0-pre1...... 331 A.36 Squish Coco v2.1.8...... 332 A.37 Squish Coco v2.1.7...... 332 A.38 Squish Coco v2.1.6...... 333
Squish Coco 5.1.0 - xiv - froglogic GmbH CONTENTS
A.39 Squish Coco v2.1.5...... 333 A.40 Squish Coco v2.1.4...... 334 A.41 Squish Coco v2.1.3...... 335 A.42 Squish Coco v2.1.2...... 335 A.43 Squish Coco v2.1.1...... 335 A.44 Squish Coco v2.1.0...... 336 A.45 Squish Coco v2.0.3...... 336 A.46 Squish Coco v2.0.2...... 337 A.47 Squish Coco v2.0.1...... 337 A.48 Squish Coco v2.0.0...... 337
XII License Information 339
B License Agreement 340
C Copyright Acknowledgements 344
Index 347
Squish Coco 5.1.0 - 1 - froglogic GmbH Part I Overview
Squish Coco 5.1.0 - 2 - froglogic GmbH Squish Coco – A code coverage tool for Tcl, QML, C# and C/C++
Chapter Squish Coco – A code coverage tool for Tcl, 1 QML, C# and C/C++
Squish Coco is a complete code coverage tool chain for Tcl, QML, C# and C/C++ programs (including SystemC programs). It runs on macOS, Linux™ and Microsoft® Windows. Squish Coco analyzes the way an application runs, e.g. as part of a test suite. The results can then be used to make the tests more efficient and complete. In particular, Squish Coco helps to • find untested code sections. • find redundant tests which can then be eliminated. Squish Coco can identify which portions of the source code are covered by a test. It can detect whether a new test covers lines in the source code that the existing tests do not cover. • find dead code by displaying code that is never executed. • calculate the optimum test execution order so as to maximize test coverage for each run. This is particularly useful for manual testing.
• analyze two separate versions of an application and compare the differences. This makes it possible to see which tests are affected by source code modifications and also to get some measure of the test coverage of a patch or hot fix. • measure the execution time of programs and tests. Squish Coco can be used at every stage of testing and for all testing methodologies(unit tests, automatic tests, manual white box tests, black box tests, etc.). A test suite can be divided into independent parts: Squish Coco makes it possible to merge multiple execution reports into a single unified report. The main components of the Squish Coco package are two separate tools and an add-in: 1. CoverageScanner – a program to analyze and instrument a Tcl, QML, C# and C/C++ application.
2. CoverageBrowser – a program to display and manage the results of the coverage analysis.
3. The Visual Studio® Coco Wizard – an optional add-in to configure code coverage for C# or C++ projects that are developed under Microsoft® Visual Studio® and similar IDEs. There are also several helper programs for report generation, coverage data manipulation and license management.
Squish Coco 5.1.0 - 3 - froglogic GmbH CoverageScanner – Instrumentation as part of the build process
CoverageScanner CoverageBrowser Instrumentation Source Code C/C++ Preprocessing (.csmes file) (.cpp file)
Executions Instrumentation (.csexe file)
Application Compiling (.exe file)
Figure 1.1: Overview of the code coverage toolchain for Tcl, QML, C# and C/C++
1.1 CoverageScanner – Instrumentation as part of the build process
CoverageScanner instruments the source code that is seen by the compiler without touching the original code. It generates an instrumented executable, shared library or plugin. The instrumentation keeps track of the code that is executed when the executable is run or when the library or plugin is used. This allows measuring e.g. how many times each line is executed, or whether a line was executed at all. When the instrumented program terminates, it produces an execution report. In order to get high-quality coverage measurements, CoverageScanner not only instruments at the function and state- ment level but also records statement coverage and decision coverage information. Another kind of instrumentation supports measurements of code execution times. Optionally it is also possible to insert the test name and execution status (i.e. “passed” or “failed”) into the report. This can be done by a script or directly from the application. CoverageScanner can then be integrated into a test framework (like CppUnit or CxxTest), where it generates separate coverage information for each test. CoverageScanner is a command line tool that replaces the compiler normally used to build the executable, library, or plugin. It inserts special instrumentation instructions into the preprocessed source code and then uses the original compiler to compile the modified code. At the same time, a database (the .csmes file) is generated for later analysis. It contains the list of instrumentations and a copy of the instrumented source code. After compilation, the instrumented executable can be run as usual (and an instrumented library or plugin can be used via the program that accesses them), but now, when the program terminates, an execution report is produced (the .csexe file). CoverageScanner works with almost any standard compiler, including GNU gcc, GNU g++, Microsoft® Visual Studio® 6.0, .NET, Express, and Embedded C++, Intel® C++, etc. For most other compilers, CoverageScanner can be easily configured. i CoverageScanner has built-in support for the Qt Framework which makes it possible to disable the instrumentation of the moc-generated code.
Squish Coco 5.1.0 - 4 - froglogic GmbH CoverageBrowser – Viewing, analyzing and managing code coverage results
1.2 CoverageBrowser – Viewing, analyzing and managing code coverage results
The execution report in the .csexe file can be analyzed and managed with the CoverageBrowser tool. This program has a GUI (graphical user interface) by which the user can browse and manage their execution reports. This makes it straightforward to locate untested or unreachable code and inefficient tests. A summary of CoverageBrowser’s abilities: • Adding comments to instrumented source code lines.
• Displaying the list of executions in a tree view. • Marking portions of code which cannot be tested as “Manually Validated”. • Browsing through the instrumented code. • Switching between the coverage levels: decision coverage, condition coverage and statement block coverage.
• Displaying a detailed explanation of the state of the instrumentation. This includes: – User comments. – The state of each instrumented statement: executed, not executed, partially executed. – The number of times an instrumented statement was executed. – A list of the tests in which an instrumented statement was executed. • Retrieving the execution status from an automated test suite. • Code coverage analysis of unit tests. • Exporting results to spreadsheets.
•A test benefit analysis mode makes it possible to see which additional code coverage a set of tests adds. • A source code and function browser which displays code coverage statistics for each file, each class, each namespace and for each Tcl, QML, C# and C/C++ function.
• Generation of a report in HTML format. It includes statistics per source file, methods, executions, and a list of unexecuted code lines. • Black box testing support. • The ability to compare coverage data for different versions of the software.
Squish Coco 5.1.0 - 5 - froglogic GmbH CoverageBrowser – Viewing, analyzing and managing code coverage results
Navigation
Class, Namespace, and Method Browser
Source Navigation and Statistics
Detailed Explanation Manual Validation
Executions Management User Comments
Figure 1.2: CoverageBrowser: Code coverage screen shot
Squish Coco 5.1.0 - 6 - froglogic GmbH Setup and Tutorials Part II
Squish Coco 5.1.0 - 7 - froglogic GmbH Installation and Setup
Chapter Installation and Setup 2
This chapter describes the installation and setup of Squish Coco on your machine and a later update of software and licenses.
2.1 Choosing a license
There are two kinds of licenses, node-locked and floating licenses. A node-locked license is bound to a specific machine and a specific user account. On that account, you can run as many Squish Coco processes as you like. Note that if you run Squish Coco from a Continuous Integration Server, the server has a separate account; in this case Squish Coco needs an additional node-locked license. Floating licenses are bound to processes, not to computers. One needs as many floating licenses as there are Squish Coco processes running at the same time. A Squish Coco process can be 1. an instance of CoverageScanner or of the compiler wrappers, 2. an instance of the CoverageBrowser, 3. each instance of the command line tools, cmmerge, cmcsexeimport and cmreport. Every compiler call that is instrumented by Squish Coco is counted separately because it requires one instance of the CoverageScanner to run. Builds with several compilations in parallel should however cause no real problem since a process that cannot get a license will wait until one is freed. Floating licenses are managed by a license server (see Chapter 29, page 175). The server must be running on a machine that is reachable from the machine on which Squish Coco runs. froglogic will provide a license file which determines the maximal number of Squish Coco processes that can be active at the same time. (The license server itself is not counted among the processes that need a license.)
Squish Coco 5.1.0 - 8 - froglogic GmbH Installation
2.2 Installation
An installer for Squish Coco can be downloaded via https://www.froglogic.com/squish/download/. The installers for the various platforms supported by Squish Coco have a common name scheme, but the way they must be executed differs. The naming scheme is SquishCocoSetup_x.y.z_hplatformi.hsuffixi, where x.y.z is the program version, hplatformi describes the operating system and other details of the installation, and hsuffixi is an operating system specific suffix. (In the noncommercial version, SquishCoco is replaced by SquishCocoNoncommercial.)
Microsoft® Windows: The installer is a file SquishCocoSetup_x.y.z_Windows_x86.exe or SquishCocoSetup_x.y.z_Windows_x64 .exe and must be executed.
Linux™: The installer is a file SquishCocoSetup_x.y.z_hplatformi.run and must be executed with bash, as in
$ bash SquishCocoSetup_3.3.2_Linux_x86_64.run
Note that the CentOS version of the installer is also valid for RedHat Linux™. macOS: The installation package is a file of the form SquishCocoSetup_x.y.z_hplatformi.pkg. Click on it and the installer will be started.
If no valid license is present for your account, the installer will run the License Wizard (see Chapter 28, page 174) at the end of the installation. The License Wizard will configure the license.
2.2.1 Installation of a license server
If you have chosen a floating license, you need to specify a machine on your local network on which the license server program runs. The license server needs a configuration file to run. It specifies the number of licenses served, the machine on which it runs, and the port used by the server. To specify the configuration file, Squish Coco must be installed first. Install therefore Squish Coco on the license server machine. After the installation, cocolicwizard will be started. Ignore it. Instead, run the command cocolicserver --server-identifier and redirect its output to a file. How this is done depends slightly on the operating system.
Microsoft® Windows: In a command window, type
c:\project> "hWindows Cocoi\cocolicserver.exe" --server-identifier > machine.txt
Linux™: Open an command shell and type
$ /opt/SquishCoco/bin/cocolicserver --server-identifier > machine.txt macOS: Open an command shell and type
$ /Applications/SquishCoco/cocolicserver --server-identifier > machine.txt
Squish Coco 5.1.0 - 9 - froglogic GmbH Updates
This will generate a file machine.txt in the directory in which the command was executed. Send it to froglogic, together with the number of requested licenses (and, if necessary, the port number).
2.2.2 Running the license server froglogic sends you a configuration file, which is usually called cocoserver.cfg. With it, the license server can run. In the simplest case, write
$ cocolicserver -c cocoserver.cfg
The server then provides licenses to other machines in the same network. For additional options of cocolicserver, see Chapter 29, page 175.
2.3 Updates
A new license can be installed with with cocolic or cocolicwizard. To update to a newer version of Squish Coco, download and install it. It will then overwrite the previous version. The license will not be touched, you can continue to use it. If a new compiler is installed, then the Squish Coco installer should be run again, so that Squish Coco becomes aware of the new compiler. This includes new versions of Microsoft® Visual Studio®: If Visual Studio 2013 was present during the Squish Coco installation and Visual Studio 2015 is added, the Squish Coco installer needs to be run. If instead the compiler is updated, Squish Coco should work mostly without changes.
One exception is the C# compiler. If it is updated, the Squish Coco setup will be overwritten. Squish ! Coco needs then to be reinstalled to make code coverage working again.
Squish Coco 5.1.0 - 10 - froglogic GmbH Instrumentation of a simple project
Chapter Instrumentation of a simple project 3
In this section we will show a small project with unit tests and show how it can be instrumented. The project is a simple expression parser, and it has not many requirements besides a C++ compiler. The project replicates in miniature an existing project that has been extend with unit tests and for which we will now use Squish Coco to find out how good the test coverage is. Instrumentation should therefore be non-intrusive and should not change the project very much. i The procedures to set up a project for instrumentation under UNIX® and macOS differ from that for Microsoft® Windows. The setup section has therefore two versions, which appear in the following sections.
3.1 UNIX® and macOS setup
3.1.1 Setup
The parser example can be found in Squish Coco’s installation directory. Under UNIX®, this is the directory /opt/ SquishCoco/ or, if you have installed Squish Coco locally, the subdirectory SquishCoco/ of your home directory.1 Under macOS, the installation directory is /Applications/SquishCoco/.
We will refer to it as the SquishCoco/ directory, wherever it is located. The Squish Coco examples, together with their supporting programs, are in SquishCoco/samples/, and the parser is in SquishCoco/samples/parser/. This directory contains three versions of the program, in the directories parser_v1/ to parser_v3/. They represent the parser in different stages of its development.
The example uses CppUnit as its unit test framework. There is a version of CppUnit in the SquishCoco/samples/ directory, and the parser example is prepared to use it.
Copy therefore now the content of the whole samples/ directory to your workspace and make parser_v1/ your working directory. If Squish Coco is installed in /opt/SquishCoco/, this is done in the following way:
$ cp -a /opt/SquishCoco/samples .
1If you have chosen a custom directory to install Squish Coco, the example will not work without modifications.
Squish Coco 5.1.0 - 11 - froglogic GmbH UNIX and macOS setup
Make the parser_v1 directory your work directory:
$ cd samples/parser/parser_v1
Make also sure that the Squish Coco tools are in your search path. If they are not so already, you can now do it by writing
$ . cocosetup.sh
(Do not forget the dot at the beginning!) Now programs like coveragebrowser can be called from the command line.
3.1.2 Structure of the parser directories
We will use samples/parser/parser_v1/ as our working directory. It contains C++ source files and header files, together with an unit test file, unittests.cpp. The makefile has been prepared for unit tests, but not for instrumentation. The instrumentation is done with the help of the bash script instrumented. (There are also some files that are needed in the Microsoft® Windows version. We will ignore them here.)
3.1.3 Compiling and testing
Run “make” to compile the program. It is a simple expression parser and calculator.
$ ./parser Enter an expression an press Enter to calculate the result. Enter an empty expression to quit.
> 2 + 2 Ans = 4 > Pi Ans = 3.14159 > sin(Pi) Ans = 1.22465e-16 > sinn(90) Error: Unknown function sinn (col 9) > sin(90) Ans = 0.893997 > cos(pi) Ans = -1 > $
We have added some unit tests for the main class, Parser. Look into the file unittests.cpp to see the tests that have been included. Execute it with make tests. You will see that 14 tests have been executed, and also that one of them has failed. This is for greater realism and also allows us to see later how Squish Coco handles test failures.
Squish Coco 5.1.0 - 12 - froglogic GmbH UNIX and macOS setup
3.1.4 Instrumentation
We have kept the instrumentation separate from the main project. The core of the instrumentation is a short shell script, instrumented. It is a simple wrapper, and calling “instrumented hcommandi” executes hcommandi with a few environment variables set. We will do this now. Enter
$ make clean $ ./instrumented make tests
The first command removes all object files, since we need everything to be recompiled. The second command then compiles the program with instrumentation and runs the tests. That’s all!
We now have a look at what the script has done and how it has done it. List the contents of your parser directory:
$ ls constants.h functions.o.csmes parser.cpp unittests.o error.cpp instrumented parser.h unittests.o.csmes error.h LICENSE parser.o variablelist.cpp error.o main.cpp parser.o.csmes variablelist.h error.o.csmes main.o unittests variablelist.o functions.cpp main.o.csmes unittests.cpp variablelist.o.csmes functions.h Makefile unittests.csexe functions.o NOTICE unittests.csmes
You see two kinds of files that do not appear as result of normal compilation. The .csmes files contain the information that is needed for coverage measurement, and the .csexe files contain the results of code execution. The files that end in .o.csmes are temporary files and are only used during compilation.
This time, the only program that was actually executed was unittests, and therefore the only .csexe file is unittests.csexe. To see the coverage results, you can therefore start the CoverageBrowser with the command
$ coveragebrowser -m unittests.csmes -e unittests.csexe
Then the CoverageBrowser will start with a modal window, "Load Execution Result". Click on the "Import" button to load the data. (This button will also appear als "Import & Delete", depending on the settings of the browser.)
By default, CoverageBrowser automatically deletes the .csexe file after it loads it. You can switch ! this behavior off by unselecting the "Delete after loading" checkbox. Or, if you do not, select the "File¡Save" menu item to save the execution report in the unittests.csexe file.
For the use of CoverageBrowser, see Part VII, page 125. We will now rather describe how the instrumentation is done.
3.1.5 How the project is instrumented
The file instrumented is a short bash script:
#!/bin/bash
. getcoco.sh # Get Coco variables
Squish Coco 5.1.0 - 13 - froglogic GmbH UNIX and macOS setup
export PATH=$COCO_WRAPPER_DIR:$PATH export COVERAGESCANNER_ARGS=’--cs-on’ export COVERAGESCANNER_ARGS+=’ --cs-mcdc’ export COVERAGESCANNER_ARGS+=’ --cs-mcc’ export COVERAGESCANNER_ARGS+=’ --cs-function-profiler=all’
"$@"
At its beginning, the shell script getcoco.sh sets the shell variable COCO_WRAPPER_DIR. It contains the name of the directory in which Squish Coco is installed. Then there are four export statements, and the final cryptic statement executes the command line parameters of instrumented. So if you call ./instrumented make tests, the command make tests is executed by the script, but in a different environment than normally.
The important part of it are therefore the four export statements. In the first one, the search path is manipulated so that the programs in /opt/SquishCoco/wrapper/bin/ are searched first. This directory contains a lot of files with the same names as the compilers2 that are supported by Squish Coco:
$ ls /opt/SquishCoco/wrapper/bin ar g++-4.9 x86_64-linux-gnu-ar c89-gcc gcc x86_64-linux-gnu-g++ c99-gcc gcc-4.6 x86_64-linux-gnu-g++-4.6 ...
These programs are the compiler wrappers. With the new PATH, they are executed instead of the real compilers. The compiler wrappers are actually symbolic links to a single program, coveragescanner (see PartIV, page 70). When executed to compile a source file, they create an instrumented version of the source and then run the original compiler to compile it.
In the other three export statements, additional flags for the compiler wrappers (see Chapter9, page 72) are set. The most important option is the first one, --cs-on. If it is not present, the compiler wrappers are inactive and just call the compilers they represent. The following options enable MC/DC (see Chapter 5.2.5, page 55) and Multiple Condition coverage (see Chapter 5.2.4, page 54). They are disabled by default because they are sometimes expensive to execute. Here they are enabled so that you can see all coverage modes later in the CoverageBrowser. The resulting script should work without changes for many simple projects. If more customization is needed, it can often be achieved by adding more options to COVERAGESCANNER_ARGS.
3.1.6 Additional changes
It is also convenient to add make targets to handle the files generated by CoverageScanner. In the parser directory, the Makefile has been changed in the following way:
clean: testclean ... -$(DEL_FILE) *.o.csmes # (added)
distclean: clean ... -$(DEL_FILE) *.csmes *.csexe # (added)
2There is also ar, which is not a compiler but takes part in the compilation process. This is the output under UNIX®; under Mac OS X, the directory /Applications/SquishCoco/wrapper/ contains a much smaller list of files.
Squish Coco 5.1.0 - 14 - froglogic GmbH Microsoft Windows setup
Since the .o.csmes files are needed only for compilation, they can be deleted whenever the .o files are deleted (which is that what make clean does). The .csmes and .csexe files are more precious and should only be deleted when all generated files are removed. Therefore we have added their deletion statements to the distclean target.
3.2 Microsoft® Windows setup
3.2.1 Setup
The parser example can be found in the directory hWindows Cocoi\parser. This directory contains three versions of the program, in the subdirectories parser_v1 to parser_v3. They represent the parser in different stages of its development.
The example uses CppUnit as its unit test framework. There is a version of CppUnit in the hWindows Cocoi\squishcoco directory, and the parser example is prepared to use it. Since these directories are write-protected, you need to create your own working copies. Copy therefore the two directories hWindows Cocoi\squishcoco\parser and hWindows Cocoi\squishcoco\cppunit-1.12.1 to a directory of your choice. Then remove the write protection of the directories and all the files contained in them.
3.2.2 Structure of the parser directories
We will use parser\parser_v1 as our working directory. It contains C++ source files and header files, together with an unit test file, unittests.cpp. The Makefile has been prepared for unit tests, but not for instrumentation. The instrumentation is done with the help of the batch file instrumented.bat. (There are also some files that are needed in the UNIX®version. We will ignore them here.)
3.2.3 Compiling and testing
We will do the compilation of the example on the command line. To get a command window, execute the batch file CocoCmd.bat that is located in the parser_v1 directory. In this window, the Microsoft® Visual Studio® command line tools (like nmake) are accessible, and also the main Squish Coco programs (like CoverageBrowser).
Run “nmake” to compile the program. It is a simple expression parser and calculator.
$ C:\code\parser\parser_v1>parser.exe Enter an expression an press Enter to calculate the result. Enter an empty expression to quit.
> 2 + 2 Ans = 4 > Pi Ans = 3.14159 > sin(Pi) Ans = 1.22465e-16 > sinn(90) Error: Unknown function sinn (col 9) > sin(90)
Squish Coco 5.1.0 - 15 - froglogic GmbH Microsoft Windows setup
Ans = 0.893997 > cos(pi) Ans = -1 > C:\code\parser\parser_v1>
We have added some unit tests for the main class, Parser. Look into the file unittests.cpp to see the tests that have been included. Execute it with nmake tests. You will see that 14 tests have been executed, and also that one of them has failed. This is for greater realism and also allows us to see later how Squish Coco handles test failures.
3.2.4 Instrumentation
We have kept the instrumentation separate from the main project. The core of the instrumentation is a short shell script, instrumented.bat. It is a simple wrapper, and calling “instrumented.bat hcommandi” executes hcommandi with a few environment variables set. We will do this now. Enter
C:\code\parser\parser_v1>nmake clean C:\code\parser\parser_v1>instrumented.bat nmake tests
The first command removes all object files, since we need everything to be recompiled. The second command then compiles the program with instrumentation and runs the tests. That’s all!
We now have a look at what the script has done and how it has done it. List the contents of your parser directory:
C:\code\parser\parser_v1>dir /D
Directory of C:\code\parser\parser_v1
[.] LICENSE unittests.exe [..] main.cpp unittests.exe.csexe constants.h main.obj unittests.exe.csmes error.cpp main.obj.csmes unittests.exp error.h Makefile unittests.lib error.obj nmake.mak unittests.obj error.obj.csmes NOTICE unittests.obj.csmes functions.cpp parser.cpp variablelist.cpp functions.h parser.h variablelist.h functions.obj parser.obj variablelist.obj functions.obj.csmes parser.obj.csmes variablelist.obj.csmes instrumented README.squishcoco instrumented.bat unittests.cpp 35 File(s) 1,079,919 bytes 2 Dir(s) 35,159,457,792 bytes free
You see two kinds of files that do not appear as result of normal compilation. The .csmes files contain the information that is needed for coverage measurement, and the .csexe files contain the results of code execution. The files that end in .obj.csmes are temporary files and are only used during compilation.
This time, the only program that was actually executed was unittests, and therefore the only .csexe file is unittests.exe.csexe. To see the coverage results, you can therefore start the CoverageBrowser with the com- mand
Squish Coco 5.1.0 - 16 - froglogic GmbH Microsoft Windows setup
C:\code\parser\parser_v1>coveragebrowser -m unittests.exe.csmes -e unittests.exe.csexe
Then the CoverageBrowser will start with a modal window, "Load Execution Result". Click on the "Import" button to load the data. (This button will also appear als "Import & Delete", depending on the settings of the browser.)
By default, CoverageBrowser automatically deletes the .csmes file after it loads it. You can switch ! this behavior off by unselecting the "Delete after loading" checkbox. Or, if you do not, select the "File¡Save" menu item to save the execution report in the unittests.csexe file.
For the use of CoverageBrowser, see Part VII, page 125. We will now rather describe how the instrumentation is done.
3.2.5 How the project is instrumented
The file instrumented.bat is a short batch file:
@echo off setlocal
set PATH=%SQUISHCOCO%\visualstudio;%PATH% set COVERAGESCANNER_ARGS=--cs-on --cs-mcdc --cs-mcc --cs-fucntion-profiler=all
call %*
endlocal
The variable SQUISHCOCO contains the name of the directory in where Squish Coco is installed. It is set by Squish Coco during installation.
At the beginning, the setlocal command ensures that the following commands change the environment variables only temporarily. Then there are two set statements, and the final call statement executes the command line parameters of instrumented. So if you call “instrumented nmake tests”, the command “nmake tests” is executed by the batch file, but in a different environment than normally. At the end, endlocal undoes the changes in the environment variables.
The important part of the script are therefore the two set statements. In the first one, the search part is manipulated so that the programs in hWindows Cocoi\squishcoco\visualstudio are searched first.3 This directory contains files with the same names as the compilers and the linker
C:\code\parser\parser_v1>dir /d "\Program Files\squishcoco\visualstudio"
Directory of C:\Program Files\squishcoco\visualstudio
[.] cl.exe link.cspro msvcr100.dll [..] lib.cspro link.exe cl.cspro lib.exe msvcp100.dll 8 File(s) 5,662,299 bytes 2 Dir(s) 35,053,502,464 bytes free
3This is the version for the 32 bit compiler. There is also a directory hWindows Cocoi\visualstudio_x64 for 64 bit compilation.
Squish Coco 5.1.0 - 17 - froglogic GmbH Beyond the minimal instrumentation
The .exe files in this directory are the compiler wrappers. With the new PATH, they are executed instead of the real compiler. The compiler wrappers are actually copies of a single program, coveragescanner.exe (see PartIV, page 70). When executed to compile a source file, they create an instrumented version of the source and then run the original compiler to compile it.
In the second set statement, additional flags for the compiler wrappers (see Chapter9, page 72) are set. The most important option is the first one, --cs-on. If it is not present, the compiler wrappers are inactive and just call the compilers they represent. The following options enable MC/DC (see Chapter 5.2.5, page 55), Multiple Condition coverage (see Chapter 5.2.4, page 54) and the function profiler. They are disabled by default because they are sometimes expensive to execute. Here they are enabled so that you can see all coverage modes later in the CoverageBrowser. The resulting script should work without changes for many simple projects. If more customization is needed, it can often be achieved by adding more options to COVERAGESCANNER_ARGS.
3.2.6 Additional changes
It is also convenient to add make targets to handle the files generated by CoverageScanner. In the parser_v1 directory, the Makefile has been changed in the following way:
clean: testclean ... -$(DEL_FILE) *.obj.csmes # (added)
distclean: clean ... -$(DEL_FILE) *.csmes *.csexe # (added)
Since the .obj.csmes files are needed only for compilation, they can be deleted whenever the .obj files are deleted (which is that what make clean does). The .csmes and .csexe files are more precious and should only be deleted when all generated files are removed. Therefore we have added their deletion statements to the distclean target.
3.3 Beyond the minimal instrumentation
In the following sections we will show additional abilities of Squish Coco. They will require small changes in the code of the project.
3.3.1 Excluding code from instrumentation
The coverage information generated so far has a problem: It covers too many files. The problematic files are those that belong to the testing framework and not to the tested program. Including them would create artificially low coverage rates.
With Squish Coco, one can exclude files from coverage by additional command line options. In parser_v2, this has been done. Look into parser_v2/instrumented (or parser_v2\instrumented.bat under Microsoft® Windows). In it, three additional command line options have been set, which we will now explain:
Squish Coco 5.1.0 - 18 - froglogic GmbH Beyond the minimal instrumentation
• The option --cs-exclude-path=../../cppunit-1.12.1 excludes the source files of a directory and all its subdirectories. Here we use it to exclude all the files of the CppUnit framework. You can use slashes or backslashes with this option—Squish Coco normalizes them internally.
• The options --cs-exclude-file-wildcard=unittests.cpp and --cs-exclude-file-wildcard=CppUnitListener.cpp exclude specific files. We use them to exclude the files unittests.cpp and CppUnitListener.cpp. (The second file is described below.)
3.3.2 Making the test names visible
For the next modification, we want to change the project such that we know not only whether a line of code is covered by tests, but also by which tests it is covered. For this we will add calls of the CoverageScanner C/C++ library (see Chapter 10.1, page 86) to the code, to tell Squish Coco the names of the tests and where they begin and end.
An updated version of the project can be found in the directory parser_v2. The greatest difference to the version in parser_v1 is that the file CppUnitListener.cpp has been added. It is copied almost verbatim from Chapter 42.1. The file contains a class CppUnitListener and a new main() function. The main() function in unittests.cpp has been removed, but the file is otherwise unchanged.
CppUnitListener.cpp provides a unit test listener which allows hooking into the framework before and after the execution of each test. One can thus record additional test information, like the name and the result of a test, in the code coverage data without modifying the test code itself. (For a listing of CppUnitListener.cpp and an explanation how it works, see Chapter 42.1) Now you can execute this program with the same way as its previous version. View the results in the CoverageBrowser. In the "Executions" subwindow you can now see the code coverage for each single test. You can especially see that it was the test testInvalidNumber that had failed. Hover with your mouse over one of the lines in the source window, and select a line that was marked in dark green. This means that it was executed by the tests4. You will see a tooltip with a list of the tests that executed this line. To keep the tooltip short, there is a maximal number of tests to display there. To see a full list, click on the line. Then the full information about the line is shown in the "Explanation" window.
3.3.3 Patch file analysis
Now consider the following scenario: In a large project, a last-minute patch has to be evaluated. It is not enough time to run the full test suite, but some risk assessment needs to be done. For situations like this, Squish Coco provides the feature of patch analysis. With it, one can specifically display the code coverage for the changed lines of code, and find the tests in a large suite that cover them. One can now see how risky the changes are. We will simulate this situation in our example. In a new version of the parser, the character classification functions in the code, isWhiteSpace(), isAlpha(), etc., have been changed and use the standard C classification functions, like isspace(), instead of strchr(). The new version of the parser can be found in the directory parser_v3.
We will now compare it with the version in parser_v2, but neither run the tests nor even compile it. Instead we need the following two pieces of information:
1. The coverage data from parser_v2, as generated in the previous section.
2. A patch file showing the differences between the two directories. There is already a diff file in the parser directory that you can use, parser.diff.
4Lines in bright green were also executed, but not instrumented by Squish Coco.
Squish Coco 5.1.0 - 19 - froglogic GmbH Beyond the minimal instrumentation
The diff file must be in the “unified” difference format. This is the standard output format of the diff functionality of many version control systems, e.g. of git diff (see Chapter 40, page 209). Under UNIX®- like systems, the patch file can also be generated by the diff utility. It would be invoked from the parser directory in the following way:
$ diff -u parser_v2 parser_v3 > parser.diff
There is also a Microsoft® Windows version of GNU diff in the parser directory, therefore the same command works in a Windows command shell too.
Start the CoverageBrowser. Then load the instrumentation database parser_v2/unittests.csmes via the menu entry "File¡Open...", and the measurements file parser_v2/unittests.csexe with "File¡Load Execution Report...".
Now select the menu entry "Tools¡Patch File Analysis...". When the "Patch File Analysis" dialog appears:
• Enter a title in the "Title" box.
• Enter the path to the patch file in the "Patch File" box.
• Enter a path (including the file name) for the report in the "Type" box.
• Set the "Tooltip Maximal Size" to 5 (or any value greater than zero). • Make any other option adjustments.
Then click "Open" to view the report in the browser.
3.3.4 The patch analysis report
The report consists of three tables that summarize the influence that the patch has on code coverage, and then an annotated version of the patch file. The two tables in the section “Overview” contain statistics about the number and kind of the lines that were influenced by the patch. The first table groups the patched lines in the code by the results of the tests that executed them. One can see here how much influence the patch has on tests that have passed (and now could fail) or failed (and could now succeed). There are also entries for manually checked tests and for those whose status is unknown. In our example, we did not register the test results and all our tests are counted as “Unknown”. The second table shows the kind of changes to expect in the test coverage after the patch has been applied. It consists of three columns, containing the statistics about removed and inserted lines and their sum. From the first two columns one can see whether the test coverage for the patched code grows or falls. (In the parser example, it stays the same.) The last line in the table is also important: It shows the number of lines which Squish Coco could not classify as inserted or removed. Patch analysis is a heuristics, after all. The section “List of tests influenced by the modifications” is a list with the names of the tests that executed the patched code, together with their result. It is helpful for a qualitative analysis of the patch. In our example, we can see that all tests execute code that is affected by the patch.
Squish Coco 5.1.0 - 20 - froglogic GmbH Beyond the minimal instrumentation
Figure 3.1: Coverage of the patched lines by the tests
More details can be found in the “Patch File” section of the report. It is an annotated version of the original patch file, with the old version of the text in red and the new version in green. Lines that did not change are shown in gray. The most important column is “Tests”, which shows for each code line the number of tests that executed it (if it is removed) or will probably execute it after the patch is applied. A tooltip shows the names of these tests.
3.3.5 Reverse patches
Now suppose that a patch has already been checked in. It has changed the application’s behavior, and you want to write more test to make sure that all new code has been covered. In this case, a reverse patch analysis is helpful.
To simulate a reverse patch analysis with the parser example, switch to the directory parser_v3 and compile the code there in the same way as before parser_v2. Then start the CoverageBrowser and load this time the instrumentation database parser_v3/unittests.csmes and the measurements file parser_v3/unittests.csexe. This can also be done on the command line, e.g. in the form
$ cd parser_v3 $ coveragebrowser -m unittests.csmes -e unittests.csexe
Generate the patch analysis report as before. Squish Coco recognizes automatically that patch.diff is a patch to the current version of the program, and generates a corresponding report. This patch report contains the same data as the report for the forward patch, but the headers have changed to reflect the new interpretation.
3.3.6 Bug location
CoverageBrowser can compute the probable location of an error in the source code by comparing the coverage data of all tests that were run on a program. The algorithm expects that at least one test had failed. In our example, the parser does not detect that the exponent of a floating point number is invalid. It is possible to write the following:
$ cd parser_v2 $ ./parser Enter an expression an press Enter to calculate the result. Enter an empty expression to quit.
> 1E+ Ans = 1
Squish Coco 5.1.0 - 21 - froglogic GmbH Beyond the minimal instrumentation
But 1E+ is not a valid number because it has no digits after the +. To find out which line in the source code causes this, we need a few additional tests. The following tests are already part of the test suite in the file unittests.cpp: Test input Expected result Execution status 1.1 1.1 Passed 1.1e1 11 Passed 1.1e+1 11 Passed 1.1e-1 0.11 Passed 1.1e+ error message Failed With these tests already contained in the test suite, we can use CoverageBrowser to find a candidate for the line that caused the bug. There is a "Bug Location" window in the CoverageBrowser, but it is by default disabled. You can enable it by checking the field "Bug Location" in the menu "View".
After the window is enabled, select all the executions in the "Executions" window and click the "Compute" button. The window will then be filled with a list of possible bug locations, with the most probable (according to Squish Coco) first. In our case there is only one candidate: Line 263 of the file parser.cpp. This line is the location where the sign ’+’ of the exponent is parsed. At this point a fix could be implemented, but since this is only a demonstration, we will not do it here.
Figure 3.2: Bug location for the parser sample
Squish Coco 5.1.0 - 22 - froglogic GmbH Beyond the minimal instrumentation
3.3.7 The function profiler
With the function profiler, Coco measures for all functions in a program how much time they require when the program is executed.
The function profiler is disabled by default. To switch it on, one needs the CoverageScanner option --cs-profiler in the compiler and linker command line arguments. As it is possible with code coverage, one can use the CoverageBrowser to measure the time consumption of a selected set of executions. One can also use it to compare the time measurements for two program versions or two test executions.
Basic use In the parser example, function profiling is already enabled. For a simple example, run the unit tests in the parser_v3 directory and load the instrumentation database into the CoverageBrowser. A profiler docking window will then be shown. (If it is not shown, select the menu entry "View¡Function Profiler".) When no tests are selected, it contains only a list of the functions for which the execution time is recorded:
Figure 3.3: The profiler window when no tests are selected
Once a test is selected, the timing information of the functions appears:
Total Duration: The cumulated execution time of the function.
Count: The number of calls.
Mean Duration: The mean execution time of a single call. All timing information refers to the selected executions. Click on the title bar of a column to sort it and to find the highest values.
Squish Coco 5.1.0 - 23 - froglogic GmbH Beyond the minimal instrumentation
Figure 3.4: The profiler window when tests are selected
Comparing two sets of executions CoverageBrowser can also compare the profiling information of two set of tests. The principle is simple: a set of reference functions is selected and compared to another set.
Activate the execution comparison mode by selecting the menu entry "Tools¡Execution Comparison Analysis". A column “Reference” will then appear in the "Executions" window. Select as reference the test CppUnit/ ParserTest/testFloatExp and as selected execution CppUnit/ParserTest/testFloat.
Figure 3.5: Executions window in test comparison mode
Use then the context menu to show the columns "Total Duration (Ratio)" and "Total Duration (Difference)". Click on the column "Total Duration (Ratio)" to sort it. You should get then a screen equivalent to the following screenshot. (The timing information may differ from that on the image because the execution time depends on the machine which runs the tests).
Squish Coco 5.1.0 - 24 - froglogic GmbH Beyond the minimal instrumentation
Figure 3.6: Profiler window in test comparison mode
The entries in the table have the form “hratioi (hreferencei->hselectedi)” and “hdiff i (hreferencei->hselectedi)”. In them, hreferencei is the runtime of the function in the reference execution and hselectedi the runtime of the seleced executions taken together. hratioi is their quotient and hdiff i is their difference.
We can see that the function Error::Error has an infinite ratio because it was not executed in the reference test. We can also see that that the function Parser::getToken is 20 times slower than the reference test. The entry in the “Difference” column is however not really meaningful because the difference is only 78 microseconds.
Comparing two versions of a program It is also possible to compare the execution times of two different versions of the same program.
i This requires that the .csexe data are already incorporated into their .csmes files. If you have not done this already, open parser_v2/unittests.csmes and parser_v2/unittests.csexe into the CoverageBrowser, select "File¡Save", and then to the same with parser_c3.
To do this, select the reference execution database with "Tools¡Compare with...". In the resulting file selection dialog, choose the file parser_v2/unittests.csmes.
Since the absolute names of the source files are not identical, it is necessary to rename them first. in the "Source" window, select the context menu by a right mouse click and in it, "Rename Sources...". As current name enter parser_v2 and as new name parser_v3 and then press “Ok”.
Now select all executions in the "Execution" window. Then CoverageBrowser compares the two program versions in the same way as the two program executions in the previous section.
Squish Coco 5.1.0 - 25 - froglogic GmbH Using the function profiler for performance improvents
Figure 3.7: Profiler window when comparing two software versions
3.4 Using the function profiler for performance improvents
This section is an example how Squish Coco can be used to find and resolve performance problems. It covers narrowing down the performance issues, identifying problematic functions, rewriting the source code and comparing the refactored code’s performance to the original version.
3.4.1 Application Building
To enable the profiler, CoverageScanner needs a flag, --cs-function-profiler. Two versions are available:
--cs-function-profiler=all: Profile all functions.
--cs-function-profiler=skip-trivial: Skip functions with only a single statement. These functions are mostly getter or setter functions. Profiling them would only slow down the program without providing more information. With these options, Squish Coco collects the profiling together with the code coverage. No additional work is necessary to analyze the results with the CoverageBrowser.
For our parser sample, the setting --cs-function-profiler=all is already part of the instrumentation script. It is therefore enough to use the instrumented version of the parser/parser_v3 directory, as before.
Squish Coco 5.1.0 - 26 - froglogic GmbH Using the function profiler for performance improvents
3.4.2 Analyzing the Performance Issues
Identifying the Test Cases Now load the files unittests.csmes and unittests.csexe into the Coverage- Browser. In the "Executions" window we see that some tests executions required a large amount of time:
Figure 3.8: Identifying tests which have performance issues.
In these tests, new parser variables are defined, from 10 in testAddVariable10 up to 40 000 new variables in testAddVariable40000. We can see that the time spent in these tests increases exponentially with the number of variables. To analyze the situation, we use the execution comparison mode in the CoverageBrowser. To acticate it, we use the menu entry "File¡Execution Comparison Analysis". We select the test with 1000 variable definitions as a reference test and compare it with the test with 10 000 definitions.
Figure 3.9: Comparing the tests with 1000 variable definitions to the test with 10 000 variable definitions.
Identifying the problematic functions The "Function Profiler" window makes a more detailed analysis possible:
Figure 3.10: Comparison using the Function Profiler window.
This window displays the execution time and number of calls for the test with 10 000 variables and the reference test with 1000 variables.
Squish Coco 5.1.0 - 27 - froglogic GmbH Using the function profiler for performance improvents
Additionally, there are two columns for the ratio of the current test to the reference. The content of the column “Total duration (Ratio)” is given by the formula
ratio = duration of test_current/duration of test_reference and there is a similar formula for the column “Count (Ratio)”. With this information it is easy to see that if the number of variable definitions in a test grows by a factor of 10, the execution times of toupper(), Variablelist::add() and Variablelist::get_id() grow by a factor of over 100. Furthermore, it seems to be necessary to reduce the number of calls of toupper() because its number of executions increases by a factor of 99 whereas the number of executions for the other functions grow by a factor that is smaller than 10.
Next we look at the code fore these functions. (It is in the file parser_v3/variablelist.cpp.) We see a typical example of source code that was originally written in C: The code uses char* as the string data type, and the variables and their values are stored as an array of structs. STL containers are used but just to replace the classic C arrays. The function get_id() finds the position of an entry for a variable by iterating over the complete table of variables. The number of search steps is thererefore proportional to the number of variables in the table, but it could be logarithmic if it were replaced by a linear search. There is also a call of toupper() in each iteration that slows down the algorithm even more.
/* * Adda name and value to the variable list */ bool Variablelist::add(const char* name, double value) { VAR new_var; strncpy(new_var.name, name, 30); new_var.value = value;
int id = get_id(name); if (id == -1) { // variable does not yet exist var.push_back(new_var); } else { // variable already exists. overwrite it var[id] = new_var; } return true; }
/* * Returns the id of the given name in the variable list. Returns -1 if name * is not present in the list. Name is case insensitive */ int Variablelist::get_id(const char* name) { // first make the name uppercase char nameU[NAME_LEN_MAX+1]; char varU[NAME_LEN_MAX+1]; toupper(nameU, name);
for (unsigned int i = 0; i < var.size(); i++) { toupper(varU, var[i].name);
Squish Coco 5.1.0 - 28 - froglogic GmbH Using the function profiler for performance improvents
if (strcmp(nameU, varU) == 0) { return i; } } return -1; }
/* * str is copied to upper and made uppercase * upper is the returned string * str should be null-terminated */ void toupper(char upper[], const char str[]) { int i = -1; do { i++; upper[i] = std::toupper(str[i]); } while (str[i] != ’\0’); }
Rewriting the Source Code The solution here is simple. We replace the array of variables with a std::map. The updated code (in parser_v4/variablelist.cpp) is then: std::map
bool Variablelist::add(const char* name, double value) { var[ toUpper( name ) ] = value ; return true; }
bool Variablelist::set_value(const char* name, const double value) { return add(name, value); }
std::string toUpper(const char str[]) { std::string upper; upper.reserve(NAME_LEN_MAX); int i = -1; do { i++; upper += std::toupper(str[i]); } while (str[i] != ’\0’); return upper; }
Here toupper() is rewritten for std::string and the function add() is only a simple affectation of std::map, which has a logarithmic complexity.
Squish Coco 5.1.0 - 29 - froglogic GmbH Using the function profiler for performance improvents
Comparing the results Now we can re-execute the tests with the program version in parser_v4 and compare the results. We open the CoverageBrowser, load the latest version of parser_v4/unittests.csmes and compare it with the previous one. The comparison is done the same way as in the previous section (see Chapter 3.3.7, page 25), except that it is now with parser_v3 and parser_v4.
In the "Executions" window of CoverageBrowser we see that the speed issue is resolved. The execution time of the tests no longer grows exponentially.
Figure 3.11: Execution of the test suite in both software versions.
The "Function Profiler" window confirms this finding. Squish Coco highlights the modified functions in bold, underlines the new functions and strikes out the removed functions. Now, by measuring the difference of the execution time (1 minute and 32 seconds), we can see that the performance gain is nearly the same as the previous execution time.
Figure 3.12: Comparing the profiler data in the Function Profiler window.
Squish Coco 5.1.0 - 30 - froglogic GmbH Code coverage for a Qt application
Chapter Code coverage for a Qt application 4
The following example is more complex. We will take Qt’s TextEdit example and use it to illustrate how Squish Coco can be used at different stages of the development process. To cover the whole coding cycle, we will first show how an instrumented application is created, perform manual tests and analyze their results. Then we will create an instrumented unit test. In a second step (see Chapter 4.2, page 41), we will cover the aspects which are more interesting to product managers: analyzing the impact of code changes (e.g. bug fixes)—in particular, tracking their test progress, externalizing testing, and collecting the code coverage analysis of a complete testing team.
i The Qt framework can be downloaded directly from https://www.qt.io/developers.
Location of the example The example can be found in a directory called “textedit”. Its location varies according to the operating system. Under Microsoft® Windows, the directory is stored directly in the installation directory. Under Linux™, it can be found under /opt/SquishCoco/samples, and under macOS it is in /Applications/ SquishCoco/samples.
The example is write-protected, so you need to create a copy of the directory textedit to work with it.
4.1 Compiling the example application
In this section, we will work with the files in the directory textedit/textedit_v1. We want to be able to build our application both normally and with generated test coverage instrumentation code, without having to change our source code. This can be achieved by making a small change to the application’s project (.pro) file. We can then use a command line option for qmake to generate an instrumented build instead of a normal one. To make the project file suitable both for normal and for instrumented builds, we create a set of definitions that can be activated by a command line switch; in qmake’s terminology this is called a scope. The following listing (see Figure 4.1, page 32) shows a minimal scope for code instrumentation. The following must be done:
Squish Coco 5.1.0 - 31 - froglogic GmbH Compiling the example application
• We must ensure that precompiled headers are disabled when the code is instrumented. qmake allows us to do this by setting the PRECOMPILED_HEADER variable to an empty value.
• It is also necessary to increase the value of QMAKE_LINK_OBJECT_MAX in order to disable the usage of the linker script. We set it here to 10000.
• Finally, qmake must be instructed to use CoverageScanner’s wrappers for compilation. This is done by prefixing the names of the compilation tools with cs.
CodeCoverage { PRECOMPILED_HEADER= QMAKE_LINK_OBJECT_MAX=10000
QMAKE_CC=cs$$QMAKE_CC QMAKE_CXX=cs$$QMAKE_CXX QMAKE_LINK=cs$$QMAKE_LINK QMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIB QMAKE_AR=cs$$QMAKE_AR QMAKE_LIB=cs$$QMAKE_LIB }
Figure 4.1: Minimal qmake configuration
These modification are sufficient for most standard C and C++ applications. For Qt applications we must use additional settings in order to ensure that CoverageScanner does not instrument the source code that is generated by Qt’s tools (e.g., by uic, qrc and the moc).
To exclude qrc resource files from instrumentation, we must tell CoverageScanner not to instrument any file with a name that begins with qrc_. This can be done with the command line option --cs-exclude-file-abs-wildcard= */qrc_*. Since we do not want to have to enter this option manually, we will put it in the .pro file. Similarly, to let CoverageScanner ignore the files generated by uic we can use the same command line option, only this time with a different file matching regular expression: --cs-exclude-file-abs-wildcard=*/ui_*. Squish Coco also supports by default applications that are built with the Qt 4 toolkit.1 This ensures that Coverage- Scanner:
• does not instrument the Q_OBJECT and Q_DECLARE_PLUGIN macros, and
• does not instrument code generated by the moc, except that signal emissions and slot receives are instrumented, since they are vital to a Qt program’s logic. We can also exercise some control over the level of instrumentation and what information is reported. For example, we can switch on the counting of code executions with the --cs-count command line option, or we can enable full instrumentation at decision/condition level with the --cs-full-instrumentation option. With the --cs-output option we can specify the file the execution report is written to when the application terminates. (By default the output is written to the file happnamei.csexe, where happnamei is the name of the program that has been executed.)
So, for a Qt 4-based application, the final Squish Coco scope in the application’s .pro file will look something like this:
1If needed, this behavior can be disabled with the option --cs-no-qt4.
Squish Coco 5.1.0 - 32 - froglogic GmbH Compiling the example application
CodeCoverage { COVERAGE_OPTIONS = --cs-output=textedit.exe COVERAGE_OPTIONS += --cs-exclude-file-abs-wildcard=*/qrc_*
QMAKE_CFLAGS += $$COVERAGE_OPTIONS QMAKE_CXXFLAGS += $$COVERAGE_OPTIONS QMAKE_LFLAGS += $$COVERAGE_OPTIONS
QMAKE_CC=cs$$QMAKE_CC QMAKE_CXX=cs$$QMAKE_CXX QMAKE_LINK=cs$$QMAKE_LINK QMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIB QMAKE_AR=cs$$QMAKE_AR QMAKE_LIB=cs$$QMAKE_LIB }
Figure 4.2: Final qmake configuration
In this form, the scope has been added to the project file textedit_v1/textedit_v1.pro. When we now run qmake without options, a Makefile for a normal build is generated; but we can also build an instrumented version of the program in the following way: • Linux™ and macOS:
$ ~/Qt/5.5/gcc/bin/qmake CONFIG+=CodeCoverage $ make
The path to the qmake program will of course vary according to the Qt version you use.
• Under Microsoft® Windows, it is also necessary to set the paths that make the nmake command accessible. All Microsoft® Visual Studio® version provide a “Native Tools Command Prompt” window, which sets the paths for a given compiler version. In such a command window, the command sequence to compile the example might look like this:
C:\textedit\textedit_v1>C:\Qt\5.5\msvc2015\bin\qmake CONFIG+=CodeCoverage C:\textedit\textedit_v1>nmake
In either case we still end up with a textedit.exe executable. But with an instrumented build we will also get an additional file, textedit.exe.csmes. It contains the instrumentation database.
4.1.1 The first code coverage results
For our very first exercise we will simply execute TextEdit and then quit the application straight away. This will generate a file called textedit.exe.csexe. The file is in a binary format, so it is not human readable. It is an instrumentation database that contains a snapshot of the most recent execution that we have just done.
To see the results we must run the CoverageBrowser tool and load the textedit.exe.csmes instrumentation database. This can be done with the menu entry "File¡Open...". After the file has been opened, no coverage information is available because no execution reports have been imported. The instrumented code lines are shown grayed out and no coverage statistics has been computed (see Figure 4.3, page 34).
Squish Coco 5.1.0 - 33 - froglogic GmbH Compiling the example application
Figure 4.3: CoverageBrowser after loading the TextEdit’s instrumentation database
In order to see an execution report, click "File¡Load Execution Report...", which invokes the import dialog. You need to enter at least the file name (including the full path) of the textedit.exe.csexe file into the field "File Name". You also need to give the test a name. Enter in the field "Name" e.g. the text “Start and Quit”. Switch the "Delete after loading" option on because the report is no longer needed after our import. It is also helpful to set the option "When file becomes modified" to “Open this dialog”, because then the file import dialog is automatically opened after each run. The new .csexe file that it created in the test can then be added to the database. After the import has finished, the code coverage information is visible: • The coverage statistics for the functions and methods of all source files of the application is shown. • The source window is now colored. It shows executed code on a green background and unexecuted code on a red background.
• The execution list now contains one selected item called “Start and Quit”, the only test execution report we have created so far.
4.1.2 Interactive testing
CoverageBrowser shows reveals, for example, that the TextEdit::fileSave() function is not executed. We will now validate this function interactively, guided by the code coverage analysis. In the source window, all unexecuted source code lines are shown on a red background (see Listing 4.4, page 35).
Squish Coco 5.1.0 - 34 - froglogic GmbH Compiling the example application
bool TextEdit::fileSave() { if (fileName.isEmpty()) { QMessageBox::warning(this, tr("No file name specified"), tr("Save first your document using ’Save As...’ from the menu"), QMessageBox::Ok); return false; }
QTextDocumentWriter writer(fileName); bool success = writer.write(textEdit->document()); if (success) textEdit->document()->setModified(false); return success; }
Figure 4.4: CoverageBrowser source view of the function TextEdit::fileSave()
To test this function, we must perform the following steps:
1. Start the TextEdit application.
2. Click on the "Save" button: TextEdit should display the error message "Save first your document using ‘Save As. . . ’ from the menu".
3. Quit the application.
After these steps have been done and the coverage report imported, CoverageBrowser shows that the “return false ;” line just after the call to QMessageBox::warning() has been executed, as indicated by the green background). However, the line “if (fileName.isEmpty())” is partially executed: This is shown by an orange background (see Listing 4.5, page 35).
bool TextEdit::fileSave() { if (fileName.isEmpty()) { QMessageBox::warning(this, tr("No file name specified"), tr("Save first your document using ’Save As...’ from the menu"), QMessageBox::Ok); return false; }
QTextDocumentWriter writer(fileName); bool success = writer.write(textEdit->document()); if (success) textEdit->document()->setModified(false); return success; }
Figure 4.5: CoverageBrowser source view after clicking TextEdit’s ‘Save’ button.
Squish Coco 5.1.0 - 35 - froglogic GmbH Compiling the example application
The explanation window (see Listing 4.6, page 36) tells us that the value of the expression “fileName.isEmpty()” was true during one execution but was never false. It is therefore considered as only partially executed. In order to fully test this expression, we must click on the ‘Save As. . . ’ button, then choose a file name, and finally click on the ‘Save’ button.
partially executed: fileName.isEmpty()
TRUE FALSE
yes no Execution Count: 1 Execution Count: 0 Executed by: - Save Clicked
Figure 4.6: CoverageBrowser explanation window after clicking on the ’Save’ button of TextEdit.
After rerunning the application and doing a “Save as”, the new execution report has only one source code line that is partially untested (see Listing 4.7, page 36). In this case, CoverageBrowser shows that the Boolean variable success was never false, which means that saving the document has never failed. We could force a write failure, and this would ensure that we had 100% code coverage for this function. But we will use a different test strategy to get complete code coverage: We will use a unit test and import the execution result into the TextEdit instrumentation database.
bool TextEdit::fileSave() { if (fileName.isEmpty()) { QMessageBox::warning(this,tr("No file name specified"), tr("Save first your document using ’Save As...’ from the menu"), QMessageBox::Ok ); return false; }
QTextDocumentWriter writer(fileName); bool success = writer.write(textEdit->document()); if (success) textEdit->document()->setModified(false); return success; }
Figure 4.7: CoverageBrowser source view after clicking TextEdit’s ‘Save As. . . ’ button and then the ‘Save’ button.
4.1.3 Writing unit tests
The unit test infrastructure can be found in the directory textedit_v1_tests/. It contains just one test, which sets an illegal filename and then tries to execute TextEdit’s fileSave() function. To do this, we use the QTestLib unit test library that is part of Qt. The test is contained in the file textedit_v1_tests/tst_textedit.cpp (see Listing 4.8, page 37).
Squish Coco 5.1.0 - 36 - froglogic GmbH Compiling the example application
#include "tst_textedit.h"
void TestTextEdit::tst_saveFile() { TextEdit textEdit; textEdit.fileName="/"; QVERIFY( !textEdit.fileSave() ); }
QTEST_MAIN(TestTextEdit);
Figure 4.8: Unit test for the TextEdit application, tst_textedit.cpp
To import the instrumentation result of this test into TextEdit’s instrumentation database, the following infrastructure is necessary:
1.A qmake project file with code coverage, configured identically to that of the TextEdit project.
2. A post-build rule, which automatically executes the test and collects the coverage information. 3. A unit test listener, which saves the code coverage data (and the test status—passed or failed) for every executed test into the unit test’s own instrumentation database.
4. A way to import the code coverage report into TextEdit’s instrumentation database.
The unit test will recompile along with textedit_v1/textedit.cpp. To make its results importable into the TextEdit instrumentation database, it is necessary that both executables, TextEdit and the unit test, are instrumented in exactly the same way. Since we use only the default instrumentation, this requirement is already fulfilled. Unfortunately, this is not everything we need to do. Squish Coco’s default behavior is to instrument only header and source files in the directory it is invoked, but here we need to instrument the TextEdit application’s sources in addition to the unit test. Therefore we must use another command line option to specify an additional path for files to instrument: --cs-include-path. As before, we do not want to have to remember these command line arguments every time, so we set them in the qmake project file, textedit_v1_tests.pro (see Figure 4.9, page 38). With these lines in the unit test project file, the qmake-generated Makefile will create the tst_textedit.exe executable which, when run, produces the execution report tst_textedit.exe.csexe. We then can use the CoverageBrowser to import this report into the file tst_textedit.exe.csmes.
Squish Coco 5.1.0 - 37 - froglogic GmbH Compiling the example application
HEADERS = ../textedit_v1/textedit.h tst_textedit.h SOURCES = ../textedit_v1/textedit.cpp tst_textedit.cpp
CodeCoverage { COVERAGE_OPTIONS = --cs-output=tst_textedit.exe COVERAGE_OPTIONS += --cs-include-path=../textedit_v1 COVERAGE_OPTIONS += --cs-exclude-file-abs-wildcard=*/qrc_*
QMAKE_CXXFLAGS += $$COVERAGE_OPTIONS QMAKE_CCFLAGS += $$COVERAGE_OPTIONS QMAKE_LFLAGS += $$COVERAGE_OPTIONS
QMAKE_CC=cs$$QMAKE_CC QMAKE_LINK=cs$$QMAKE_LINK QMAKE_CXX=cs$$QMAKE_CXX }
Figure 4.9: An extract from the unit test qmake project file
We can also execute the unit test automatically and import the execution report with a post-build rule. Squish Coco provides an extra command line tool, cmcsexeimport (see Chapter 25, page 162), which imports an execution report into an instrumentation database. The post-build rule first deletes any previous execution report, then executes the test itself, and finally imports the results into the application’s execution database, tst_textedit.exe.csexe (see Listing 4.10, page 38).
CodeCoverage { win32: MAINDIR=$$replace(PWD,"/","\\") !win32:MAINDIR=$$PWD
unix { QMAKE_POST_LINK = rm $$MAINDIR/tst_textedit_v1.exe.csexe ; QMAKE_POST_LINK += $$MAINDIR/tst_textedit_v1.exe ; QMAKE_POST_LINK += cmcsexeimport -m $$MAINDIR/tst_textedit_v1.exe.csmes \ -e $$MAINDIR/tst_textedit_v1.exe.csexe -t UnitTest } win32 { QMAKE_POST_LINK = del /F $$MAINDIR\\tst_textedit_v1.csexe & QMAKE_POST_LINK += $$MAINDIR\\tst_textedit_v1.exe & QMAKE_POST_LINK += cmcsexeimport -m $$MAINDIR\\tst_textedit_v1.exe.csmes \ -e $$MAINDIR\\tst_textedit_v1.csexe -t UnitTest } }
Figure 4.10: Post-build rules for the import of the execution report into the unit test’s instrumentation database
By default, the coverage data is imported without any information about the executed tests. Instead, an execution report called ‘UnitTest’ is created, which does not describe which test was executed or whether its execution was successful. To provide the missing information, we must use the CoverageScanner API and generate an execution report for each test that is executed. An example that shows how this is done is available in the chapter 42.2. In the example, the API is used in the following way:
Squish Coco 5.1.0 - 38 - froglogic GmbH Compiling the example application
• Two Squish Coco source files are added to the qmake project file (see Listing 4.11, page 39).
• The unit test class, TestTextEdit, inherits from TestCoverageObject instead of directly from QObject (see Listing 4.12, page 39).
HEADERS += testcoverageobject.h SOURCES += testcoverageobject.cpp
Figure 4.11: Including the CoverageScanner listener in textedit_v1_tests.pro
#include "testcoverageobject.h" #include "../textedit_v1/textedit.h" #include
class TestTextEdit : public TestCoverageObject { Q_OBJECT private slots: void tst_saveFile(); };
Figure 4.12: The TextEdit unit test header file, tst_textedit.h
The source code of the file testcoverageobject.cpp (see Listing 4.13, page 40) is simple to understand. The file adds a single cleanup() function to QTestLib, which is executed after each unit test item. The code between #ifdef __COVERAGESCANNER__ and #endif is only compiled when CoverageScanner is invoked2. This extra code creates a test name by combining the test class’s object name and the test function’s name. In the example, an execution item called unittest/TestTextEdit/tst_saveFile is generated. The slash is used to support the organization of tests in a tree view. The current test status (“PASSED” or “FAILED”) is also recorded and added to the execution report by the __coveragescanner_save() function.
2The symbol __COVERAGESCANNER__ is defined automatically by CoverageScanner and so does not need to be defined manually.
Squish Coco 5.1.0 - 39 - froglogic GmbH Compiling the example application
... void TestCoverageObject::cleanup() { cleanupTest(); #ifdef __COVERAGESCANNER__ QString test_name="unittest/"; test_name += metaObject()->className(); test_name += "/"; test_name += QTest::currentTestFunction(); __coveragescanner_testname(test_name.toLatin1()); if (QTest::currentTestFailed()) __coveragescanner_teststate("FAILED"); else __coveragescanner_teststate("PASSED") ; __coveragescanner_save(); #endif }
Figure 4.13: An extract from the TestCoverageObject’s source code
At this stage we could start CoverageBrowser, load the TextEdit instrumentation database, and import the unit test’s instrumentation database by clicking "File¡Import Unit Tests...". An even more convenient alternative is to use the cmmerge tool to automate this step. The cmmerge program is designed to import one instrumentation database’s execution report into another instrumentation database. This means that we can extend our post-build rules to use the cmmerge program to import the coverage information automatically from the unit test into the TextEdit program’s instrumentation database (see Listing 4.14, page 41).
Squish Coco 5.1.0 - 40 - froglogic GmbH Working with code coverage data
CodeCoverage { # Merge coverage database into TextEdit database unix { QMAKE_POST_LINK += ; QMAKE_POST_LINK += cmmerge -o $$MAINDIR/../textedit_v1/textedit.tmp \ -i $$MAINDIR/../textedit_v1/textedit.exe.csmes \ $$MAINDIR/./tst_textedit_v1.exe.csmes && QMAKE_POST_LINK += rm $$MAINDIR/../textedit_v1/textedit.exe.csmes && QMAKE_POST_LINK += mv $$MAINDIR/../textedit_v1/textedit.tmp \ $$MAINDIR/../textedit_v1/textedit.exe.csmes } win32 { QMAKE_POST_LINK += & QMAKE_POST_LINK += echo Merging unit test result into the main application & QMAKE_POST_LINK += cmmerge -o $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes \ -i $$MAINDIR\\..\\textedit_v1\\textedit.exe.csmes \ $$MAINDIR\\tst_textedit_v1.exe.csmes & QMAKE_POST_LINK += COPY /Y $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes \ $$MAINDIR\\..\\textedit_v1\\textedit.exe.csmes & QMAKE_POST_LINK += DEL /F $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes } }
Figure 4.14: Post build rules: merging instrumentation results into the TextEdit instrumentation database
With all these changes to the .pro file in place, we can once again build and run the unit test. Now the Coverage- Browser shows the fileSave() function to be 100% covered, with the execution list containing our three original manual tests and the single unit test (see Figure 4.15, page 41).
Figure 4.15: The execution list after all the tests have been executed
4.2 Working with code coverage data
The most common ways in which code coverage is used are for developers to use it to find untested code, and for managers to use it to produce test status reports (e.g., as diagrams). In addition to fully supporting the common use cases, Squish Coco also provides additional features which make it possible to go beyond these fundamentals and extend what can be achieved with code coverage. This will be discussed in the current subsection.
Squish Coco 5.1.0 - 41 - froglogic GmbH Working with code coverage data
4.2.1 Post mortem analysis
Recording each test’s coverage data makes it possible to compare their data to answer the question, “What does this test cover that the others do not?” This is particularly useful if just one test fails, since it can help us to identify which part of the code is involved.
To see how this works in practice, let us return to the TextEdit example. If we click ‘Save’, we will get an error message that no file name is defined. This is not very convenient for users—we should have designed TextEdit to handle this particular case by opening the ‘Save As. . . ’ dialog rather than by producing an error.
To identify where in the code this problem arises, we simply compare the ‘Save Clicked’ execution with all other executions that involve the ‘Save’ button. To do this we must first switch to the “Execution Comparison Analysis” mode (Menu: "Tools¡Execution Comparison Analysis"). Select the checkboxes in the “Reference” column for the “tst_saveFile” and “SaveAs clicked before Save clicked” tests. This will make the execution comparison symbol appears in front of the affected names (see Figure 4.16, page 42). In the “Executions” column, click on “Save clicked” checkbox.
Figure 4.16: The execution list being used to compare different executions
In this mode, the coverage analysis is based only on source code lines which are not executed by “tst_saveFile” and “SaveAs clicked before Save clicked”. This is why the overall coverage decreases to 1.29%: It means that “Save clicked” executes 1.29% more code than the selected tests. Using cmreport it is possible to generate a HTML report which displays the same information:
cmreport --csmes=textedit_v1/textedit.exe.csmes \ --html=textedit.html \ --section=execution \ --select-reference=".*tst_saveFile" \ --select-reference="SaveAs clicked before Save clicked"
If we now look at the source code itself, we will see that only two lines of the TextEdit::fileSave() function (see Listing 4.17, page 43) are not grayed: the lines which pop up the error message. These are the lines that must be modified to change the “Save” button’s behavior.
Squish Coco 5.1.0 - 42 - froglogic GmbH Working with code coverage data
bool TextEdit::fileSave() { if (fileName.isEmpty()) { QMessageBox::warning(this,tr("No file name specified"), tr("Save first your document using ’Save As...’ from the menu"), QMessageBox::Ok ); return false; }
QTextDocumentWriter writer(fileName); bool success = writer.write(textEdit->document()); if (success) textEdit->document()->setModified(false); return success; }
Figure 4.17: CoverageBrowser source view of the comparison of the “Save clicked” execution.
In this case, changing the fileSave() function is easy—we simply replace the QMessageBox::warning() call with a call to the fileSaveAs() method. (see Listing 4.18, page 43)
bool TextEdit::fileSave() { if (fileName.isEmpty()) return fileSaveAs();
QTextDocumentWriter writer(fileName); bool success = writer.write(textEdit->document()); if (success) textEdit->document()->setModified(false); return success; }
Figure 4.18: The TextEdit’s improved fileSave() function.
4.2.2 Evaluating the impact of a hot fix
Before committing a change or starting to test a hot fix, it is possible to estimate the impact of the code modification. CoverageBrowser is able to perform an analysis on the difference between two source sets and can list the tests that will be affected (and those which will not).
Start CoverageBrowser and load the modified TextEdit example’s instrumentation database. Now click on "Tools¡Compare with..." and select the original version of the TextEdit instrumentation database. Cover- ageBrowser will now display the source code like a text comparison application does (e.g., diff).
Click on "Tools¡Analysis of Modified Methods" to exclude all unmodified functions from the coverage analysis. In the TextEdit case, doing this will mean that only one function, TextEdit::fileSave(), will be treated as being instrumented since that is the only method we have changed (see Figure 4.19, page 44). This also
Squish Coco 5.1.0 - 43 - froglogic GmbH Working with code coverage data affects the statistic calculations since execution coverage statistics will now be limited to just this function. The test executions whose coverage statistic is not zero are the ones that are affected by the code modifications we have made. In our case we have: • “Save clicked”, • “SaveAs clicked before Save clicked” and • “tst_saveFile” (our unit test). The “Start and Exit” case has a coverage of 0% and so does not execute our modified code. It is for this reason no longer visible in the execution list. All entries in the ‘Execution’ column are struck through to inform us that these tests are not executed in the newest version and only present in the reference database. In other words, only the two manual tests and the unit test listed above must be re-executed to ensure that no regressions have been introduced by our code changes.
Figure 4.19: List of tests affected by a code modification
4.2.3 Black-box testing/distributed testing
Up to now we have done white-box testing, i.e. testing where we have access to the source code and which makes use of our knowledge of the code. It is also possible to use Squish Coco for black-box testing. In other words, we can still do code coverage analysis without having access to or even knowledge of the source code. If we use this approach, the generated instrumentation database will, of course, contain no source code.
To use black-box testing we must create a suitable instrumentation database by clicking "File¡Generate Black-Box Configuration...". This database, along with the TextEdit executable, can be given to the test team which can then use them with a simplified version of CoverageBrowser (see Figure 4.20, page 45). This version of
Squish Coco 5.1.0 - 44 - froglogic GmbH Working with code coverage data
CoverageBrowser only supports the importing and managing of execution reports since it does not have access to the application’s source code.
Figure 4.20: Black-box testing results as shown by CoverageBrowser
Once all the tests are finished, the black-box database can be merged into the original TextEdit instrumentation database using CoverageBrowser’s merge facilities (Menu: "File¡Merge with...").
4.2.4 Verifying if a bug fix is correctly tested
Often, when a small bug fix is made, the effects are very localized and leave most of the source code unchanged. In view of this, it is often unnecessary to retest the entire application with the whole test suite. Squish Coco makes it possible to avoid unnecessary testing. We can tell it to restrict itself to the source code that has changed between the original and fixed version of the application (see Chapter 4.2.2, page 43). This allows us to focus purely on the analysis of the fix. To achieve this, simply load the fixed application’s freshly generated instrumentation database (e.g., for the modified TextEdit application), and compare it with the earlier database for the unfixed version, using Squish Coco’s facility for analyzing modified functions.
Squish Coco 5.1.0 - 45 - froglogic GmbH Conclusion
Figure 4.21: Coverage of the patched function
We have done just such a comparison and the results are shown in in 4.21: The two tests, “Save clicked” and “Start and Exit”, cover 85% of the TextEdit::fileSave() function, the only method that was modified for our fix. From this we know exactly what additional testing is necessary to achieve 100% code coverage for our tests for the fixed version of the application. CoverageBrowser continues to display the list of missing tests (which are only executed using the first version of TextEdit) in strikeout style. This gives a hint of what remains for testing effort. Using cmreport it is possible to generate a HTML report which displays the same information:
cmreport --csmes=textedit_v2/textedit.exe.csmes \ --csmes-reference=textedit_v1/textedit.exe.csmes \ --html=textedit.html \ --section=execution
4.3 Conclusion
Squish Coco provides code coverage analysis which can be applied to all the usual testing techniques: unit, manual, and black-box testing. Squish Coco can easily be told to ignore the generated code produced by the Qt library’s tools (moc, qrc, and uic), so that only the code written by developers is instrumented. Test results can be collected into a database and can be used to evaluate how much code coverage our tests achieve and to show which statements are not currently tested. With this information we can target our testing efforts towards 100% test code coverage. In addition, Squish Coco makes it possible to see what effect a code modification would have in terms of test code coverage without having to test the entire application.
Squish Coco 5.1.0 - 46 - froglogic GmbH Conclusion
Overall, Squish Coco can help us target our tests to ensure that our applications have as much test coverage as possible, while avoiding or minimizing test coverage duplication. Furthermore, Squish Coco can help us see what effects changes to our code have on test coverage, so that we can adapt our test suites accordingly.
Squish Coco 5.1.0 - 47 - froglogic GmbH About Code Part III Coverage
Squish Coco 5.1.0 - 48 - froglogic GmbH Squish Coco’s code coverage analysis
Chapter Squish Coco’s code coverage analysis 5
This chapter describes the code analysis done by Squish Coco in detail. In order to analyze the code coverage of e.g. a test suite, it is necessary to compile a version of the application in which statements are inserted that record the execution of each part of the source code. The generation of such a modified version of a program is called instrumentation.
5.1 Coverage metrics supported by Squish Coco
Several kinds of code coverage are possible with Squish Coco. The most common are:
Statement Block Coverage: With this metric, Squish Coco verifies that all statements are executed. For this it groups the statements of the program to blocks. Statements belong to the same block if they are always executed together. (Statement blocks roughly correspond to the “blocks” of most programming languages.) The coverage of a program is the number of executed statement blocks in it, divided by the total number of blocks.
Decision (or Branch) Coverage: Here Squish Coco verifies that all statements are executed and all decisions have all possible results. The coverage of a program is the number of executed statement blocks and decisions divided by the total number of statements and decisions – where each decision counts twice, once for the true case and one for the false case.
Condition Coverage: This is like decision coverage, except that the decisions are split into elementary subexpressions (or conditions) that are connected by “and” or “or” operators. The coverage of a program is the number of executed statement blocks and conditions divided by the total number of them. Here each condition counts twice, which may result in a large number of possible outcomes in a complex decision.
Modified Condition/Decision Coverage (MC/DC): This is like condition coverage, but every condition in a decision must be tested independently to reach full coverage. For each condition there must be two executions that differ only in the results of this conditions, while
Squish Coco 5.1.0 - 49 - froglogic GmbH Coverage metrics supported by Squish Coco
the other conditions have the same result. In addition, it needs to be shown that each condition independently affects the decision. The coverage of a program is the number of executed statement blocks and of conditions that were tested independently divided by the number of statement blocks and conditions in the program.
Multiple Condition Coverage: In this coverage metric, all statements must be executed and all combinations of truth values in each decision must occur at least once to reach full coverage. The coverage of a program is the number of executed statement blocks and condition combinations divided by their total number in the program. Other, simpler coverage metrics are also supported.
Function coverage: With this metric, Squish Coco counts which functions were called and how often. “Functions” includes also the member functions of objects, as always with Squish Coco. The coverage of a program is then the number of functions that were called at least once, divided by the total number of functions. Relevance for safety standards: IEC 61508 highly recommends 100% structural test coverage of entry points (i.e. functions) for SIL 1, 2, 3 and 4.
Line coverage: Here the number of executed code lines is counted and how often each one of them was executed. Only lines that contain executable statements are instrumented, not those with pure declarations. The coverage of a program is then the number of executed lines divided by the number of instrumented lines. Line coverage is however an unstable coverage measure since it depends strongly on the way the code is formatted (see Chapter 5.6, page 61). We do not recommend line coverage.
5.1.1 Coverage metrics and safety standards
In many safety standards, specific coverage levels are required. • IEC 61508 is a standard for the functional safety of electrical and electronic programmable systems. • ISO 26262 is a standard for the functional safety of electrical and electronic systems in cars.
• EN 50128 is a standard for safety-relevant software in railway systems. • DO 178 is a safety standard for commercial software-based aerospace systems. The following table shows which coverage levels are required by these safety standards.
IEC 61508 ISO 26262 EN 50128 DO 178 Function Coverage (Entry Points) X Statement Block Coverage XXXX Decision Coverage (Branch Coverage) XXXX Modified Condition/Decision Coverage XXXX Multiple Condition Coverage X
Depending on the security levels the coverage requirement is either just recommended, highly recommended or required. More detailed information can be found at the end of the descriptions of the coverage metrics in the following sections.
Squish Coco 5.1.0 - 50 - froglogic GmbH Description of the coverage metrics
5.2 Description of the coverage metrics
In this section we describe the coverage metrics supported by Squish Coco in greater detail; the code that is inserted during the instrumentation process is described in more detail in Section 48, page 294. In the next sections, we will use the following function to illustrate the coverage metrics and the instrumentation process. It is written in C++; coverage with other languages it is similar.
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i<100) && (!found); ++i) {
5 if (i==50)
6 break;
7 if (i==20)
8 found=true;
9 if (i==30)
10 found=true;
11 }
12 printf("foo\n");
13 }
Figure 5.1: Example function for simple coverage metrics
5.2.1 Statement Block Coverage
The most elementary kind of instrumentation records the statements in a program that are executed when it runs. It is however not necessary to record the execution of every statement to get this information. If several statements form a sequence, it is enough to record how often the last statement is executed, since they all form a block that is either executed as a whole or not at all. Squish Cocotherefore inserts instrumentation statements only at the end of each block, and the resulting coverage metric is called Statement Block Coverage. In the following listing, the instrumented statements are underlined:
1 void foo()
2 {
3 bool found=false; // not instrumented
4 for (int i=0; (i<100) && (!found); ++i) {
5 if (i==50)
6 break;
7 if (i==20)
8 found=true;
9 if (i==30)
10 found=true;
11 }
12 printf("foo\n"); // not instrumented
13 }
Figure 5.2: Code coverage at statement block level
Squish Coco 5.1.0 - 51 - froglogic GmbH Description of the coverage metrics
One can see that not all lines are instrumented. The conditions of the if and for statements are not covered since they are no statements themselves. Some other statements need not to be instrumented because they are part of a block. In the example, these are the statements at lines 3 and 12: they belong to the block that begins at line 2 and ends with line 13. It is therefore enough to put an instrumentation point at line 13 to verify that all of them were executed.
Relevance for safety standards ISO 26262 highly recommends Statement Coverage for ASIL A and B. ASIL C and D recommend it as well but the more strict Branch Coverage and Modified Condition/Decision Coverage are highly recommended instead.
5.2.2 Decision Coverage
A more detailed coverage metric also records the values of the Boolean conditions in branch and loop statements, like if, while, for, etc. To reach full coverage, the decision in such a construct must have evaluated to true and to false at least once. This kind of code coverage is called Decision Coverage or sometimes Branch Coverage.
Decision coverage also handles switch statements. In them, full coverage is reached if every case label in the code was reached at least once during the execution of the program. Note that Decision coverage also include the coverage of statements, as in Statement Block Coverage. In the following listing, the conditions instrumented for Decision Coverage are displayed with a gray background. The instrumented statements are underlined as before. They are the same as in the previous listing.
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i<100) && (!found) ; ++i) {
5 if (i==50)
6 break;
7 if (i==20)
8 found=true;
9 if (i==30)
10 found=true;
11 }
12 printf("foo\n");
13 }
Figure 5.3: Code coverage at decision level
Relevance for safety standards ISO 26262 recommends Branch Coverage for ASIL A and highly recommends the method for ASIL B, C and D. EN 50128 recommends Branch Coverage for SIL 1 and 2. For SIL 3 and 4 this level is even highly recommended. DO 178 mandates that Decision Coverage should be satisfied with independence for Software Level A and B. IEC 61508 recommends Branch Coverage for SIL 1 and 2 and highly recommends this level for SIL 3 and 4.
Squish Coco 5.1.0 - 52 - froglogic GmbH Description of the coverage metrics
5.2.3 Condition Coverage
Often one wants to analyze the Boolean decision in the if, while, for, etc. statements in greater detail. For this one can use Condition Coverage. In this coverage metric, each decision is decomposed into simpler statements (or conditions) that are connected by Boolean operators like ’~’, ’||’ and ’&&’. Then for full coverage of the decision, each of the conditions must evaluate to true and to false when the program is executed.
Note that ’||’ and ’&&’ are shortcut operators and that therefore in a complex decision, not all conditions are always executed. An example is the code fragment ‘if (a || b)return 0;’. In it, the conditions are the variables a and b. To get them both evaluate to true and false, one has to run the code fragment once with a == true and b arbitrarily (since it is not evaluated), and also two times with a == false: in one run b is set to true, in the other run it is set to false. This is in contrast to the situation with decision coverage, where it is enough for full coverage that the whole decision, ’a || b’, evalutes to true and to false. With Condition Coverage, our example function is instrumented in the following way. As before, the instrumented Boolean expressions are gray:
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i<100) && (!found) ; ++i) {
5 if (i==50)
6 break;
7 if (i==20)
8 found=true;
9 if (i==30)
10 found=true;
11 }
12 printf("foo\n");
13 }
Figure 5.4: Code coverage at condition level
We can see here that the decision of the for statement has been split into two separately instrumented conditions.
Instrumentation of assignments CoverageScanner also instruments the assignment of Boolean expressions to a variable. Constant or static expressions are not instrumented. This is because their values are computed at compile time or only once during program initialization, so full coverage cannot be reached anyway. In the following listing, the instrumented Boolean expressions are again gray:
Squish Coco 5.1.0 - 53 - froglogic GmbH Description of the coverage metrics
1 void foo()
2 {
3 static bool a = x && y;
4 bool b = x && y;
5 bool c = b;
6 int d = b ?0:1;
7 }
Figure 5.5: Code coverage of assignments
We see that the statements at lines 3 and 5 are not instrumented – the first is evaluated at compile time and the second does not involve a Boolean operation.
Potential problems with Boolean overloading Since Squish Coco works on a syntactic level, it cannot distinguish between expressions that involve only Booleans and those with other data types that support Boolean operators – all of them are instrumented. The instrumentation tries to have no effect on the program, but sometimes this is not possible. One example are older versions of C#. They need to be instrumented with the option --cs-no-csharp-dynamic: Then the instrumented code converts the operands of a Boolean operator like ’||’ and ’&&’ first to Boolean, before the operator is applied. (Short circuit evaluation still works, of course.) This is different from the way in which an uninstrumented program would do it. In C# programs that are compiled with Microsoft® Visual Studio® versions before 2010, it is therefore necessary that both cast operators true and false are defined for the objects that are arguments of Boolean operators. For newer C# versions, the default settings of the CoverageScanner can be used and the instrumented code does not have this problem.
5.2.4 Multiple Condition Coverage
The properties of Multiple Condition Coverage and MC/DC become only visible when a program contains a complex decision with many conditions. Our previous example therefore cannot be used anymore. Instead, we will use the following program, which replaces some letters in a string with underscores: those between ‘a’ and ‘e’ (inclusive) in the alphabet, and those that follow the full stop at the end of the sentence. (There is no character after the full stop in the example text, but that is on purpose: We need some conditions that fail.)
Squish Coco 5.1.0 - 54 - froglogic GmbH Description of the coverage metrics
1 #include
2
3 int main()
4 {
5 char text[] = "The quick brown dog jumps over the lazy fox.";
6 7 for (char *p = text; *p; p++) { 8 if ((*p >= ’a’ && *p <= ’e’) || (p != text && *(p - 1) == ’.’)) 9 *p = ’_’; 10 }
11
12 puts(text);
13 return 0;
14 }
Figure 5.6: Example program for Multiple Condition Coverage and MC/DC
We will concentrate on the complex condition in line 8. When the coverage of this program is shown in the CoverageBrowser, one can move the mouse over that line and see the following table, which describes the coverage data. (Or one can click on the line and see a similar table in the "Explanation" window.)
*p >= ’a’ *p <= ’e’ p != text *(p - 1) == ’.’ Description TRUE FALSE TRUE TRUE never executed FALSE TRUE TRUE never executed TRUE FALSE TRUE FALSE executed 27 times by 1 test FALSE TRUE FALSE executed 9 times by 1 test TRUE FALSE FALSE never executed FALSE FALSE executed 1 time by 1 test TRUE TRUE executed 7 times by 1 test
Figure 5.7: Multiple Condition Coverage table for line 8
In this table, each line contains a combination of condition results. The first four columns contain the results of a single condition. Since C uses shortcut operators, not all conditions are shown executed in each row. Rows that were executed have a green background, rows that were not have a background in red. From the current table we see that three combinations of conditions were not executed and are missing from full condition coverage. To execute e.g. the condition combination in the second red line, it would be necessary to make the condition *p >= ’a’ false and the conditions p != text and *(p - 1)== ’’ true. To reach this, one could put a capital letter (for the first condition) behind the full stop (for the two others) in text.
Relevance for safety standards EN 50128 recommends MCC (or Modified Condition/Decision Coverage) for SIL 1 and 2. For SIL 3 and 4 this level is even highly recommended.
5.2.5 MC/DC
Modified Condition/Decision Coverage (MC/DC) is a variant of Multiple Condition Coverage that requires fewer tests. Its goal is to make sure that for each condition in a complex decision there are two executions that differ only in the result of that condition and the condition result.
Squish Coco 5.1.0 - 55 - froglogic GmbH Description of the coverage metrics
The condition table of MC/DC is a more complex version of the table for Multiple Condition Coverage:
*p >= ’a’ *p <= ’e’ p != text *(p - 1) == ’.’ Decision Description TRUE FALSE TRUE TRUE TRUE never executed FALSE TRUE TRUE TRUE never executed TRUE FALSE TRUE FALSE FALSE executed 27 times by 1 test FALSE TRUE FALSE FALSE executed 9 times by 1 test FALSE FALSE FALSE executed 1 time by 1 test TRUE TRUE TRUE executed 7 times by 1 test TRUE FALSE FALSE FALSE never executed
Figure 5.8: MC/DC table for line 8
Its large-scale structure is the same as with Multiple Condition Coverage, but there are some new features: • The column headers for the conditions and the decision have a colored background. If a condition has a green background, the MC/DC conditions for this condition are already fulfilled, otherwise it is red.
• The red rows contain combinations of truth values that did not occur when the program was executed. Executing one of them would increase the code coverage. • The light green rows contain combinations of truth values that did occur in the execution of the program. If they contribute to the coverage of a condition, they contain fields with a dark green background. To see which rows contributed to the coverage of a covered condition, search in the column that belongs to this condition for dark green fields. Any two rows with a dark green field contribute together to the coverage of the condition if one of the dark green fields has a TRUE entry and the other one a FALSE entry.
In the example, we see that the first condition, ‘*p >= ’a’’, is covered in two different ways: by the second green row together with the fourth green row, and also by the third and fourth green row together. The second condition, ‘*p <= ’e’’, is covered by the first and fourth green row together. One can also see that these two rows only differ in the second column and in the decision; the other columns contain the same results or at least one empty field. • The light red rows contain combinations of truth valued that were not executed either, but executing a single of them will not increase the code coverage. You may have noted that the table is sorted in a different way from that one for Multiple Condition Coverage: First there are the red rows, then the green rows and then the light red ones. This is always the case.
Relevance for safety standards ISO 26262 recommends MC/DC for ASIL A, B, C and highly recommends this method for ASIL D. EN 50128 recommends MC/DC (or Multiple Condition Coverage) for SIL 1 and 2. For SIL 3 and 4 this level is even highly recommended. DO 178 mandates that MC/DC should be satisfied with independence for Software Level A. IEC 61508 recommends MC/DC for SIL 1, 2 and 3 and highly recommends this level for SIL 4.
Squish Coco 5.1.0 - 56 - froglogic GmbH Display of the results
5.3 Display of the results
The CoverageBrowser is a graphical user interface program to display the analyzed results of the instrumentation. It uses a color coding to indicate the status of the statements. In this manual we use the same colors as in the program. There are two kinds of statements in an instrumented program. Some contain an instrumentation point, i.e. a piece of code inserted by Squish Coco which increments a counter when it is executed. If a line contains an instrumentation point, it is shown on a dark-colored background by the CoverageBrowser and in the HTML reports. For reasons of efficiency, not all statements contain instrumentation points. If a line does not contain an instrumenta- tion point but its coverage status can be inferred from other statements, it is shown on a light-colored background. The resulting color scheme is then:
• Lines in dark green and in light green have been executed.
• Lines in dark red and in light red have not been executed.
• Lines in orange contain boolean expressions that have been partially executed. These expressions occur in general either as part of control structures like while and if or as the right side of an assignment to a boolean variable. Boolean expressions are always instrumented, therefore a light orange background is not necessary. The output for our example function illustrates this:
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100) && (!found); ++i)
5 {
6 if (i==50)
7 break;
8 if (i==20)
9 found=true;
10 if (i==30)
11 found=true;
12 }
13 printf("foo\n");
14 }
Figure 5.9: Code coverage output
The for statement in line 4 is shown in orange since it contains the expression ’i<100’, which is only partially executed: it always takes the value true when the function is executed. The expressions ’i==50’ and ’i==30’ were also partially executed, and we can see from the way the statements ’break;’ and ’found=true;’ are colored that in both cases the condition evaluates to false. (In CoverageBrowser, the value of the condition is also shown in a tooltip.) Lines 3, 5, and 13 are not directly included in the coverage measurements and are therefore shown on a light background. Their coverage states are inferred by Coco from other statements that were (or were not) executed later. In our example, lines 3 and 13 must have been executed because the closing brace in line 14 has been executed, and line 3 has been executed because of line 12. Therefore all the lines with light background are shown in green.
Squish Coco 5.1.0 - 57 - froglogic GmbH Performance
5.4 Performance
The insertion of code during the instrumentation increases the code size and also affects performance of the instrumented application. It will use more memory and run slower. For non-conditional expressions, the instrumentation code is only a write instruction to a counter at a fixed memory location. However, for conditional expressions, more detailed analysis is needed and this is more computationally expensive. Overall, an instrumented application will be 60% to 90% larger and will run 10% to 30% slower. i Detailed measurements are available in Chapter 49, page 298.
5.5 Statistics
Some developers write more lines of code than others simply by using a particular coding style—for example by putting opening braces on a line of their own rather than on the same line as an if statement. Therefore Squish Coco uses by default a coverage metric that is not susceptible to such differences in coding style. Its calculations are based on the number of executed instrumented instructions compared with the total number of instrumented instructions.
Every instrumented simple statement (like return, break, the last instruction of a function, etc.) is recorded by a single instrumentation counter. A fully instrumented condition in an IF...THEN...ENDIF block uses has two instrumentation counters: one for the true case and one for the false case. If the code is only partially instrumented, only one condition is recorded (either the false case or the true case). The statistics itself depends of the type of instrumentation. It is in general not possible to compare code coverage at statement block level with that at decision level: having 80% coverage at decision level does not tell us anything about the coverage at statement block level, which could be bigger or smaller. We can only be sure that reaching 100% coverage is more difficult with condition coverage than with decision or statement block coverage. In our example code, the coverage is between 60% and 75%. The following table shows the details for each instrumentation type.
Code coverage type Instrumented Executed Coverage statement block (see Figure 5.10, page 59) 5 3 60% Decision partial (see Figure 5.12, page 60) 9 7 77% Decision full (see Figure 5.11, page 59) 13 9 69% Condition partial (see Figure 5.14, page 61) 12 9 75% Condition full (see Figure 5.13, page 60) 15 10 66%
In the examples below, the details of the calculations are displayed with subscripts. The first number in a subscript shows how many instrumentated statements were executed; the second is the number of instrumented statements in total.
Squish Coco 5.1.0 - 58 - froglogic GmbH Statistics
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100) && (!found); ++i)
5 {
6 if (i==50)
7 break;0/1 [not executed] 8 if (i==20)
9 found=true;1/1 [executed] 10 if (i==30)
11 found=true;0/1 [not executed] 12 }1/1 [executed] 13 printf("foo\n");
14 }1/1 [executed]
Figure 5.10: Code coverage statistics for full statement block instrumentation
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100) && (!found)2/2 [was false and true]; ++i) 5 {
6 if (i==50)1/2 [was false but not true] 7 break;0/1 [not executed] 8 if (i==20)2/2 [was false and true] 9 found=true;1/1 [executed] 10 if (i==30)1/2 [was false but not true] 11 found=true;0/1 [not executed] 12 }1/1 [executed] 13 printf("foo\n");
14 }1/1 [executed]
Figure 5.11: Code coverage statistics for full decision level instrumentation
Squish Coco 5.1.0 - 59 - froglogic GmbH Statistics
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100) && (!found)1/1 [was false]; ++i) 5 {
6 if (i==50)1/1 [was false] 7 break;0/1 [not executed] 8 if (i==20)1/1 [was false] 9 found=true;1/1 [executed] 10 if (i==30)1/1 [was false] 11 found=true;0/1 [not executed] 12 }1/1 [executed] 13 printf("foo\n");
14 }1/1 [executed]
Figure 5.12: Code coverage statistics for partial decision level instrumentation
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100)1/2 [was true but not false] && (!found)2/2 [was false and true]; ++i) 5 {
6 if (i==50)1/2 [was false but not true] 7 break;0/1 [not executed] 8 if (i==20)2/2 [was false and true] 9 found=true;1/1 [executed] 10 if (i==30)1/2 [was false but not true] 11 found=true;0/1 [not executed] 12 }1/1 [executed] 13 printf("foo\n");
14 }1/1 [executed]
Figure 5.13: Code coverage statistics for full condition level instrumentation
Squish Coco 5.1.0 - 60 - froglogic GmbH Problems with line coverage
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100)1/2 [was true but not false] && (!found)2/2 [was false and true]; ++i) 5 {
6 if (i==50)1/1 [was false] 7 break;0/1 [not executed] 8 if (i==20)1/1 [was false] 9 found=true;1/1 [executed] 10 if (i==30)1/1 [was false] 11 found=true;0/1 [not executed] 12 }1/1 [executed] 13 printf("foo\n");
14 }1/1 [executed]
Figure 5.14: Code coverage statistics for partial condition level instrumentation
5.6 Problems with line coverage
Line coverage is a natural metric which allows easily to see which lines of code are executed. But it is less accurate than the instrumentation at statement block level and its results rely on the developer’s coding style. The following example illustrates this problem:
int main() { if (true) return 1; foo(); return 0; }
Executing it would produce the following result:
int main() { if (true) return 1; // Executed foo(); // Not executed return 0; // Not executed }
This execution correspond to a coverage of 33%.
Since the first line of main contains two executed statements, splitting it in two increases the number of executed lines, and so the test coverage. So if we reformat the main function as follows,
int main() { if (true) return 1; foo();
Squish Coco 5.1.0 - 61 - froglogic GmbH Problems with line coverage
return 0; } the execution of this code results in a code coverage of 66%:
int main() { if (true) // Executed return 1; // Executed foo(); // Not executed return 0; // Not executed }
Another way to increase the coverage by reformatting is to hide an uncovered statement behind an executed one. To do it it is only necessary to write the whole code of this main function in one line:
int main() { if (true) return 1; foo(); return 0; }
This code has a line coverage of 100%:
int main() { if (true) return 1; foo(); return 0; // Executed }
This small example illustrate how the result depends on how the source code is formatted. Therefore, Squish Coco provides the line coverage as additional measurement to the decision and the condition coverage and so does not allow that a source code is only instrumented at line level.
Squish Coco 5.1.0 - 62 - froglogic GmbH Testing Methodologies
Chapter Testing Methodologies 6
This chapter discusses the various testing strategies that can be carried out so as to make the best use of the Squish Coco tools.
6.1 Coverage Hits vs. Counts
If we want to set very stringent standards for an application to ensure that it is of the highest possible quality, executing just portions of its code, or executing code just once, is insufficient. To address this issue the Squish Coco tools provide two possible approaches:
Code Coverage Count: We can require that each portion of instrumented code is executed a minimum number of times. This can be achieved by adding to the instrumented code so that it counts each execution.1 Naturally, setting such minimums, and the extra bookkeeping itself, can make the application consume more memory and run a bit slower. An important disadvantage of this approach is that loops in the source code end up having a high count with only a single execution, so counting this way is not always useful.
Test Coverage Count: We can require that each portion of instrumented code is executed by a minimum number of test items. This avoids the code coverage count disadvantage regarding loops, since by setting a minimum number of test executions for each portion of instrumented code, we give equal value to loops, recursions, and to instructions that are executed only once.
6.2 Testing Strategies
Squish Coco can be adapted to fit in with many different testing strategies, from unit tests during early development to complete product validation (e.g., acceptance tests). CoverageBrowser supports the merging of code coverage results at each stage of testing and so provides a precise overview of the software’s overall quality.
1This can be done using CoverageScanner’s --cs-on command line option.
Squish Coco 5.1.0 - 63 - froglogic GmbH Strategies
6.2.1 Manual White Box Tests
White box testing (a.k.a. clear box testing, glass box testing or structural testing) uses an internal perspective of the system to design test cases based on internal structure. It requires programming skills to identify all paths through the software. The tester chooses test case inputs to exercise paths through the code and determines the appropriate outputs.
Definition from Wikipedia
Interactive white box tests are easy to do using Squish Coco: once the application has been compiled with Cov- erageScanner, the test engineer can execute the it, and after each execution they can import the execution report into CoverageBrowserfor analysis.
6.2.2 Manual Black Box Tests
Black box testing takes an external perspective of the test object to derive test cases. These tests can be functional or non-functional, though usually functional. The test designer selects valid and invalid input and determines the correct output. There is no knowledge of the test object’s internal structure.
Definition from Wikipedia
Black box tests assume that test engineers have no knowledge of the application’s internals—they work purely in terms of inputs and the corresponding expected outputs (whether of data or of error reports for invalid input). For this kind of testing it is possible to follow the same pattern as for white box testing, but without providing the test engineer with the instrumentation database (.csmes file) generated by CoverageScanner. The application itself will generate an execution report which can be analyzed after the testing cycle. However, this approach has the disadvantage that test engineers cannot manage their own execution reports (e.g., to add comments). Squish Coco has a facility which makes black box testing possible without disadvantaging test engineers. The CoverageBrowser tool can generate an instrumentation database (.csmes file) which does not contain any source code or source browsing information. This allows test engineers to manage the list of tests and to view the statistics, but it does not provide any source level coverage information.
6.2.3 Unit Tests
Unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit could be an entire module but is more commonly an individual function or procedure. In object-oriented programming a unit is often an entire interface, such as a class, but could be an individual method.
Definition from Wikipedia
CoverageScanner supports the saving of the code coverage data generated by each unit test using its library. (see Chapter 34.2, page 190) As long as all objects are well compiled2, it is possible to merge the unit tests instrumentation database into those of the real application. And so see the code coverage of the unit tests on the main application.
2Well compiled means that a source file is not be compiled twice or at least it is not be compiled with different preprocessor options.
Squish Coco 5.1.0 - 64 - froglogic GmbH Strategies
6.2.4 Automatic Tests
Squish Coco allows us to annotate the execution report with the execution name and state. This does not require any specific library—the information can be inserted directly by editing the execution report before and after the execution of a test item. This makes it possible to integrate the report into an automatic testing framework or for use with a test executing script. (see Chapter 34.1, page 189)
Squish Coco 5.1.0 - 65 - froglogic GmbH Code metrics
Chapter Code metrics 7
This chapter explains the code metrics that are supported by Squish Coco.
7.1 eLOC – Effective lines of code
The eLoc metric measures the effective number of lines in a piece of code. An effective line is a line which contains statements that produce executable code. When computing effective lines of code, Squish Coco ignores • empty lines, • comment lines, and • declarations. This metric is more precise than simply counting the number of lines in a source code project since it skips non-functional code.
7.2 McCabe metric – Cyclomatic complexity
The cyclomatic complexity is a code complexity measure proposed by Thomas J. McCabe, Sr. It is a strictly positive number that expresses the complexity of a program. A function with a cyclomatic complexity higher than 10 is usually considered to be difficult to maintain: It contains too many branches, switch/case statements or loops.
7.2.1 Definition
In structured programming languages like C++ and C#, the McCabe complexity of a piece of code is
McCabe = Branchings + Functions.
Squish Coco 5.1.0 - 66 - froglogic GmbH McCabe
In this equation, Branchings is the number of binary branch statements (see below), and Functions is the number of functions in the code. The binary branching statements in the code are counted in the following way:
• An “if (...)... else ...” statement counts as one binary branch, and also a single “if (...)...” statement: Both contain a choice between two different execution paths.
• A loop construct, like “while (...)... ” or “for (...)... ”, also counts a one binary branch statement, since it contains a condition that offers the choice to stay in the loop or exit it.
• A“switch (...){ case ...: ... }” construct with n case labels counts as as n − 1 binary branches because it can be simulated by n − 1 “if (...)... else ...” statements.
7.2.2 Properties
The McCabe complexity has a several properties which distinguish it from eLOC: • McCabe is independent to the number of source lines of a function: It does not matter how many statements are present in the code, only how many branches are into it.
• The metric is completely independent of the coding style. Reformatting the code does not affect it. This is a big difference to eLoc and other metrics that measure the number of lines of code.
• The metric has a linear growth with the function complexity. For example, adding an “if (...)... else ...” statement somewhere in a function increments the metric by one. It does not matter if the selection statement is nested or at the beginning of the function and it does not grow exponentially.
• The McCabe metric is easy to interpret for a single function. But its value for a source file, a class or an application is difficult to interpret. The reasons for this is that function calls are not represented in the flow diagram. The McCabe metric is computed independently for each function and it is as if there were no interactions between the functions present in the source code. • The McCabe metric is not adapted to measure the complexity of the software architecture. Or, to say it differently, it does not work on a complex C++/C# design patterns which splits the complexity over several classes. • And, like all metrics, the McCabe metric measures only the code complexity and not the complexity of the data structures.
7.2.3 eLOC vs McCabe – Which metric to choose?
In general, for measuring the complexity of a function, the McCabe approach is more useful because the measurement is only dependent of the structure of the code. The code style and comments has no impact on it. But, as soon as a complete file or a class is analysize, even it is difficult to interpret. The reason is that the McCabe value grows with the amount of functions in a piece of code no matter if the functions in a class are complex or not. In this case, the number of effective lines of code is certainly easier to interpret.
7.2.4 Variants of McCabe metrics
Squish Coco generate alternate variants of McCabe metrics to handle the complexity of the switch/case statement.
Squish Coco 5.1.0 - 67 - froglogic GmbH McCabe
7.2.4.1 McCabe with cases grouped
McCabe is a metric which rely on an statement graph but this graph is in general not unique. Specially for switch/ case, the way to handle consecutive cases is undefined. Let take an example:
1 switch( a )
2 {
3 case 0:
4 case 1:
5 case 2:
6 return true;
7 default:
8 return false;
9 }
The McCabe complexity of this statement is 4 if we consider that each case is a separate branch. But it is also allowed to consider that ’case 0:’, ’case 1:’ and ’case 2:’ are in fact on branch of the statement graph. In this case the complexity of the whole code would only be 2. Squish Coco support this interpretation of McCabe and call this metric “McCabe grouped cases”.
7.2.4.2 McCabe with condensed switches
Many developer are using a switch/case statement to do some very simple computations which are not really complex. But the usage of this statement have then the consequence that the function complexity is then higher as the feeling of the developers. For example:
1 switch( day )
2 {
3 case 0: return "Monday";
4 case 1: return "Tuesday";
5 case 2: return "Wednesday";
6 case 3: return "Thursday";
7 case 4: return "Friday";
8 case 5: return "Saturday";
9 case 6: return "Sunday";
10 }
The McCabe complexity is 7 but this code is not really more complex as a single if...then...else... statement. To handle thjis use case, Squish Coco provides a metric called “McCabe condensed switches” which considers that a switch/case has a complexity of 1 regardless of the number of cases. (So in this case, the complexity of this sample is 1)
“McCabe condensed switches” is not conform to the definition of McCabe and should for this ! reason not be used if analysis the McCabe metric is required for a project.
Squish Coco 5.1.0 - 68 - froglogic GmbH CoverageScanner Part IV Reference
Squish Coco 5.1.0 - 69 - froglogic GmbH Introduction
Chapter Introduction 8
CoverageScanner is a C++ program that—in effect—replaces the usual compiler. It uses the native compiler’s preprocessor program to process the source code, then inserts instrumentation code into the preproecessed program, and at the end it compiles and links the project file, in the same way as in a normal build. CoverageScanner’s instrumentation process consists of: 1. declaring a static table that contains the of instrumentation results.
2. generating a library which produces an execution report when the program terminates or receives a UNIX® signal or a Microsoft® Windows event. 3. adding the instrumentation code itself for each relevant statement or Boolean expression. The compiler, linker and preprocessor that are used are from the native environment (gcc, Visual Studio, etc.). The Squish Coco tools are used transparently and the developer has only to prepend ’cs’ to the name of the compiler executable to activate the code coverage1. For example, using ’csgcc’ instruments the source code and uses the ’gcc’ compiler to generate objects. Another way to achieve coverage is to add a wrapper directory to the search path. It contains compiler wrappers with the same name as the native tools. They are disabled by default but can be switched on with help of the COVERAGESCANNER_ARGS environment variable.
8.1 Invoking CoverageScanner
CoverageScanner works like a pre-processor which calls after instrumenting a source file the native compiler (Microsoft® cl.exe, GNU g++, . . . ). Several invocation method are possible: 1. Using the dedicated CoverageScanner executable for a specific compiler: CoverageScanner executable for a specific compiler is called to cs+’native compiler name’ or alternatively ’native compiler name’+-cs. (example: cl.exe is the Microsoft® C++ compiler and the corresponding CoverageScanner executable is cscl.exe or cl-cs.exe).
1 For Microsoft® Visual Studio®, an additional wrapper is provided which directly uses the Microsoft® ’cl’ and ’link’ (see Chapter 41.11, page 230)
Squish Coco 5.1.0 - 70 - froglogic GmbH Invoking CoverageScanner
CoverageScanner aims to work then as the native compiler with the difference that the code get instrumented. Example:
csg++ source.cpp -o application.exe
2. Using CoverageScanner compiler wrapper and activating the instrumentation using --cs-on in the command line arguments: Squish Coco provides also a replacement of the native compiler called wrapper which extends its command line argument to support the code coverage feature. To use it it is necessary to add at the first position in the PATH variable the path of CoverageScanner wrapper. The instrumentation get activated, when adding --cs-on to the command line argument. Example:
export PATH=/opt/SquishCoco/wrapper/bin:$PATH g++ --cs-on source.cpp -o application.exe
Or, without modifying globally the PATH variable:
PATH=/opt/SquishCoco/wrapper/bin:$PATH g++ --cs-on source.cpp -o application.exe
3. Using CoverageScanner compiler wrapper and activating the instrumentation through COVERAGESCANNER_ARGS environment variable: It is also possible to set the command line arguments of the wrapper using the COVERAGESCANNER_ARGS environment variable (see Chapter 9.1, page 72). In this case it is no longer necessary to modify the command line arguments. Example:
export PATH=/opt/SquishCoco/wrapper/bin:$PATH export COVERAGESCANNER_ARGS=’--cs-on’ g++ source.cpp -o application.exe
Or, to instrument a complete project built using GNU make:
COVERAGESCANNER_ARGS=’--cs-on’ PATH=/opt/SquishCoco/wrapper/bin:$PATH make all
4. Using --cs-compiler command line option of CoverageScanner: --cs-compiler specifies which native compiler should be used. All other command line arguments, except those specific to CoverageScanner, are passed through to the native compiler. Example:
coveragescanner --cs-compiler=g++ source.cpp -o application.exe
i The native compiler path needs to appear in the PATH variable, after the path to the compiler wrappers. This is necessary for finding the location of the native compiler after insertion of the instrumentation code.
Compiler support is done with the help of a profile. It contains declarations which make it possible to adapt CoverageScanner to C, C++ and C# preprocessors, compilers and linkers. For a description of the compilers that are currently supported see Chapter 47, page 292. For a description of the profile syntax see Chapter 50, page 301
Squish Coco 5.1.0 - 71 - froglogic GmbH Command Line Arguments
Chapter Command Line Arguments 9
Syntax:
cshcompileri [hcs-optionsi] hcompiler argumentsi coveragescanner --cs-compiler=hSTRINGi [hcs-optionsi] hcompiler argumentsi coveragescanner --cs-profile=hSTRINGi [hcs-optionsi] hcompiler argumentsi where hcompileri: One of the compilers supported by Squish Coco. Calling cshcompileri results in a call of CoverageScanner as a wrapper of hcompileri. hcs-optionsi: One or more of the options that are described in the following sections. hcompiler argumentsi: All the other options. They are copied to the command line of the compiler when it is called by Coverage- Scanner.
For the use of --cs-compiler and --cs-profile see Chapter 9.2.9, page 83. i The command line arguments are prioritized by their order in the command line: the latest entered option has a higher priority than the first one. Example: cscl --cs-include=foo.h --cs-exclude=foo.h will exclude the header file foo.h from the coverage analysis.
9.1 Environment variables
CoverageScanner also reads options from the environment variable COVERAGESCANNER_ARGS. The options must be separated by spaces or tab characters (‘ ’ or ‘\t’). If both command line arguments are present and the variable COVERAGESCANNER_ARGS is set, the command line options take precedence over the options in the environment variable. This variable should only be set temporarily, e.g. in a script, because otherwise it would influence all builds.
Squish Coco 5.1.0 - 72 - froglogic GmbH List of options
The location of the temporary files that CoverageScanner generates can be changed by setting the environment variable SQUISHCOCO_TEMP_DIR to the path to another directory (see Chapter 46.3, page 290).
9.2 List of options
9.2.1 Coverage methods
For a description of the coverage methods, see Part III, page 49.
--cs-off: Disable all code coverage analysis. CoverageScanner then calls the compiler or linker unchanged.
--cs-on: Enable code coverage analysis. This is the default if CoverageScanner is called directly or as a wrapper that starts with cs, like csgcc. If the wrapper has however the same name as the program (e.g. gcc), code coverage is by default switched off.
--cs-off=hpatterni: Disable all code coverage analysis when compiling a source with an absolute file name that matches the wildcard expression hpatterni or if the string hpatterni is identical to a command line argument. CoverageScanner then calls the compiler or linker unchanged.
--cs-on=hpatterni: Enable all code coverage analysis when compiling a source with an absolute file name that matches the wildcard expression hpatterni or if the string hpatterni is identical to a command line argument. It is possible to have several --cs-on=hpatterni and --cs-off=hpatterni options. The last option that is matching decides whether code coverage analysis is enabled or disabled.
If several sources are compiled with the same compiler command (as it occurs with C#), disabling ! the coverage analysis for one source file with --cs-off=hpatterni will disable the analysis for all others.
--cs-hit: Set the instrumentation mode to hit instrumentation. It then only matters whether an instrumentation point (a line, a condition or a block) was executed at least once, not how many times.
--cs-count: Set the instrumentation mode to count instrumentation. (Default) With this mode, the number of times an instrumentation point is executed is counted.
--cs-statement-block: Enable statement block coverage.
--cs-decision: Enable statement block and decision coverage.
--cs-condition: Enable statement block, decision and condition coverage. (Default)
--cs-mcc: Enable Multiple condition coverage.
Squish Coco 5.1.0 - 73 - froglogic GmbH List of options
--cs-mcdc: Enable MC/DC coverage.
--cs-function: Enable function coverage. (Default)
--cs-no-function: Disable function coverage instrumentation.
--cs-line: Enable line coverage. (Default) If this option is set, decision and condition coverage are automatically enabled too and full instrumentation mode (--cs-full-instrumentation) is switched on.
--cs-no-line: Disable line coverage. The other instrumentation modes that are automatically enabled by --cs-line are not disabled.
Summary of the effects of the options for coverage metrics Most of the command line options that switch a certain coverage metric on also enable the data collection for other metrics. The following table shows which option enables which metrics.
Sometimes, the coverage measurements are also influenced by the option --cs-partial-instrumentation. For each of them, the table has two lines. The second line shows what happens when --cs-partial-instrumentation is present too. It is marked “(+partial)” in the table. Function Line Statement Decision Condition MC/DC MCC CoverageScanner Flags Block --cs-statement-block • • --cs-decision • • • • --cs-decision (+partial) • • • --cs-condition • • • • --cs-condition (+partial) • • • --cs-mcc • • • • • • • --cs-mcdc • • • • • • •
9.2.2 Fine-tuning of the instrumentation
--cs-no-exceptions: Do not instrument the catch block of a try... catch statement.
--cs-no-assignments: Do not instrument Boolean expressions in assignments.
--cs-no-execution-time: Disable the usage of performance measurements of the application to instrument. (By default, execution time measurement is enabled.)
--cs-execution-time: Enable the usage of performance measurements of the application to instrument.
--cs-no-function-profiler: Disable Squish Coco profiler.
Squish Coco 5.1.0 - 74 - froglogic GmbH List of options
--cs-function-profiler=option: Enable Squish Coco profiler. ’option’ can be on of the followings: all: instrument all functions. skip-trivial: Disable Squish Coco profiler for function having only one statement.
--cs-no-returns: Do not instrument Boolean expressions in return statements.
--cs-combine-switch-cases: With this option, decision and condition coverage do not distinguish between case labels that lead to the same code. This means that in the following code, the cases i == 3 and i == 4 are not distinguished. If the option is not set, the two cases are counted separately.
switch (i) { case 3: case 4: i = 0; break; ... }
The option is valid for C++ and C#.
--cs-partial-instrumentation: Suppression of redundant conditions. Setting this option also disables line coverage. The option influences the instrumentation of if statements without an else part, like
if (cond) { i += 1; }
Without the option, an invisible else branch is created and its coverage is measured too. If the option is set, this is no longer done and the coverage of the case in which cond is false is no longer measured. The instrumented code then becomes slightly smaller, which may be necessary on small embedded systems.
--cs-full-instrumentation: No suppression of redundant conditions. (Enabled by default)
--cs-boost: Specific handling of Boost library constructs. Currently this only means that the code behind the macro BOOST_FOREACH is not instrumented. (Enabled by default.)
--cs-no-boost: No specific handling of Boost library constructs.
--cs-qt4: No instrumentation for class members generated by moc for the Qt4 and Qt5 libraries (Q_OBJECT, qt_metacall , qt_metacast, . . . ) . The code behind the macro Q_FOREACH stays also uninstrumented. Qt signals remain instrumented. (Enabled by default.)
--cs-no-qt4: No specific handling of the Qt4 and Qt5 libraries.
--cs-qt3: No instrumentation for class members generated by moc for the Qt3 library( Q_OBJECT, className, tr, . . . ). Qt signals remain instrumented. (Enabled by default.)
Squish Coco 5.1.0 - 75 - froglogic GmbH List of options
--cs-no-qt3: No specific handling of the Qt3 library.
--cs-no-annotations: No extraction of annotations (comments and manual validations) from the source files.
--cs-no-purecov-annotations: No extraction of purecov’s annotations from the source files.
--cs-vs2010-lambda: Instrument the expression in the return statement of a lambda function. This brings a better support of lambda expression on Microsoft® Visual Studio® 2010. The option is only valid for C++11 compilers and Microsoft® Visual Studio® 2010. With a C compiler or an older C++ compiler, this option has no effect.
--cs-exit-function=hfunctioni: Declare hfunctioni as the name of an exit function. An exit function is a command that ends the control flow in a function, like a return statement or the call of the exit() function, or a function that always throws an exception. Any code directly after such a statement is then unreachable. With this option, it is possible to inform Squish Coco that there are user-defined functions that have the same effect as exit(). The option can be repeated; then several exit functions are declared.
--cs-no-line-directive: Remove all #line directives from the code generated by the preprocessor for C++and C#. The preprocessor is called by the CoverageScanner before instrumenting a file; afterwards the original compiler is called. On some systems, compiling a file this way causes problems that do not occur when the compiler is called directly. Removing the #line statements then sometimes helps.
--cs-counter-size=hinti: Size of the internal execution counters. Possible values are 1 (byte), 4 (32bit integer) or 8 (64bit integer). The counters are used inside the instrumented program to count the number of times an instrumentation point has been executed. Large internal counters prevent counter overflows, smaller counters may be necessary on small embedded systems to save space. The default value is 8.
--cs-mcc-max=hsizei | --cs-mcdc-max=hsizei: Set the maximal size of the instrumentation tables that are used for Multiple Condition coverage or MC/DC. The default for hsizei is 8000. For each decision, a truth table is generated under Multiple Condition coverage and MC/DC. hsizei is the maximal number of lines in the table. If a decision requires a truth table that is longer than hsizei, Squish Coco falls back to condition coverage for that decision.
9.2.3 File and path inclusion
The following options specify which files are included in the code coverage. The options apply to files on the command line and to those that are specified on #include statements.
One can use wildcards and regular expressions. In contrast to the common usage, the ‘*’ in wildcard expressions matches also the path separation characters. The syntax of regular expressions is described in Section 9.4, page 84. Options accumulate. If more than one of the file inclusion/exclusion option is present, the last one that matches a given file determines whether it is included or excluded. In the next options, name of the file in question is first converted to an absolute path.
Squish Coco 5.1.0 - 76 - froglogic GmbH List of options
--cs-include-file-abs-wildcard=hpatterni: Include a file in the coverage analysis if its absolute path matches the wildcard expression hpatterni.
--cs-exclude-file-abs-wildcard=hpatterni: Exclude a file from the coverage analysis if its absolute path matches the wildcard expression hpatterni.
Example: --cs-exclude-file-abs-wildcard=*.h excludes all files with the extension .h.
--cs-include-file-abs-regex=hpatterni: Include a file in the coverage analysis if its absolute path matches the regular expression hpatterni.
--cs-exclude-file-abs-regex=hpatterni: Exclude a file from the coverage analysis if its absolute path matches the regular expression hpatterni.
Example: --cs-exclude-file-abs-regex=^.*\.h$ excludes all files with the extension .h.
--cs-include-path=hpathi: Include all files in the directory hpathi and its subdirectories in the coverage analysis. Wildcards or regular expressions cannot be used. Example: --cs-include-path=c:\include includes all files located in c:\include.
--cs-exclude-path=hpathi: Exclude all files in the directory hpathi and its subdirectories from the coverage analysis. Wildcards or regular expressions cannot be used. Example: --cs-exclude-path=c:\include excludes all files located in c:\include and its subdirectories.
Examples for complex inclusion patterns:
• --cs-exclude-file-abs-wildcard=* --cs-include-file-abs-wildcard=*.cpp instruments only .cpp files.
• --cs-exclude-file-abs-wildcard=* --cs-include-file-abs-wildcard=*.cpp --cs-include-file-abs-wildcard=*.cxx instruments only .cpp and .cxx files.
• --cs-exclude-file-abs-wildcard=*/3rdparty/* does not instrument any files from the directory 3rdparty.
• --cs-exclude-file-abs-wildcard=*/3rdparty/* --cs-include-file-abs-wildcard=* instru- ments all files: The first filter rule has no effect.
Relative paths The following options refer to the name under which a file occurs in an #include statement or on the command line.
File paths and wildcard expression are converted to a canonical form which does not contain ‘.’, ‘..’, or a duplicated path separation character. (A ‘..’ at the beginning is however not removed.) On Microsoft® Windows, characters are also converted to lower case and every slash (‘/’) is replaced with a backslash (‘\’). Regular expressions are untouched. It is e.g. necessary to ensure that they do not contain duplicated path separators.
--cs-include-file-wildcard=hpatterni: Include a file in the coverage analysis if its normalized path matches the wildcard expression hpatterni.
Example: --cs-include-file-wildcard=../include/*.h includes all files with the extension .h located in the directory ../include/ and its subdirectories.
--cs-exclude-file-wildcard=hpatterni: Exclude a file from the coverage analysis if its normalized path matches the wildcard expression hpatterni.
Squish Coco 5.1.0 - 77 - froglogic GmbH List of options
--cs-include-file-regex=hpatterni: Include a file in the coverage analysis if its normalized path matches the regular expression hpatterni. Example: --cs-include-file-regex=^\.\./include/[^/]*\.h$ includes all files with the extension .h located in ../include/.
--cs-exclude-file-regex=hpatterni: Exclude a file from the coverage analysis if its normalized path matches the regular expression hpatterni.
Relative file names are computed relatively to the current working directory. For example, depending on the working directory one has to use different parameters in order to inhibit the instrumentation of a file src/test.c: If one is outside the directory src, one may use
$ csgcc src/test.c --cs-exclude-file-wildcard=src/test.c but when switching to src, the parameters must change:
$ cd src $ csgcc test.c --cs-exclude-file-wildcard=test.c
Setting inclusion/exclusion expressions on absolute paths should be preferred on complex build ! systems, since modern build system often change the current working directory. Working with absolute paths make it easier to set global rules for a project.
9.2.4 Function inclusion
--cs-include-function-wildcard=hstringi: This command line option enables the user to include a function in the coverage analysis. The function name is specified using a wildcard expression and must include for C++ code the class name and namespace. Example: --cs-include-function-wildcard=C::* includes all members of the class C. --cs-exclude-function-wildcard=hstringi: This command line option enables the user to exclude a function from the coverage analysis. The function name is specified using a wildcard expression and must include for C++ code the class name and namespace. Example: --cs-exclude-function-wildcard=*::Get* excludes all members of any classes which starts with Get.
--cs-include-function-regex=hstringi: This command line option enables the user to include a function in the coverage analysis. The function name is specified using a regular expression and must include for C++ code the class name and namespace. (see chapter 9.4, page 84 for the syntax)
--cs-exclude-function-regex=hstringi: This command line option enables the user to exclude a function from the coverage analysis. The function name is specified using a regular expression and must include for C++ code the class name and namespace. (see chapter 9.4, page 84 for the syntax)
If more than one of these inclusion/exclusion option is present, the last one that matches a given function determines whether it is included or excluded. The following option works indepentent of the previous ones:
--cs-record-empty-functions: Enable instrumentation of functions with an empty body. (By default this is disabled.)
Squish Coco 5.1.0 - 78 - froglogic GmbH List of options
9.2.5 Debugging
--cs-verbose-source-lines=hinti: Maximum number of source code lines dumped in an error message.
--cs-verbose=hstringi: Generate verbose debug messages. Possible options are: cmd: During the build, CoverageScanner displays the command line arguments of the programs it exe- cutes. This includes especially the command line arguments of the compiler when it translates the instrumented source code. build: During the build, the instrumentation by CoverageScanner is traced. This includes e.g. the informa- tion about which files are instrumented and which are excluded from instrumentation. This option automatically implies --cs-verbose=cmd. api: When the instrumented program is run, the calls to the CoverageScanner library are traced. By default, the output is written to stderr.
--cs-verbose-file=hfilei: Define the file used for generating a CoverageScanner log file. This option has no effect when --cs-verbose is not used.
--cs-keep-instrumentation-files: Do not delete temporary instrumentation files. They will be left under random names in the Squish Coco temporary directory (see Chapter 46.3, page 290).
--cs-warnings=hstringi: Instrumentation warning behaviour: none: Warnings will not be displayed. display: Warnings will be displayed via stderr. error: Warnings will trigger a compilation error.
--cs-compilation-warnings=hstringi: Compilation warning behaviour: none: Warnings will not be displayed. display: Warnings will be displayed via stderr. error: Warnings will trigger a compilation error.
--cs-annotation-warnings=hstringi: Source code annotations warning behaviour: none: Warnings will not be displayed. display: Warnings will be displayed via stderr. error: Warnings will trigger a compilation error.
9.2.6 Execution report
--cs-output=hstringi: Set the file name for the execution report. The extension .csexe is added automatically. By default, hstringi has the value “%F” and CoverageScanner uses the application name without the path as name for the .csexe file.
Squish Coco 5.1.0 - 79 - froglogic GmbH List of options
The value of hstringi may be an absolute path, like “/my/dir/%F”. The directories in this path must already be present when the instrumented program runs, since CoverageScanner does not create them by default.1 Otherwise, hstringi is interpreted as a path relative to the working directory at the time the program starts. hstringi may contain several format specifiers. The values of the following specifiers are computed at compile time: %T: Current local time as an integer. (Example: 163355 for 16h33 and 55s) %D: Current local date as an integer. (Example: 20140130 for January 30, 2014) %F: Application/library file name without path. %A: Absolute application/library file name. %B: Application/library file name without path and extension. %P: Application/library file path. %/: File path separator (on Unix a slash, ‘/’, and on Windows a backslash, ‘\’). The values of the following format specifiers are computed at runtime: %c: Current working directory. The value is is computed before the main() function is started. If the current working directory changes, this path remains identical. %p: Process identifier. If for any reason it cannot be computed, “%p” will be replaced with an empty string. %t: Current local time. If for any reason it cannot be computed, “%t” will be replaced with an empty string. %d: Current local date. If for any reason it cannot be computed, “%d” will be replaced with an empty string.
--cs-file-name-format-specifier: Enable the file format specifiers support.
--cs-no-file-name-format-specifier: Disable the file format specifiers support.
--cs-lock-csexe: Create a lock file which lock the access of the .csexe file during its generation.
--cs-nolock-csexe: Inverse option to --cs-lock-csexe. The preprocessor output is transmitted to CoverageScanner through a temporary file.
--cs-dump-on-signal=hsignali: Generate an execution report when a specific signal was received. The signal can be specified by a number from 1 to 63, or by the conventional signal name, like SIGINT, SIGTERM, etc. (see Chapter 37.1, page 194).
--cs-dump-on-event=hstringi: Generate an execution report when a specific Microsoft® Windows event was received (see Chapter 37.2, page 194).
--cs-trigger-function=htriggerfunctioni: Make that at the end of each call of htriggerfunctioni, coverage data will be saved. This switch may be repeated to handle more than one trigger function. hfunctioni is specified according to the C++ convention. This means that A::foo stands for the function foo insider the A namespace or class. By default, the save function that is called when htriggerfunctioni runs is __coveragescanner_save(), but it can be changed with the following switch.
1This behaviour may change if the user provides their own output functions. (See e.g. chapters 10.1.13, page 91 and 10.2.10, page 94.)
Squish Coco 5.1.0 - 80 - froglogic GmbH List of options
--cs-coverage-save-function=hsavefunctioni: Set the custom function that is inserted by --cs-trigger-function. hsavefunctioni must be a function without parameters, since it replaces __coveragescanner_save(). Squish Coco automatically inserts a declaration for hsavefunctioni at the beginning of each source file. This makes it possible to define hsavefunctioni in a file that is compiled with --cs-custom-library-source (see Section 9.2.8, page 81).
9.2.7 Command line modification
--cs-compiler-option=hoptioni: Add hoptioni to the end of the compiler command line when compiling the instrumented code. When code coverage is disabled, nothing happens. The option can be repeated if more than one option is needed.
--cs-libgen=hoptioni: Put an additional command line option to the compiler call which generates the code coverage library. The option can be repeated if more than one option is needed. The options are appended and become the content of $LIBGEN$, a variable that may occur in the replacement text for the profile option COMPILER_CMD (see Chapter 50.4, page 303).
--cs-library-after=hoptioni: Put the linker argument for the CoverageScanner library after a specified option. If used in the form “--cs-library-after=-lmylib”, the linker parameter for the CoverageScanner library will be inserted directly after the option -lmylib.
--cs-library-at-start: Add the linker argument for the CoverageScanner library before all other command line options.
--cs-library-at-end: Add the linker argument for the CoverageScanner library after all other command line options.
9.2.8 Generated code
--cs-no-csharp-dynamic: If set, the C# dynamic type is not used in the instrumentation code generated by Squish Coco. With the dynamic type, Squish Coco can better handle Boolean expressions that contain non-Boolean objects with overloaded true and false functions. Dynamic types are however only available for Microsoft® Visual Studio® 2010 and later; for earlier C# versions, this option is needed.
--cs-csharp-dynamic: If set, the C# dynamic type is used in the instrumentation code generated by Squish Coco.
--cs-enable-coveragescanner-library-abs-wildcard=hstringi: Enable the generation of the CoverageScanner library when the absolute file name of the generated binary matches the given wildcard expression.
--cs-disable-coveragescanner-library-abs-wildcard=hstringi: Disable the generation of the CoverageScanner library when the absolute file name of the generated binary matches the given wildcard expression.
--cs-pointer-type=hstringi: Type which can be used to cast a pointer to an integer. This might be ’int’ or ’long long’ depending of the target platform.
Squish Coco 5.1.0 - 81 - froglogic GmbH List of options
--cs-memory-pool=hinti: Generate a CoverageScanner library which uses a memory pool bytes instead of calling malloc()/cfunctionfree. The size of the pool is hinti bytes. This option is useful only for operating systems which do not support dynamic memory allocation.
--cs-memory-pool-code=hstringi: Declaration code for the memory pool. This is optional and permits a custom declaration of the array containing the head used by the CoverageScanner library. The default implementation is a static array of character declared as followings: static char %S[%I]. The following placeholder are available: %S: the symbol name for the custom heap. %I: the heap size in bytes.
--cs-memory-alloc-failure-function=hstringi: C-function name which is called when a malloc() call fails. The prototype of the function should be ’void func(int size)’ and should not return. The parameter size is the requested memory size passed as argument of the malloc() function. This permits to catch an out of memory issue of the CoverageScanner’s code through a custom function.
--cs-link-instrumentation-tables: Chain all instrumentation tables at link time and not at run time. (This is the default.) The instrumentation tables are internal arrays generated by Squish Coco which contain the counter variables for code coverage.
--cs-register-instrumentation-tables: Chain all instrumentation tables at run time and not at link time. This option has an effect only on plat- forms for which the profile parameter INSTRUMENTATION_TABLES_LINKED_DURING_RUNTIME_SUPPORT (see Chapter 50.4, page 303) has the value “Yes”.
--cs-dotnet-exit-handler: Specific handling of .NET Core exit. This lets you save the execution report event if Environment.Exit is called.
--cs-no-exit-handler-installation: Disable the automatic installation of the handler which saves the code coverage report. If this handler is not installed, no executions report will be generated when the application exits or when a custom signal or Windows event is received. Call __coveragescanner_install() to install the handler manually.
--cs-minimal-api: With this command line option, a CoverageScanner API is generated that has fewer dependencies to external libraries than the default API. This may be necessary for embedded systems or when compiling a C# application with the command line switch /noconfig.
--cs-custom-library-source=hfilei: Compile hfilei as an additional source. The object code for hfilei is then linked to the code of the program. The option has an effect only for the instrumentation of C or C++ programs. The code in hfilei must then be written in the same language. This is to include larger pieces of code into the instrumented code for a program without the need to change the sources of the program. The result is that code in the program can use the functions defined in hfilei. But note that the new functions still need to be declared in the source files that use them. When however hfilei is used to define the hsavefunctioni that is set by --cs-coverage-save-function= hsavefunctioni (see Section 9.2.8, page 81), an explicit declaration is not necessary.
Squish Coco 5.1.0 - 82 - froglogic GmbH List of options
9.2.9 Miscellaneous
--cs-compiler=hstringi: This command option enables the user to select a profile. This option does not normally need to be specified, due to the fact that the profile name is usually extracted from the command line name. (cscl.exe implicitly selects the profile cl.cspro) Example: --cs-compiler=cl selects the profile ’cl.cspro’.
--cs-profile=hstringi: This option sets the path of CoverageScanner profile. Using this option is not compatible with --cs-compiler. Example: --cs-profile=%SQUISHCOCO%\cl.cspro is equivalent to --cs-compiler=cl on Microsoft® Win- dows.
--cs-architecture=hstringi: Internal option to specify the target architecture. It sets the variable $ARCHITECTURE$ in the replacement text for the profile parameter PREPROCESSOR_CMD (see Chapter 50.3, page 302). The target architecture is also used to select alternate profile parameters (see Chapter 50, page 301).
--cs-parallel-build=hinti: Maximum number of concurrent code generation/compilation.
--cs-cpp-ext=hlisti: List of file extensions corresponding to C++ files (ex: .cpp,.cxx)
--cs-c-ext=hlisti: List of file extensions corresponding to C files (ex: .c)
--cs-native-toolchain=hstringi: Adds a directory in the search path of the native compiler. This command can be repeated. In this case the first one is the preferred location.
--cs-option-file=hfilei: Specify command line options from a file. The file may contain all valid CoverageScanner command line options and the command line options of the native tool. It must contain one argument per line. Leading and trailing spaces are removed and empty lines are ignored, but no other input processing is done. The option can be repeated.
--cs-file-name-lower-case: Convert absolute file names to lower case. (For backward compatibility with the version 4.1)
--cs-no-pch: If set, precompiled headers are deactivated during the compilation.
--cs-no-pch=hpatterni: Disable the precompiled header usage when compiling a source with an absolute file name that matches the wildcard expression hpatterni or if the string hpatterni is identical to a command line argument.
--cs-no-cspch: If set, the files with the extension .cspch are not generated. This files are only used to enhance the compilation speed when using precompiled headers.
--cs-squish: Enable the automatic Squish integration. (Default).
--cs-no-squish: Disable the automatic Squish integration.
Squish Coco 5.1.0 - 83 - froglogic GmbH Instrumenting using preprocessor symbols
9.3 Instrumenting using preprocessor symbols
CoverageScanner also activates the instrumentation if some symbols are present in the command line option. This allows you to instrument code from an IDE which does not permit adding custom command line options to the compiler or linker. Example: Defining COVERAGESCANNER_COVERAGE_ON is equivalent of adding --cs-on to the command line. Preprocessor symbol Equivalence COVERAGESCANNER_RUNTIME_LOG --cs-verbose=api COVERAGESCANNER_KEEP_INSTRUMENTATION_FILES --cs-keep-instrumentation-files COVERAGESCANNER_VERBOSE --cs-verbose=build COVERAGESCANNER_DUMP_ON_EVENT --cs-dump-on-event=Global\COVERAGE COVERAGESCANNER_COVERAGE_ON --cs-on COVERAGESCANNER_COVERAGE_OFF --cs-off COVERAGESCANNER_COVERAGE_HIT --cs-hit COVERAGESCANNER_COVERAGE_COUNT --cs-count COVERAGESCANNER_COVERAGE_BASIC_BLOCK --cs-statement-block COVERAGESCANNER_COVERAGE_DECISION --cs-decision COVERAGESCANNER_COVERAGE_CONDITION --cs-condition COVERAGESCANNER_COVERAGE_MCC --cs-mcc COVERAGESCANNER_NO_CSHARP_DYNAMIC --cs-no-csharp-dynamic COVERAGESCANNER_NO_LINE_DIRECTIVE --cs-no-line-directive COVERAGESCANNER_CSHARP_DYNAMIC --cs-csharp-dynamic COVERAGESCANNER_COVERAGE_LOCK_CSEXE --cs-lock-csexe COVERAGESCANNER_COVERAGE_NOLOCK_CSEXE --cs-nolock-csexe COVERAGESCANNER_COVERAGE_PARTIAL_INSTRUMENTATION --cs-partial-instrumentation COVERAGESCANNER_COVERAGE_FULL_INSTRUMENTATION --cs-full-instrumentation COVERAGESCANNER_COVERAGE_NO_EXCEPTIONS --cs-no-exceptions COVERAGESCANNER_COVERAGE_NO_ASSIGNMENTS --cs-no-assignments COVERAGESCANNER_MINIMAL_API --cs-minimal-api COVERAGESCANNER_COVERAGE_COMBINE_SWITCH_CASES --cs-combine-switch-cases COVERAGESCANNER_OUTPUT_ABS --cs-output=Global\%A COVERAGESCANNER_PROFILER_ALL --cs-function-profiler=all COVERAGESCANNER_PROFILER_SKIP_TRIVIAL --cs-function-profiler=skip-trivial COVERAGESCANNER_DOTNET_EXIT_HANDLER --cs-dotnet-exit-handler
9.4 Regular Expressions
Regular expressions are composed of items (see table 9.1, page 85 for the full list) and combining operators. These can be:
• [... ]: The brackets are used to define a set of characters. For example, [123] corresponds to the characters 1, 2 or 3. It is possible to define a range using the minus sign: [a-z] corresponds to all characters between a and z.
• [^... ]: The caret defines a set of characters which should not match. For example, [^0-9] corresponds to any character except a digit.
• ?: The question mark after a regular expression item makes it optional.
Squish Coco 5.1.0 - 84 - froglogic GmbH Regular Expressions
Example: ab? matches the string a or ab.
• +: The plus operator makes it possible to repeat the regular expression item. Example: ab+ matches every strings ab... b.
• *: The star operator after a regular expression item signifies that the item is optional and can be repeated. Example: ab* matches every string ab... b and a.
• |: The pipe operator specifies an alternative. Example: a|b matches the character a or b.
• (... ): The parenthesis groups expressions into sub-expressions. Example: (true)|(false) matches the string true or false. • Regular Expression Items: The regular expression items are ASCII or magic characters (See table 9.1, page 85 for the complete list). If a magic character needs to be used as an ASCII character, it has to be escaped with a backslash (\).
Element Description Equivalence Reverse Element ^ The caret marks the beginning of the string. For example, ^c: will only match file names with an absolute path on the drive C:. If you wish to match a literal ^ you must escape it by writing \^. $ The dollar marks the end of the string. For example \.cpp$ will match every C++ file name. If you wish to match a literal $ you must escape it by writing \$. \< Match the start of a word. \> Match the end of a word. . Any character \ Escape character. Example: \. corresponds to the character ’.’ and not any character \m Alphanumeric character [0-9a-zA-Z] \M \a Alphabetic character [a-zA-Z] \A \b Blank character [ t] \B \c Control character \C \d Digit [0-9] \D \g Printable character except space [] \G \l Lower-case character [a-z] \L \p Printable character \P \n Printable character except space or alphanumeric character \N \s White-space characters (space, form-feed , newline , carriage [ fnrtv] \S return , horizontal tab and vertical tab) \u Uppercase letter [A-Z] \U \x Hexadecimal digits [0-9a-fA-F] \X \w Alphanumeric character or underscore [0-9a-zA-Z_] \W
Table 9.1: Non ASCII Regular Expression Items
Squish Coco 5.1.0 - 85 - froglogic GmbH Library calls
Chapter Library calls 10
10.1 C and C++ Library
The CoverageScanner library is generated during instrumentation and enables the user
1. to initialize the measurement system and select the name of the .csexe file, 2. to install a handler which saves the execution report on exit, 3. in a unit test scenario, to pass information about the tests to the execution report, especially the name of the test, comments and the test result, 4. to handle dynamically loaded libraries, and 5. to customize the output system for special needs.
The library functions are accessible from any instrumented file, without the need for an #include directive. When a file is instrumented, a preprocessor symbol __COVERAGESCANNER__ is defined: It can be used to exclude calls to the library functions from the compilation when instrumentation is switched off.
10.1.1 __coveragescanner_install()
Syntax:
void __coveragescanner_install(const char *appname)
i This function is not needed for gcc, clang or Microsoft® Visual Studio®.
Initialize the CoverageScanner library. It should be the first function call in the main() function. The parameter appname contains the name of the executable. It is used to set the name of the measurement file to happnamei.csmes.
__coveragescanner_install() registers an exit handler via atexit() in order to save the execution report whenever the application terminates. The handler calls the function __coveragescanner_save() to store the execution traces at the end of the program. It is called when the program terminates regularly and also if the signals SIGABRT, SIGTERM, SIGFPE, SIGILL, SIGINT, SIGSEGV and SIGTERM are received. Suggested usage:
Squish Coco 5.1.0 - 86 - froglogic GmbH C and C++ Library
int main(argc,char *argv[]) { #ifdef __COVERAGESCANNER__ __coveragescanner_install(argv[0]); #endif ... }
Platform dependencies With gcc, clang or Microsoft® Visual Studio®, CoverageScanner automatically installs a minimal handler. It saves the execution in the file hprogram_namei.csmes on a normal exit of the application. Calling __coveragescanner_install() allows setting the file name of the execution report and also saves the report at an abnormal exit, like a crash or an interrupt by a signal.
__coveragescanner_install() is not available for a compiler if its profile parameter CUSTOM_SETUP is set to NONE (see Chapter 50, page 301).
10.1.2 __coveragescanner_testname()
Syntax:
void __coveragescanner_testname(const char *name)
Set the name of the current test. It will be saved in the execution report and be displayed in the "Executions" window (see Chapter 19.1, page 129) when the .csmes file is loaded into CoverageBrowser.
All code that is executed until the end of the program or the next call of __coveragescanner_save() is considered part of the test name. If tests get the same name, CoverageBrowser disambiguates them by numbers, like “MyTest (2)”.
Test names may be hierarchical, with hierarchy levels separated by slashes: If a program defines two tests, “group/ test1” and “group/test2”, CoverageBrowser displays them as two tests, “test1” and “test2” under a common heading, “group”.
i This and the following functions until __coveragescanner_save() are especially intended for the use with unit test frameworks (see Chapter 34, page 189).
10.1.3 __coveragescanner_teststate()
Syntax:
void __coveragescanner_teststate(const char *state)
Set the state of the current test. The parameter state may have the following values:
"PASSED": to indicate that the test was successful.
"FAILED": to indicate that the test was not successful.
Squish Coco 5.1.0 - 87 - froglogic GmbH C and C++ Library
"INCIDENT": to indicate that the test was not successful. (similar to failed)
"CHECK_MANUALLY": to indicate that it was not possible to determine whether the test was successful.
"SKIPPED": to indicate that the test was skipped. If the function is called twice, the first state is overwritten. The test state will be saved in the execution report and be displayed in the "Executions" window (see Chapter 19.1, page 129) when the .csmes file is loaded into CoverageBrowser.
10.1.4 __coveragescanner_add_html_comment()
Syntax:
void __coveragescanner_add_html_comment(const char *comment)
Set an HTML comment to the current execution. There is only one HTML comment active at any time, but it can be put together with several calls of __coveragescanner_add_html_comment(). In this case, the values of the various comment parameters are concatenated.
The full comment must follow the HTML syntax, but only the content of the
tag is used. A minimal comment might therefore be something like “My comment”.10.1.5 __coveragescanner_clear_html_comment()
Syntax:
void __coveragescanner_clear_html_comment()
Remove the comment added by calls of the function __coveragescanner_add_html_comment().
10.1.6 __coveragescanner_save()
Syntax:
void __coveragescanner_save()
Save the execution report and reset the status of all instrumentations. The coverage counters for each line of code and the execution status of the current test are reset, but the HTML comment stays unchanged.
10.1.7 __coveragescanner_clear()
Syntax:
void __coveragescanner_clear()
Squish Coco 5.1.0 - 88 - froglogic GmbH C and C++ Library
Reset the coverage counters for all lines of code.
This is useful in a unit testing framework. After the framework has started, __coveragescanner_clear() can be called. Then the activity of the framework at startup does not become part of the code coverage.
10.1.8 __coveragescanner_filename()
Syntax:
void __coveragescanner_filename(const char *name)
Set the file name of the execution report. The extension ’.csexe’ is added automatically. If name is NULL, the generation of execution reports is disabled. name may contain the following escape sequences:
%p: The process identifier, if it is provided by the platform, otherwise the empty string.
%w: The working directory. The value is computed when the CoverageBrowser library is initialized; it does not change afterwards.
%t: The current time, or the empty string if it cannot be computed.
%d: The current date, or the empty string if it cannot be computed.
Platform dependencies The escape sequences are only available for a compiler if its profile parameter FILE_FORMAT_SPECIFIER is set to YES (see Chapter 50, page 301). Date and time may not be available on some embedded platforms.
10.1.9 __coveragescanner_get_filename()
Syntax:
const char * __coveragescanner_get_filename()
Function which returns the ’.csexe’ file name.
10.1.10 __coveragescanner_register_library()
i For the use of this and the following function, see Chapter 33, page 185. Syntax:
int __coveragescanner_register_library(const char *library_name)
Register a shared library that was loaded on demand. The function must be called after a library has been loaded with dlopen() (UNIX®) or LoadLibrary() (Microsoft® Windows). After it is registered, the code coverage of the library is saved in the same coverage report file as the coverage of the main application.
Squish Coco 5.1.0 - 89 - froglogic GmbH C and C++ Library
It is possible to call __coveragescanner_register_library() more than once, but every call must be matched by a corresponding call of __coveragescanner_unregister_library(). For each library there is a use counter to keep track of the registrations. Returned values:
0: Success.
-1: Error: library could not be loaded.
-2: Error: invalid library name.
1: Success, but the library is already registered. The use count of the library has been incremented.
2: Success, but the library is not instrumented.
Platform dependencies __coveragescanner_register_library() and __coveragescanner_unregister_library() are not available for a compiler if its profile parameter PLUGIN_REGISTRATION_API is set to NO (see Chapter 50, page 301).
10.1.11 __coveragescanner_unregister_library()
Syntax:
int __coveragescanner_unregister_library(const char *library_name)
De-register a shared library that was loaded on demand. The function must be called before a library is unloaded with dlclose() (UNIX®) or CloseHandle() (Microsoft® Windows). Returned values:
0: Success.
-1: Error: library was not registered.
-2: Error: invalid library name.
1: Success, but the library was not unregistered because its use count was not null.
2: Success, but the library is not instrumented.
10.1.12 __coveragescanner_register_squish()
Syntax:
void __coveragescanner_register_squish()
Try to register the instrumented application with Squish. This makes most of the Squish Coco library functions accessible to Squish, so that it can control the writing of the coverage measurements. It is rarely necessary to use this function explicitly since it is automatically called at start of the application (or by __coveragescanner_install(), if calling it is necessary). Calling the function more than once has the same effect as calling it once. If the application was not started by Squish, the function has no effect.
Squish Coco 5.1.0 - 90 - froglogic GmbH C and C++ Library
10.1.13 __coveragescanner_set_custom_io()
Syntax:
void __coveragescanner_set_custom_io( char *(*csfgets)(char *s, int size, void *stream), int (*csfputs)(const char *s, void *stream), void *(*csfopenappend)(const char *path), void *(*csfopenread)(const char *path), void *(*csfopenwrite)(const char *path), int (*csfclose)(void *fp), int (*csremove)(const char *filename) )
Define your own set of I/O functions. Parameters: csfgets: Read at most size - 1 characters from stream and store them into the buffer pointed to by s. Reading stops after an EOF or a newline is read. If a newline is read, it is stored in the buffer. A ‘\0’ is stored after the last character in the buffer. csfputs: Write the string s to stream, without its trailing ’\0’. csfopenappend: Open the file path for appending text at the end. csfopenread: Open the file path for reading. csfopenwrite: Open the file path for writing. csfclose: Flush and close the stream pointed to by fp. csremove: Remove the file filename from the file system.
By default, CoverageScanner writes the execution report (the .csexe file) to the file systems with help of common C library functions. In some situations, e.g. with embedded systems, it is necessary to use other methods. For this reason, CoverageScanner provides __coveragescanner_set_custom_io(), which lets you replace the I/O functions used by __coveragescanner_save(). Default implementation:
char *csfgets(char *s, int size, void *stream) { return fgets(s,size,(FILE *)stream); } int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); } void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); } void *csfopenread(const char *path) { return (void*)fopen(path,"r"); } void *csfopenwrite(const char *path) { return (void*)fopen(path,"w"); } int csremove(const char *filename) { return remove(filename); } int csfclose(void *fp) { return fclose((FILE*)fp); }
Examples are available in Chapter 38, page 198.
Squish Coco 5.1.0 - 91 - froglogic GmbH C# Library
10.2 C# Library
CoverageScanner library is generated during the link process and enables the user to: 1. Generate the execution report. 2. Give a name to the executed tests. 3. Install a signal handler which saved the execution report on exit.
Additionally CoverageScanner sets the preprocessor symbol __COVERAGESCANNER__ which excludes this function from a normal compilation.
10.2.1 CoverageScanner.__coveragescanner_init()
Syntax:
void CoverageScanner.__coveragescanner_init()
CoverageScanner.__coveragescanner_init() initialize explicitly the CoverageScanner library. Calling this function is only necessary on C# DLL which are not calling the static module initializer automatically.
10.2.2 CoverageScanner.__coveragescanner_testname()
Syntax:
void CoverageScanner.__coveragescanner_testname(string name)
CoverageScanner.__coveragescanner_testname() sets the name of the test which is currently being executed. It will be saved to the execution report and is displayed in the "Executions" window (see Chapter 19.1, page 129) when the loaded in CoverageBrowser.
10.2.3 CoverageScanner.__coveragescanner_teststate()
Syntax:
void CoverageScanner.__coveragescanner_teststate(string state)
CoverageScanner.__coveragescanner_teststate() sets the state of the test which is currently being executed. The string parameter state can have the following values:
"PASSED": indicates that the test was successfully executed.
"FAILED": indicates that the test was not successfully passed.
"INCIDENT": indicates that the test was not successfully executed. (similar to failed)
"CHECK_MANUALLY": indicates that it was not possible to determinate if the test was successfully executed.
Squish Coco 5.1.0 - 92 - froglogic GmbH C# Library
"SKIPPED": indicates that the test was skipped.
It will be saved to the execution report and is displayed in the "Executions" window (see Chapter 19.1, page 129) when the loaded in CoverageBrowser.
10.2.4 CoverageScanner.__coveragescanner_memory_pool_stat()
Syntax:
void __coveragescanner_memory_pool_stat(int *size, int *used, int * max_used )
__coveragescanner_memory_pool_stat() permits to retrieve the current memory consumption of the memory pool. The following parameters are returned: size: the overall size of the memory pool. used: the current memory usage. max_used: the peak memory usage. This function makes only sense in the case of using a memory pool.
10.2.5 CoverageScanner.__coveragescanner_add_html_comment()
Syntax:
void __coveragescanner_add_html_comment(string comment)
__coveragescanner_add_html_comment() append a comment to the current execution. The comments needs to follow the HTML syntax but only the body is parsed. (The contents of the “
” tag is ignored). This function can be called several times, in this case the content is appended.10.2.6 CoverageScanner.__coveragescanner_clear_html_comment()
Syntax:
void __coveragescanner_clear_html_comment()
__coveragescanner_add_html_comment() clear comments added using void __coveragescanner_add_html_comment (string comment).
10.2.7 CoverageScanner.__coveragescanner_save()
Syntax:
Squish Coco 5.1.0 - 93 - froglogic GmbH C# Library
void CoverageScanner.__coveragescanner_save()
CoverageScanner.__coveragescanner_save() saves the execution report and resets the status of all instrumen- tations.
10.2.8 CoverageScanner.__coveragescanner_clear()
Syntax:
void CoverageScanner.__coveragescanner_clear()
CoverageScanner.__coveragescanner_clear() resets the status of all instrumentations.
10.2.9 CoverageScanner.__coveragescanner_filename()
Syntax:
void CoverageScanner.__coveragescanner_filename(string name) this function sets the file name of the execution report. The extension ’.csexe’ is added automatically. i Setting the execution report file name is necessary if the initialization function __coveragescanner_install() is not used.
10.2.10 CoverageScanner.__coveragescanner_set_custom_io()
Syntax:
void CoverageScanner.__coveragescanner_set_custom_io( __cs_fgets_delegate cs_fgets, __cs_fputs_delegate cs_fputs, __cs_fopenappend_delegate cs_fopenappend, __cs_fopenread_delegate cs_fopenread, __cs_fopenwrite_delegate cs_fopenwrite, __cs_fclose_delegate cs_fclose, __cs_remove_delegate cs_remove)
Parameters: csfgets: Type:public delegate string __cs_fgets_delegate(System.IO.Stream stream) csfgets reads in at most one less than size characters from stream and stores them into the buffer pointed to by s. Reading stops after an EOF or a newline. If a newline is read, it is stored into the buffer. A ’\0’ is stored after the last character in the buffer. csfputs: Type:public delegate void __cs_fputs_delegate(string s, System.IO.Stream stream) csfputs writes the string s to stream, without its trailing ’\0’.
Squish Coco 5.1.0 - 94 - froglogic GmbH C# Library csfopenappend: Type:public delegate System.IO.Stream __cs_fopenappend_delegate(string path) csfopenappend open the file path for writing at the end. csfopenread: Type:public delegate System.IO.Stream __cs_fopenread_delegate(string path) csfopenread open the file path for reading. csfopenwrite: Type:public delegate System.IO.Stream __cs_fopenwrite_delegate(string path) csfopenwrite open the file path for writing. csfclose: Type:public delegate void __cs_fclose_delegate(System.IO.Stream fp) The csfclose function will flush and close the stream pointed to by fp. csremove: Type:public delegate void __cs_remove_delegate(string n) csremove removes filename from the file system.
CoverageScanner writes the execution report (.csexe file) into file using the common C# file IO functions. In some situations, for embedded systems for example, it is necessary to use an other way for recording it. For this reason, CoverageScanner provides CoverageScanner.__coveragescanner_set_custom_io() which replace the IO functions used by CoverageScanner.__coveragescanner_save().
Squish Coco 5.1.0 - 95 - froglogic GmbH Controlling the instrumentation from the source code
Chapter Controlling the instrumentation from the source 11 code
Sometimes it is necessary to apply CoverageScanner settings only to certain regions of the source code, e.g. to exclude parts of a file from instrumentation. Several methods to do this are described in this chapter.
11.1 Annotations
Source code annotations are specific comments which let you select source code parts to validate. Two kinds of annotations exist:
Comments: Comments are simple code comments which can be used for giving advice to the tester. They do not have any impact on the code coverage
Manual Validations: Manual validations indicate that some lines of code should be considered as validated. Manual validations have the same effect on code coverage as executing the source lines. Source code annotations can directly be edited with CoverageBrowser or generated by writing specific comments into the source code.
11.1.1 Comments
To generate comments for a single source line it is only necessary to start a C or C++ comment with the keyword “coco:hcommenti”. This will insert a comment for all instrumentations in the current line. Example: In the following example, the comment “To test the failure, change the current working directory to a read-only directory” will be added to the instrumentations of the line 4.
1 void write_log( const char *text ) 2 { 3 FILE * f = fopen( "log.txt", "a+" );
Squish Coco 5.1.0 - 96 - froglogic GmbH Annotations
4 if ( f ) /* coco: To test the failure, 5 change the current working 6 directory toa read-only directory */ 7 {
8 fputs( text, f ):
9 fclose( f );
10 }
11 }
Comments can also be generated for multiples source lines using the keywords “coco begin:hcommenti” and “coco end”. Example: In the following example, the comment “To test the failure, change the current working directory to a read-only directory” will be added to the instrumentations of the lines of the function write_log().
1 void write_log( const char *text ) 2 { 3 /* coco begin: To test the failure, 4 change the current working 5 directory toa read-only directory */ 6 FILE * f = fopen( "log.txt", "a+" ); 7 if ( f )
8 {
9 fputs( text, f ):
10 fclose( f );
11 } 12 /* coco end */ 13 }
11.1.2 Manual Validation
Manually validated instrumentation can be generated with a comment in the code that starts with the keywords “coco validated: hcommenti”. The hcommenti will later occur in the coverage report. To validate a multi-line region of code, one can use the keywords “coco begin validated: hcommenti” and “coco end”. In order to be compatible with PureCoverage adjustments, CoverageScanner also recognizes “purecov:” line comments with the keywords “inspected” or “tested”, or alternatively “begin inspected”/“begin tested” and “end inspected”/“end tested” for multi-line annotations. Example: In the following example, the comment “This function does not need to be tested, only used for debugging.” will be added to the instrumentations of the function write_log().
1 void write_log( const char *text ) 2 {
3 // coco begin validated: This function does not need to be tested, only used for debugging. 4 FILE * f = fopen( "log.txt", "a+" ); 5 if ( f )
6 {
7 fputs( text, f ):
8 fclose( f );
9 }
10 }
Squish Coco 5.1.0 - 97 - froglogic GmbH C and C++ Pragmas
11 // coco end
Note that the “coco end” comment is placed after the end of the function. This is because Coco measures code coverage of a function at its very end, there where the final closing brace is located. When we want to mark a complete function a covered, we need to include the last closing brace in the region between the manual validation comments.
Including the function header by placing the “coco begin” comment in front of the function is however not important.
11.2 C and C++ Pragmas
CoverageScanner defines C and C++ pragmas to control the instrumentation during the compilation. All control prag- mas are handled in a stack: They are valid until the end of the source file or until a #pragma CoverageScanner(pop) is encountered. The pragmas have the following syntax: #pragma CoverageScanner ( string ) or: __pragma ( CoverageScanner ( string )) or: _Pragma ( "CoverageScanner ( string )" )
List of supported pragmas:
#pragma CoverageScanner(cov-on) : Enable code coverage instrumentation.
#pragma CoverageScanner(cov-off) : Disable code coverage instrumentation.
#pragma CoverageScanner(cov-hit) : Instruments using code coverage hit.
#pragma CoverageScanner(cov-count) : Instruments using code coverage count.
#pragma CoverageScanner(cov-statement-block) : Select statement block coverage as instrumentation method.
#pragma CoverageScanner(cov-decision) : Select decision coverage as instrumentation method.
#pragma CoverageScanner(cov-condition) : Select condition coverage as instrumentation method.
#pragma CoverageScanner(cov-partial-instrumentation) : Partial instrumentation of conditions and decisions.
#pragma CoverageScanner(cov-full-instrumentation) : Full instrumentation of conditions and decisions.
#pragma CoverageScanner(cov-assignment-on) : Instrument Boolean expressions in assignments.
Squish Coco 5.1.0 - 98 - froglogic GmbH C# regions
#pragma CoverageScanner(cov-assignment-off) : Do not instrument Boolean expressions in assignments.
#pragma CoverageScanner(cov-return-on) : Instrument Boolean expressions in return statements.
#pragma CoverageScanner(cov-return-off) : Do not instrument Boolean expressions in return statements.
#pragma CoverageScanner(cov-switch-case-off) : Do not instrument all cases of a switch statement.
#pragma CoverageScanner(cov-switch-case-on) : Instrument all cases in a switch statement.
#pragma CoverageScanner(pop) : Restore the instrumentation options from before the call of the latest CoverageScanner pragma.
11.3 C# regions
CoverageScanner defines C# extensions of the #region keyword which, at compile-time, controls the generation of the instrumentation. The #region extension has the following syntax: #region CoverageScanner ( string )
The keyword #endregion restores the generation options. List of supported regions:
#region CoverageScanner(cov-on) : Enable code coverage instrumentation.
#region CoverageScanner(cov-off) : Disable code coverage instrumentation.
#region CoverageScanner(cov-hit) : Instrument using code coverage hit.
#region CoverageScanner(cov-count) : Instrument using code coverage count.
#region CoverageScanner(cov-statement-block) : Select statement block coverage as instrumentation method.
#region CoverageScanner(cov-decision) : Select decision coverage as instrumentation method.
#region CoverageScanner(cov-condition) : Select condition coverage as instrumentation method.
#region CoverageScanner(cov-partial-instrumentation) : Partial instrumentation of conditions and decisions.
#region CoverageScanner(cov-full-instrumentation) : Full instrumentation of conditions and decisions.
Squish Coco 5.1.0 - 99 - froglogic GmbH C# regions
#region CoverageScanner(cov-assignment-on) : Instrument Boolean expressions in assignments.
#region CoverageScanner(cov-assignment-off) : Do not instrument Boolean expressions in assignments.
#region CoverageScanner(cov-switch-case-off) : Do not instrument all cases in a switch statement.
#region CoverageScanner(cov-switch-case-on) : Instrument all cases in a switch statement.
Squish Coco 5.1.0 - 100 - froglogic GmbH QML Coverage Part V
Squish Coco 5.1.0 - 101 - froglogic GmbH QML coverage
Chapter QML coverage 12
QML is a user interface markup language that is used in the Qt Framework. Squish Coco has an add-on that supports code coverage for QML. The add-on is delivered separately and requires an additional license. It consists of two parts, a program to instrument the QML code and a QML plugin that is needed at runtime.
The instrumentation program is called cocoqmlscanner and it inserts code into the QML sources of a program. The plugin implements a QML object that at runtime collects the coverage data and writes them to a file. The plugin is delivered in source form and must be compiled by the customer.
12.1 Setup
The CocoQML package is a ZIP archive. When unpacked, it generates a directory with two subdirectories: bin: The directory with the cocoqmlscanner executable and some libraries that are needed by it. trackerplugin: The source code of the tracker plugin. Both directories are independent of each other and it is possible to copy them to different places of a test installation.
12.1.1 Compilation of the plugin
The plugin is a qmake project. It must be compiled for the same Qt version as the application for which the coverage should be measured. We recommend that you make it part of your project and include it in the repository, so that it is automatically compiled correctly. Under UNIX®, the compilation looks like this:
$ /Qt/5.11.3/gcc_64/bin/qmake ../trackerplugin Info: creating stash file /home/user/trackerplugin-build/.qmake.stash $ make g++ -c -pipe -O2 -std=gnu++11 -D_REENTRANT -Wall -W -fPIC -DQT_NO_DEBUG -DQT_PLUGIN - DQT_GUI_LIB -DQT_QML_LIB -DQT_NETWORK_LIB -DQT_CORE_LIB -I../trackerplugin -I. -I/
Squish Coco 5.1.0 - 102 - froglogic GmbH Use of CocoQML
Qt/5.11.3/gcc_64/include -I/Qt/5.11.3/gcc_64/include/QtGui -I/Qt/5.11.3/gcc_64/ include/QtQml -I/Qt/5.11.3/gcc_64/include/QtNetwork -I/Qt/5.11.3/gcc_64/include/ QtCore -I. -isystem /usr/include/libdrm -I/Qt/5.11.3/gcc_64/mkspecs/linux-g++ -o csexeapi.o ../trackerplugin/csexeapi.cpp [...] mv -f libcocoqmltracker.so QmlJsCoverage/libcocoqmltracker.so cp -f /home/user/trackerplugin/qmldir QmlJsCoverage $
When compiled this way, the build directory contains a subdirectory QmlJsCoverage which contains the plugin:
$ ls QmlJsCoverage/ libcocoqmltracker.so qmldir $
The Microsoft® Windows build is similar; the plugin directory then contains of course a DLL.
12.2 Use of CocoQML
12.2.1 Instrumentation
To measure QML coverage in a software project, its QML files must be be instrumented. This is done by the cocoqmlscanner command, in the form
$ cocoqmlscanner hproject directoryi
All QML files in subdirectories of hproject directoryi are then (reversibly) instrumented and a program database (or .csmes file) is written. By default, the program database file is cocoqmlscanner_result.csmes. After the QML files are instrumented, it is recommended to recompile the project. This is especially important in two cases:
• The QML files are resources of the program (i.e. specified in a .qrc file). • The QML compiler is used. (This is the default for release builds that are compiled with Qt Creator). In both cases, the compilation makes the instrumented files part of the executable.
12.2.2 Measuring the coverage
After instrumentation, an application needs the tracker plugin to run. To find the plugin, it uses the environment vari- able QML2_IMPORT_PATH. This variable must be set and its value must be the directory which has QmlJsCoverage/ as a subdirectory. If this variable is set correctly, the instrumented AUT can run.
Debugging If something goes wrong, it is useful to set the variables QML_IMPORT_TRACE and COCOQML_VERBOSE to 1. With the first one set, Qt shows where it searches for modules. With the second one, the plugin logs its activity.
Squish Coco 5.1.0 - 103 - froglogic GmbH Reference
12.3 Reference
12.3.1 cocoqmlscanner
Syntax:
cocoqmlscanner [hoptionsi] hdirectoryi... where the options stand for: hdirectoryi: is the path to a directory that contains files that will be instrumented or restored. The action depends on whether the option -r (see below) is set. If -r is not set, the hdirectoryi is scanned recursively and each file that ends with “.qml” or “.js” is read and instrumented. A backup copy of each file is kept; it has the suffix “.qmlbak” or “.jsbak”, respectively. If there are already backup copies, nothing is done and the program returns with an error. If -r is set, all the backup files are renamed back to their old names, replacing the instrumented files. If there are no backup copies, the program does nothing and returns with an error.
-r | --restore: Enable the restore mode.
-c | --csmes-name=hfilenamei: If this option is used, the resulting .csmes file is saved as hfilenamei. A suffix “.csmes” is appended if it does not exist already. Without this option, the resulting data is saved as cocoqmlscanner_result.csmes.
-b | --blacklist=hfilenamei: Exclude certain paths from instrumentation. The hfilenamei is the path to a text file which contains in each line a regular expression. If one of the regular expressions matches a file name, the file is excluded from the instrumentation. Lines beginning with a “#” are ignored.
-n | --no-tracker: Do not insert an import statement for the tracker plugin into the instrumented code. With this option, cocoqmlscanner behaves the same way as it did before the tracker plugin was introduced. The option was introduced only for compatibility with the previous version of cocoqmlscanner and is rarely, if ever needed. In particular, it will break the tracker plugin functionality.
-v | --version: Print the version number of the program and finish.
-h | --help: Print a help message and finish.
12.3.2 The tracker plugin
The tracker plugin is provided in source form because the Qt version with which it is compiled must be the same as the Qt version of the QML application. The following variables control various debug output generated by the tracker. All log messages are prefixed with “CocoQML: ” (without the quotes).
Squish Coco 5.1.0 - 104 - froglogic GmbH Reference
COCOQML_QUIET: Set this variable to 1 if you want to disable all messages generated by the add-on. Usually all messages will be printed to standard error, but if you rely on parsing standard error in your application you can use this to avoid interference. This setting overrides the verbose mode, see below.
COCOQML_VERBOSE: Set this variable to 1 if you want to enable verbose mode. If the verbose mode is active, the CocoQML extension will print additional messages that might be helpful for debugging purposes. Usually you will not need to enable verbose mode. This settings can be overridden by the quiet mode, see above.
COCOQML_LOGFILE: If this variable is set, the CocoQML add-on writes its log messages to a file. The value of COCOQML_LOGFILE must be an absolute or relative file path. A relative file path is interpreted relative to the working directory of the application when it starts. If the log file already exists, the new data is appended to it. If the path contains directories, they must already exist, otherwise no log file will be created. Example: Under Unix, one could set COCOQML_LOGFILE=/tmp/cocoqml.log. Since the directory /tmp/ exists in any normal Unix installation and is writable, all log messages are then written to the file /tmp/ cocoqml.log
Squish Coco 5.1.0 - 105 - froglogic GmbH QML coverage with a library
Chapter QML coverage with a library 13
i This version of the QML coverage add-in is now deprecated. Please use the version of the add-in in Chapter 12 instead.
13.1 Overview
QML is a user interface markup language that is used in the Qt Framework. Squish Coco has an add-on that supports code coverage for QML. The add-on is delivered separately and requires an additional license.
It consists of a replacement for a Qt shared library file. By default the new library acts the same way as the version supplied by Qt, unless QML coverage is activated by setting an environment variable. i There is one exception from this rule. It concerns the QML disk cache, which interferes with correct instrumentation when switched on. In Qt 5.8 and later, where a disk cache exists, it is therefore always deactivated in the replacement library, even when instrumentation is not enabled.
13.2 Installation
The coverage add-on consists of a replacement of the Qt shared library Qt5Qml and is delivered in a ZIP archive. There exist several versions of the add-on, each for a different combination of Qt version and platform. To install it, one needs to download the right version of the archive, unpack it and put its content in place of the original library. It is useful to keep a backup version of the original library files.
Squish Coco 5.1.0 - 106 - froglogic GmbH Installation
13.2.1 General remark
In the following sections we describe the installation for the most common case, where a Qt development kit has been installed in the system and an application is compiled to use it. Please note that even if you have installed such a development kit, its library is not always used. Sometimes the application you want to trace has its own version of the Qt libraries. Then you need to find the copy of the Qt5Qml library that it uses and replace this library with the froglogic version.
13.2.2 Finding the right version of the archive
The archives have names of the form “cocoqml-hcoco-versioni-qthqt-versioni-hcompileri-hplatformi.zip”, with the following components: hcoco-versioni: The version of Coco to which the add-on belongs. It has the form x.y.z, like e.g. 4.0.3. All versions of Squish Coco with a release number later than that of hcoco-versioni can then read the coverage data that are generated by the add-on. However, since each Squish Coco release can read the data generated by earlier releases, it is not necessary to update the add-on when a newer version of Squish Coco is installed. hqt-versioni: The Qt version for which the library is compiled. It has the form a.b.c, like e.g. 5.8.0. Note that a new version of the add-on is necessary whenever the patch level (the third number in the version string) of the Qt installation changes. hcompileri: A string that identifies the compiler for which the Qt library is built. Examples are gcc_64 or msvc2013. hplatformi: Either linux, linux_64, mac or windows.
13.2.3 Linux™ installation
Under Linux™, the archive contains a library file with a name like libQt5Qml.so.5.8.0. The numbers at the end are those of the Qt version.
To find the right version of the archive to download, you can usually use the command qmake that is used for the compilation of your project. If Qt is installed in the /opt/Qt/ directory, the full path to qmake might be /opt/Qt /5.8/gcc_64/bin/qmake. This means that the hcompileri is gcc_64 and the hqt-versioni is a version number that begins with 5.8. To get the full version number, run qmake in the form
$ /opt/Qt/5.8/gcc_64/bin/qmake -query QT_VERSION 5.8.0
This then tells us that the full hqt-versioni must be 5.8.0.
The correct package is then, for a 64-bit Linux system, “cocoqml-4.1.0-qt5.8.0-gcc_64-linux_64.zip”. Unpack it and copy the file libQt5Qml.so.5.8.0 contained in it to the directory /opt/Qt/5.8/gcc_64/lib.
There is also a file cocoqmlscanner (see Chapter 13.3.1, page 109), which you can copy wherever you like.
i Apart from libQt5Qml.so.5.8.0, there are also other files in the directory /opt/Qt/5.8/gcc_64/ lib that begin with libQt5Qml.so. These files are only symbolic links and need not to be changed.
Squish Coco 5.1.0 - 107 - froglogic GmbH CocoQML Tutorial
13.2.4 macOS installation
Under macOS, the archive contains a library in the form of a directory, with the name QtQml.framework/.
To find the right version of the archive to download, you can usually use the command qmake that is used for the compilation of your project. If Qt is installed in the /Users/someone/Qt/ directory, the full path to qmake might be /Users/someone/Qt/5.8/clang_64/bin/qmake. This means that the hcompileri is clang_64 and the hqt-versioni is a version number that begins with 5.8. To get the full version number, run qmake in the form
$ /Users/someone/Qt/5.8/clang_64/bin/qmake -query QT_VERSION 5.8.0
This tells us that the full hqt-versioni must be 5.8.0.
The correct package is then “cocoqml-4.1.0-qt5.8.0-clang_64-mac.zip”. Unpack it and copy the directory QtQml.framework/ contained in it to the directory /Users/someone/Qt/5.8/clang_64/lib/. (You need to rename the original QtQml.framework/ directory.)
The file cocoqmlscanner (see Chapter 13.3.1, page 109) can be copied to any appropriate place.
13.2.5 Microsoft® Windows installation
Under Microsoft® Windows, the archive contains one Qt library file, Qt5Qmld.dll, together with some more DLL files that are needed by the enhanced QML libraries. There is also a separate executable, cocoqmlscanner.exe, which is needed only in specific cases (see Chapter 13.3.1, page 109).
To find the right version of the archive to download, you can usually use the command qmake that is used for the compilation of your project. If Qt is installed in the C:\Qt\ directory, the full path to qmake might be C:\Qt\5.8\ msvc2013\bin\qmake. This means that the hcompileri is msvc2013 and the hqt-versioni is a version number that begins with 5.8. To get the full version number, run qmake in the form
C:\Users\Someone>C:\Qt\5.8\msvc2013\bin\qmake -query QT_VERSION 5.8.0
This tells us that the full hqt-versioni must be 5.8.0.
This then means that the correct package is “cocoqml-4.1.0-qt5.8.0-msvc2013-windows.zip”. Unpack it and copy the DLL files contained in it to the directory C:\Qt\5.8\msvc2013\bin\.
The file cocoqmlscanner.exe can be copied to any appropriate place.
13.3 CocoQML Tutorial
The following tutorial provides examples how to use CocoQML. It also shows how to detect whether CocoQML is installed correctly.
i The following examples are written for a UNIX® system. Microsoft® Windows is similar.
Squish Coco 5.1.0 - 108 - froglogic GmbH CocoQML Tutorial
13.3.1 Compile a Qt application
We will use the calqlatr program as our example. It is part of the Qt distribution and can be found in the Examples/Qt-5.8/quick/demos/calqlatr/ subdirectory of your Qt installation.
The easiest way to set it up is to use the Qt Creator. Start the Qt Creator, select the "Welcome" mode and then "Examples". It is then possible to search for “calqlatr” and open the project. There are some points one needs to take care of:
• Make sure that the compiler toolkit used for compiling the calqlatr project agrees with that of the CocoQML add-on. If you have e.g. downloaded and installed the package cocoqml-4.0.3-qt5.8.0-gcc_64-linux_64 .zip, you need to select the Qt 5.8.0 GCC 64bit kit in the Qt Creator to compile it.
• Under Microsoft® Windows, you need to select a release build since the patched version of Qt5Qml.dll exists only as a release build. Under UNIX® and macOS, this is not necessary. • In newer versions of Qt Creator, a release build has automatically enabled the QML compiler. It must be switched off because otherwise CocoQML will not work. To do this, one needs to select the "Projects" mode of Qt Creator, and there the qmake build step entry. There, one can see a checkbox labled, “Enable Qt Quick Compiler” (Figure 13.1). It must be diabled.
Figure 13.1: The QML compiler is switched off.
Compile the program and run it. Check whether it runs correctly.
13.3.2 Enable code coverage
Now you can enable code coverage by setting the COCOQML variable to 1. For the first experiments with CocoQML it is also useful to make the output a bit more verbose by setting COCOQML_VERBOSE to 1, too. To do this, select "Projects" in the Qt Creator and add COCOQML=1 and COCOQML_VERBOSE=1 to the "Run Environment" settings. Then run calqlatr again.
In the "Application Output" window you now should see lines like the following:
Squish Coco 5.1.0 - 109 - froglogic GmbH Resolving instrumentation problems
CocoQML: Initializing tracker object CocoQML: Creating QmlJsCoverageTracker CocoQML: Checking: qrc:///demos/calqlatr/calqlatr.qml CocoQML: Checking: file:qrc:///demos/calqlatr/content/calculator.js CocoQML: Checking: qrc:///demos/calqlatr/content/Display.qml CocoQML: Checking: qrc:///demos/calqlatr/content/NumberPad.qml CocoQML: Checking: qrc:///demos/calqlatr/content/Button.qml
If it does not, there is an error in the setup of CocoQML.
Now do some computations with the calqlatr, then close the program window. (Do not kill it from the Qt Creator, because then it cannot write the coverage data.) You should see an additional output similar to this:
CocoQML: Destroying QmlJsCoverageTracker CocoQML: Handle destroyed tracker object CocoQML: Saved coverage: /home/user/calqlatr/calqlatr_qml.csexe /home/user/calqlatr/calqlatr exited with code 0
From this output we can see that the files calqlatr_qml.csmes and calqlatr_qml.csexe have been created in the directory /home/user/calqlatr. You can now open them with the CoverageBrowserand view the results.
When everything works well, you can now disable the debug output and set COCOQML_VERBOSE=0 (or remove the variable completely from the settings).
13.3.3 Excluding source files from coverage
There is a common use case when some source files need to be excluded from instrumentation – maybe because they cause instrumentation errors or because coverage data is not needed for them. We will now show how this is done and exclude the file Button.qml from the code coverage measurements.
Source code exclusion is controlled by the environment variable COCOQML_BLACKLIST, the value of which must be the path to a text file. So we need to create he text file first.
Create a file with the name cocoqml.txt. It should contain one line of text,
Button.qml
Then add the variable COCOQML_BLACKLIST to the "Run Environment" section in the Qt Creator. Its value should be the full path to the cocoqml.txt file, like “/home/user/cocoqml.txt”.
Then the file Button.qml (and in fact all files with a name that contains the string “Button.qml”) will be excluded from instrumentation. Run the calqlatr program again. In the "Application Output" window you should now see a similar output as before, but with the Button.qml line missing. Similarly, the file Button.qml will be no longer be visible in the CoverageBrowser.
13.4 Resolving instrumentation problems
The following is a short recipe that tells what to do when the instrumented program does not run or behaves not the same way as the uninstrumented program.
Squish Coco 5.1.0 - 110 - froglogic GmbH Environment Variables
1. If logging was disabled, enable it by running the instrumented program with the environment variable COCOQML_VERBOSE set to 1. The output may contain hints about files that cause problems. 2. If the output shows that a source file causes problems, you can exclude it from instrumentation with the COCOQML_BLACKLIST mechanism (see Chapter 13.3.3, page 110).
3. If no error is logged but the instrumented program still behaves strange, you can try to find the problem file (or files) by selectively excluding files. The idea is to exclude selectively files to see whether they cause problems. In a small project, one can enter the name of a single source file into the blacklist file and then run the program again. If it now runs without problems, that file was the cause. Otherwise, repeat this procedure with another file, until the error is found. With larger projects, it is better to start with excluding whole subdirectories. If the name of a directory is entered in the COCOQML_BLACKLIST file, all files in that directory are excluded from instrumentation. If the program runs without problems with a directory excluded from instrumentation, the problem must be a file in this directory. You then need to repeat the test with its subdirectories or, if it is small enough, with its files. Otherwise, repeat the test with the other directories. This way, all files that cause instrumentation problems can usually be found and blacklisted. 4. If there are still problems, mail the log file together with a description of the problem to froglogic support.
13.5 Environment Variables
When the Coco add-on is present, QML coverage is determined by environment variables. By default, instrumentation and coverage are switched off. The following variables can be used to control code coverage:
COCOQML: Set this variable to 1 to enable the instrumentation of your QML source code. If the variable is not set or set to a different value, no instrumentation will be done: The Qt installation then behaves as usual.
COCOQML_OUTPUT: This variable contains the path to the directory in which the output files are generated. If the variable is not set, the current working directory of the program is used. Setting this variable is especially useful if you do not have sufficient permissions in the automatically determined paths or want to gather all intermediate instrumentation databases in a single file.
COCOQML_CACHE: If this variable is set, CocoQML uses a cache to store the instrumentations of the source files. Its value must be the path to an empty directory which CocoQML uses to store the cache files. With the cache enabled, CocoQML instruments a source file only if it cannot find it in the cache or if the cache contains a different version of the file. The new instrumented version of the file is then stored in the cache.
COCOQML_BLACKLIST: Set this variable to exclude certain paths from instrumentation. The value of the variable is the path to a text file which contains in each line a regular expression. If one of the regular expressions matches a file name, the file is excluded from the instrumentation. Lines beginning with a “#” are ignored. The following variables control various debug output generated by the extension, especially the log messages. All log messages are prefixed with “CocoQML: ” (without the quotes).
Squish Coco 5.1.0 - 111 - froglogic GmbH The CocoQML scanner
COCOQML_DEFAULT_BLACKLIST: This variable controls the default blacklist of CocoQML. By default, Qt system include files are excluded from instrumentation. This is so because in general, only the coverage of the application should be measured and not that of the functions in the system files. If however coverage of the system files is needed, COCOQML_DEFAULT_BLACKLIST must be set to 0.
COCOQML_QUIET: Set this variable to 1 if you want to disable all messages generated by the add-on. Usually all messages will be printed to standard error, but if you rely on parsing standard error in your application you can use this to avoid interference. This setting overrides the verbose mode, see below.
COCOQML_VERBOSE: Set this variable to 1 if you want to enable verbose mode. If the verbose mode is active, the CocoQML extension will print additional messages that might be helpful for debugging purposes. Usually you will not need to enable verbose mode. This settings can be overridden by the quiet mode, see above.
COCOQML_LOGFILE: If this variable is set, the CocoQML add-on writes its log messages to a file. The value of COCOQML_LOGFILE must be an absolute or relative file path. A relative file path is interpreted relative to the working directory of the application when it starts. If the log file already exists, the new data is appended to it. If the path contains directories, they must already exist, otherwise no log file will be created. Example: Under Unix, one could set COCOQML_LOGFILE=/tmp/cocoqml.log. Since the directory /tmp/ exists in any normal Unix installation and is writable, all log messages are then written to the file /tmp/ cocoqml.log
COCOQML_DUMPLOC: If this variable is set, the CocoQML add-on writes the instrumented versions of the QML files it instruments to the disk. The value of COCOQML_DUMPLOC must be the path to a directory, either absolute or relative. A relative path is interpreted relative to the working directory at start time. If the directory does not exist, it is created.
13.6 The CocoQML scanner
There is a second tool for the coverage of QML. The cocoqmlscanner instruments source files directly and creates a .csmes file for them. A use case for this program is when one wants to be certain that the code for all QML source files is contained in the .csmes file. This does not occur automatically when instrumenting with the patched dynamic library: Then only those source files that are actually executed are instrumented and occur in the .csmes file, which might lead to too high coverage rates.
13.6.1 Syntax
Syntax:
cocoqmlscanner [hoptionsi] hdirectoryi...
Squish Coco 5.1.0 - 112 - froglogic GmbH The CocoQML scanner where the options stand for: hdirectoryi: is the path to a directory that contains files that will be instrumented or restored. The action depends on whether the option -r (see below) is set. If -r is not set, the hdirectoryi is scanned recursively and each file that ends with “.qml” or “.js” is read and instrumented. A backup copy of each file is kept; it has the suffix “.qmlbak” or “.jsbak”, respectively. If there are already backup copies, nothing is done and the program returns with an error. If -r is set, all the backup files renamed back to their old names, replacing the instrumented files. If there are no backup copies, the program does nothing and returns with an error.
-r | --restore: Enable the restore mode.
-c | --csmes-name=hfilenamei: If this option is used, the resulting .csmes file is saved as hfilenamei. A suffix “.csmes” is appended if it does not exist already. Without this option, the resulting data is saved as cocoqmlscanner_result.csmes.
-v | --version: Print the version number of the program and finish.
-h | --help: Print a help message and finish. The program also respects all CocoQML environment variables (see Chapter 13.5, page 111).
Squish Coco 5.1.0 - 113 - froglogic GmbH Tcl Coverage Part VI
Squish Coco 5.1.0 - 114 - froglogic GmbH Command Line Arguments
Chapter Command Line Arguments 14
Syntax:
coveragescannertcl [hcs-optionsi] happlication argumentsi
Where: happlication argumentsi: Tcl-based application including optional arguments hcs-optionsi: Any of the following options: --cs-output=hSTRINGi: Base name of the generated .csmes and .csexe files --cs-exclude-path=hSTRINGi: Exclude files located in specified directory --cs-include-path=hSTRINGi: Override exclusion of files based on specified directory --cs-exclude-file-abs-regexp=hSTRINGi: Exclude file if absolute file name matches regular expression --cs-exclude-file-abs-regexp=hSTRINGi: Override exclusion of file if absolute name matches regular expression --cs-exclude-file-abs-wildcard=hSTRINGi: Exclude file if absolute name matches wildcard --cs-exclude-file-abs-wildcard=hSTRINGi: Override exclusion of file if absolute name matches wild- card --cs-include-code-regexp=hSTRINGi: Regular expression of code that will be instrumented even if it is excluded --cs-exclude-code-regexp=hSTRINGi: Regular expression of code that will not be instrumented --cs-include-code-wildcard=hSTRINGi: Wildcard of code that will be instrumented even if it is ex- cluded --cs-exclude-code-wildcard=hSTRINGi: Wildcard of code that will be instrumented even if it is ex- cluded --cs-include-init-scripts: Enables code instrumentation for standard Tcl initialization scripts --cs-include-unnamed: Instrument Tcl code even if it has no or just an ambiguous file name set. By default such code is ignored as it is typically just temporary code generated for e.g. an Tk event handler or results from nested eval calls.
Squish Coco 5.1.0 - 115 - froglogic GmbH Command Line Arguments
@hpathi: Read command line options from the file at hpathi and insert them at the position of this option. The option file is a text file with one option per line. Leading and trailing blanks and empty lines are ignored. CoverageScannerTcl is a utility which collects code coverage information of a Tcl-based execution. It will collect all scripts parsed by the embedded Tcl interpreter and determine which parts were actually executed.
Once the application exits a pair of .csmes and .csexe files will be written out that contain the code instrumentation information and execution data, respectively.
If the --cs-output switch is omitted the base name of the output files will the same as the happlicationi name.
See (see Chapter 16, page 119) for a recommended set of --cs-exclude-code-regexp switches for typical Tcl/Tk applications.
Squish Coco 5.1.0 - 116 - froglogic GmbH Getting started
Chapter Getting started 15
i At the moment, this example only works with Tcl v8.5 or earlier. See Section 15.1, page 118 below for details.
Create a file hello.tcl with the following content:
if { $argc == 0 } { puts "Hello World!" } else { for {set i 0} {$i<$argc} {incr i} { if { $i == 0 } { set hello "Hello [lindex $argv $i]" } elseif { $i == $argc - 1 } { set hello "$hello and [lindex $argv $i]" } else { set hello "$hello, [lindex $argv $i]" } } puts $hello }
Execute hello.tcl using the tclsh interpreter:
$ coveragescannertcl tclsh hello.tcl Jim John James
Two files are generated: tclsh.csmes: The instrumentation database of hello.tcl tclsh.csexe: The execution report. To import the execution report into its instrumentation database, execute:
$ cmcsexeimport -m tclsh.csmes --title="Jim John James" tclsh.csexe
Squish Coco 5.1.0 - 117 - froglogic GmbH Using more than one Tcl version on one system
After the import, it is possible to analyze the code coverage data directly with CoverageBrowser or generate a HTML report:
$ cmreport --title="Hello application" -m tclsh.csmes --html=tclsh.html
15.1 Using more than one Tcl version on one system
Squish Coco v3.0 does not yet support the use of tclsh under Tcl v8.6. If you want to use coveragescannertcl with tclsh, Tcl v8.5 or earlier is necessary. In some UNIX®systems (like Debian Linux™), more than one Tcl version can be present at the same time. The executables are then distinguished by their names. On such a system, the tclsh executable of Tcl v8.5 may be called tclsh8.5.
We can then run CoverageScannerTcl such that it calls tclsh8.5. On its own, this would however change the names of the generated files into tclsh8.5.csmes and tclsh8.5.csexe. We can correct this with another commandline option, --cs-output=tclsh. The first command in the example above now becomes:
$ coveragescannertcl --cs-output=tclsh tclsh8.5 hello.tcl Jim John James
Afterwards, the rest of the example can be done as before.
Squish Coco 5.1.0 - 118 - froglogic GmbH Coco TCL Tutorial
Chapter Coco TCL Tutorial 16
16.1 Analysis of Tcl script coverage
16.1.1 Demo application build
Copy the sample to your working diretcory
$ cp -r /opt/SquishCoco/doc/demoapp_tcl ./ $ cd demoapp_tcl
Edit Makefile and adjust the following variables
TCLLIBS=-ltk8.5 -litcl3.4 -ltcl8.5 TCLINCLUDES=-I/usr/include/tcl8.5 to match your system’s library names and include paths. Build the C++ executable with:
$ make
Test the simple Tk GUI with an invocation of the launcher script:
$ ./proto
16.1.2 Ad-hoc Tcl coverage run
Launch the application under test via the Tcl coverage scanning tool :
$ coveragescannertcl --cs-output=tcl ./proto
The GUI will show up as normal while code coverage is monitored. At this point the application might already be shut down via the "File¡Exit" menu entry or the [X] button in the window’s title bar. Observe the two files
Squish Coco 5.1.0 - 119 - froglogic GmbH Analysis of Tcl script coverage tcl.csmes and tcl.csexe being generated. The latter contains data about the code covered in the last execution. The former is a measurement database containing information about the instrumented code and possibly imported previous executions. In this case the base name of the output files was set via the --cs-output switch. If omitted it will be deduced from the application name.
16.1.3 Result analysis
The results can be view interactively in the GUI provided by Coco. Run
$ coveragebrowser to launch it. Select "File¡Open..." to open the tcl.csmes file. You’ll see all Tcl files and procedures listed in the Sources and Methods tab, respectively. When selecting either of them a source code viewer will open that shows instrumented code with a gray background color. Note that the coloring is still limited to instrumented, i.e. seen code only. To check which code got actually executed (or not) select "File¡Load Execution Report..." and open tcl.csexe. Now red and green colors denoting positive and negative coverage will show up for files, procedures and the code itself.
16.1.4 Filtered Tcl coverage run
As you might have noticed that above ad-hoc run produced results for several unnamed and internal files. These stem from internal Tcl/Tk files or e.g. mouse event handlers that result in on-the-fly eval calls. Compared to the application-specific Tcl files the coverage information for these internal pieces of code is of little interest.
The easiest way to hide the non-relevant pieces of code is through usage of the various --cs-exclude-code options. A recommended list of switches will be shown in the repetition of the previous run with filtering options applied:
$ coveragescannertcl --cs-output=tcl --cs-exclude-code-regex="^\s*[^\n]*\s*$" --cs-exclude-code-regex="(tcl|tk)Init" --cs-exclude-code-regex="namespace eval :: itcl" --cs-exclude-code-regex="^# This file is generated" --cs-exclude-code-regex="tk::Menu(Motion|Leave)" ./proto
When viewing the new tcl.csmes in CoverageBrowser you’ll notice that the list of files is reduced to the relevant canvas.tclx, input.tclx, mainwindow.tclx, point.itcl and rect.itcl scripts.
16.1.5 Report Generation
All of text, XML and HTML reports can be generated from the CoverageBrowser GUI. With future automation of nightly test runs or Continuous Integration (CI) in mind we’ll create the report from the command line. And for this very purpose we’ll also pretend that the execution data has not been imported into the measurement database, yet. This step can be accomplished like this from the command line:
$ cmcsexeimport --title="First run" -m tcl.csmes -e tcl.csexe
The resulting proto.csmes file now contains everything we need to generated a report:
$ cmreport --title="Coverage Data" -m tcl.csmes --html=report.html
Squish Coco 5.1.0 - 120 - froglogic GmbH Analysis of C/C++ code coverage
The resulting report.html file (including a report_html/ directory) can now be visited with a Web browser. It includes a listing of executions, source files, functions and metrics on a global level, per file and per functions.
16.2 Analysis of C/C++ code coverage
16.2.1 Instrumentation
While Tcl code can be instrumented for coverage analysis on-the-fly this step requires a rebuild for C or C++ applications. But this can be accomplished with almost no change to your regular application build. Here are the two possible approaches: • Modify your build system to make use of the code scanning tools csgcc, csg++, etc. that will both instrument the code and later use the original compiler to compile the code as usual. • Sneak Coco’s compiler wrappers named gcc, g++, etc. into the executable search PATH. They will again instrument the code and then let the real compiler do its job. We’ll make use of approach number 2. First, clean the existing object files:
$ make clean
Then, recompile the application with a changed PATH and a switch that enables the compiler wrappers. For brevity, we’ll do this all in a single line:
$ PATH=~/SquishCoco/wrapper/bin:$PATH COVERAGESCANNER_ARGS="--cs-on" make
As a result we’ll not only get the main executable (proto.exe) but also a corresponding instrumentation file proto. exe.csmes. Information about the instrumented code (gray background color) can optionally be reviewed at this point
$ coveragebrowser -m proto.exe.csmes
16.2.2 Execution
Let’s run the application
$ ./proto and quit it via its GUI right after. The current working directory will now contain a proto.exe.csexe containing information about completed execution.
16.2.3 Result analysis
Same as with Tcl the results can now be reviewed in CoverageBrowser. A short cut invocation that opens both the instrumentation and execution files is shown here:
Squish Coco 5.1.0 - 121 - froglogic GmbH Analysis of C/C++ code coverage
$ coveragebrowser -m proto.exe.csmes -e proto.exe.csexe
16.2.4 Report Generation
The process of a report generation works analogous to the approach shown for Tcl code above.
16.2.5 Analysis of mixed Tcl and C/C++ code coverage
Coverage data for the Tcl and C/C++ parts of a hybrid application can be gathered independently as shown above. Either in a single or multiple executions. The only extra step needed is the merging of the data into a single measurement database. Which then can be used to generate a single, combined report taking both languages into account.
16.2.6 Combined coverage run
Assuming the instrumentation of the C/C++ code is still in place we can gather coverage data for both types of languages with a single run: $ coveragescannertcl --cs-output=tcl ./proto
With the proto.exe.csmes file still in place from the application rebuild we’ll now have two sets of files:
1. tcl.csexe
2. tcl.csmes
3. proto.exe.csexe
4. proto.exe.csmes
At this point the coverage data for Tcl and C/C++ could be analysed independently using cmcsexeimport. But we want to gain an overall picture and will therefore merge the results in the next section.
16.2.7 Merging measurement data
Instead of using the Coverage Browser GUI we’ll again use the command line tools. In the first step we’ll merge the instrumentation files stemming from the two languages into one: cmmerge -o all.csmes tcl.csmes proto.exe.csmes
To save some disk space the --append option can be used rather than creating the new all.csmes file.
In the second step the two execution files are merged using the merge policy for the 2nd language: $ cmcsexeimport -m all.csmes -e tcl.csexe --title="Hybrid" --passed $ cmcsexeimport -m all.csmes -e proto.exe.csexe -p merge --title="Hybrid" --passed
For the merge to actually happen the title has to be identical.
Squish Coco 5.1.0 - 122 - froglogic GmbH Analysis of C/C++ code coverage
16.2.8 Report Generation
An HTML report encompassing coverage for both application parts can be generated with a single command line call:
$ cmreport --title="Hybrid Tcl/C++ Coverage" -m all.csmes --html=report.html
Select "Source Files" to see both Tcl (various files) and C++ code (main.cpp) being listed in a combined report.
Squish Coco 5.1.0 - 123 - froglogic GmbH CoverageBrowser Part VII Reference
Squish Coco 5.1.0 - 124 - froglogic GmbH Introduction
Chapter Introduction 17
CoverageBrowser is a graphical user interface program which enables the user to analyze the test coverage. CoverageBrowser is typically used in the following way:
1. Load an instrumentation database (a .csmes file) that was generated by CoverageScanner.
2. Load a corresponding execution report (a .csexe file). There may be several reports to choose from: Cover- ageBrowser displays them in a tree view where they can be selected or deselected individually for coverage analysis. 3. Search for untested code segments. 4. Mark dead code or code which cannot be tested as “manually validated”. 5. Add comments to the instrumented code.
CoverageBrowser saves all these data (execution reports, comments, etc.) to the instrumentation database.
17.1 Command Line Arguments
Syntax:
coveragebrowser coveragebrowser -m hcsmes_filei ... coveragebrowser hcsmes_filei ...
Where: hcsmes_filei |-m hcsmes_filei | --csmes=hcsmes_filei: Load an instrumentation database from the hcsmes_filei. This option can be used at most once.
-e hcsexe_filei | --csexe=hcsexe_filei: After the CoverageBrowser has started, and import dialog is opened to import hcsexe_filei. This option can be used only once, and only if a hcsmes_filei is given.
Squish Coco 5.1.0 - 125 - froglogic GmbH Command Line Arguments
@hpathi: Read command line options from the file at hpathi and insert them at the position of this option. The option file is a text file with one option per line. Leading and trailing blanks and empty lines are ignored. If no option is given, the CoverageBrowser tries to open the hcsmes_filei again that it had shown in its previous run.
Squish Coco 5.1.0 - 126 - froglogic GmbH Black box and white box testing
Chapter Black box and white box testing 18
CoverageBrowser can be used both for white box testing and black box testing: If no source code information is available in the instrumentation database (i.e., in the .csmes file), CoverageBrowser will switch to black box testing mode. In this mode, CoverageBrowser has a simpler user interface that does not provide the functionality that is possible only with access to the source code. Nevertheless, even with this reduced functionality, it is still possible to import and manage executions.
A black box instrumentation database can be generated by clicking on "File¡Generate Black-Box Configuration...". It is possible to merge such a database into a white box configuration at a later stage.
Squish Coco 5.1.0 - 127 - froglogic GmbH Black box and white box testing
Figure 18.1: CoverageBrowser’s simplified user interface for black box tests
Squish Coco 5.1.0 - 128 - froglogic GmbH The windows of CoverageBrowser
Chapter The windows of CoverageBrowser 19
19.1 The Executions Window
This window is present in both black box and white box testing mode. It shows all application executions that have been done, including details of their code coverage.
19.1.1 Principles
Executions of the instrumented application are displayed in a tree view in the "Executions" window. Coverage- Browser uses a slash ‘/’ as a separator for grouping measurements together. For example, the tests shown in Figure 19.1, page 129 have the following names:
• SubTest/test1 • SubTest/test2 (12) • SubTest/test2 (2)
Code Coverage Execution State Statistics
Selection using Selection Wildcard Expressions
or Regular Expressions
Figure 19.1: The Executions View
The checkbox next to each item can be used to select executions. The "Sources", "Functions" and "Source Viewer" windows only display the code coverage status of the selected executions.
The code coverage percentage of the tests is visualized by gray horizontal bars in the "Coverage" column. The input field allows filtering the output with regular expressions (see Chapter 20.1, page 141). Click the
Squish Coco 5.1.0 - 129 - froglogic GmbH The Executions Window button to select all the visible executions (i.e., all those that have not been filtered out). Or click the button to unselect all the executions.
For a more finely controlled filter, click the "..." button: This will pop up a dialog by which it is possible to set filtering depending on the execution state and the comments.
Note that the text which is filtered is a test execution’s full name—for example, SubTest/test1.
- Click the + button to switch to the execution comparison analysis mode (see Chapter 19.1.3, page 133).
The user can set the state of the executed test by clicking into the "State" field of a test. The new state can be any of the following:
"Unknown": Default state.
" Passed ": This state is used to mark the test as passed. The background colour of the "State" field is then green.
" Failed ": This state is used to mark the test as failed. The background colour of the "State" field is then red.
" Need manual check ": This state is used to indicate that the test must be done manually. The background colour of the "State" field is then orange. i The name of the test item and its state can also be defined by an external test suite. (see Chapter 34, page 189)
It is possible to rename, delete or merge executions, or add comments to them through the use of CoverageBrowser’s context menus and dock windows1. One can use regular expressions to identify the executions to which these modifications are applied. (The regular expression syntax is described in Chapter 20.1, page 141.) Before regular expression-driven actions are carried out, CoverageBrowser shows a preview of what effects the change would have.
To delete executions, right-click into the "Executions" window and select "Delete multiple executions..." from the context menu. A window appears into which the name of the executions can be entered. Here are some examples: • To delete all execution using the wildcard syntax: Execution Name: *
• To delete all executions in TESTS using the wildcard syntax: Execution Name: TESTS/*
• To delete all executions in TESTS using the regular expression syntax: Execution Name: =TESTS/.* To rename executions, select "Rename multiple executions..." from the context menu. A window appears (see Figure 19.2, page 131) into which expressions for the old and new names of the executions can be entered. Here are some examples:
• To move all executions to the directory TESTS, set: Actual Execution Name: =.* New Execution Name: TESTS/&
• To move all executions in the directory TESTS to the directory OLD, set: Actual Execution Name: =TESTS/(.*) New Execution Name: OLD/\1
1After merging, the executions used as the source of the new merged execution are deleted.
Squish Coco 5.1.0 - 130 - froglogic GmbH The Executions Window
• To rename all executions in all directories in testname [directory], set: Actual Execution Name: =([^/]*)/([^/]*) New Execution Name: \2 [\1]
Regular Expression
Executions to Rename
New Execution Name
Preview
Error
Error Message
Figure 19.2: Renaming with regular expressions
19.1.2 Loading an Execution Report
An execution report is produced when an instrumented application finishes its execution. It contains the list of all executed code segments for each application that was run. The execution report is never overwritten; execution data are always appended. Its file name is defined by the initialization function __coveragescanner_install() of the CoverageScanner library (see Chapter 10.1.1, page 86) and always ends with ".csexe".
To load an execution report, click on "File¡Load Execution Report..." or the icon on the toolbar. A dialog like that in Figure 19.3, page 131 appears.
Execution result file name
Default execution name
Default execution status
Figure 19.3: Execution report loading dialog
Import from a file The report can be imported directly or via a script. To load it directly, select "File" in the top left menu. Enter the path of the .csexe file into the free form input box, or use the browse button.
The "Name" field specifies the name of the imported instrumentation if it is not already specified in the execution report (see Chapter 10.1.2, page 87). It is also possible to set the status (“Passed”, “Failed” or “Requires Manual Checking”) of all imported executions for which it was not set at runtime. By default the status is “Unknown”. Invalid executions are not imported. If more than one instrumentation is imported with the same name, an index number is appended to the name to make it unique. An execution with the name “Main Test” may then become “Main Test (2)”.
To see which executions will be imported, press the button "Next", which shows a list of all executions in the file you want to load, together with a description whether they will be imported and why. (see Figure 19.4, page 132)
Squish Coco 5.1.0 - 131 - froglogic GmbH The Executions Window
Figure 19.4: Import overview
At this point, no import yet has taken place. One still needs to press the "Import" (or "Import & Delete") button to load the .csexe data. Or one can press the "Back" to change the settings, or "Cancel" to stop it all.
Import by a script If the execution report is not accessible through the file system, a script can be used to import it. To do this, select "Script:" in the menu at the top left of the dialog. Two input fields become visible:
"Fetch Command": Enter here the script that imports the execution report, together with its parameters. The script must print the content of the execution report to the standard output (stdout). If it writes to the standard error stream (stderr), the output occurs in a window and can be used for debugging. On success, the script must return the value 0.
"Delete Command": Enter here the script that deletes the execution report, together with its parameters. This field needs only to be filled in when the "Delete execution report after loading" mode is se- lected. In both input fields, quotation marks (single and double quotes) can be used to group arguments, and backslashes can be used as escape characters.
Pressing the "Next" button executes the fetch command. If it fails, an error report is shown. If it is successful, a preliminary list of imported executions is shown as before. To finish the import, it is again necessary to press the "Import" (or "Import & Delete") button. Additional options grouped in the advanced parameters section can also be specified:
• If the checkbox "Delete execution report after loading" is enabled, the execution report file will be automatically deleted after it has been loaded.
• If the checkbox "Open this dialog automatically when the file is modified" is selected, the im- port dialog reappears whenever the .csmes file changes, and it can be loaded again.
Squish Coco 5.1.0 - 132 - froglogic GmbH The Executions Window
A workflow to collect test results interactively is to have this and the previous checkbox selected and then run the instrumented program repeatedly. Each time the program ends, a .csmes file is created and can be imported in the CoverageBrowser and then be deleted. This way the test results accumulate.
• If the checkbox "Check output for scripts" is selected and the execution reports are loaded by scripts, an output window is shown when the script is executed. It displays the output of the script while it runs, and finally its return value.
• The option "Import Preprocessing" selects the behaviour in case of conflicts or redundant executions: "Ignore Duplicate Executions": Executions are ignored if they have executed the same code as an execution that was already imported. "Import Duplicate Executions": Executions are imported if at least one instrumented source code line is executed. "Import Duplicate and Empty Executions": All executions are imported. "Merge executions with the same name": All executions with the same name are merged. The execution counts of all merged instrumentations are added.
19.1.3 The Execution Comparison Analysis Mode
- The Execution Comparison Analysis mode is activated by clicking the button + . In this mode, one or more executions must be selected as references and others as those to analyze. CoverageBrowser then shows only those lines as covered that were executed in the analyzed executions but not in the reference executions. Similarly, the coverage statistics displayed in the "Sources" list contain only the percentage of instrumented statements that were executed only by the executions to analyze. With Execution Comparison Analysis mode it is therefore possible to find changes in the coverage between different runs of a program.
To select the reference executions, use the checkboxes in the "Reference" column at the right of the "Executions" window. They must be selected first. Then select the executions to analyze as before in the "Execution" column. i If the execution to analyse is present in the list of reference executions, it is implicitly removed from the list. So if execution A is compared to executions A and B, CoverageBrowser actually compares execution A only with B, since comparing A with itself will provide no useful information.
Squish Coco 5.1.0 - 133 - froglogic GmbH The Source Browser Window
Activation
Reference List
Coverage Statistic of Execution of lines which Analysed are only coveraged by the selected execution
Instrumented lines which are only coveraged by the selected execution are displayed
Figure 19.5: Execution Comparison Analysis Mode
19.2 The Source Browser Window
i This feature is not available for black box testing.
The "Sources" window can be displayed by selecting the menu entry "View¡Sources".
Source
Header and Statistics Source Files per File Compiled
Source excluded from the instrumentation
Figure 19.6: Source Browser Window
In each line, the "Source" column contains the name of a source file. When a line is clicked, the "Source Window" is displayed. For C and C++, there are sub-entries for included header files which have been instrumented. If a file has been compiled more than once but with different preprocessor options, the file gets more than one entry. To distinguish them, the entries get numbers attached. In the example, the file context.h could then become the two entries “context.h #1” and “context.h #2”. When coverage percentages are computed, each variant counts as a separate file.
Squish Coco 5.1.0 - 134 - froglogic GmbH The Function Browser Window
The "Coverage" column displays a rudimentary code coverage statistics for each source code file. There is an underlying horizontal bar in each field whose length represents the fraction of the code that is covered. The color of the main part of the bar is selected according to code coverage statistics for each file and the value of the thresholds (see Chapter 22.3, page 155). If parts of the code have been manually validated, then there is a second, gray, horizontal bar at the left of the "Coverage" column; its width represents the number of tests that have been only manually validated. The full length of the column still represents the fraction of all tests that have been validated. When the window however also contains a "Manual Validations" column, the gray horizontal bar is only displayed there. The checkbox in front of each source file lets you exclude or include the source file to the statistics computation. If excluded, the file is treated as if it and its functions had not been instrumented. The input field allows filtering the content of the window with regular expressions (see Chapter 20.1, page 141). The filter expression refers to the full path of the source file. (Example: c:\directory\file.cpp)
Icon Shortcut Description Ctrl+Shift+F Previous source file Ctrl+F Next source file
Table 19.1: Source Browser - Shortcuts
19.3 The Function Browser Window
i This feature is not available for black box testing.
The "Functions" window can be opened by selecting the menu entry "View¡Functions".
Prototype
Namespace or Class
Statistics Method
Figure 19.7: Function Browser Window
It displays the code coverage statistics for all functions, classes and namespaces. A click on an item in the window shows the code for the corresponding object in the "Source Viewer" window and highlights it.
The "Method" columns contains the names of the function or class method whose coverage is shown. If hierarchical display is enabled, it also displays classes and name spaces. If a function is defined in a file that is compiled more than once and with different preprocessor options, it gets several entries. To distiguis them, the name of the file and a number is attached. A function get_context() in the file context.h could then have the two entries “get_context()[error.h #1]” and “get_context()[error.h #2]”. The names and the numbers of the files are the same as those in the "Sources" window (see Chapter 19.2, page 134).
Squish Coco 5.1.0 - 135 - froglogic GmbH The Source Viewer Window
The "Coverage" column displays a rudimentary code coverage statistics for each function. There is an underlying horizontal bar in each field whose length represents the fraction of the code that is covered. The color of the main part of the bar is selected according to code coverage statistics for each file and the value of the thresholds (see Chapter 22.3, page 155). If parts of the code have been manually validated, then there is a second, gray, horizontal bar at the left of the "Coverage" column; its width represents the number of tests that have been only manually validated. The full length of the column still represents the fraction of all tests that have been validated. When the window however also contains a "Manual Validations" column, the gray horizontal bar is only displayed there. The input field lets you filter the output with regular expressions (see Chapter 20.1, page 141). The filter expression refers to the full names of the items, including the class name and the namespace. (Example: MyNamespace:: MyClass::MyProc)
19.4 The Source Viewer Window
i This feature is not available for black box testing.
19.4.1 Source Display
The "Source Viewer" window can be displayed by clicking on "View¡New Source Window".
Selected Folding Instrumentation Filter
Unselected
Comment Instrumentation Mark
Test and Code Coverage Count
Figure 19.8: Source Window
CPP The "Source Viewer" window displays the source file or its C or C++ preprocessed view. Clicking on C++ enables the user to toggle between the 2 different views. The source code is colored with code coverage instrumentations. The colors used are described in section 19.4.2, page 137. By selecting an area with the mouse, corresponding instrumentations are highlighted and a detailed description of them is displayed in the "Explanation" window (see Chapter 19.5, page 139). It is possible to navigate between instrumentations using the navigation buttons and . Navigation buttons in yellow, blue, red
Squish Coco 5.1.0 - 136 - froglogic GmbH The Source Viewer Window and green jump to the next or previous comments, manually validated instrumentations, non-executed code parts or executed code parts. Clicking on the source code selects the nearest instrumentation. If a comment is entered for an instrumentation, the icon is displayed in the margin. On the right side, CoverageBrowser displays the test coverage count2 or the code coverage count3 for each line. If a source code line contains more than one instrumentation, CoverageBrowser display the range of their counts.
Mouse Wheel Description Wheel Scroll up/down Ctrl+Wheel Zoom in/out Shift+Wheel Next/previous instrumentation
Table 19.2: Source Display - Mouse Wheel
Icon Shortcut Description Ctrl+Shift+B Previous comment Ctrl+B Next comment Ctrl+Shift+U Previous unexecuted code Ctrl+U Next unexecuted code Ctrl+Shift+T Previous executed code Ctrl+T Next executed code Ctrl+Shift+V Previous manually validated instrumentation Ctrl+V Next manually validated instrumentation Ctrl+Shift+I Previous instrumentation Ctrl+I Next instrumentation Ctrl+Shift+F Previous source file Ctrl+F Next source file Ctrl+J Open a new source window
CPP C++ Ctrl+Shift+J Switch between the preprocessor view and the original source Ctrl+K Add/Edit Comments Ctrl+Shift+K Remove Comments ok Ctrl+D Mark as Validated ok Ctrl+Shift+D Clear Validation Flag Ctrl+Z Undo Ctrl+Shift+Z Redo
Table 19.3: Source Display - Shortcuts
19.4.2 Color Convention
Instrumentations are displayed in a source window using different colors:
Green - "Executed": An instrumentation is displayed in green when the code has been executed.
Orange - "Partially Executed": An instrumentation is marked as "Partially Executed" when it is not completely executed. This occurs when a Boolean expression was only true or false for example. In the case of a source code line which contains more than one instrumentation, the line is marked as "Partially Executed" when one of its instrumentations
2The test coverage count is obtained by generating the instrumentation with CoverageScanner using the option --cs-hit 3The code coverage count is obtained by generating the instrumentation with CoverageScanner using the option --cs-count
Squish Coco 5.1.0 - 137 - froglogic GmbH The Source Viewer Window
has not been "Executed". A detailed information is displayed in the "Explanation" window (see Chapter 19.5, page 139).
Red - "Never Executed" or "Execution count too low": An instrumentation is displayed in red when the code is never executed or when the execution count is lower that than the execution count requested.
Magenta - "Dead-Code": An instrumentation is displayed in magenta when the code cannot be executed.
Blue - "Manually Set To Be Executed": The user has the possibility to mark an instrumentation as ’Manually Validated’. This is usually to exclude dead code or code which cannot be tested for code coverage statistics. This state is only relevant if executions are in a "Never Executed" or "Partially Executed" state.
Gray - "Unknown" or "Hidden": Gray is used when no information about instrumentation is available. This occurs when no executions are selected or when comparing executions of tests (see Chapter 19.1.3, page 133).
19.4.3 Comments
i This feature is not available for black box testing.
19.4.3.1 Editing Comments
It is possible to add a comment by selecting an instrumentation and clicking on the context menu entry "Add/Set Comment", the main menu entry "Instrumentation¡Add/Set Comment" or the icon on the toolbar.
The "Comment" Window 19.9, page 138 appears and allows a comment to be edited. The most recently entered comments can be retrieved by clicking on the "Last Comments" selection field. Basic text formatting is possible using the integrated toolbar buttons (see 19.4, page 139).
Edit Field Text Formating
Last Comment
Comments Length
Figure 19.9: Comment Editing
i If a minimal length for a comment is set, the comment can only be entered if this is reached (see Chapter 22.2, page 155).
The comment is printed in the explanation in a yellow box and the icon ( ) is displayed in the source window near the line number.
Squish Coco 5.1.0 - 138 - froglogic GmbH The Explanation Window
Icon Shortcut Description Ctrl+S Strikeout Ctrl+B Bold Ctrl+I Italic Ctrl+U Underline Ctrl+J Justify Ctrl+R Align Right Ctrl+L Align Left Ctrl+M Center Ctrl+Z Undo Ctrl+Shift+Z Redo
Table 19.4: Comments - Shortcuts
19.4.3.2 Removing Comments
It is possible to remove a comment by selecting an instrumentation and clicking on the context menu entry "Clear Comments", the main menu entry "Instrumentation¡Clear Comment" or the icon on the toolbar.
19.5 The Explanation Window
i This feature is not available for black box testing.
The "Explanation" Window 19.10, page 140 is a docking window which is automatically updated with a detailed description of the selected instrumentations of the source window. For each instrumentation, the following information is displayed: 1. A short description of the instrumentation state (see Chapter 19.4.2, page 137). 2. The preprocessed source code which is concerned by the instrumentation. 3. For Boolean expressions, the truth-table which shows executed and unexecuted states.
4. The list of executions which are executing the portion of code. 5. User comments.
Squish Coco 5.1.0 - 139 - froglogic GmbH The Statistics Window
Truth-Table
Preprocessed
State Source
Comment
Figure 19.10: Explanation Window
CoverageBrowser displays the truth-table in the case of a Boolean expression which is partially executed. The truth-table indicates which value the expression has or has not reached during execution. Example: the truth-table 19.5, page 140 indicates that the expression was false but not true.
TRUE FALSE no yes
Table 19.5: Truth-Table Example
19.6 The Statistics Window
The "Statistics" Window 19.11, page 140 is a docking window which is automatically updated with the code coverage statistic for the whole project.
Figure 19.11: Statistics Window
If parts of the code are manually validated, then their percentage is also displayed in the coverage statistics. The bar chart has then two regions: the percentage of the manually validated code at the left and then the percentage of the code that is covered by the automatic tests. The numbers in the bar chart refer to all validated code, that which was manually validated together with that covered by tests.
By clicking the "..." button, the two kinds of validation are split into two separate bars.
Figure 19.12: Statistics Window with Manual Validation, Split
Squish Coco 5.1.0 - 140 - froglogic GmbH Working with CoverageBrowser
Chapter Working with CoverageBrowser 20
20.1 Filtering with wildcards or regular expressions
CoverageBrowser provides a generic filtering mechanism of rows using wildcard or regular expressions. Wildcard expressions are activated per default and regular expressions are selected when the expression starts with an equal sign (’=’). Clicking on the filter icon converts the expression from wildcard into regular form as far as this is possible and vice versa.
Icon Description .* The filter uses regular expression syntax. ?* The filter uses wildcard syntax. ! Syntax error. More information are displayed in the status bar.
Table 20.1: Filter States
20.1.1 Wildcard Expressions
Element Meaning * any characters (0 or more) ? any character [...] set of character
Example: foo*bar match any tests containing the string foo followed by bar.
20.1.2 Regular Expression
The first character must be ’=’ to activate the regular expressions.
Squish Coco 5.1.0 - 141 - froglogic GmbH Filtering
20.1.2.1 Pattern matching
Element Meaning c Any character represents itself unless it has a special regexp meaning. Thus c matches the character c. \c A character that follows a backslash matches the character itself except where mentioned below. For example if you wished to match a literal caret at the beginning of a string you would write \^. \a This matches the ASCII bell character (BEL, 0x07). \f This matches the ASCII form feed character (FF, 0x0C). \n This matches the ASCII line feed character (LF, 0x0A, Unix newline). \r This matches the ASCII carriage return character (CR, 0x0D). \t This matches the ASCII horizontal tab character (HT, 0x09). \v This matches the ASCII vertical tab character (VT, 0x0B). \xhhhh This matches the Unicode character corresponding to the hexadecimal number hhhh (between 0x0000 and 0xFFFF). \0ooo (i.e., zero ooo) matches the ASCII/Latin1 character corresponding to the octal number ooo (between 0 and 0377). . (dot) This matches any character (including newline). \d This matches a digit. \D This matches a non-digit. \s This matches a whitespace. \S This matches a non-whitespace. \w This matches a word character. \W This matches a non-word character. ^ The caret negates the character set if it occurs as the first character, i.e. immediately after the opening square bracket. For example, [abc] matches ’a’ or ’b’ or ’c’, but [^abc] matches anything except ’a’ or ’b’ or ’c’. - The dash is used to indicate a range of characters, for example [W-Z] matches ’W’ or ’X’ or ’Y’ or ’Z’. E? Matches zero or one occurrence of E. This quantifier means "the previous expression is optional" since it will match whether or not the expression occurs in the string. It is the same as E{0,1}. For example dents? will match ’dent’ and ’dents’. E+ Matches one or more occurrences of E. This is the same as E{1,}. For example, 0+ will match ’0’, ’00’, ’000’, etc. . . E* Matches zero or more occurrences of E. This is the same as E{0,}. The * quantifier is often used by a mistake. Since it matches zero or more occurrences it will match no occurrences at all. For example if we want to match strings that end in whitespace and use the regexp \s *$ we would get a match on every string. This is because we have said find zero or more whitespace followed by the end of string, so even strings that do not end in whitespace will match. The regexp we want in this case is \s +$ to match strings that have at least one whitespace at the end. E{n} Matches exactly n occurrences of the expression. This is the same as repeating the expression n times. For example, x{5} is the same as xxxxx. It is also the same as E{n,n}, e.g. x{5,5}. E{n,} Matches at least n occurrences of the expression. E{,m} Matches at most m occurrences of the expression. This is the same as E{0,m}. E{n,m} Matches at least n occurrences of the expression and at most m occurrences of the expression. () groups expressions into sub-expressions. | Alternative. Example:"aaa|bbb" matches the string "aaa" or "bbb".
Squish Coco 5.1.0 - 142 - froglogic GmbH Code Coverage Level
20.1.2.2 String substitution
Element Meaning & Matched expression \n sub-expression number n. Example: the regular expression is "(.*):([0-9]*)" matches the string "joe:18". The replacement string "\1 is \2" will produce the result: "joe is 18"
20.2 Code/Test Coverage Level
The menu entry "Instrumentation¡Level:x" sets the targeted code coverage count or, if compiled with instru- mentation hit support1, the targeted test coverage count. The level determines the number of executions/test coverage runs necessary to consider that an instrumented code is executed. Example: Setting the level to 10, will make it necessary to execute 10 times each line of the source code if compiled with code coverage count. If compiled with code coverage hit, 10 execution runs need to execute each line of the source code. The menu entry "Tools¡Test Coverage Count Mode" and the button 1.2.3. switch between code coverage count and test coverage count analysis. This provides the behaviour of code coverage hit analysis 2 when the project is compiled with code coverage count support3.
20.3 Code Coverage Algorithm
CoverageBrowser displays the code coverage analysis (statement block, decision or condition) generated by Cov- erageScanner. But "Instrumentation¡Coverage Method¡Statement Block" lets you reduce the analysis to the code coverage of statement blocks. This produces the same result as compiling with the --cs-statement-block of CoverageScanner. Similarly, "Instrumentation¡Coverage Method¡Decision" shows the code coverage analysis at the Decision level. Here is a short overview of the command line options necessary for each code coverage analysis method:
Coverage analysis CoverageScanner command line option Statement Block --cs-statement-block Decision with full instrumentation --cs-decision Decision with partial instrumentation --cs-decision --cs-partial-instrumentation Condition with full instrumentation (default) Condition with partial instrumentation --cs-partial-instrumentation
1Command line option --cs-hit from CoverageScanner. To compile with code coverage count support, the option --cs-count must be added to the command line. 2CoverageScanner command line option --cs-hit 3CoverageScanner command line option --cs-count
Squish Coco 5.1.0 - 143 - froglogic GmbH Optimized execution order
20.4 Optimized execution order
CoverageBrowser can calculate an execution order for tests with which high code coverage can be reached fast with a small number of tests. In this execution order, the test with the highest coverage comes first. The second test is the that one which makes the additional code coverage as high as possible, and so on. This feature is meant for cases where the full test suite cannot be executed, e.g. because there is not enough time or there are many manual tests. Then one can run a number of tests from the beginning of the list, say the first 20, and still get a high coverage fast. To calculate the execution order proceed as follows:
1. Select a set of executions in the "Executions" window.
2. Click on "View¡Optimized Execution Order...". The "Optimized Execution Order" window (see Figure 20.1, page 144) is then displayed.
3. Click on the "Compute" button to start the analysis.
Execution name
Execution Order
Coverage of the 'n'
first executions
Figure 20.1: Optimized execution order
20.5 Bug Location
20.5.1 Theory
To locate a bug, CoverageBrowser simulates the behaviour of a human programmer searching for a single error in the source code. It simplifies the programmer’s behaviour to a stochastic process that goes from source code line to source code line. After each step, the process tries to jump to the next better error candidate. After an infinite time, we can then look on the probability that a source code line was chosen as the best location of the failure. Because instructions in a program are have a strong dependence, CoverageBrowser forms groups from instructions that are always executed together, because they cannot be distinguished from code coverage data. The bug location algorithms works with these groups of instruction instead of individual statement. At the beginning of the process, a covered source code line is selected at random. Then we select an other instrumented source code line with the following rules: 1. Select a test which covers the current line.
2. Then select the next source line as follows:
Squish Coco 5.1.0 - 144 - froglogic GmbH Bug Location
(a) If the test had passed, the line that caused the failure is not expected to be among the source lines executed by the selected test. Therefore select any instrumented line that is not executed by this test. (b) If the test had failed, select any source code line that is executed by this test. We repeat this process until a set of pertinent source code lines are identified.
An example We will use a trivial example to illustrate how the algorithm works. The following function computes the inverse of a number:
float inv( float x ) { if ( x != 0 ) x = 1 * x ; // <- here is the bug else x = 0; return x; }
The bug itself is easy to understand; a multiplication is used instead of a division. Our test suite is: Name Test State INV(0) inv(0)== 0 Passed INV(1) inv(1)== 1 Passed INV(2) inv(2)== 0.5 Failed INV(3) inv(3)== 0.3333333 Failed INV(4) inv(4)== 0.25 Failed We will now simulate the bug location algorithm step by step. i The following is a simplified version of the algorithm that Squish Coco uses. It would return the same results as the actual algorithm, but be to slow in practice. For better precision and better performance, CoverageBrowser therefore computes the probabilities directly and does not use a sampling method as below.
First we note that it is not possible to distinguish between the lines “if ( x != 0 )” and “return x;” with a test. If one of these lines is executed, the other one is also. We group them together and view them as a single line. This means that if we estimate that these lines are a good error candidate, we cannot determinate which of this line is responsible of the bug. To simplify the explanation, we omit now the “return;” statement. The algorithm starts by randomly selecting a source code line as an error candidate. We will use here the line “if ( x != 0 )” as our starting point. The algorithm then searches the list of the tests that execute this line and chooses one at random; assume that it selects INV(2):
Squish Coco 5.1.0 - 145 - froglogic GmbH Bug Location
if ( x != 0 ) INV(0)
x = 1 * x ; INV(1)
else INV(2)
x = 0 ; INV(3)
INV(4)
Figure 20.2: Bug location of INV example – Step 1
The test INV(2) has failed, and we suppose that one of the source code lines executed by this test is responsible of the error. The algorithm selects then as error candidate a line that is executed by INV(2); we assume it is “x = 1 * x”:
if ( x != 0 ) INV(0)
x = 1 * x ; INV(1)
else INV(2)
x = 0 ; INV(3)
INV(4)
Figure 20.3: Bug location of INV example – Step 2
The algoritm then select randomly the test INV(1) in the set of tests executed by the line “x = 1 * x” (that are: INV(1), INV(2), INV(3) and INV(4)):
Squish Coco 5.1.0 - 146 - froglogic GmbH Bug Location
if ( x != 0 ) INV(0)
x = 1 * x ; INV(1)
else INV(2)
x = 0 ; INV(3)
INV(4)
Figure 20.4: Bug location of INV example – Step 3
The test INV(1) was passed, so we suppose that a source code line which is not executed by this test is responsible of the error. We select then “x = 0;” as the next candidate:
if ( x != 0 ) INV(0)
x = 1 * x ; INV(1)
else INV(2)
x = 0 ; INV(3)
INV(4)
Figure 20.5: Bug location of INV example – Step 4
We iterate then the process infinitely and compute the probabilities that a source line is chosen as error candidate: Line Probability x = 1 * x 0.528 if ( x != 0 ) 0.283 x = 0; 0.188 As expected, the line “x = 1 * x” has the highest probability of having a bug.
Squish Coco 5.1.0 - 147 - froglogic GmbH Patch Analysis
20.5.2 Usage
To calculate the bug location proceed as follows:
1. Select a set of executions in the "Executions" window. At least one execution should have failed.
2. Click on "View¡Bug Location...". The window (see Figure 20.6, page 148) will be displayed.
3. Click on the "Compute" button to start the analysis.
Source code fragment
Rank
Location in the
source code
Figure 20.6: Bug Location
20.6 Patch Analysis
With the menu entry "Tools¡Patch File Analysis..." one can generate a report about the influence of a patch on the test coverage of a project, without running the test suite for the patched version.
Prerequisites are a project for which .csmes and .csexe files exist and are loaded into CoverageBrowser, and a diff file. Patch analysis works best with programs that have automatic tests and which are instrumented in such a way that the names of the tests are known to Squish Coco (see Chapter 34.1, page 189). Line coverage (--cs-line) and statement block coverage (--cs-statement-block) should not be disabled. (They are on by default.)
The diff file must be in the “unified” format. It is generated by the Linux™ diff utility with the option -u, and is also the default output format of several version control systems (see Chapter 40, page 209).
Clicking the menu entry "Tools¡Patch File Analysis..." lets the dialog "Patch File Analysis" appear. It has the following entries:
Title: The title of the report, both for HTML and CSV.
Patch File: Path of the patch file that contains the changes to the project.
Output: To select the output file and its type: Type: Either HTML or the Excel CSV format. The field to the right of the "Type" field contains the name and path of the report file that is generated.
Source Code Parameter: For HTML reports, to select the display of the annotated source code:
Squish Coco 5.1.0 - 148 - froglogic GmbH Comparing Releases
Tooltip Maximal Size: The annotated patch file in the HTML report has a tool tip which displays the tests that have executed a certain line of code. This parameter sets the maximal number of tests that can appear in a tooltip. With a value of 0, no tooltips appear.
CSV Parameter: To select the format of the CSV report: Column Separator: The column separator symbol can be either a comma or a semicolon.
Sections: To choose the content of the report: Execution Statistics: Create a table that groups the tests by their results. It shows how many of the tests have passed, failed, need manual testing, and whose execution status is unknown. Executions: Create a list of the tests that execute code which is affected by the patch. For each test, the name and the execution result is shown. Source Code Statistics: Create a table that shows the influence of the patch on the test coverage. It shows how many lines in the patch are covered by a test, how many are not, and for how many lines Squish Coco could not find out whether they are covered. These numbers are shown for the lines that were removed, or added, and for all lines of the patch. Annotated Patch Source: Create an annotated version of the patch file. Each line of code in the patch is shown in red if it is removed, green if it is added, or otherwise gray. Also shown are the line numbers of the code lines, both before and after the patch is applied, and the number of tests that cover a line. This last field also has a tooltip that shows which tests cover the specific line. (The tooltip is only visible if "Tooltip Maximal Size" is set to a non-zero value.)
To generate the report, one clicks either the button "OK" or "Show": The second button also opens a browser window to show the generated report. Clicking "Apply" saves the values of the dialog entries without generating a report, while "Cancel" closes the dialog without saving anything.
20.7 Comparing Code Coverage of Two Software Releases
CoverageBrowser is able to instrumentation database together in order to: 1. check is the modified/unmodified code is correctly tested. 2. find which tests are impacted by a source code modification. This feature is particularly adapted to compare two releases together which contains small modifications (bug fixes only) and to limit the tests of the modified code only. In this mode CoverageBrowser uses the following typographic rules:
Rule Source Window Method List Source List Execution List Normal font Identical4 source Identical4 methods Identical4 files Executions available part in both releases Bold Modified methods Modified files Bold+Underline New text inserted New methods New files New executions Bold+Strike Deleted text Deleted methods Deleted files Missing executions
CoverageBrowser comparison and difference algorithm is particularly designed for C and C++ source code and ignore white spaces and modifications in comments.
Squish Coco 5.1.0 - 149 - froglogic GmbH Changing the Instrumentation Database
20.7.1 Reference Database
The reference database is the base instrumentation database which are used for the comparison. To select it click on "Tools¡Compare with..." and select a .csmes database. Switching the working database with the reference database can be performed by clicking on "Tools¡Switch databases".
Once the reference file loaded, additional filter possibilities are available in the "Executions", the "Sources" and the "Methods" window. These filters let you show/hide, modified, new, deleted or identical procedures and source files.
The "Executions" window displays a mix between the executions of the reference and the current release: • stroked executions are only available in the reference release. The statistic computed is corresponding to the coverage for the reference release. This value can be interpreted as an expected code coverage when this tests get re-executed on the current release.
• underlined executions are new tests. • executions available in both version are not highlighted.
20.7.2 Coverage analysis of modified/unmodified source code
CoverageBrowser is able to limit the code coverage analysis to the modified (resp. unmodified) functions. When selecting the coverage analysis on the modified (resp. unmodified) functions only, CoverageBrowser treat all unmodified (resp. modified) functions as if they are not instrumented. Limiting the code coverage analysis to modified functions can be a practical way to verify that the new features are tested and to identify the list of tests which have are impacted by a modification. To limit the code coverage to modified function (resp. unmodified functions) click on "Tools¡Analysis of Modified Methods" (resp. "Tools¡Analysis on Identical Methods").
20.8 Changing the Instrumentation Database
20.8.1 Merging Instrumentations
Clicking on the menu entry "File¡Merge with..." lets you import the executions, the source code, and the instrumentations from other .csmes files. Comments and code marked as validated are merged together.
20.8.2 Importing Unit Tests
Clicking on the menu entry "File¡Import Unit Tests..." lets you import the execution report of unit tests into the current application. Only execution reports of source files present in the main application are imported. Executions of other source files (for example test code) are ignored.
Squish Coco 5.1.0 - 150 - froglogic GmbH Function Profiler
20.8.3 Importing Reviewer Comments
Clicking on the menu entry "File¡Import Reviewer Comments..." lets you import comments and manual validations of a previous version of the current instrumentation database. Comments and manual validations of unmodified functions will be imported even if the source code is modified.
20.9 Function Profiler
The function profiler is activated as soon as the --cs-function-profiler=option (option can be ’all’ or ’skip- trivial’) is added to the compiler and linker command line arguments. It lets you view the time spent for each function in CoverageBrowser. Like for the code coverage, the profiler lets you analyze the time consumed for each procedure of each selected group of tests. It also lets you compare the timing between two product versions or between executions.
The profiling informations are displayed in the docking window "Function Profiler" which displays:
Total Duration: the cumulated execution time of the function.
Count: the number of function calls.
Mean Duration: the mean execution time of a single call. All timing information are for the selecting executions. It is possible to exclude or include interactively from the profiling analysis tests by selecting it into the "Execution" window. Clicking on the title bar of a column permits also to sort it and so quickly find the highest values. i The ticks used for the profiler is different to those used for computing the execution time of the application. The first one is able to measure short durations but has not the same absolute precision. For this reason, the timing displayed may differ a bit between the profiler window and the execution window.
20.9.1 Comparing Executions Together
CoverageBrowser permits also to compare the profiling information between two set of tests. The principle is simple: a set of reference functions is selected and compared to an other set. The comparison is realized by a difference and a radio computed in an additional column:
Difference: the difference (selected − re f erence) lets you compare the absolute time and counts between the 2 sets.
Ratio: the radio (selected/re f erence) lets you compare a relative difference between the 2 sets. For all 3 kind of measurements provided by the function profiler (count, duration and mean duration) 3 additional columns are provided with the values of the reference set, the difference and the ratio. Through the possibility to sort each of these columns, it is possible to quickly identify the difference in term of the computation resources between 2 tests.
Squish Coco 5.1.0 - 151 - froglogic GmbH Function Profiler
20.9.2 Comparing Two Software Versions Together
Exactly like for the comparisons of the executions of a binary, it is also possible to compare the executions of two different binaries. This lets you analyze the difference in terms of the performance between two software versions.
CoverageBrowser provides then the same computation as for the execution comparison in the "Profiler" window. It provide also some additional columns which permits also to see the functions which are differently instrumented from one version to an other.
Squish Coco 5.1.0 - 152 - froglogic GmbH Generation of Reports
Chapter Generation of Reports 21
i This feature is not available for black box testing.
21.1 HTML/CSV Report
The menu entry "Reports¡Generate HTML/CSV Report..." allows to export code coverage statistics (per meth- ods, source files, executions, . . . ) of the selected executions in HTML format. It allows also to list the manually validated and unexecuted code parts.
21.2 EMMA-XML Report
Selecting the menu entry "Reports¡Export Statistics in EMMA-XML Format..." allows exporting code cov- erage statistics of in EMMA-XML format. The output contains global statistics in a format that is compatible with EMMA. This allows using Squish Coco in tools that provide support for EMMA, notably to give an easy way to use Squish Coco with continuous integration servers like Jenkins CI.
EMMA defines four categories for coverage: classes, methods, blocks, and lines. With Squish Coco they have the following meaning:
EMMA category Meaning classes A class is considered executed if one of its method is called. Code which is not in a class is located in the class "" (empty). methods A method is covered if it was called. blocks Code coverage at statement block level. lines Line code coverage (if compiled with line coverage support). conditions Decision/Condition coverage (if compiled with Decision/Condition coverage support).
Squish Coco 5.1.0 - 153 - froglogic GmbH Cobertura XML Report
21.3 Cobertura XML Report
The menu entry "Reports¡Export Statistics in Cobertura-XML Format..." allows exporting code cover- age statistics in Cobertura-XML format. The output contains global statistics in a format that is compatible with Cobertura. This allows the use of Squish Coco in tools that provide support for Cobertura, notably giving an easy way to use Squish Coco with continuous integration servers like Jenkins CI and SonarQube.
The statistics in the Cobertura report is computed a little bit differently from the usual way due to the limitations of the report format. The report is in fact a combination of line and condition coverage.
Every statement block, decision and condition instrumentation is counted as a Cobertura-condition. The end of a function is e.g. marked as one condition to fulfill. The report format also requires that each line with a condition is also instrumented at line coverage level. This is not always the case (e.g. with empty functions), so in this case we add artificially some instrumentation values. Finally, if an instrumented statement is written in more than one line, it is necessary to generate instrumentation data for each line. All this has an impact on the computation of the statistics for classes, methods and sources. But the resulting values are comparable to those in the other report formats.
21.4 JUnit Report
The menu entry "Reports¡Export JUnit Report..." allows exporting the test result as JUnit report. This report does not contain coverage data and only list the test execution result (i.e. “passed” or “failed”) for each test item.
21.5 Text Report
Clicking on the menu entry "Reports¡Generate Text Report..." generates a small text report in the form of one line per executed/unexecuted item. A distinct line format can be specified for executed or unexecuted lines. The following escape sequences are recognized:
%f: Absolute source code file name
%r: Relative to the current build directory source code file name
%l: Line number
%c: Column number
%m: Explanation
Example: Setting the field "Unexecuted Code Fragments" to "%f:%l: %m" will create a text file which contains all unexecuted code parts. Each line will look like as follows:
foo.cpp:55: Unexecuted: ’return;’
Squish Coco 5.1.0 - 154 - froglogic GmbH Preferences
Chapter Preferences 22
22.1 Save/Load Project
"Save/Restore window position": If this option is selected, the position of all windows and toolbars will be restored upon application restart.
"Reload automatically the last project": If this option is selected, the last project opened will automatically be reloaded upon application restart.
"Saves project automatically on exit": Saves the project file automatically without asking on application exit.
22.2 Comments
"Minimum Comment Size": The minimum comment size, is the minimum length requested for a comment.
"Do not request a comment when setting an item to the ’manually validated’ state": This option is used to allow the user to manually modify the state of an instrumentation without entering a comment. i Enabling this option should be avoided because modifying the state of an instrumentation should be performed with a valid reason which should be recorded as a comment.
22.3 Thresholds
Thresholds are trigger values that control the background color of:
• the instrumented source files in in the "Sources" window.
Squish Coco 5.1.0 - 155 - froglogic GmbH Cache
• the instrumented classes or namespaces in the "Methods" window.
• the instrumented functions, methods or procedures in the "Methods" window. Description:
"Medium/High Coverage Level": If the statistic is above this value, the background color is set to green. Otherwise, the color is orange.
"Low/Medium Coverage Level": If the statistic is below this value, the background color is set to red. Otherwise, the color is orange.
22.4 Cache
Description:
"Execution": Maximum number of executions loaded into the RAM.
"Source": Maximum number of source files loaded into the RAM.
Squish Coco 5.1.0 - 156 - froglogic GmbH Part VIII Other Coco Tools
Squish Coco 5.1.0 - 157 - froglogic GmbH cmedit – Edit the paths in a csmes file
Chapter cmedit – Edit the paths in a csmes file 23
cmedit is a tool to modify the paths of the files in a .csmes file. Syntax:
cmedit [hoptionsi] hinfilei [-o houtfilei] where: hinfilei: The .csmes file to be edited. Unless the option -o is set, the changes in the path names are written to this file too. (Required parameter)
-o houtfilei | --output=houtfilei: The .csmes output file. If this parameter is set, the original hinfilei is unchanged and the edited version of hinfilei is written too houtfilei. hoptionsi: Any of: -c hdepthi | --cut=hdepthi: Remove a common initial segment of all paths in hinfilei. hdepthi is a non-negative integer and describes the number of path components that should be removed. A path component is every sequence of characters that lies between slashes or backslashes. The drive letter (plus colon) at the beginning of a Windows path name is also counted as a path component. Therefore the command --cut=1 will transform the path “/home/someone/main.cpp” into “/someone/main.cpp”, and “C:\mydir\main.cpp” into “\mydir\main.cpp”. If hdepthi is too large, cmedit exits with an error and does nothing. This occurs either if there is a path with less than hdepthi components or if there are two paths that differ in the first hdepthi components. If hinfilei contains the two paths “/home/someone/myproject/main.cpp” and “/ home/someone/library/header.h”, the command --cut=3 will not succeed. The --cut command is always executed before all --rename commands. -C | --cut-all: Remove the largest common initial segment of the paths in hinfilei. The option is equivalent to calling -c hdepthi with the highest possible value of hdepthi. The --cut-all command is always executed before all --rename commands. You may either use --cut or --cut-all, not both. -r hcommandi | --rename=hcommandi: Rename all paths that match a given pattern. The parameter hcommandi has the form “hpatterni,hreplacementi” or “hpatterni,hreplacementi, hflagsi”, where:
Squish Coco 5.1.0 - 158 - froglogic GmbH cmedit – Edit the paths in a csmes file
• hpatterni is the pattern to replace. cmedit searches for it in every path in hinfilei and tries to replace it with the hreplacementi string. If hpatterni occurs more than once in a path, all occurrences are replaced. By default, hpatterni uses the wildcard syntax, with “?” standing for a single character and “*” standing for any number of characters. By setting the “r” flag (see below), one can use regular expression syntax instead. (Note that with regular expressions, the backslashes in Microsoft® Windows path names must be doubled to distinguish them from the backslashes of the regular expression syntax.) • hreplacementi is the replacement string. • hflagsi can be the letters “i” or “r” or both. The flag “i” enables case-insensitive comparison, while “r” enables regular expression matching. --rename-function=hcommandi Rename all functions that match a given pattern. The patters are identical to the source file renaming. If hpatterni or hreplacementi should contain a comma, it can be written as “\,”. The option can be repeated to set more than one hcommandi. The commands are then executed in sequence in the order in which they are written. But before they are executed, cmedit checks whether the resulting transformation is possible at all. If there are two paths that would be transformed into the same new path, cmedit exits with an error and does nothing. -l | --list-sources: List the paths of all sources in hinfilei. The paths are left unchanged. --list-sources: List the name of all functions in hinfilei that can be renamed. -I | --case-sensitivity-fix: Rename all source files whose file name are identical but with a different case. -f | --force: Rename source files even if errors are reported. -n | --dry-run: Do not change the paths, only describe what would be done. -v | --verbose: Make the output more verbose. @hpathi: Read command line options from the file at hpathi and insert them at the position of this option. The option file is a text file with one option per line. Leading and trailing blanks and empty lines are ignored.
Exit values cmedit exits with status 0 if it succeeds. It returns -1 is the Squish Coco license is invalid, and 1 if the requested renamings cannot be done.
Squish Coco 5.1.0 - 159 - froglogic GmbH cmmerge – Merging Utility
Chapter cmmerge – Merging Utility 24
cmmerge is a small utility which lets you merge several instrumentation databases (.csmes file) together. It behaves like the CoverageBrowser menu entry "File¡Merge with..." (see Chapter 20.8.1, page 150). Syntax:
cmmerge [hoptionsi] -o houtfilei hinfilei... cmmerge --blackbox -o houtfilei hinfilei where:
-o houtfilei | --output=houtfilei: .csmes output file. (Required.) hinfilei: .csmes input file. Only one input file is allowed if --blackbox is selected.
-b | --blackbox: Generate a .csmes file for black-box testing. hoptionsi: Any of: -a | --append: Merge the input file(s) with the existing content of the output file. houtfilei must already exist. -i hreferencei | --instrumentation-and-execution=hreferencei: Include only execution results that be- long to the .csmes file hreferencei. With this option, cmmerge merges hreferencei with the hinfileis, but includes the instrumentations and executions of an hinfilei only if they also occur in hreferencei. If --append is set, houtfilei is viewed as a part of hreferencei. This option is useful for unit tests, where the hreferencei is the program for which the test is written, while the hinfileis contain the execution results of the unit tests. -r hreferencei | --reviews-only=hreferencei: Merge hreferencei with the comments and manual valida- tions in the hinfileis. The file hreferencei is a .csmes file. Only validations and comments of unmodified functions are imported. If there are comments for the same line of code, they are merged. The original comments are then separated by a horizontal bar. If however the same comment occurs more than once in the input files, only one copy is kept. -s | --strict: Require that the preprocessed output for each C++ file is identical across the build.
Squish Coco 5.1.0 - 160 - froglogic GmbH cmmerge – Merging Utility
--delete: Delete the input files from disk after importing them. -v | --verbose: Verbose output. @hpathi: Read command line options from the file at hpathi and insert them at the position of this option. The option file is a text file with one option per line. Leading and trailing blanks and empty lines are ignored.
Squish Coco 5.1.0 - 161 - froglogic GmbH cmcsexeimport – Command Line Import Utility
Chapter cmcsexeimport – Command Line Import Utility 25
cmcsexeimport is a utility to import an execution report (.csexe file) into an instrumentation data base (.csmes file). It behaves like the menu entry "File¡Load Execution Report..." in the CoverageBrowser (see Chapter 19.1.2, page 131). Syntax:
cmcsexeimport -m hcsmes_filei -t htitlei [hoptionsi] hcsexe_filei... where
-m hfilenamei | --csmes=hfilenamei: Set the name of the .csmes file into which the .csexe files are imported. This parameter is mandatory.
-t hstringi | --title=hstringi: Set a default name for the executions in the hcsexe_fileis. This parameter is mandatory. Executions that already have a name will be imported unchanged. Others get the name hstringi, with a number attached in braces if there is more than one of them. hcsexe_filei: Name of one of the .csexe files that should be imported. hoptionsi: Any of: -e hfilenamei | --csexe=hfilenamei: Set the name of a .csexe file to be imported. This argument can be repeated to process more than one file. -l hfilenamei | --csexe-list=hfilenamei: Read the list of .csexe files to import from a text file. The text file must have one file name per line, without leading or trailing spaces. -p hpolicyi | --policy=hpolicyi: Set the import policy. The possible import policies are: ignore_duplicates: Executions are ignored if they have executed the same code as an execution that was already imported. (Default.) import_duplicates: Executions are imported if at least one instrumented source code line is executed. import_duplicates_and_empty: All Exections are imported. merge: All executions with the same name are merged. The execution counts of all merged instrumentations are added. -I | --incident: Mark the executions in the imported files as “INCIDENT”.
Squish Coco 5.1.0 - 162 - froglogic GmbH cmcsexeimport – Command Line Import Utility
-S | --skipped: Mark the executions in the imported files as “SKIPPED”. -P | --passed: Mark the executions in the imported files as “PASSED”. -F | --failed: Mark the executions in the imported files as “FAILED”. -C | --check-manually: Mark the executions in the imported files as “CHECK_MANUALLY”. -f hnumi | --csmes-save-frequency=hnumi: After every hnumi exection reports have been read, the current state of the instrumentation database is written to disk. Without this option, the it is written to the disk only at the end. -d | --delete: Delete the imported files after processing them. -v | --verbose: Write progress messages to the standard error output (stderr). -D | --debug: Write debugging information to the standard error output (stderr). @hpathi: Read command line options from the file at hpathi and insert them at the position of this option. The option file is a text file with one option per line. Leading and trailing blanks and empty lines are ignored.
Squish Coco 5.1.0 - 163 - froglogic GmbH cmreport – Code coverage report generation
Chapter cmreport – Code coverage report generation 26
cmreport is a utility which generates Text, HTML, XML or CSV reports from an instrumentation database (a .csmes file). It generates exactly the same reports as these generated by CoverageBrowser (see Chapter 21, page 153). Example: The simplest use case is the generation of an HTML report from a single instrumentation database. If the name of the file is project.csmes, we may write:
$ cmreport --csmes=project.csmes --html=report
This command creates a file report.html and a directory report_html. The file report.html is the main page of the report and can be opened with the browser, while report_html contains all the other files. Most of the other command line options just modify the content of such a report. Syntax:
cmreport -m hcsmes_filei ...
Where:
-m hfile_namei | --csmes=hfile_namei: Set the name of the main .csmes file (mandatory).
-p hfile_namei | --patch=hfile_namei: Enable patch/diff file analysis. hfile_namei is the name of a file im "unified diff" format (see Chapter 40, page 209). The file must contain the differences between the files in the directory tree covered by the .csmes file and a modified version of the same directory tree. Patch reports can be generated in the CSV and in the HTML format. With the option --section (see Chapter 26.4, page 167) one can select which sections of the report are displayed.
--source-type=hargumenti: Source type used for the analysis. Possible values: preprocessed: The preprocessed code is analyzed. original: The original source code is analyzed. C macros are ignored in this mode.
-l hnumberi | --level=hnumberi: Set the code coverage level. hnumberi must be an integer greater than 0.
Squish Coco 5.1.0 - 164 - froglogic GmbH File selection options
--max-threads=hnumberi: Maximal number of threads used for the report computation. By default, there as many threads as there are CPUs on the system.
--stat: Print global coverage statistics to the standard output
--max-lines-per-page=hnumberi: Maximal number of lines per page for HTML tables.
@hpathi: Read command line options from the file at hpathi and insert them at the position of this option. The option file is a text file with one option per line. Leading and trailing blanks and empty lines are ignored.
26.1 File selection options
The following options specify the source files shown in the report.
--include-file-abs-regex=hregexi: Include the source files that match the regular expression hregexi in the report.
--exclude-file-abs-regex=hregexi: Exclude the source files that match the regular expression hregexi from the report.
--include-file-abs-wildcard=hwildcardi: Include the source files that match the wildcard pattern hwildcardi in the report.
--exclude-file-abs-wildcard=hwildcardi: Exclude the source files that match the wildcard pattern hwildcardi from the report. If two or more of these options match a file, the last one decides whether it is included or excluded. For the syntax of regular expressions and wildcard patterns see Chapter 20.1, page 141.
26.2 Execution selection options
The following options specify the executions that occur in the report.
-s hregexi | --select=hregexi: Select executions using regular expression. If not used, all executions are implicitly selected.
-d hregexi | --deselect=hregexi: Deselect executions using regular expression
--deselect-to-check: Deselect all executions with the status “CHECK_MANUALLY”
--select-passed: Select all executions with the status “PASSED”
--deselect-passed: Deselect all executions with the status “PASSED”
Squish Coco 5.1.0 - 165 - froglogic GmbH Release comparison
--select-failed: Select all executions with the status “FAILED”
--deselect-failed: Deselect all executions with the status “FAILED”
--select-unknown: Select all executions with unknown status
--deselect-unknown: Deselect all executions with unknown status
--executions-reference-from-csmes=hargumenti: Select all executions present in the specified .csmes file.
26.3 Release comparison
With the following options, one can compare the coverage of two releases.
-mr hfile_namei | --csmes-reference=hfile_namei: Set the name of the .csmes reference file. With this option, cmreport compares two software releases.
--release-comparison=hargumenti: Comparison mode when comparing two software releases. Possible values: none: Display only the differences between two software releases. modified_functions: Display code coverage only for modified functions. unmodified_functions: Display code coverage only for identical functions.
-sr hregexi | --select-reference=hregexi: Select reference executions using regular expression. This option activates the comparison of executions.
-dd hregexi | --deselect-reference=hregexi: Deselect reference executions using regular expression. This option activates the comparison of executions.
--deselect-to-check-reference: Deselect all reference executions with the status “CHECK_MANUALLY”
--select-passed-reference: Select all reference executions with the status “PASSED”
--deselect-passed-reference: Deselect all reference executions with the status “PASSED”
--select-failed-reference: Select all reference executions with the status “FAILED”
--deselect-failed-reference: Deselect all reference executions with the status “FAILED”
--select-unknown-reference: Select all reference executions with unknown status
--deselect-unknown-reference: Deselect all reference executions with unknown status
Squish Coco 5.1.0 - 166 - froglogic GmbH Options for HTML or CSV reports
26.4 Options for HTML or CSV reports
Command line arguments:
-h hfile_namei | --html=hfile_namei: Generate a multi-page HTML report. The report consists of a file hfile_namei.html, which directs to the index file of the reort, and a directory hfile_namei_html, which contains all other files in the report. If hfile_namei ends with .html, the ending is ignored, so that no file name or directory is generated that ends with a double “html”.
--html-single=hfile_namei: Generate a single HTML report as file hfile_namei. If hfile_namei does not end with .html, it is added automatically.
--csv-excel=hfile_namei: Generate a CSV report. The report is in a format that is used by Excel and OpenOffice. If hfile_namei does not end with .csv, it is added automatically.
--title=hstringi: Set the title of the report.
--execution-level=hnumberi: Set the required level of code coverage to hnumberi. A piece of code is counted as covered in the generated report only if it is at least hnumberi times executed. The default level is 1.
--section=hsection_namei: Select the sections of the generated report. This option can be repeated to select more than one section type. If nothing is selected, cmreport generates all relevant sections automatically. The following sections are possible: global: Global statistics function: Function statistics function-tree: Function tree with statistics class: Class/namespace statistics execution: Execution statistics source: Source file statistics source-tree: Source tree with statistics directory: Directory file statistics manually-validated: Manually validated code fragments unexecuted: Unexecuted code fragments dead-code: Dead code fragments executed: Executed code fragments If the option --patch is used, the following sections are also possible. patch-execution-statistic: Global execution statistics of a patch patch-source-statistic: Global source code statistics of a patch
Squish Coco 5.1.0 - 167 - froglogic GmbH Options for HTML or CSV reports
patch-execution: Executions impacted by a patch patch-source: Annotated patch file patch-source-instrumented: Annotated patch file for instrumented files only For the content of these sections, see Chapter 20.6. By default, if no --section option is set, all them occur in the report.
--coverage-mcc: Report multiple condition code coverage measurements.
--coverage-mcdc: Report MC/DC coverage measurements.
--coverage-condition: Report condition coverage measurements.
--coverage-decision: Report decision coverage measurements.
-b | --coverage-statement-block: Report statement (block) coverage measurements.
--line-coverage: Report line coverage measurements.
--function-coverage: Report function coverage measurements.
-t | --test-coverage: Test count mode
-D | --debug: Debug flag
--no-line-metrics: Do not report the eLOC (Effective Lines of Code) metric.
--no-mccabe-metrics: Do not report the McCabe (or Cyclomatic Complexity) metric.
--mccabe-group-case-metrics: Report the McCabe metric (or Cyclomatic Complexity) with consecutive cases of a switch grouped.
--mccabe-condensed-switches-metrics: Report the McCabe metric (or Cyclomatic Complexity) with condensed switches.
26.4.1 Options only for HTML
--css=hfile_namei: CSS style sheet
--icon=hfile_namei: Icon
--global-threshold-low-medium=hnumi: Global threshold setting
Squish Coco 5.1.0 - 168 - froglogic GmbH Options for text reports
--global-threshold-medium-high=hnumi: Global threshold setting
--source-threshold-low-medium=hnumi: Source/directory threshold setting
--source-threshold-medium-high=hnumi: Source/directory threshold setting
--function-threshold-low-medium=hnumi: Function/class/namespace threshold setting
--function-threshold-medium-high=hnumi: Function/class/namespace threshold setting
26.4.2 Options only for CSV
--csv-field-separator=hchari: Field separator for a CSV file
--csv-comma=hchari: Separator (, or .) used for floats in a CSV file
26.5 Options for text reports
Command line arguments:
-t hfile_namei | --text=hfile_namei: Text report output file name
--format-executed=hargumenti: Line format for the executed for code fragments
--format-unexecuted=hargumenti: Line format for the unexecuted for code fragments
--format-manually=hargumenti: Line format for the manually validated for code fragments
--format-dead-code=hargumenti: Line format for dead code fragments The following escape sequences are recognized:
%f: Absolute source code file name
%r: Relative to the current build directory source code file name
%l: Line number
%c: Column number
%m: Explanation
It is allowed to to set the output file to an empty string (i.e.: --text=), in this case the standard output of the console is used.
Squish Coco 5.1.0 - 169 - froglogic GmbH Options for EMMA-XML reports
26.6 Options for EMMA-XML reports
The following options are used if a report in the EMMA-XML format is generated:
--emma=hfilenamei: Generate an EMMA-XML report. hfilenamei is the name of the XML file that is generated.
--emma-source-pattern=hargumenti: Source file pattern for EMMA-XML reports
--emma-package-pattern=hargumenti: Package name pattern for EMMA-XML reports The patterns have the following format:
%f: file name (without path)
%b: file name without path and extension
%e: file extension
%p: file path (without file name)
%P: absolute file path (without file name) i For Jenkins-CI users, it is not recommended to use the last two parameters. The default values are chosen to work with the EMMA-XML plugin.
26.7 Options for JUnit reports
Command line arguments:
-j hfile_namei | --junit=hfile_namei: Generate an JUnit report JUnit reports does not contain any code coverage information. The report list only the tests executed, their status (passed or failed) and their comments.
26.8 Options for Cobertura reports
To generate a Cobertura report, one needs to specify either the first two or the last one of the following options:
--cobertura=hfilei: Generate a Cobertura-XML report. hfilei is the path to the generated output file.
--cobertura-azure-devops: Generate a Cobertura-XML report for Azure DevOps. https://azure.microsoft.com/services/devops/
--cobertura-manual-validation-hits=hinti: For manual validations, set the line coverage hits with the parameter in the Cobertura-XML report.
Squish Coco 5.1.0 - 170 - froglogic GmbH Options for Cobertura reports
--cobertura-source=hdirectoryi: Root directory of the source code when generating a Cobertura-XML report. This option is required when --cobertura is present; otherwise it is meaningless.
--sonarqube-project=hpathi: Generate a Cobertura-XML report for a SonarQube project. hpathi is here the path to the sonar-project.properties file. The root path of the source tree is then ex- tracted from its content: It is the value of the sonar.cxx.coverage.reportPath, the sonar.cxx.coverage. itReportPath or the sonar.cxx.coverage.overallReportPath property. Setting any of these properties has the same effect.
Squish Coco 5.1.0 - 171 - froglogic GmbH cocolic – License activation
Chapter cocolic – License activation 27
Command line alternative to cocolicwizard. The program can be used to activate and configure the Squish Coco license on a computer without a graphical interface. Syntax:
cocolic hargumentsi
Where:
--fetch-license-key=hcodei: Use an activation code to retrieve the license key for a node-locked license. The activation code is provided by froglogic. For this, cocolic needs a network connection to froglogic. It can also be configured to connect via a proxy server (see below).
--license-key=hkeyi: Set the key for a node-locked license. The license key is generated by froglogic from the output of cocolic -- machine-identifier.
--machine-identifier: Display the machine identifier on the standard output.
--license-server=hserveri:hporti: Set the address and the port of the license sever for a floating license. The port can be omitted; then the default port of 49344 is used.
--check: Check the license validity. The program prints a message about the status of the license to stderr. It then exits with a return value of 0 if the license is valid, or 1 if it is invalid.
--http-proxy=hserveri:hporti: Use a HTTP proxy sever to retrieve the license key.
--socks5-proxy=hserveri:hporti: Use a SOCKS5 proxy sever to retrieve the license key.
--proxy-user=hnamei: Set the user name for the proxy server.
Squish Coco 5.1.0 - 172 - froglogic GmbH cocolic – License activation
--proxy-password=hpasswordi: Set the password for the proxy server.
--no-proxy: Disable the use of a proxy server.
--disable-node-locked: Disable the node locked license. This option is meaningful if a node-locked license and a license server use are present on the same machine. Squish Coco will then use the license server, but the node-locked license is still present and can be enabled later.
--enable-node-locked: Re-enable a previously disabled node locked license. Do nothing if the license is already enabled.
@hpathi: Read command line options from the file at hpathi and insert them at the position of this option. The option file is a text file with one option per line. Leading and trailing blanks and empty lines are ignored.
Squish Coco 5.1.0 - 173 - froglogic GmbH cocolicwizard – GUI for license activation
Chapter cocolicwizard – GUI for license activation 28
cocolicwizard provides a graphical user interface to configure the license. It lets you enter a license activation code, send an activation email, enter a license key, or configure the connection to a license server. The program also displays information about the currently active license. Syntax:
cocolicwizard [--check]
Where:
--check: check whether there is a valid license. Show the GUI window only if there is no valid license, otherwise do nothing.
Squish Coco 5.1.0 - 174 - froglogic GmbH cocolicserver – License Server
Chapter cocolicserver – License Server 29
cocolicserver can be used to serve Squish Coco licenses from a central computer; clients can connect to the server and cocolicserver will make sure that no more than the available number of licenses will be used concurrently. Syntax:
cocolicserver hoptionsi where:
-s | --server-identifier: Print the identifier of the machine on which cocolicserver should run. This identifier must then be sent to froglogic to get a license file (see Chapter 2.2.1, page9).
-c hpathi | --config=hpathi: Absolute path to the configuration file (provided by froglogic).
-l hpathi | --logfile=hpathi: Write log messages to the file at the absolute hpathi. The file name may contain the following escape sequences: %Y: current year, in four digits, %M: current month number, in two digits, %D: current day, in two digits, %h: current hour, in two digits, %m: current minute, in two digits, %s: current second, in two digits. New log files will be opened according to the values of the escape sequences. This means that if hpathi is e.g. cocoserver_%Y_%M_%D.log, every day a new log file is created, and the files have names like cocoserver_2016_01_01.log, cocoserver_2016_01_02.log, etc.
@hpathi: Read command line options from the file at hpathi and insert them at the position of this option. The option file is a text file with one option per line. Leading and trailing blanks and empty lines are ignored. The following options are available only on Linux™and macOS:
--syslog: Write log messages to the UNIX® system logging service.
Squish Coco 5.1.0 - 175 - froglogic GmbH cocolicserver – License Server
--daemon: On Linux™and macOS: Launch cocolicserver as a daemon background process. The following options are available only on Microsoft® Windows:
--service-install: Install cocolicserver as a Windows service that starts automatically at boot time. For this option cocolicserver needs administrator privileges. It must be run in the form
C:\somedir> cocolicserver --service-install hoptionsi
Then the service is installed and starts automatically. It will run as if called directly in the form cocolicserver hoptionsi. The visible name of the server is “Squish Coco License Server”. If the server is already installed as a service, the hoptionsi are updated. The new options have effect only after the service is restarted.
--service-uninstall: Uninstalls the cocolicserver Windows service. For this option cocolicserver needs administrator privileges.
Squish Coco 5.1.0 - 176 - froglogic GmbH The Visual Studio® Coco Wizard
Chapter The Visual Studio® Coco Wizard 30
The Visual Studio® Coco Wizard is an application to edit the CoverageScanner options for C/C++ and C# projects on Microsoft® Visual Studio® and related systems.
The add-in reads Solution Files (*.sln) and Project Files(*.proj and *.vcxproject) and modifies them. These files are used by Microsoft® Visual Studio®, Microsoft® MSBuild, .NET Core, Visual Studio Code and other software.
30.1 How it works
From Microsoft® Visual Studio®, the add-in can be started from the menu entry "Tools¡Coco Wizard...". It is then automatically configured to edit the Solution file of the currently active project.
The add-in can also be started on its own from the Microsoft® Windows start menu, under "Squish Coco¡Visual Studio Build Mode Configuration". It then requires as a first step the selection of the Solution or Project file that should be modified.
30.2 The dialog steps of the Coco Wizard
The Visual Studio® Coco Wizard has the following dialog steps: 1. File selection. In this step, the user can select a Solution or Project File to edit – either from the file system or from a list of files that were used earlier. This step does not occur when the add-in is started from Microsoft® Visual Studio®. 2. Modification / Copying. Here one can choose whether an existing build mode is edited or it is copied with a new name. It is useful to have separate build modes for build with and without coverage. Therefore, at the beginning, one should copy an existing build mode and add coverage flags to it.
Squish Coco 5.1.0 - 177 - froglogic GmbH The dialog steps of the Coco Wizard
3. Creation / Editing / Removal. Here one can choose whether a new build mode should be created, an existing mode edited, or a build mode removed. In general, it is recommended to create a separate build mode for coverage. The build mode can be copied from a debug build mode or a release mode, depending only on the requirements of the project.
4. Selecting the options. This is a two-column display with CoverageScanner options (see Chapter9, page 72). The list at the left side contains all possible options, that on the right, the options that are activated for the build mode. The options can be dragged from the left to the right side to activate them or dragged to the right to deactivate them. If an option requires a parameter, a dialog window to set it appears when it is dragged from the right. To change the parameter of an option at the right column, double-click on it. Note that the order of the options matters. They can be rearranged with drag and drop. 5. Summary. In this step, the actual list of selected options is shown. If Microsoft® Visual Studio® is running at this time, it will reload the .sln file and the project can be recompiled.
Squish Coco 5.1.0 - 178 - froglogic GmbH Howtos Part IX
Squish Coco 5.1.0 - 179 - froglogic GmbH Creating an instrumented project
Chapter Creating an instrumented project 31
This chapter contains recipes how a software project can be set up for instrumentation. The methods depend on the languages and the development environments. Here we show how to create a new software project so that it is prepared for instrumentation. This is another simple way to get acquainted with Squish Coco. A description of a larger project can be found in Chapter4, page 31.
31.1 C++ on Microsoft® Visual Studio® using the Visual Studio® Coco Wizard
There is an add-in for Microsoft® Visual Studio® (see Chapter 30, page 177) that supports the work with Squish Coco.
31.2 C# on Microsoft® Visual Studio®
Start Microsoft® Visual Studio® and create a new C# application:
1. Click on "File¡New¡Project..." to pop up the new project wizard.
2. Choose a project type of "Visual C#¡Windows" and the "Console Application" template.
3. Enter a project name of squishcoco_sample, then click the "OK" button.
4. When the wizard’s second page appears, click the "Finish" button. To activate the instrumentation, use the project properties:
1. In the "Solution Explorer", double click "squishcoco_sample¡Properties" to open the property dia- log.
2. Click on the "Build" tab.
3. Select "Conditional compilation symbols" and enter COVERAGESCANNER_COVERAGE_ON.
Squish Coco 5.1.0 - 180 - froglogic GmbH Command Line Tools
The code coverage analysis is enabled when the symbol COVERAGESCANNER_COVERAGE_ON is defined from the command line.
Build the squish_coco project. This will cause the executable squishcoco_sample.exe to be built and the code coverage instrumentation file squishcoco_sample.exe.csmes to be generated. Double click on squishcoco_sample .exe.csmes to inspect this file in CoverageBrowser. Right now there are no code coverage statistics to be seen in CoverageBrowser: this is because the application has not yet been executed. Click on Program.cs in the source list to display the main function. All the instrumented lines are shown grayed out to indicate that nothing has been executed.
Now execute squishcoco_sample.exe by double clicking it. This will result in a file called squishcoco_sample .exe.csexe being generated. This file contains a code coverage snapshot which can be imported into Coverage- Browser:
1. Click "File¡Load Execution Report...".
2. Select the "File" item and enter the path of the squishcoco_sample.exe.csexe file.
3. Click on the "Import" button. This will cause the code coverage statistics to be updated. Now, in the source code window, the main function’s return statement will be colored green to indicate that this line has been executed.
31.3 Command Line Tools
Open a console window (MS-DOS Prompt or Command Prompt window on Windows) and make sure that the Microsoft® Visual Studio® compiler (cl.exe) or GCC is installed on your system.
Create a simple hello.c source file that contains the following code:
#include
int main(int argc, char *argv[]) { printf("hello\n"); return 0; }
Compile this program using the CoverageScanner compiler wrapper instead of the native compiler. This is done by prepending cs to the command line compiler’s name: With Microsoft® Visual Studio® With GCC $ cscl.exe hello.c /Fehello.exe $ csgcc hello.c -o hello.exe Then the executable hello.exe and the file hello.exe.csmes, which contains the code coverage instrumentation, will be generated.
Execute hello.exe; this will cause the file hello.exe.csexe to be generated. It contains a code coverage snapshot which can be imported into hello.exe.csmes using cmcsexeimport:
$ cmcsexeimport --title="Hello execution" -m hello.exe.csmes -e hello.exe.csexe
Once imported, the hello.exe.csexe file is no longer needed and can be deleted. Now hello.exe.csmes contains a single execution record. We can generate an HTML report with cmreport:
Squish Coco 5.1.0 - 181 - froglogic GmbH Command Line Tools
$ cmreport --title="Hello application" -m hello.exe.csmes --html=hello.html
Here is a summary of the command line options that were used in this command:
--title="Hello application": Set a title for the report. It appears on all generated HTML pages.
-m hello.exe.csmes: Select the instrumentation database.
–html=hello.html: Specify the name of the HTML file that the report should be written to.
Squish Coco 5.1.0 - 182 - froglogic GmbH Generating instrumentations without modifying projects
Chapter Generating instrumentations without modifying 32 projects
Squish Coco can also generate code coverage information for a project without the need to modify it. The principle is to prepend to the PATH variable the path of the CoverageScanner compiler wrappers and to set the instrumentation parameters with the COVERAGESCANNER_ARGS environment variable. To activate the instrumentation, --cs-on must be present in COVERAGESCANNER_ARGS. If this is not the case, CoverageScanner is completely deactivated.
The variable COVERAGESCANNER_ARGS should only be set locally, e.g. in a script or on the command ! line. If it is set globally, it will influence every build.
32.1 GNU Make
On a UNIX® system, proceed as follows to instrument a project which can be generated using GNU Make:
export PATH=/opt/SquishCoco/wrapper/bin:$PATH export COVERAGESCANNER_ARGS=--cs-on make clean make
For macOS, replace the first line with
export PATH=/opt/SquishCoco/wrapper:$PATH
32.2 Microsoft® NMake
Proceed as follows to instrument a project that is generated with NMake:
set PATH=%SQUISHCOCO%\visualstudio;%PATH%
Squish Coco 5.1.0 - 183 - froglogic GmbH Microsoft Visual Studio
set COVERAGESCANNER_ARGS=--cs-on nmake clean nmake
32.3 Microsoft® Visual Studio®
Proceed as follows to instrument a project that is generated with Microsoft® Visual Studio®:
set PATH=%SQUISHCOCO%\visualstudio;%PATH% set COVERAGESCANNER_ARGS=--cs-on devenv /useenv myproject.sln /Rebuild
32.4 Microsoft® MSBuild
Proceed as follows to instrument a project that is generated with Microsoft® MSBuild
set PATH=%SQUISHCOCO%\visualstudio;%PATH% set COVERAGESCANNER_ARGS=--cs-on msbuild /p:UseEnv=true myproject.sln /t:ReBuild
32.5 Mono C# XBuild
Proceed as follows to instrument a project that is generated with Mono C# XBuild
export COVERAGESCANNER_ARGS=--cs-on msbuild \ /p:UseEnv=true \ /p:UseHostCompilerIfAvailable=true \ /p:CscToolPath="${SQUISHCOCO}/wrapper" \ myproject.sln /t:ReBuild
The environment variable SQUISHCOCO contains the path to the CoverageScanner executable.
Squish Coco 5.1.0 - 184 - froglogic GmbH Code Coverage of Libraries
Chapter Code Coverage of Libraries 33
33.1 Code Coverage of Static/Shared Libraries and DLL
During the linking operation, CoverageScanner includes all instrumentations of the shared libraries (if these are compiled with CoverageScanner). CoverageBrowser displays the code coverage of the complete application (executable and its libraries) in one view. i To get an analysis of the code coverage of a library only, it is necessary to compile the main application and exclude its sources from the code coverage (by adding in the command line --cs-exclude-file-regex=.* for example.)
33.2 Code Coverage of Plugins/Manually Loaded Shared Libraries
Libraries loaded dynamically can also be instrumented, but it is necessary to handle the generation of the execution report in the main application or in the plugin code itself.
33.2.1 Generating Code Coverage Information directly from the Main Application
Handling the plugins into the main application can be easily performed using the register/unregister mechanism of the CoverageScanner API: It is just necessary to call __coveragescanner_register_library() after loading a library and call __coveragescanner_unregister_library() just before unloading it. Example:
#include
Squish Coco 5.1.0 - 185 - froglogic GmbH Code Coverage of Plugins/Manually Loaded Shared Libraries
int main(int argc, char **argv) { void *handle; double (*cosine)(double); char *error;
handle = dlopen("libm.so", RTLD_LAZY); #ifdef __COVERAGESCANNER__ __coveragescanner_register_library("libm.so"); #endif if (!handle) { fprintf(stderr, "%s\n", dlerror()); exit(EXIT_FAILURE); }
dlerror(); /* Clear any existing error */
/* Writing: cosine=(double( *)(double)) dlsym(handle,"cos"); would seem more natural, but the C99 standard leaves casting from"void *" toa function pointer undefined. The assignment used below is the POSIX.1-2003(Technical Corrigendum 1) workaround; see the Rationale for the POSIX specification of dlsym(). */
*(void **) (&cosine) = dlsym(handle, "cos");
if ((error = dlerror()) != NULL) { fprintf(stderr, "%s\n", error); exit(EXIT_FAILURE); }
printf("%f\n", (*cosine)(2.0)); #ifdef __COVERAGESCANNER__ __coveragescanner_unregister_library("libm.so"); #endif dlclose(handle); exit(EXIT_SUCCESS); }
i Calling __coveragescanner_register_library() or __coveragescanner_unregister_library() on an non-instrumented library is allowed.
33.2.2 Generating Code Coverage Information directly from the Plugin
CoverageScanner cannot handle the instrumentation of plugins (i.e. shared libraries loaded manually) during the linking phase. In this case, the library must initialize and store itself the executions. Therefore, the shared library needs to call __coveragescanner_filename() to set the name of the execution file during its initialization and __coveragescanner_save() to save the instrumentations when its becomes unloaded.
Squish Coco 5.1.0 - 186 - froglogic GmbH Code Coverage of Plugins/Manually Loaded Shared Libraries
33.2.2.1 Code Coverage of Plugins Generated with Microsoft® Visual Studio®
The function DllMain is called on the initialization and the termination of the DLL generated using Microsoft® Visual Studio®. When the ’reason’ field is equal to DLL_PROCESS_ATTACH the function __coveragescanner_filename() should be called. To save the measures on exit the function __coveragescanner_save() shall be called when ’reason’ is DLL_PROCESS_DETACH.
Example:
extern "C" BOOL WINAPI DllMain(HINSTANCE hInstance, DWORD dwReason, LPVOID /*lpReserved*/) { switch( dwReason ) { case DLL_PROCESS_ATTACH: #ifdef __COVERAGESCANNER__ /* Initialization of the CoverageScanner library. */ /* Replace"mylib" with your filename without extension */ __coveragescanner_filename("mylib"); #endif ... break; case DLL_PROCESS_DETACH: ... #ifdef __COVERAGESCANNER__ /* Saves the execution report */ __coveragescanner_save(); #endif break; } return TRUE; }
33.2.2.2 Code Coverage of Plugins Generated with GNU gcc
The GNU compiler offers 2 attributes which let you execute a function when a library becomes loaded or unloaded:
__attribute__ ((constructor)) my_init(void);: This attribute specifies a function to be called when the library is loaded. Call the function __coveragescanner_filename() in the custom initialization function of the library.
__attribute__ ((destructor)) my_fini(void);: This attribute specifies a function to be called when the library is unloaded. Call the function __coveragescanner_save() on the termination of the library. Example:
static void plugin_load(void) __attribute__ ((constructor)) ; static void plugin_unload(void) __attribute__ ((destructor)) ;
static void plugin_load(void) {
Squish Coco 5.1.0 - 187 - froglogic GmbH Code Coverage of Plugins/Manually Loaded Shared Libraries
#ifdef __COVERAGESCANNER__ /* Initialization of the CoverageScanner library. */ /* Replace"mylib" with your filename without extension */ __coveragescanner_filename("mylib"); #endif ... }
static void plugin_unload(void) { ... #ifdef __COVERAGESCANNER__ /* Saves the execution report */ __coveragescanner_save(); #endif }
Squish Coco 5.1.0 - 188 - froglogic GmbH Test suites and Squish Coco
Chapter Test suites and Squish Coco 34
34.1 Execution comment, name and status
Test suites are able to provide to CoverageBrowser the name of the test, a comment in HTML format and also the status of their execution (passed or failed). Two methods are possible:
• The functions __coveragescanner_testname(), __coveragescanner_add_html_comment() and __coveragescanner_teststate() can be used to specify the test name, its comment and its status. This is particularly useful for unit tests in which the test framework is compiled into the application to test (see Chapter 10.1.2, page 87).
• One can write these data directly into the .csexe file. (See below.)
The .csexe file is a simple text file on which is it possible to append additional lines to extend the information provided to CoverageBrowser.
To set the name of a test, simply add before executing it the following line: “*hname of the testi\n” The character “*” must be the first character of the line. The name of the test is placed directly after it and the line is terminated with a carriage return.1
To set the status of a test, simply add after executing it the following line: “!hstatusi\n” The character “!” must be the first character of the line. The status of the test is placed directly after it and the line is terminated with a carriage return. The status can be one of the following strings:
"PASSED": to indicate that the test was successfully executed,
"FAILED": to indicate that the test was not successfully passed, and
"CHECK_MANUALLY": to indicate that it was not possible to determinate whether the test was successfully executed. To append an execution comment, insert the contents of an HTML document just after the execution of the application.
Example: The following batch file execute the test First Test and set the status of the execution to CHECK_MANUALLY.
1 It is also possible to begin the line with a paragraph sign (“§”) instead of a “*”. The “§” must be the in Latin-1 encoding and has the numerical value of 167. Since this may lead to internationalization problems, we strongly dicourage its use. It is only supported for compatibility with older versions of the program.
Squish Coco 5.1.0 - 189 - froglogic GmbH Unit testing
echo *First Test >> myapp.csexe myapp echo "
Execution of myapp" >> myapp.csexe echo !CHECK_MANUALLY >> myapp.csexeThere may be more than one HTML comment, and the comments and the status declaration may occur in any order, but they all must be appended to the .csexe file after the test has been executed.
34.2 Unit testing
CoverageBrowser imports execution results for all object files that are part of an application. This means that, if an unit test uses the same object files as the application, CoverageBrowser can import the execution results of the unit tests and merge them into the code coverage of the compiled application. Example: An application consists of three files: app.cpp: This file contains the main() function of the application. library.cpp: This file contains functions that are called by app.cpp. testlibrary.cpp: This file contains the test code for the functions in library.cpp. It has its own main() function. Under Microsoft® Windows, the application is compiled with the following commands:
cscl app.cpp /Foapp.obj cscl library.cpp /Folibrary.obj cscl library.obj app.obj /Feapp.exe
The following commands compile the unit test program:
cscl testlibrary.cpp /Fotestlibrary.obj cscl library.obj testlibrary.obj /Fetestlibrary.exe
To import the execution report of the unit tests into the instrumentation database of the main application, two methods can be used:
• The execution of testlibrary.exe generates the execution report testlibrary.exe.csexe. This report is then loaded into the measurement database app.exe.csmes.
• The execution of testlibrary.exe generates the execution report testlibrary.exe.csexe. This report is then loaded into the measurement databaseof the test code, testlibrary.exe.csmes. This can be done with the function "File¡Import Unit Tests..." of the CoverageBrowser(see Chapter 20.8.2, page 150).
In both cases, only the code coverage analysis of the file library.cpp is loaded and the execution report of the test code ignored.
Squish Coco 5.1.0 - 190 - froglogic GmbH Special compiler versions for cross-compiling
Chapter Special compiler versions for cross-compiling 35
If a GNU compiler is used for cross-compiling, it often has a special name, like ‘arm-linux-gcc’ instead of ‘gcc’. The installation script of Squish Coco tries to create automatically the compiler wrappers for every GNU compiler installed on the machine. In general, all necessary compiler wrappers are therefore present in the system. A cross- compiler may however also reside in a non-standard directory in the system, where it will not be found by the Squish Coco installer. If a compiler configuration is missing, the following can be done:
1. Copy ’coveragescanner’ to the file ’cshcompileri’. Example: On Microsoft® Windows, this would be something like:
C:\Program Files\squishcoco>copy coveragescanner.exe csarm-linux-gcc.exe
2. Copy the profile ’gcc.cspro’ or g++.cspro’ to ’hcompileri.cspro’. Example:
C:\Program Files\squishcoco>copy gcc.cspro arm-linux-gcc.cspro
The new GNU cross-compiler ’cshcompileri’ can now be used. It instruments the code and then calls ’hcompileri’ for compilation.
Squish Coco 5.1.0 - 191 - froglogic GmbH Multi-Platform Application
Chapter Multi-Platform Application 36
36.1 Principle
Squish Coco has strict checks to detect that the coverage data is coherent with the source code. It is, for example, not possible to import an execution report (.csexe file) of an earlier project version or to import the coverage information of an old version into a newer. This prevents some usage errors and the generation of incoherent test metrics. The main principle of the coherence analysis consists simply of verifying that every instrumented source file in the project is not modified. This is done during the build and when coverage data are imported or merged. If the C preprocessor produces different files across the build – which can occur when some file are built in debug mode and others in release mode – Squish Coco detects it, adds a suffix like #1 or #2 to the file name, and requires that both versions are tested to achieve full coverage. It is also possible to compile a program on several platforms and then merge the coverage data. This requires that the source code is identical for each build. If that is not the case, Squish Coco refuses to merge. Let us take a small toy example to understand how it works: We suppose that the project consists of only one source file project.cpp and that we build it on Microsoft® Windows and Linux™. On Linux™, we compile it with the command
$ csg++ /home/me/project.cpp -o project-unix
A file project-unix.csmes is generated and the tests can be executed and imported as usual into it. On Microsoft® Windows, it is compiled with the command
C:\code> cscl C:\Home\me\project.cpp /Feproject-windows.exe
A file project-windows.exe.csmes is generated. Merging the coverage information is possible with cmmerge:
$ cmmerge -o project.csmes project-unix.csmes project-windows.exe.csmes
Unfortunately this operation is not enough because the file project.csmes will then contain two project.cpp files:
Squish Coco 5.1.0 - 192 - froglogic GmbH Restrictions
• /home/me/project.cpp
• C:\Home\me\project.cpp It is in fact necessary to indicate that both files are identical. To do that, we use cmedit to rename this two absolute file names to an identical one:
$ cmedit project.csmes --rename="/home/me/,/PRJ/" --rename="C:\Home\me\,/PRJ/" -- verbose
These two rules rename the base directories /home/me/ and C:\Home\me\ to /PRJ/ to a single source file: /PRJ/ project.cpp. The coverage information is merged as if the code was tested identically on both platforms.
36.2 Restrictions
36.2.1 Code Generators
In some build processes, code generators are used, like flex/bison or, for Qt products, moc. Then the code generators need to produce identical source code on each platform to make the editing work. If this is not the case, cmedit will refuse the renaming operation because it assumes that the project is not produced from the same source code. The solution is then to skip explicitly the files that are automatically generated.
Let us take the case of moc. qmake generates files with the name moc_*.cpp. The trick could consist of using the switch --force which skips the renaming operation for files with conflicts:
$ cmedit project.csmes --force --rename="/home/me/,/PRJ/" --rename="C:\Home\me\,/PRJ/" --verbose
As result, all source files are merged together in a directory PRJ except the moc files on which a conflict is detected. These moc files need then to be covered separately on each platforms.
36.2.2 Platform depending macros
In general, C macros expand to platform dependant code and should therefore be avoided. In the case that macro expansions are different between the platforms, Squish Coco will create duplicates by appending a #1 or #2 to the source files. Both versions of the sources need then to be covered. There are some common pitfalls that the developer need to be careful about to avoid conflicts:
• The __FILE__ macro can generate unwanted differences because it may contain the absolute source file name. To solve this issue, since this macro is often used in asserts, the easiest solution is simply to work with the release build. Another solution is to ensure that the source file names passed to the compiler are not absolute file names. In this case, on most platforms __FILE__ is not expended to be the source’s absolute path.
• The NULL macro way also be different on each platform. Sometimes it is expanded as (void*)0 and sometimes as __null. The proper solution, is in this case to avoid the comparison with NULL and to use the C++ extension std::nullptr.
Squish Coco 5.1.0 - 193 - froglogic GmbH Control of execution report generation
Chapter Control of execution report generation 37
By default, the of CoverageScanner library produces an execution report when the instrumented application exits. This might not be enough in the case of unit test suites (where it is preferable to generate a report after each single test) or for applications like daemons which never terminate. For these purposes, the CoverageScanner library also lets you: • embed C code into the instrumented application to control the report generation (34.2, page 190), and
• generate a coverage report when a Microsoft® Windows event or a UNIX® signal has been received. This chapter is about the second method.
37.1 Execution report generation with UNIX® signals
To enable execution report generation with UNIX® signals, use the option --cs-dump-on-signal=hsigi, where hsigi is the number of a signal or its common name (like e.g. SIGUSR1).
With the UNIX® kill command it is possible to trigger the report generation. (ex: kill -SIGUSR1
37.2 Execution report generation with Microsoft® Windows events
To enable execution report generation with Microsoft® Windows events, use the option --cs-dump-on-event= heventi, where heventi is a string which identifies the event.
Since there is no standard Windows equivalent to the UNIX® kill command, Squish Coco offers a replacement which is described in the following sections.
Squish Coco 5.1.0 - 194 - froglogic GmbH Execution report generation with Microsoft Windows events
37.2.1 A program to send events to applications
Squish Coco contains a tool to send events to Windows applications, for the use with --cs-dump-on-event. It is needed not very often, therefore you need to compile it yourself.
The program can be found in the directory hWindows Cocoi\dump_on_event. It exists in two versions, one in C++ and one in C#. They are called dump_on_event and dump_on_event_cs, respectively. To compile the program and see how it works, follow the instructions below. They refer to the C++ case – the C# version is quite similar and the differences are listed at the end.
1. Copy the folder hWindows Cocoi\dump_on_event to some other place, say to C:\dump_on_event, since the original version is write protected.
2. Double-click on the file build_cpp. This starts the compilation of dump_on_event.cpp and of the example application event_sample_cpp.cpp. When the file event_sample_cpp.cpp is compiled, it is instrumented with the option --cs-dump-on-event=COVERAGE. 3. After the programs have been compiled, three windows become visible. (a) The window of the CoverageBrowser, with the contents of the file event_sample_cpp.exe.csmes loaded. It contains the source of the program event_sample_cpp, but not yet any coverage data. (b) A window with a command prompt from which the program dump_on_event can be run later. (c) A command window in which event_sample_cpp runs. Every second it prints a dot, and it will do so for five minutes.
4. Now you can send the event COVERAGE to event_sample_cpp. Enter in the command prompt window:
C:\dump_on_event>dump_on_event COVERAGE
When the program event_sample_cpp receives the event, it creates a file event_sample_cpp.exe.csexe, which contains the coverage data that were generated so far. Send the command a second time. The file event_sample_cpp.exe.csexe then grows, since the new coverage data are appended to it. There is no danger that the same data are written twice.
5. You can import the execution report from the CoverageBrowser with "File¡Load Execution Report...". Each data dump appears as a separate execution, so if you have sent the event twice, you will see two executions in the executions window of the bowser.
The same recipe also works with the C# versions of the programs. You only need to replace “build_cpp” with “build_cs”, “dump_on_event” with “dump_on_event_cs” and “event_sample_cpp” with “event_sample_cs”.
37.2.2 Global events
In general, Windows events are created for a specific account and not accessible from other accounts on the same machine. But Windows supports also system-wide events. They have names that begin with Global\. So to generate a code coverage report with Windows events from an application on a different account, system-wide events are needed.
In the following example we use the program dump_on_event to generate the event:
1. Compile your application with the argument --cs-dump-on-event=Global\COVERAGE. 2. Start your application. 3. On another account, execute the command
Squish Coco 5.1.0 - 195 - froglogic GmbH Execution report generation with Microsoft Windows events
C:\dump_on_event>dump_on_event Global\COVERAGE
The application then writes its coverage report to the disk.
37.2.3 Write your own program
You can also use dump_on_event as an example for sending events from your test setup. It is in fact only a wrapper for the Microsoft® Windows function SetEvent():
#include
void DisplayError( LPTSTR lpszMessage ) { // Retrieve the system error message for the last-error code
LPSTR lpMsgBuf; DWORD dw = GetLastError();
FormatMessageA( FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, NULL, dw, MAKELANGID( LANG_NEUTRAL, SUBLANG_DEFAULT ), ( LPSTR ) &lpMsgBuf, 0, NULL );
// Display the error message and exit the process std::cerr << lpszMessage << " (error " << dw << "): " << lpMsgBuf << ’\n’;
LocalFree( lpMsgBuf ); }
int main( int argc, char *argv[] ) { if ( argc != 2 ) { std::cerr << "Usage: " << argv[0] << "
const char * eventName = argv[1] ;
HANDLE ghEvent = OpenEventA( EVENT_MODIFY_STATE, FALSE,
Squish Coco 5.1.0 - 196 - froglogic GmbH Execution report generation with Microsoft Windows events
eventName );
if ( ghEvent == NULL ) { DisplayError( "Opening event failed" ); return 1; }
if ( ! SetEvent( ghEvent ) ) { DisplayError( "Raising event failed" ); CloseHandle( ghEvent ); return 1; }
CloseHandle( ghEvent ); return 0; }
Squish Coco 5.1.0 - 197 - froglogic GmbH Customizing I/O of CoverageScanner library
Chapter Customizing I/O of CoverageScanner library 38
38.1 Custom I/O using C file access
The following example shows how to generate the execution report using the standard C file API. To compile the example on Microsoft® Windows:
cscl customiofile.c
To compile the example on Linux™ or macOS:
csgcc customiofile.c -o customiofile
Source code:
#include
static int csfputs(const char *s, void *stream) { #if VERBOSE fprintf(stderr,"csfputs:%s\n",s); #endif return fputs(s, (FILE *)stream); }
static void *csfopenappend(const char *path) { #if VERBOSE fprintf(stderr,"csfopenappend:%s\n",path); #endif return (void*)fopen(path,"a+"); }
static void *csfopenread(const char *path)
Squish Coco 5.1.0 - 198 - froglogic GmbH Custom I/O using C file access
{ #if VERBOSE fprintf(stderr,"csfopenread:%s\n",path); #endif return (void*)fopen(path,"r"); }
static void *csfopenwrite(const char *path) { #if VERBOSE fprintf(stderr,"csfopenwrite:%s\n",path); #endif return (void*)fopen(path,"w"); }
static char *csfgets(char *s, int size, void *stream) { char * ret; ret = fgets(s, size, (FILE *)stream); #if VERBOSE fprintf(stderr,"csfgets:%s\n",s); #endif return ret; }
static int csremove(const char *filename) { #if VERBOSE fprintf(stderr,"csremove:%s\n",filename); #endif return remove(filename); }
static int csfclose(void *fp) { #if VERBOSE fprintf(stderr,"csfclose\n"); #endif return fclose((FILE*)fp); }
int main() { char location[256]; int lg_location;
printf(".csexe file name (without extension}:"); fflush(stdout); fgets(location,sizeof(location),stdin); lg_location=strlen(location); if (lg_location) location[lg_location-1]=’\0’; // strip\n #ifdef __COVERAGESCANNER__ __coveragescanner_set_custom_io( csfgets,
Squish Coco 5.1.0 - 199 - froglogic GmbH Custom I/O using SFTP protocol
csfputs, csfopenappend, csfopenread, csfopenwrite, csfclose, csremove); __coveragescanner_install(location); #endif }
38.2 Custom I/O using SFTP protocol
The following example shows how to generate the execution report directly on a SFTP server. The SFTP server is part of SSH v2 and is available on most of the Unix platforms. On Microsoft® Windows, a free SSH server can be downloaded from http://www.freesshd.com. To compile the example on Microsoft® Windows:
1. Download libSSH2 from http://www.libssh2.org. Generate the library and set the environment variable LIBSSH2 to the location of the libSSH2 source code. 2. To compile the example:
cscl %LIBSSH2%\win32\debug_dll\libssh2.lib -DWIN32 --cs-libgen=/MTd /MTd -I %LIBSSH2%\include ws2_32.lib customiosftp.c
3. Execute custom_io_sftp.exe. To compile the example on Linux™: 1. Install the development package of libssh2. 2. To compile the example:
csgcc -lssh2 customiosftp.c -o customiosftp
3. Execute custom_io_sftp. Source code:
#define VERBOSE 1
#ifdef WIN32 # include
Squish Coco 5.1.0 - 200 - froglogic GmbH Custom I/O using SFTP protocol
#endif #include
#include
static LIBSSH2_SESSION *session=NULL; static LIBSSH2_SFTP *sftp_session=NULL; static int sock=0;
static void extract_location(const char *location, char *server, char *user, char* passwd, char *file) { int i,j; int lg_location; lg_location=strlen(location); for (i=0;i for (j=0;i for (j=0;i for (j=0;i Squish Coco 5.1.0 - 201 - froglogic GmbH Custom I/O using SFTP protocol file[j]=’\0’; } static void close_sftp_session() { if (sftp_session) libssh2_sftp_shutdown(sftp_session); sftp_session=NULL; if (session) { libssh2_session_disconnect(session, "Normal Shutdown, Thank you for playing"); libssh2_session_free(session); } session=NULL; if (sock) { #ifdef WIN32 Sleep(1000); closesocket(sock); #else sleep(1); close(sock); #endif } sock=0; } static int open_sftp_session(const char *server,const char *user,const char *passwd) { struct sockaddr_in sin; int rc; sock = socket(AF_INET, SOCK_STREAM, 0); sin.sin_family = AF_INET; sin.sin_port = htons(22); sin.sin_addr.s_addr = inet_addr(server); if (connect(sock, (struct sockaddr*)(&sin), sizeof(struct sockaddr_in)) != 0) { close_sftp_session(); return 0; } /* Createa session instance */ session = libssh2_session_init(); if(!session) { close_sftp_session(); return 0; Squish Coco 5.1.0 - 202 - froglogic GmbH Custom I/O using SFTP protocol } /* ... start it up. This will trade welcome banners, exchange keys, * and setup crypto, compression, and MAC layers */ rc = libssh2_session_startup(session, sock); if(rc) { close_sftp_session(); return 0; } libssh2_session_set_blocking(session, 1); if (libssh2_userauth_password(session, user, passwd)) { close_sftp_session(); return 0; } sftp_session = libssh2_sftp_init(session); if (!sftp_session) { close_sftp_session(); return 0; } return 1; } static int csfputs(const char *s, void *stream) { #if VERBOSE fprintf(stderr,"csfputs:%s\n",s); #endif return libssh2_sftp_write((LIBSSH2_SFTP_HANDLE*)stream,s,strlen(s)); } static void *csfopenappend(const char *location) { LIBSSH2_SFTP_HANDLE *handle; char server[1024]; char user[1024]; char passwd[1024]; char file[1024]; LIBSSH2_SFTP_ATTRIBUTES attrs; extract_location(location,server,user,passwd,file); #if VERBOSE fprintf(stderr,"csfopenappend %s:%s:%s\n",server,user,file); #endif if (open_sftp_session(server,user,passwd)) { Squish Coco 5.1.0 - 203 - froglogic GmbH Custom I/O using SFTP protocol handle = libssh2_sftp_open(sftp_session, file, LIBSSH2_FXF_CREAT|LIBSSH2_FXF_WRITE , LIBSSH2_SFTP_S_IRUSR|LIBSSH2_SFTP_S_IWUSR| LIBSSH2_SFTP_S_IRGRP|LIBSSH2_SFTP_S_IROTH ); } else return NULL; if (handle) { if (libssh2_sftp_fstat(handle,&attrs)==0) { /* Go to the end of the file */ libssh2_sftp_seek(handle,attrs.filesize); } } return handle; } static void *csfopenread(const char *location) { char server[1024]; char user[1024]; char passwd[1024]; char file[1024]; extract_location(location,server,user,passwd,file); #if VERBOSE fprintf(stderr,"csfopenread %s:%s:%s\n",server,user,file); #endif if (open_sftp_session(server,user,passwd)) return (void*) libssh2_sftp_open(sftp_session, file, LIBSSH2_FXF_READ, 0); else return NULL; } static void *csfopenwrite(const char *location) { char server[1024]; char user[1024]; char passwd[1024]; char file[1024]; extract_location(location,server,user,passwd,file); #if VERBOSE fprintf(stderr,"csfopenwrite %s:%s:%s\n",server,user,file); #endif if (open_sftp_session(server,user,passwd)) return (void*) libssh2_sftp_open(sftp_session, file, LIBSSH2_FXF_CREAT| LIBSSH2_FXF_WRITE, LIBSSH2_SFTP_S_IRUSR|LIBSSH2_SFTP_S_IWUSR| LIBSSH2_SFTP_S_IRGRP|LIBSSH2_SFTP_S_IROTH ); else return NULL; Squish Coco 5.1.0 - 204 - froglogic GmbH Custom I/O using SFTP protocol } static char *csfgets(char *s, int size, void *stream) { size_t ss; ss = libssh2_sftp_read((LIBSSH2_SFTP_HANDLE*)stream,s,size-1); if (ss) { s[ss]=’\0’; #if VERBOSE fprintf(stderr,"csfgets:%s\n",s); #endif return s; } else return NULL; } static int csremove(const char *location) { int ret; char server[1024]; char user[1024]; char passwd[1024]; char file[1024]; extract_location(location,server,user,passwd,file); #if VERBOSE fprintf(stderr,"csremove %s:%s:%s\n",server,user,file); #endif if (open_sftp_session(server,user,passwd)) { ret = libssh2_sftp_unlink(sftp_session,file); close_sftp_session(); return ret; } else return -1; } static int csfclose(void *fp) { #if VERBOSE fprintf(stderr,"csfclose\n"); #endif return libssh2_sftp_close((LIBSSH2_SFTP_HANDLE*)fp); } int main() { char location[1024]; char tmp[1024]; #ifdef WIN32 Squish Coco 5.1.0 - 205 - froglogic GmbH Custom I/O using SFTP protocol WSADATA wsadata; WSAStartup(WINSOCK_VERSION, &wsadata); #endif location[0]=’\0’; printf("server IP:"); fflush(stdout); fgets(tmp,sizeof(tmp),stdin); strcat(location,tmp); printf("user:"); fflush(stdout); fgets(tmp,sizeof(tmp),stdin); strcat(location,tmp); printf("passwd:"); fflush(stdout); fgets(tmp,sizeof(tmp),stdin); strcat(location,tmp); printf(".csexe file name (without extension}:"); fflush(stdout); fgets(tmp,sizeof(tmp),stdin); strcat(location,tmp); location[strlen(location)-1]=’\0’; #ifdef __COVERAGESCANNER__ __coveragescanner_set_custom_io( csfgets, csfputs, csfopenappend, csfopenread, csfopenwrite, csfclose, csremove); __coveragescanner_install(location); #endif } Squish Coco 5.1.0 - 206 - froglogic GmbH Using the CoverageScanner library with a memory pool Chapter Using the CoverageScanner library with a 39 memory pool The CoverageScanner library can use an internal memory pool to handle memory allocations. This is necessary on platforms which do not provide the standard C functions malloc() and free(). In this case, it is necessary to allocate a C array at program start which will serve these functions. Doing this requires to estimate the memory consumption. The CoverageScanner library also provides a way to monitor the usage and to detect memory overflow. 39.1 Estimation of the size of the memory pool The size of the memory pool depends on the number of files instrumented and on the size of the temporary output buffer and are used for dynamically loaded library support and for string manipulations. For a code using only static libraries and which produce a single executable, it is possible to use the following approximation: 512 + 100 ∗ number_of_instrumented_sources number_of_instrumented_sources includes the headers and the source files. The size of the temporary buffer used for the generation of the execution report file will be added automatically to this size. 39.2 Monitoring of the size of the memory pool The following example shows how to monitor the memory usage of the memory pool: #include #ifdef __COVERAGESCANNER__ void memory_failure( int s ) { int used = -1; int max_used = -1; Squish Coco 5.1.0 - 207 - froglogic GmbH Monitoring of the size of the memory pool int size = -1; __coveragescanner_memory_pool_stat( &size, &used, &max_used ); printf( "Memory Failure : Requested:%i Actual Size=%i Actually Used=%i Peak Usage =%i\n", s, size, used, max_used ); exit( 2 ); } #endif int main( int argc, char *argv[] ) { #ifdef __COVERAGESCANNER__ int used = -1; int max_used = -1; int size = -1; __coveragescanner_memory_pool_stat( &size, &used, &max_used ); printf( "Memory Pool Usage: Size=%i Used=%i Peak=%i\n", size, used, max_used ); #endif return 0 ; } To compile the example on Microsoft® Windows: cscl memory-pool.c --cs-memory-pool=1000 --cs-memory-alloc-failure-function= memory_failure To compile the example on Linux™ or macOS: csgcc memory-pool.c -o customiofile --cs-memory-pool=1000 --cs-memory-alloc-failure- function=memory_failure The size of the memory pool is 1000 bytes plus the size of the temporary buffer that is used for serializing the execution report. To monitor the actual usage, the function CoverageScanner.__coveragescanner_memory_pool_stat() can be used. If a memory overflow occurs (to simulate it, one can set the size of the memory pool to 5 bytes), the C function memory_failure() is called. Squish Coco 5.1.0 - 208 - froglogic GmbH Generation of diff files Chapter Generation of diff files 40 Squish Coco supports patch file analysis in the CoverageBrowser (see Chapter 20.6, page 148) and in cmreport with the option --patch (see Chapter 26, page 164). For this, a diff file in the “unified” format is needed. It is provided by the UNIX® diff tool and by most version control systems. The following examples work both on the UNIX® shell and with a Microsoft® Windows command window. 40.1 Comparing directories If the two versions of the source are in different directories of the file system, the diff command can be used. It is the standard UNIX® file comparison program which gives the functionality its name. The “unified” format must be set with the -u flag, and the result must be redirected to a file, so that the command line will look like $ diff -u project/version_1 project/version_2 > project.diff Then the directories project/version_1 and projects/version_2 are compared and the output is written to the file project.diff. Of the two directories, project/version_1 is viewed as the earlier version. 40.2 Version control systems In a version control system, the diff functionality is used to compare two revisions of the source tree. For most of the newer version control systems exist GUI wrappers, like Tortoise SVN, especially for Windows. If the underlying version control system supports unified diffs by default, the same can be expected from the GUI wrappers. git: With git, one needs to specify two revisions and to redirect the output to a file. git uses the unified format by default. One can compare two revisions with $ git diff hrev1i hrev2i > project.diff Squish Coco 5.1.0 - 209 - froglogic GmbH Version control systems This command creates by default a diff of the whole source tree. For a list of ways to specify the revisions under git, see the manpage of gitrevisions (7). Mercurial: The distributed version system Mercurial generates by default diff files in the unified format. One has to write $ hg diff -r hrev1i -r hrev2i > project.diff to compare the Mercury change sets hrev1i and hrev2i. Subversion: The version control system Subversion generates unified diffs by default. To generate the diff, write $ svn diff -r hrev1i -r hrev2i > project.diff where hrev1i and hrev2i may be tags or revision numbers. cvs: The Concurrent Versions System is the oldest of the revision control systems described here. It does not use the unified format by default. With it, one has to write $ cvs diff -u -N -r hrev1i -r hrev2i > project.diff to compare two revisions. The flag -u switches unified diffs on, and -N is necessary to track changes in which a file is added or removed. The revisions hrev1i and hrev2i must be tags or dates, since cvs does not have global version numbers. The command then creates a lot of additional console output, but the differences between the two revisions are still written to project.diff. Note that the command only lists the differences in the current directory and its subdirectories. Squish Coco 5.1.0 - 210 - froglogic GmbH Coco Integration Part X Handbook Squish Coco 5.1.0 - 211 - froglogic GmbH Support for IDEs and toolkits Chapter Support for IDEs and toolkits 41 In this chapter we give examples how Squish Coco can be combined with several IDEs and build environments. 41.1 GNU Makefiles Mostly, in makefiles, the C and C++ compiler and the linker are defined using the environment variables CC, CXX and LINK. This can be substituted by CoverageScanner by setting CC, CXX and LINK in the command arguments of make. Example: make LINK=csg++ CXX=csg++ CC=csgcc 41.2 CygWin To install the CoverageScanner compiler wrapper for GCC and G++ on CygWin, proceed as follows: 1. Open the Build Environment Selection application. (hWindows Cocoi\toolselector.exe) 2. Select the item “CygWin - www.cygwin.com”. 3. Click on “Install CygWin Support”. The build environment selection dialog then displays the list of generated compiler wrappers. Open then the CygWin console and compile your application with csgcc instead of gcc (or csg++ instead of g++). Squish Coco 5.1.0 - 212 - froglogic GmbH Scratchbox 41.3 Scratchbox If Squish Coco is installed on the root file system1, a compiler wrapper is created for each compiler supported by Scratchbox. To invoke CoverageScanner, prepend cs to the name of the cross-compiler. 41.4 CMake CMake is a platform independent build tool from Kitware which can be downloaded from http://www.cmake.org. When Squish Coco is used with CMake, the changes are partially dependent on the tool chain that is used for compilation. We will now first describe the addition of a new build type, which is independent from the tool chain, and then the additional changes for Microsoft® Visual Studio® and GNU GCC. 41.4.1 Adding new build type for instrumented compilation The first step is independent of the toolchain that is used. Its purpose is to declare the CMake variables that are used to specify the instrumented compilation. In CMake this is done by declaring a build type, which we will here call COVERAGE. To do this, add to the to CMakeLists.txt file the following lines. (The variable COVERAGE_FLAGS in the first line specifies the CoverageScanner command line options. Change its value to fit you needs. Only --cs-on must always be present.) SET(COVERAGE_FLAGS "--cs-on --cs-count") SET(CMAKE_CXX_FLAGS_COVERAGE "${CMAKE_CXX_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING "Flags used by the C++ compiler during coverage builds." FORCE ) SET(CMAKE_C_FLAGS_COVERAGE "${CMAKE_C_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING "Flags used by the C compiler during coverage builds." FORCE ) SET(CMAKE_EXE_LINKER_FLAGS_COVERAGE "${CMAKE_EXE_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING "Flags used for linking binaries during coverage builds." FORCE ) SET(CMAKE_SHARED_LINKER_FLAGS_COVERAGE "${CMAKE_SHARED_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING "Flags used by the shared libraries linker during coverage builds." FORCE ) SET( CMAKE_STATIC_LINKER_FLAGS_COVERAGE "${CMAKE_STATIC_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING "Flags used by the static libraries linker during coverage builds." FORCE ) MARK_AS_ADVANCED( CMAKE_CXX_FLAGS_COVERAGE CMAKE_C_FLAGS_COVERAGE 1Installation option: "1) Installation on the local machine (need to be root)" Squish Coco 5.1.0 - 213 - froglogic GmbH CMake CMAKE_EXE_LINKER_FLAGS_COVERAGE CMAKE_SHARED_LINKER_FLAGS_COVERAGE CMAKE_STATIC_LINKER_FLAGS_COVERAGE COMPILE_DEFINITIONS_COVERAGE ) These commands take the compiler and linker flags of the “Release” build type and add to them the coverage flags. If you want to use instead the flags of another build type, replace the suffix “_RELEASE” in this code with the name of another build type, e.g. with “_DEBUG”. 41.4.2 Microsoft® Visual Studio® Under Microsoft® Visual Studio®, we need to make the new Coverage build type visible to the IDE. To do this, add to CMakeLists.txt the lines if(CMAKE_CONFIGURATION_TYPES) set(CMAKE_CONFIGURATION_TYPES Debug Release MinSizeRel RelWithDebInfo Coverage) set(CMAKE_CONFIGURATION_TYPES "${CMAKE_CONFIGURATION_TYPES}" CACHE STRING "Reset the configurations to what we need" FORCE) endif() If necessary, customize the list of configuration types according to your needs. To make the changes visible to Microsoft® Visual Studio®, a complex procedure is apparently needed:2 1. Compile CMakeLists.txt with Ctrl-F7. If warnings occur, they can be ignored. 2. Make a trivial change in CMakeLists.txt, like adding a space. 3. Compile again. Then a dialog, "File modification detected", should appear. Click "Reload". Then the list of solution configurations is updated and one can compile in coverage mode. 41.4.3 Compilation with Microsoft® NMake In a project that is compiled whith NMake, the following must be done. 1. Create a toolchain definition file cl.cmake which replaces the compiler and linker with their CoverageScanner wrappers. Example: # this one is important SET(CMAKE_SYSTEM_NAME Windows) # specify the cross compiler FILE(TO_CMAKE_PATH "$ENV{SQUISHCOCO}/visualstudio" SQUISHCOCO) SET(CMAKE_C_COMPILER ${SQUISHCOCO}/cl.exe CACHE FILEPATH "CoverageScanner wrapper" FORCE) SET(CMAKE_CXX_COMPILER ${SQUISHCOCO}/cl.exe CACHE FILEPATH "CoverageScanner wrapper" FORCE) SET(CMAKE_LINKER ${SQUISHCOCO}/link.exe 2This applies at least to CMake 3.2 and Microsoft® Visual Studio® 2013. Squish Coco 5.1.0 - 214 - froglogic GmbH CMake CACHE FILEPATH "CoverageScanner wrapper" FORCE) 2. Create a Makefile project. Set the toolchain to the CoverageScanner wrapper and the build mode to COVERAGE. Example: cmake.exe -DCMAKE_TOOLCHAIN_FILE=cl.cmake -DCMAKE_BUILD_TYPE=COVERAGE \ -G "NMake Makefiles" hpath of cmake projecti 3. Build the project with nmake. 41.4.4 Compilation with GNU GCC The following must be done in a project that is comiled with gcc. 1. Create a toolchain definition file gcc.cmake which replaces the compiler and linker with their CoverageScanner wrappers. Example: find_program(CODE_COVERAGE_GCC gcc PATHS /opt/SquishCoco/wrapper/bin "$ENV{HOME}/SquishCoco/wrapper/bin" NO_DEFAULT_PATH ) find_program(CODE_COVERAGE_GXX g++ PATHS /opt/SquishCoco/wrapper/bin "$ENV{HOME}/SquishCoco/wrapper/bin" NO_DEFAULT_PATH ) find_program(CODE_COVERAGE_AR ar PATHS /opt/SquishCoco/wrapper/bin "$ENV{HOME}/SquishCoco/wrapper/bin" NO_DEFAULT_PATH ) # specify the cross compiler SET(CMAKE_C_COMPILER "${CODE_COVERAGE_GCC}" CACHE FILEPATH "CoverageScanner wrapper" FORCE) SET(CMAKE_CXX_COMPILER "${CODE_COVERAGE_GXX}" CACHE FILEPATH "CoverageScanner wrapper" FORCE) SET(CMAKE_LINKER "${CODE_COVERAGE_GXX}" CACHE FILEPATH "CoverageScanner wrapper" FORCE) SET(CMAKE_AR "${CODE_COVERAGE_AR}" CACHE FILEPATH "CoverageScanner wrapper" FORCE) 2. Create a Makefile project. Set the toolchain to the CoverageScanner wrapper and the build mode to COVERAGE. Example: cmake -DCMAKE_TOOLCHAIN_FILE=gcc.cmake -DCMAKE_BUILD_TYPE=COVERAGE \ -G "Unix Makefiles" hpath of cmake projecti 3. Build the project with make. Squish Coco 5.1.0 - 215 - froglogic GmbH Qt 41.5 Qt Framework 41.5.1 Qt Creator The following steps are needed to add instrumentation to a Qt Creator project. • Switch to the Projects mode using the corresponding button in the mode pane on the left side of the IDE’s interface. • Create a custom build mode called CodeCoverage: – Select the Debug or Release field in the Edit build configuration combo box. – Click on Add¡Clone Selected. – Enter CodeCoverage as new configuration name. • Adjust the Build directory such that the CodeCoverage configuration uses a directory different from that of the other builds. • In the Build Environment section, expand the System Environment pane and modify the PATH variable by prepending the path of the CoverageScanner binary: Platform new Path macOS /Applications/SquishCoco:hold pathi Linux™ /opt/SquishCoco/bin:hold pathi Microsoft® Windows c:\Program Files\SquishCoco;hold pathi Under Linux™, Squish Coco may have been installed locally. If it has been your account name is usr, use the path /home/usr/SquishCoco/bin:hold pathi instead of that one in the table. • Select the build step qmake and add to the Additional arguments the text CONFIG+=CodeCoverage. This allows specifying additional code coverage settings by modifying the qmake’s profile file (41.5.2, page 217). • Select the build step Make and enter in the Additional arguments box the following text: Platform Additional Arguments macOS LINK=csclang++ AR=csar CXX=csclang++ CC=csclang Linux™ LINK=csg++ AR=csar CXX=csg++ CC=csgcc Microsoft® Windows LINK=cscl LIB=cslib CXX=cscl CC=cscl With these settings, a new build mode CodeCoverage has been created that creates an instrumented version of your program. If the project has unit tests, it is possible to execute them at every build and also generate a coverage report for them. In order to do this, the following additional steps are needed. • In the Build Steps section, click on Add Build Step and select Custom Process Step. A new build step is created to execute the unit test the code coverage of which you are interested in. Fill in the Command box the name of the program that runs the unit tests and into the Working directory box the directory in which it should be executed. If necessary, also fill in the command line arguments of the program. • Add another Custom Process Step. For this step enter the full path of the cmcsexeimport tool in the Command box. In Working directory, enter the same directory as in before for the unit test program. As for the arguments of cmcsexeimport, make sure the csmes and csexe files are included. An entry in the Arguments could therefore look like Squish Coco 5.1.0 - 216 - froglogic GmbH Qt --title="Execution" -m unit_test.csmes -P -e unit_test.csexe. Remember to replace the word unit_test in unit_test.csexe and unit_test.csmes with the actual name of your unit test. • The final Custom Process Step that needs to be added is the one for the cmreport tool, in case that report generation was wanted. Again set the Working directory as that of the unit test, and set cmreport’s absolute path as the Command. Then set the arguments, e.g.: --title="Execution" -m unit_test.csmes --html=report Remember again to replace unit_test with the name of your unit test. Now, every time you click on Build (not just Run) with this configuration, the project will be built, instrumented and optionally a report will be generated. Note that in case the project has been built before this configuration was performed, it will be required to execute the Clean Build step before building, so that the coverage measurement data will be generated. Also keep in mind that when there was an error and one of the build steps exits with a non-zero exit status code, Qt Creator stops the build and displays the error message of the executable that failed, instead of continuing the build. This means that if a unit test fails, instead of cmcsexeimport running and importing the execution as a failed execution, cmcsexeimport does not run at all until the test case failure was fixed. 41.5.2 qmake qmake is a tool to generate a Makefile in a platform-independent way, using as specification a so-called “project file”. By default, qmake chooses the programs that are used for compilation. We can however use the Squish Coco wrappers by setting some of qmake’s variables to new values. This can be done via command line parameters or by editing the project configuration files. Command line. In simple cases, we can do this on the command line. There are two methods. • The first method works with older versions of Qt. In it, one passes the names of the Squish Coco wrappers as parameters to qmake. Under Windows this looks like C:\code>qmake QMAKE_LINK=cslink QMAKE_CXX=cscl QMAKE_CC=cscl Here we redefine the variables QMAKE_CC, QMAKE_CXX and QMAKE_LINK: They contain the name of the C compiler, the C++ compiler and the linker that are used in the Makefiles generated by qmake. For other platforms, the values of these variables must be replaced with the wrappers for the compilers and linker that are used there. • In some newer versions of Qt there is however a bug in qmake that makes the previous approach unusable. In this case, one can overwrite the compiler versions later, when the Makefile generated by qmake is used for compilation. In every call of the make program, certain Makefile variables must be overwritten. Under Microsoft® Windows, the resulting command line is: C:\code>nmake LINK=cslink CXX=cscl CC=cscl hother optionsi (Under UNIX®-like systems, make must be called instead of nmake.) Squish Coco 5.1.0 - 217 - froglogic GmbH Qt Project configuration. In more complex cases it is better to enable coverage by putting some declarations in the qmake project files. Since one needs to build the project with and without coverage, the new definitions must be put into a scope. (A scope is a region in a qmake project file that can be activated on demand.) The following listing shows a template for such a scope, named CoverageScanner, which should be sufficient for most most projects. CodeCoverage { COVERAGE_OPTIONS = QMAKE_CFLAGS += $$COVERAGE_OPTIONS QMAKE_CXXFLAGS += $$COVERAGE_OPTIONS QMAKE_LFLAGS += $$COVERAGE_OPTIONS QMAKE_CC=cs$$QMAKE_CC QMAKE_CXX=cs$$QMAKE_CXX QMAKE_LINK=cs$$QMAKE_LINK QMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIB QMAKE_AR=cs$$QMAKE_AR QMAKE_LIB=cs$$QMAKE_LIB } Here we also set the variables QMAKE_LINK_SHLIB and QMAKE_AR, which contain the names of the command to link shared libraries and that to generate archives. Furthermore, can use COVERAGE_OPTIONS to set coveragescanner commandline options (see Chapter 9.2, page 73) to customize the project. An empty value for COVERAGE_OPTIONS will also work and results in a default instrumentation. In a small project, the CodeCoverage scope may then be copied to all profile files of the project, i.e. those that end in “.pro”. (In fact, it is enough to insert them only into those files that actually compile code and not those that only include other files.) If the project is larger, it has very often a file with common settings that is included by all profiles: This is then the most convenient place to insert the CodeCoverage scope only once. The new coverage scope is by default inactive. To enable code coverage in a project you want to built, just add the name of the scope to the CONFIG variable when configuring it with qmake: $ qmake CONFIG+=CodeCoverage 41.5.3 moc The Meta-Object Compiler moc adds automatically new methods to each classes derived from QObject. For example, the translation function tr, the source code for all signals, the cast operator qt_cast, . . . In order to instrument the code using the Qt Framework and not the Qt code itself, CoverageScanner provides the command line options --cs-qt3 for Qt3 and --cs-qt4 for Qt4 and Qt5. They are enabled by default. In this case: • Q_OBJECT macro is no more instrumented. • All signal are instrumented in order to track their emission. • The glue code necessary for the signal/slot mechanism is not instrumented. • The code of Q_FOREACH macro is not instrumented on Qt4. Squish Coco 5.1.0 - 218 - froglogic GmbH SCons 41.5.4 qbs To use CoverageScanner with the Qt Build Suite (qbs) it can be set up as a toolchain. This toolchain can then be used for all qbs projects. In order to set up CoverageScanner as a toolchain, issue the following command: qbs setup-toolchains --type gcc /opt/SquishCoco/bin/csgcc csgcc For Unix-based operating systems some additional configuration steps are necessary: qbs config profiles.csgcc.cpp.archiverPath /opt/SquishCoco/bin/csar qbs config profiles.csgcc.cpp.linkerName csg++ qbs config profiles.csgcc.cpp.nmPath /opt/SquishCoco/bin/csgcc-nm The csgcc toolchain can then also be used as base profile for Qt projects: qbs setup-qt /opt/Qt/bin/qmake qt-csgcc qbs config profiles.qt-csgcc.baseProfile csgcc 41.6 SCons To use Squish Coco with SCons, proceed as followings: • Prepend the path of CoverageScanner’s wrapper (csgcc, cscl, . . . ) to the PATH environment variable. The PATH environment variable should be set to be able to execute CoverageScanner wrappers . • Set CC, AR and LINK variable to CoverageScanner corresponding wrapper. Example: when using Microsoft® Visual Studio® compiler, set CC to cscl.exe, set LINK to cslink.exe and set AR to cslib.exe. Do no use absolute file paths to the compiler wrapper since some versions of SCons do not properly ! handle spaces in file names. • Add additional code coverage settings to the variables CCFLAGS, ARFLAGS and LINKFLAGS to exclude for example files from the instrumentation. Here is a code snippet which can be used for Microsoft® Visual Studio® command line tools: import os from os.path import pathsep env = Environment() # Add the path of Squish Coco compiler wrapper env[ ’ENV’ ][ ’PATH’ ] = os.environ[ ’SQUISHCOCO’ ] + pathsep + env[ ’ENV’ ][ ’PATH’ ] # TEMP variable need to be defined env[ ’ENV’ ][ ’TEMP’ ] = os.environ[ ’TEMP’ ] env[ ’ENV’ ][ ’INCLUDE’ ] = os.environ[ ’INCLUDE’ ] # Set the compiler to Squish Coco wrappers Squish Coco 5.1.0 - 219 - froglogic GmbH ARM Keil env[ ’CC’ ] = ’cs’ + env[ ’CC’ ] ; env[ ’AR’ ] = ’cs’ + env[ ’AR’ ] ; env[ ’LINK’ ] = ’cs’ + env[ ’LINK’ ] ; # Code coverage settings coverageflags = [ ’--cs-count’ ] env[ ’CCFLAGS’ ] = env[ ’CCFLAGS’ ] + coverageflags ; env[ ’ARFLAGS’ ] = env[ ’ARFLAGS’ ] + coverageflags ; env[ ’LINKFLAGS’ ] = env[ ’LINKFLAGS’ ] + coverageflags ; In order to set correctly the build environment, it is necessary to start the build from Visual Studio’s developper prompt (is available through the start menu) or Squish Coco’s console provided by the "Build Environment Seletction". 41.7 ARM® Keil® µVision To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapper for ARM and enable the code coverage analysis during the compilation. Processed as followings: 1. To install CoverageScanner compiler wrapper for the ARM® Keil® µVision tool chain, proceed as follows: (a) Open the Build Environment Selection application. (hWindows Cocoi\toolselector.exe) (b) Select the item “ARM-Keil”. (c) Select the directory o which the ARM® Keil® µVision compilers are installed (ex: C:\Keil_v5\ARM) (d) Click on “Install ARM-Keil Support” and wait for the confirmation dialog that the tools are generated. 2. Activate the code coverage analysis during the compilation (a) Click on "Project¡Option for Target..." (b) On the "C/C++" tab, add --cs-on into the "Misc Controls" field. Squish Coco 5.1.0 - 220 - froglogic GmbH ARM Keil Figure 41.1: Activating the code coverage analysis for an ARM® compiler (c) On the "Linker" tab, add --cs-on into the "Misc Controls" field. Figure 41.2: Activating the code coverage analysis for an ARM® linker Squish Coco 5.1.0 - 221 - froglogic GmbH ARM-DS 41.8 Arm® DS To enable the code coverage analysis for an Arm® DS project, it is first necessary to compile it with Squish Coco compiler wrapper for ARM and enable the code coverage analysis during the compilation. Processed as followings: 1. To install CoverageScanner compiler wrapper for the Arm® DS tool chain, proceed as follows: (a) Open the Build Environment Selection application. (hWindows Cocoi\toolselector.exe) (b) Select the item “ARM-DS”. Figure 41.3: Installation of Arm® DS support in Build Environment Selection. (c) Select the directory o which the Arm® DS compilers are installed (ex: C:\Program Files\DS-5 v5.29.3) (d) Click on “Install ARM-DS Support” and wait for the confirmation dialog that the tools are generated. Figure 41.4: Comfirmation of the installation of Arm® DS support. 2. Activate the code coverage analysis during the compilation (a) Click on "Project¡Properties" (b) On the "Arm C/C++ Compiler 5" tab, add --cs-on into the "Miscellaneous" field. Squish Coco 5.1.0 - 222 - froglogic GmbH ARM-DS Figure 41.5: Activating the code coverage analysis for an ARM® compiler (c) On the "Arm Linker 5" tab, add --cs-on into the "Misc Controls" field. Squish Coco 5.1.0 - 223 - froglogic GmbH ARM-DS Figure 41.6: Activating the code coverage analysis for an ARM® linker In order to upload the coverage data from the target to the host, it is necessary to define some custom I/O functions and trigger the coverage generation by calling __coveragescanner_save(). Due to the fact that Arm® DS supports semi-hostings extensions, the simplest way is to use it to generate the execution report directly on the host file system. For that modify the main() function to add a call to __coveragescanner_set_custom_io(), __coveragescanner_filename() and __coveragescanner_save() as in the following sample: Syntax: #include #ifdef __COVERAGESCANNER__ static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); } static void *csfopenappend(const char *path) Squish Coco 5.1.0 - 224 - froglogic GmbH ARM-DS { return (void*)fopen(path,"a+"); } static int csfclose(void *fp) { return fclose((FILE*)fp); } #endif int main() { #ifdef __COVERAGESCANNER__ __coveragescanner_set_custom_io( NULL, csfputs, csfopenappend, NULL, NULL, csfclose, NULL); __coveragescanner_filename( "c:\\tmp\\test_arm5_csexe" ); /* destination file on the host file system */ #endif ...... #ifdef __COVERAGESCANNER__ __coveragescanner_save(); /* saves the execution report */ #endif return 0; } The project can now be rebuilt and a project file with the extension .afx.csmes will the generated. This file can be opened in CoverageBrowser. Execute then the target application and wait until __coveragescanner_save() is called. A file c:\tmp\test_arm5_csexe will be generated (hard coded in our sample). Import it using Coverage- Browserand the coverage can be analyzed as in the following screenshot: Squish Coco 5.1.0 - 225 - froglogic GmbH Green Hills Figure 41.7: Code coverage of an Arm® DS project i The extension of an execution report is usually .csexe but the semi-hosting implementation provided by ARM remove the file extensions. It is necessary to remove the default file filter in the import dialog of CoverageBrowser to be able to import it. 41.9 Green Hills® Software MULTI Launcher 41.9.1 Installation To install CoverageScanner compiler wrapper for the Green Hills® Software tool chain, proceed as follows: 1. Open the Build Environment Selection application. (hWindows Cocoi\toolselector.exe) 2. Select the item “Green Hills”. 3. Select the directory o which the Green Hills® Software compilers are installed 4. Click on “Install Green Hills Support” and wait for the confirmation dialog that the tools are generated. Squish Coco 5.1.0 - 226 - froglogic GmbH Green Hills 41.9.2 Command Line Tools The command line tools for the Green Hills® Software compiler are installed in the folder squishcoco of the installed native tool chain. Example: if the native tool chain is installed under c:\ghs\comp_201426, the Squish Coco compiler wrapper are installed under c:\ghs\comp_201426\squishcoco. The compiler wrappers are replacing completely the Green Hills® Software tool chain. To activate the code coverage analysis, it is necessary to add the parameter --cs-on to the compiler arguments. 41.9.3 CoverageScanner Library In order to save an execution report, it is necessary to: 1. provide to the CoverageScanner library a list of I/O functions which let you upload the coverage information to the host. 2. Save manually the coverage report on a specific trigger. To save a coverage report the following code snippet can be used (an this code in an event handler which should trigger the report generation): 1 #ifdef __COVERAGESCANNER__ 2 __coveragescanner_save(); 3 __coveragescanner_clear(); 4 #endif To provide the i/O functions it is necessary to call __coveragescanner_set_custom_io() in the main() function of the application. At least three functions need to be provided by __coveragescanner_set_custom_io(): 1. An equivalent function of fopen() which opens the execution report file (with the extension .csexe) in append mode. 2. An equivalent function of fclose() 3. An equivalent function of fputs() to transfer the contain. Example: The following code write an execution report to the local file system using the C file API. 1 #ifdef __COVERAGESCANNER__ 2 static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); } 3 static void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); } 4 static int csfclose(void *fp) { return fclose((FILE*)fp); } 5 #endif 6 7 int main() 8 { 9 #ifdef __COVERAGESCANNER__ 10 __coveragescanner_set_custom_io( NULL, 11 csfputs, 12 csfopenappend, 13 NULL, 14 NULL, 15 csfclose, 16 NULL); 17 #endif Squish Coco 5.1.0 - 227 - froglogic GmbH Visual DSP 18 .... 19 } 41.10 VisualDSP® To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapper and enable the code coverage analysis during the compilation in the VisualDSP® configuration. Processed as followings: 1. Click on "Project¡Project Options..." 2. On the "Compile" tab, add --cs-on into the "Additional options" field. Figure 41.8: Activating the code coverage analysis for the compiler 3. On the "Link" tab, add --cs-on into the "Additional options" field. Squish Coco 5.1.0 - 228 - froglogic GmbH Visual DSP Figure 41.9: Activating the code coverage analysis for the linker 4. Rebuild your project. On most embedded targets, it is necessary to provide a dedicated I/O since no file systems are available for storing the code coverage information. Also since embedded applications generally never exit, it is often necessary to implement an event handler which saves the execution report upon the reception of a specific trigger. The simulator emulates the support of a file system. To save the coverage report on the current build directory, it is only necessary to register in the first lines of the main() a custom file I/O which uses the standard C file API. Example: 1 #ifdef __COVERAGESCANNER__ 2 static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); } 3 static void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); } 4 static int csfclose(void *fp) { return fclose((FILE*)fp); } 5 #endif 6 7 int main() 8 { 9 #ifdef __COVERAGESCANNER__ 10 __coveragescanner_set_custom_io( NULL, 11 csfputs, 12 csfopenappend, 13 NULL, 14 NULL, 15 csfclose, 16 NULL); 17 #endif 18 .... 19 } To record a coverage report when a specific trigger occurs, add the following source code lines in its handler: Squish Coco 5.1.0 - 229 - froglogic GmbH Microsoft Visual Studio 1 #ifdef __COVERAGESCANNER__ 2 __coveragescanner_save(); 3 __coveragescanner_clear(); 4 #endif 41.11 Microsoft® Visual Studio® Squish Coco provides a wrapper for link.exe and cl.exe located on the %SQUISHCOCO%\visualstudio directory. It behaves exactly like the corresponding Microsoft® wrapper except that the code coverage analysis becomes activated when the option --cs-on is added to the command arguments. These wrappers call the Microsoft® tools for compilation or for linkage. 41.11.1 Microsoft® Visual Studio® .NET C# Compiler To activate the instrumentation of C# source code, it is only necessary to add the symbol COVERAGESCANNER_COVERAGE_ON in the properties of the Microsoft® Visual Studio® .NET project. Other symbols can be appended to select additional instrumentation options. The full list can be found on the chapter 9.3, page 84. 41.11.2 Microsoft® Visual Studio® .NET C and C++ Compiler To use Squish Coco with Microsoft® Visual Studio® .NET proceed as follows: 1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories. For Microsoft® Visual Studio® 2005 or 2008: (a) Start Microsoft® Visual Studio® 2005 or 2008. (b) Open the option dialog: click on "Tools¡Preferences...". (c) Select the item "Projects¡VC++ Directories". (d) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Fig- ure 41.10, page 231) Squish Coco 5.1.0 - 230 - froglogic GmbH Microsoft Visual Studio Add the path of the compiler wrapper for Visual Studio to the first position Figure 41.10: Installation of CoverageScanner on Microsoft® Visual Studio® 2005 and 2008: Setting the path of CoverageScanner. For Microsoft® Visual Studio® 2010: (a) Start Microsoft® Visual Studio® 2010. (b) Open a C++ project. (c) Open the project properties using the context menu of the laded project. (d) Select the item "Configuration Properties¡VC++ Directories". (e) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of executable directories. (see Figure 41.11, page 231) Add the path of the compiler wrapper for Visual Studio to the first position Figure 41.11: Installation of CoverageScanner on Microsoft® Visual Studio® 2010: Setting the path of CoverageScanner. 2. To activate code coverage analysis: (a) Open a Visual C or C++ project. (b) Edit the project settings (click on "Project¡Properties"). (c) Add to the option --cs-on to the additional command line arguments of the C or C++ compiler and linker. (see Figure 41.12, page 232) Squish Coco 5.1.0 - 231 - froglogic GmbH Microsoft Visual Studio (d) In the additional arguments of the linker, add the --cs-libgen which specifies which library should be used for the generation of the CoverageScanner library. The table 41.1, page 232 contains the list of recommended settings. (e) For Microsoft® Windows CE applications, append to the linker arguments the command line option --cs-architecture which lets you specify the target platform. The table 41.2, page 232 contains the list of available architectures. Library Library File Command line option Single Threaded LIBC.LIB --cs-libgen=/ML Static MultiThread LIBCMT.LIB --cs-libgen=/MT Dynamic Link (DLL) LIBCRT.LIB --cs-libgen=/MD Debug Single Threaded LIBCD.LIB --cs-libgen=/MLd Debug Static MultiThread LIBCMTD.LIB --cs-libgen=/MTd Debug Dynamic Link (DLL) LIBCRTD.LIB --cs-libgen=/MDd Table 41.1: CoverageScanner library settings for Microsoft® Visual Studio® Targeted Architecture Command line option ARM Microprocessor --cs-architecture=ARM ARM Microprocessor (Thumb code) --cs-architecture=THUMB x86 Microprocessor --cs-architecture=IX86 MIPS16 Microprocessor --cs-architecture=MIPS16 MIPS Microprocessor --cs-architecture=MIPS MIPS Microprocessor with FPU --cs-architecture=MIPSFPU SH3 Microprocessor with FPU --cs-architecture=SH3 SH4 Microprocessor with FPU --cs-architecture=SH4 Table 41.2: List of target architectures Add the command line option '--cs-on' here Figure 41.12: Activation of the instrumentation under Visual Studio® .NET. Squish Coco 5.1.0 - 232 - froglogic GmbH Microsoft Visual Studio Add the command line option '--cs-on' here Figure 41.13: Activation of the instrumentation under Visual Studio® .NET. 41.11.3 Microsoft® Visual C++ Express To use Squish Coco with Microsoft® Visual C++ Express proceed as follows: 1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories. (a) Start Microsoft® Visual C++ Express. (b) Open the option dialog: click on "Tools¡Preferences...". (c) Select the item "Projects¡VC++ Directories". (d) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Fig- ure 41.14, page 233) Add the path of the compiler wrapper for Visual Studio to the first position Figure 41.14: Installation of CoverageScanner on Visual C++ Express: Setting the path of CoverageScanner. 2. The activation of the code coverage analysis is similar to Microsoft® Visual Studio® .NET. (see Chapter 41.11.2, page 230) Squish Coco 5.1.0 - 233 - froglogic GmbH Microsoft Visual Studio 41.11.4 Microsoft® Visual Studio® 6.0 To use Squish Coco with Microsoft® Visual Studio® 6.0 proceed as follows: 1. Add the location of the CoverageScanner wrappers to the first position in the executable directories. (a) Start Microsoft® Visual Studio® 6.0. (b) Open the option dialog: click on "Tools¡Preferences...". (c) Select the item "Directories". (d) Select "Executable files" in the combobox "Show directories for:". (e) Add the path of the directory visualstudio of the Squish Coco installation3 to the first position in the list of directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the path c:\programme\SquishCoco\visualstudio, (see Figure 41.15, page 234)) Add the path of the compiler wrapper for Visual Studio to the first position Figure 41.15: Installation of CoverageScanner on Visual Studio® 6.0: Setting the path of CoverageScanner. 2. To activate the code coverage analysis: (a) Open a Visual C or C++ project. (b) Edit the project settings (click on "Project¡Properties"). (c) Add the option --cs-on to the additional command line arguments of the C or C++ compiler and linker. (see Figure 41.16, page 234) (d) In the additional arguments of the linker, add the --cs-libgen which specifies the name of the generated CoverageScanner library. The table 41.1, page 232 contains the list of recommended settings. Add the command line option '--cs-on' here Figure 41.16: Activation of the instrumentation under Visual Studio® 6.0. 3Microsoft® Visual Studio® 6.0 does not handle system variables in the path list. So the %SQUISHCOCO% variable needs to be expanded. Squish Coco 5.1.0 - 234 - froglogic GmbH eMbedded Visual C++ Add the command line option '--cs-on' here Figure 41.17: Activation of the instrumentation under Visual Studio® 6.0. 41.12 Microsoft® eMbedded Visual C++® To use Squish Coco with Microsoft® eMbedded Visual C++®proceed as follows: 1. Add the location of the CoverageScanner wrappers to the first position in the executable directories. (a) Start Microsoft® eMbedded Visual C++®. (b) Open the option dialog: click on "Tools¡Preferences...". (c) Select the item "Directories". (d) Select "Executable files" in the combobox "Show directories for:". (e) Select "Platform" and the targeted "CPUs". (f) Add the path of the directory WinCE of the Squish Coco installation4 to the first position in the list of directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the path c:\programme\SquishCoco\WinCE, (see Figure 41.18, page 235)) Select the Platform and the CPU Add the path of the compiler wrapper for Windows CE to the first position Figure 41.18: Installation of CoverageScanner on eMbedded Visual C++®: Setting the path of CoverageScanner. 2. To activate the code coverage analysis: (a) Open a Visual C or C++ project. 4Microsoft® eMbedded Visual C++®does not handle system variables in the path list. So the %SQUISHCOCO% variable needs to be expanded. Squish Coco 5.1.0 - 235 - froglogic GmbH Eclipse (b) Edit the project settings (click on "Project¡Properties"). (c) Add the option --cs-on to the additional command line arguments of the C and C++ compiler and linker. (see Figure 41.19, page 236) Add the command line option '--cs-on' here Figure 41.19: Activation of the instrumentation under eMbedded Visual C++®. Add the command line option '--cs-on' here Figure 41.20: Activation of the instrumentation under eMbedded Visual C++®. 41.13 Eclipse™ IDE for C/C++ In Eclipse™ IDE for C/C++, code coverage is enabled for a configuration by replacing the names of the compilers it uses with the names of the Squish Coco compiler wrappers. This can be done in the following way: 1. Start Eclipse™. 2. Load the C or C++ project that should be instrumented. 3. Open the property window (Project¡ Properties). 4. Click on "C/C++ Build/Settings". 5. Create a new configuration by clicking on "Manage Configurations...", and select it. 6. Click on "Tools Settings" tab. Squish Coco 5.1.0 - 236 - froglogic GmbH Eclipse 7. Click on "GCC C++ Compiler" and prepend cs to the name of the compiler. 8. Click on "GCC C Compiler" and prepend cs to the name of the compiler. 9. Click on "C++ Linker" and prepend cs to the name of the linker. 10. If it is a library project, also click on "GCC archiver" and replace the name of the archiver command ar with csar. Now the Eclipse will use the Squish Coco wrappers instead of the compilers when compiling. Append 'cs' to the compiler/linker executable name Figure 41.21: Eclipse™ settings However, the Squish Coco wrappers are not by default in the search part and can not yet be found during compilation. To change this, a copy of the PATH variable needs to be added to the "C/C++ Build¡Environment" section of the property window, and the path of the Squish Coco binaries added. Under UNIX®, and with Squish Coco installed at the default location, PATH will then have a value like “/opt/SquishCoco/bin:/usr/local/bin:/usr/bin:/bin”. Almost always it will be necessary to modify the behavior of Squish Coco by setting command line options. The easiest way to do this is by adding the variable COVERAGESCANNER_ARGS in "C/C++ Build¡Environment". Its value then consists of command line options (see Chapter 9.2, page 73). Code instrumentation is however already activated if this variable is not set or empty. i If the value of COVERAGESCANNER_ARGS has changed, it is necessary to compile the whole project again; otherwise the new options have no effect. If everything is done correctly and the project is compiled, one will see in the console window that csg++ is used instead of g++, etc., and that a .csmes file is created next to the place of the newly-built binary. When the binary then is run, a .csexe file is created in its working directory, which is typically a different directory from that of the .csmes file. Squish Coco 5.1.0 - 237 - froglogic GmbH Xcode 41.14 Apple® Xcode To use Squish Coco with Apple® Xcode proceed as follows: To activate the code coverage analysis: 1. Open a terminal window and set the CPLUSPLUS, LDPLUSPLUS, LD and CC to CoverageScanner compiler wrapper. The path of native compiler (clang, clang++, gcc or g++) need to be present in the PATH environment variable. Start Xcode using the open command. If GCC is used as compiler: SQUISHCOCO=/Applications/SquishCoco/wrapper export CC=$SQUISHCOCO/gcc export LD=$SQUISHCOCO/gcc export CPLUSPLUS=$SQUISHCOCO/g++ export LDPLUSPLUS=$SQUISHCOCO/g++ open /Developer/Applications/Xcode.app If clang is used as compiler: SQUISHCOCO=/Applications/SquishCoco/wrapper export CC=$SQUISHCOCO/clang export LD=$SQUISHCOCO/clang export CPLUSPLUS=$SQUISHCOCO/clang++ export LDPLUSPLUS=$SQUISHCOCO/clang++ export XCODE_TOOLCHAIN_DIR=/Applications/Xcode.app/Contents/Developer/Toolchains export XCODE_TOOLCHAIN=$XCODE_TOOLCHAIN_DIR/XcodeDefault.xctoolchain/usr/bin/ export PATH=$XCODE_TOOLCHAIN:$PATH open /Applications/Xcode.app 2. Open a Xcode C or C++ project. 3. Edit the project settings (click on "Project¡Edit Project Settings"). 4. Add the option --cs-on to the additional command line arguments of the C and C++ compiler (fields Other C Flags and Other C++ Flags) and linker (field Other Linker Flags). (see Figure 41.22, page 239) 5. Disable the usage of the precomiled header: Open the settings of the active target ("Project¡Edit Active Target") and remove the contains of Prefix Header. Squish Coco 5.1.0 - 238 - froglogic GmbH Xcode Add the linker command line option '--cs-on' here Add the compiler command line option '--cs-on' here Figure 41.22: Activation of the instrumentation under Apple® Xcode. 41.14.1 VxWorks™ support on Linux™ To activate Squish Coco on Wind River’s Workstation proceed as follows: 1. Start Wind River’s Workbench 2. Select your project in the "Project Explorer" window and click on the entry "Properties" in the context menu. 3. Select the build properties, the compiler tool chain and click on the "Variables" tab. 4. Search for the TOOL_PATH variable and replace the /bin string at the end with /squishcoco: Squish Coco 5.1.0 - 239 - froglogic GmbH Xcode Figure 41.23: Setting the TOOL_PATH variable After this change, the project can be compiled with CoverageScanner or the native tool chain. To use Coverage- Scanner, it is necessary to add the option --cs-on to the command line arguments of the compiler and the linker. To do this, proceed as followings: 1. Start the Wind River Workbench 2. Select your project in the "Project Explorer" window and click on "Properties" of the context menu. 3. Select the build properties, the compiler tool chain and click on the "Tools" tab. 4. Select the entry "C-Compiler" in the "Build Tool" combo box and add to the content of the "Tool Flags..." field the argument --cs-on: Squish Coco 5.1.0 - 240 - froglogic GmbH IAR Embedded Workbench Figure 41.24: Activating the code coverage analysis 5. Do the same for the “C++-Compiler” and the “Linker” tool. With these settings, a file with the extension .vxe.csmes is generated when the code is compiled. It contains the complete instrumented code and can be inspected with CoverageBrowser. The resulting target application will then create a file with the suffix .csexe after it has run. This file is created on the target file system. It can them be transferred to the host and imported into the .vxe.csmes file to create a report. A video that illustrates the content of this chapter is available on https://youtube.com/watch?v=bMxMV6qHsYU. 41.15 IAR Embedded Workbench Only Windows OS supported. The IAR Embedded Workbench should be already installed. Execute the "Build Environment Selection" program after the installation of Coco. The "Build Environment Selection" program can be found in the Coco install folder with the name "toolselector.exe". In the main window select the correct installation folder for your IAR binary folder and click Install IAR Support Squish Coco 5.1.0 - 241 - froglogic GmbH IAR Embedded Workbench Figure 41.25: Build Environment Selection The next step is to adjust your Embedded Workbench to execute Coco. To this by clicking Extra Options in the project settings, found in: Project > Options > C/C++ Compiler > Extra Options Figure 41.26: Compiler Extra Options Enable the checkbox and type “--cs-on” into the field. Also add this to the linker by clicking the following in the menu. Project > Options > Linker > Extra Options Squish Coco 5.1.0 - 242 - froglogic GmbH TASKING Support Figure 41.27: Linker Extra Options Enable the checkbox and type “--cs-on” into the field. To deactivate Coco just uncheck the two checkboxes again. 41.16 TASKING Support The "TriCore Eclipse IDE" from TASKING should already be installed. Execute the "Build Environment Selection" program after the installation of Coco. The "Build Environment Selection" program can be found in the Coco install folder with the name "toolselector.exe". In the main window select "TASKING VX-Toolset for TriCore" and click on "Install TASKING Support" Squish Coco 5.1.0 - 243 - froglogic GmbH TASKING Support Figure 41.28: Build Environment Selection The next step is to activate the code coverage support in the Eclipse IDE. Open "TriCore Eclipse IDE", click on the top of your project and with the contect menu, select "Properties". The project’s configuration dialog will be opened and then select "C/C++ Build->Setting" on the side bar. The complete compiler and linker settings will be displayed. Select then "C/C++ Compiler->Miscellaneous". You shoudl see then an additional argument as in the following screenshot: Squish Coco 5.1.0 - 244 - froglogic GmbH TASKING Support Figure 41.29: Compiler Extra Options Add “--cs-on” to enable the code coverage analysis for the compiler. Then do the same for the linker, select "Linker->Miscellaneous" and add “--cs-on” to the additional arguments. Squish Coco 5.1.0 - 245 - froglogic GmbH TASKING Support Figure 41.30: Linker Extra Options It is then possible to instrument the embedded application by rebuilding it. A .csmes file is then generated upon the compilation. But to see the coverage it is necessary to provide a set of functions that permist to upload it to the host. The TASKING simulator support ARM’s semi-hosting function which permits to write on file directly from the target to the host’S file system. We can use this to generate the execution report (.csexe file) but on real hardware, it may be necessary to use other implementations. For teh setup, add the following lines to the source containing the main function: #include #ifdef __COVERAGESCANNER__ static int csfputs(const char *s, void *stream) { Squish Coco 5.1.0 - 246 - froglogic GmbH TASKING Support return fputs(s, (FILE *)stream); } static void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); } static int csfclose(void *fp) { return fclose((FILE*)fp); } #endif Then in the first lines of the main() function, configure these functions as handler: int main( int argc, char *argv[] ) { #ifdef __COVERAGESCANNER__ __coveragescanner_set_custom_io( NULL, csfputs, csfopenappend, NULL, NULL, csfclose, NULL ); #endif ...... } Then at the location where the execution report should be saved, add the following lines: #ifdef __COVERAGESCANNER__ __coveragescanner_save(); #endif Squish Coco 5.1.0 - 247 - froglogic GmbH Support for specific test frameworks Chapter Support for specific test frameworks 42 42.1 CppUnit CppUnit1 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test. The following code is an example how this can be done: #include class CoverageScannerListener : public CppUnit::TestListener { public: CoverageScannerListener() {} void startTest( CppUnit::Test *test ) { m_testFailed = false; #ifdef __COVERAGESCANNER__ int pos; // Adjusting the name of the test to display the tests // ina tree view in CoverageBrowser std::string testname = "CppUnit/" + test->getName(); while ( ( pos = testname.find( "::", 0 ) ) != std::string::npos ) testname.replace( pos, 2, "/" ); // Reset the code coverage data to get only the code coverage 1Project page of CppUnit: http://cppunit.sourceforge.net Squish Coco 5.1.0 - 248 - froglogic GmbH CppUnit // of the actual unit test. __coveragescanner_clear(); __coveragescanner_testname( testname.c_str() ) ; #endif } void addFailure( const CppUnit::TestFailure &failure ) { m_testFailed = true; } void endTest( CppUnit::Test *test ) { #ifdef __COVERAGESCANNER__ // Recording the execution state in the coverage report if ( m_testFailed ) __coveragescanner_teststate( "FAILED" ); else __coveragescanner_teststate( "PASSED" ); // Saving the code coverage report of the unit test __coveragescanner_save(); __coveragescanner_testname( "" ); #endif } private: bool m_testFailed; // Prevents the use of the copy constructor and operator. CoverageScannerListener( const CoverageScannerListener © ); void operator =( const CoverageScannerListener © ); }; int main( int argc, char* argv[] ) { #ifdef __COVERAGESCANNER__ __coveragescanner_install( argv[0] ); #endif // Create the event manager and test controller CPPUNIT_NS::TestResult controller; // Adda listener that colllects test result CPPUNIT_NS::TestResultCollector result; controller.addListener( &result ); // Adda listener that print dots as test run. CPPUNIT_NS::BriefTestProgressListener progress; controller.addListener( &progress ); // Adda listener that saves the code coverage information CoverageScannerListener coveragescannerlistener; controller.addListener( &coveragescannerlistener ); Squish Coco 5.1.0 - 249 - froglogic GmbH QTestLib // Add the top suite to the test runner CPPUNIT_NS::TestRunner runner; runner.addTest( CPPUNIT_NS::TestFactoryRegistry::getRegistry().makeTest() ); runner.run( controller ); return result.wasSuccessful() ? 0 : 1; } In the example, we have done the following steps: 1. We write a CppUnit listener class which records the code coverage of unit each test after it is completed. We want to be able to run the program with and without Squish Coco. Therefore we use in the code the macro __COVERAGESCANNER__ for conditional compilation. The macro is defined in every file that is instrumented by Squish Coco, without the need to #include anything. In the listener class, CppUnitListener, we use the following menber functions: startTest(): This function is called before each test begins. In it, we compute a test name with the information provided by CppUnit and pass it to the Squish Coco library with __coveragescanner_testname(). We also call the function __coveragescanner_clear(): It empties the internal database and so makes sure that the coverage of the code that was executed before this test is ignored. addFailure(): This function is called after a test fails. It just sets a flag that is used by the other functions. endTest(): This function is called after a test has ended. It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with __coveragescanner_save(). 2. We call __coveragescanner_install() in the main() function. 3. We add this listener in the test manager of CppUnit, the class CPPUNIT_NS::TestResult. In the example above, this is done by the following lines: CoverageScannerListener coveragescannerlistener; controller.addListener( &coveragescannerlistener ); 42.2 QTestLib QTestLib is a unit test framework for Qt. It can easily be adapted to get the code coverage for each unit test. Proceed as follows: 1. Call __coveragescanner_install() in the main() function. 2. Write a subclass of QObject, named TestCoverageObject. It must record the code coverage at the end of every unit test. 3. Instead of inheriting from QObject, let all your test cases inherit from TestCoverageObject. Squish Coco 5.1.0 - 250 - froglogic GmbH QTestLib 4. The TestCoverageObject class provides its own init() and cleanup() slots, which use the Coverage- Scanner API to save the code coverage report. If these slots are also declared in the test case classes, it is necessary to rename them to initTest() and cleanupTest(). 5. Compile your project with code code coverage enabled. TestCoverageObject header: #ifndef _TEST_COVERAGE_OBJECT_H #define _TEST_COVERAGE_OBJECT_H #include TestCoverageObject source: #include "testcoverageobject.h" #include void TestCoverageObject::init() { #ifdef __COVERAGESCANNER__ __coveragescanner_clear(); #endif initTest(); } void TestCoverageObject::cleanup() { cleanupTest(); #ifdef __COVERAGESCANNER__ QString test_name="unittest/"; test_name+=metaObject()->className(); test_name+="/"; test_name+=QTest::currentTestFunction(); __coveragescanner_testname(test_name.toLatin1()); if (QTest::currentTestFailed()) __coveragescanner_teststate("FAILED"); else __coveragescanner_teststate("PASSED") ; __coveragescanner_save(); Squish Coco 5.1.0 - 251 - froglogic GmbH GoogleTest __coveragescanner_testname(""); #endif } 42.3 GoogleTest GoogleTest2 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test. Simply proceed as follows: 1. Call __coveragescanner_install() in the main() function. 2. Write a TestEventListener class which records the code coverage report upon every unit test completion. The listener should set the name (using __coveragescanner_testname()) and clear the instrumentation (using __coveragescanner_clear()) before executing a test item (class member startTest()) to ensure to get only the coverage data of the concerned test. When an test item is executed, the instrumentation and the execu- tion status should be saved (using __coveragescanner_teststate() and __coveragescanner_save()) in the class member endTest(). The class CodeCoverageListener give an implementation example. 3. Add this listener in the Append function of the GoogleTest listener (function ::testing::UnitTest::GetInstance()->listeners().Append()). 4. Compile the unit test using CoverageScanner. #include class CodeCoverageListener : public ::testing::TestEventListener { public: // Fired before any test activity starts. virtual void OnTestProgramStart(const ::testing::UnitTest& unit_test) { } // Fired before each iteration of tests starts. There may be more than // one iteration if GTEST_FLAG(repeat) is set. iteration is the iteration // index, starting from 0. virtual void OnTestIterationStart(const ::testing::UnitTest& unit_test, int iteration) { } // Fired before environment set-up for each iteration of tests starts. virtual void OnEnvironmentsSetUpStart(const ::testing::UnitTest& unit_test) { } 2Project page of GoogleTest: http://code.google.com/p/googletest/ Squish Coco 5.1.0 - 252 - froglogic GmbH GoogleTest // Fired after environment set-up for each iteration of tests ends. virtual void OnEnvironmentsSetUpEnd(const ::testing::UnitTest& unit_test) { } // Fired before the test case starts. virtual void OnTestCaseStart(const ::testing::TestCase& test_case) { } // Fired before the test starts. virtual void OnTestStart(const ::testing::TestInfo& test_info) { #ifdef __COVERAGESCANNER__ __coveragescanner_clear(); std::string test_name= std::string(test_info.test_case_name()) + ’/’ + std::string(test_info.name()); __coveragescanner_testname(test_name.c_str()); #endif } // Fired aftera failed assertion ora SUCCESS(). virtual void OnTestPartResult(const ::testing::TestPartResult& test_part_result) { } // Fired after the test ends. virtual void OnTestEnd(const ::testing::TestInfo& test_info) { #ifdef __COVERAGESCANNER__ __coveragescanner_teststate("UNKNOWN"); if (test_info.result()) { if (test_info.result()->Passed()) __coveragescanner_teststate("PASSED"); if (test_info.result()->Failed()) __coveragescanner_teststate("FAILED"); } __coveragescanner_save(); #endif } // Fired after the test case ends. virtual void OnTestCaseEnd(const ::testing::TestCase& test_case) { } // Fired before environment tear-down for each iteration of tests starts. virtual void OnEnvironmentsTearDownStart(const ::testing::UnitTest& unit_test) { Squish Coco 5.1.0 - 253 - froglogic GmbH CxxTest } // Fired after environment tear-down for each iteration of tests ends. virtual void OnEnvironmentsTearDownEnd(const ::testing::UnitTest& unit_test) { } // Fired after each iteration of tests finishes. virtual void OnTestIterationEnd(const ::testing::UnitTest& unit_test, int iteration) { } // Fired after all test activities have ended. virtual void OnTestProgramEnd(const ::testing::UnitTest& unit_test) { } }; int main(int argc, char **argv){ //initialize CoverageScanner library #ifdef __COVERAGESCANNER__ __coveragescanner_install(argv[0]); #endif ::testing::FLAGS_gtest_output = "xml"; ::testing::UnitTest::GetInstance()->listeners().Append(new CodeCoverageListener); ::testing::InitGoogleTest(&argc, argv); //init google test framework return RUN_ALL_TESTS(); //run all tests which are listed in this project } 42.4 CxxTest CxxTest3 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test. Proceed as follows: 1. Call __coveragescanner_install() in the main() function. 2. Create a CxxTest TestListener class CoverageScannerListener by subclassing an existing listener. In the example below this is ErrorPrinter. It will record the code coverage report upon every unit test completion. To ensure to get only the coverage data of the concerned test, the listener should set the name with __coveragescanner_testname() and clear the instrumentation with __coveragescanner_clear() before executing a test item (class member enterTest()). 3Project page of CxxTest: http://cxxtest.com Squish Coco 5.1.0 - 254 - froglogic GmbH CxxTest When an test item is executed, the instrumentation and the execution status should be saved in the member function leaveTest() with __coveragescanner_teststate() and __coveragescanner_save(). Finally, all test failure members of this class must be reimplemented to record the test failures. 3. In the main() function call the run() function of CoverageScannerListener instead of CxxTest:: ErrorPrinter().run(). 4. Compile the unit test with CoverageScanner activated. Example: #include class CoverageScannerListener : public CxxTest::ErrorPrinter { public: CoverageScannerListener(std::ostream &o=std::cout, const char *preLine = ":", const char *postLine = "") : CxxTest::ErrorPrinter( o, preLine , postLine ) {} int run() { return CxxTest::ErrorPrinter::run(); } void enterTest( const CxxTest::TestDescription & desc) { test_passed=true; #ifdef __COVERAGESCANNER__ // Adjust the name of the test to display the tests // ina tree view in CoverageBrowser std::string testname="CxxTest/"; testname += desc.suiteName(); testname += "/"; testname += desc.testName(); // Reset the code coverage data to get only the code coverage // of the actual unit test. __coveragescanner_clear(); __coveragescanner_testname(testname.c_str()); #endif return CxxTest::ErrorPrinter::enterTest( desc ); } void leaveTest( const CxxTest::TestDescription & desc) { #ifdef __COVERAGESCANNER__ Squish Coco 5.1.0 - 255 - froglogic GmbH CxxTest // Record the execution state in the coverage report if (test_passed) __coveragescanner_teststate("PASSED"); else __coveragescanner_teststate("FAILED"); // Save the code coverage report of the unit test __coveragescanner_save(); #endif return CxxTest::ErrorPrinter::leaveTest( desc ); } void failedTest(const char *file, int line, const char *expression) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedTest( file, line, expression ); } void failedAssert(const char *file, int line, const char *expression) { // Only record that the test fails test_passed = false; return CxxTest::ErrorPrinter::failedAssert( file, line, expression ); } void failedAssertEquals(const char *file, int line, const char *xStr, const char *yStr, const char *x, const char *y) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertEquals( file, line, xStr, yStr, x, y ); } void failedAssertSameData(const char *file, int line, const char *xStr, const char *yStr, const char *sizeStr, const void *x, const void *y, unsigned size) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertSameData( file, line, xStr, yStr, sizeStr, x, y, size ); } void failedAssertDelta(const char *file, int line, const char *xStr, const char *yStr, const char *dStr, const char *x, const char *y, const char *d) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertDelta( file, line, xStr, yStr, dStr, x, y, d ); } Squish Coco 5.1.0 - 256 - froglogic GmbH CxxTest void failedAssertDiffers(const char *file, int line, const char *xStr, const char *yStr, const char *value) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertDiffers(file, line, xStr, yStr, value ); } void failedAssertLessThan(const char *file, int line, const char *xStr, const char *yStr, const char *x, const char *y) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertLessThan(file, line, xStr, yStr, x, y ); } void failedAssertLessThanEquals(const char *file, int line, const char *xStr, const char *yStr, const char *x, const char *y) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertLessThanEquals( file, line, xStr, yStr, x, y ); } void failedAssertRelation(const char *file, int line, const char *relation, const char *xStr, const char *yStr, const char *x, const char *y) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertRelation( file, line, relation, xStr, yStr, x, y); } void failedAssertPredicate(const char *file, int line, const char *predicate, const char *xStr, const char *x ) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertPredicate( file, line, predicate, xStr, x); } void failedAssertThrows(const char *file, int line, const char *expression, const char *type, bool otherThrown) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertThrows( file, line, expression, type, otherThrown ); Squish Coco 5.1.0 - 257 - froglogic GmbH boost::test } void failedAssertThrowsNot(const char *file, int line, const char *expression) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertThrowsNot( file, line, expression ); } private: bool test_passed; }; int main() { #ifdef __COVERAGESCANNER__ __coveragescanner_install(argv[0]); #endif // Use"CoverageScannerListener().run()" instead of"CxxTest::ErrorPrinter().run()" return CoverageScannerListener().run(); } 42.5 boost::test boost::test4 is a unit test framework for C++ which is part of the boost libraries. Proceed as follows: 1. Implement a TestObserver, which derives from ’boost::unit_test_framework::test_observer’ and store as "BoostTestObserver.hpp". 2. Also implement (within the same file) a ’boost::test::fixture’ and define it to be executed in the beginning and in the end of each executed boost::test by: BOOST_GLOBAL_FIXTURE(FixtureName). The file then should look something similar to this: #ifndef COVERAGE_TEST_OBSERVER_INCLUDED #define COVERAGE_TEST_OBSERVER_INCLUDED #include #ifdef __COVERAGESCANNER__ class BoostTestObserver : public boost::unit_test_framework::test_observer { 4Project page of Boost::Test: http://www.boost.org/doc/libs/1_64_0/libs/test Squish Coco 5.1.0 - 258 - froglogic GmbH boost::test public: BoostTestObserver(const char* moduleName) : test_observer() , m_testSuiteName(moduleName) , m_isAnyTestcaseOpen(false) {} ~BoostTestObserver() { if (m_isAnyTestcaseOpen) close_current_testcase(); } /////////////////////////////////////////////////////////////////////////////// // test suite related events: virtual int priority() { return 1; } virtual void test_finish() { if (m_isAnyTestcaseOpen) { close_current_testcase(); } } virtual void test_aborted() { if (m_isAnyTestcaseOpen) { close_current_testcase(); } } const char* get_testsuite_name() { return m_testSuiteName.data(); } // called in the very beginning of the test suite: virtual void test_start( boost::unit_test_framework::counter_t NumberOfTestCases) { m_currentTestNumber = -1; m_numberOfTestCases = NumberOfTestCases; __coveragescanner_install(m_testSuiteName.c_str()); } /////////////////////////////////////////////////////////////////////////////// // test case related events: Squish Coco 5.1.0 - 259 - froglogic GmbH boost::test // called before each testcase’s start virtual void test_unit_start( boost::unit_test_framework::test_unit const &unit) { // skip any calls which may being triggered by the’TestWrap’ fixture if(unit.full_name().find(’/’) == -1) { return; } if (++m_currentTestNumber) close_current_testcase(); m_currentCaseHasFailed = false; __coveragescanner_clear(); m_currentTestName.assign( unit.full_name().c_str()); __coveragescanner_testname(m_currentTestName.c_str()); m_isAnyTestcaseOpen = true; } // should be called after each testcase has finished... // but which definitveley, for unknown reason is NOT virtual void test_unit_finish( boost::unit_test_framework::test_unit const &unit, unsigned number) { close_current_testcase(); } // workaround for missing’test _unit_finish’ event being not triggered: void close_current_testcase() { if (m_isAnyTestcaseOpen) { __coveragescanner_teststate(m_currentCaseHasFailed ? "FAILED" : "PASSED"); __coveragescanner_save(); m_currentTestName.append("_invalid"); m_isAnyTestcaseOpen = false; } } // called once on each FAILED or PASSED result virtual void assertion_result(boost::unit_test::assertion_result result) { if (result == boost::unit_test::AR_FAILED) m_currentCaseHasFailed = true; } protected: // Depricated boost event: virtual void assertion_result(bool result) { if (!result) m_currentCaseHasFailed = true; } Squish Coco 5.1.0 - 260 - froglogic GmbH boost::test /////////////////////////////////////////////////////////////////////////////// // member variables: std::string m_testSuiteName; std::string m_currentTestName; int m_currentTestNumber; int m_numberOfTestCases; bool m_verboseLogging; bool m_currentCaseHasFailed; bool m_isAnyTestcaseOpen; }; ////////////////// // boost fixture: // will be called in the very beginning and in the end of executing test /////////////////////////////////////////////////////////////////////////////// struct TestWrap { BoostTestObserver observer; TestWrap() : BoostTestObserver( boost::unit_test::framework::master_test_suite().argv[0]) { boost::unit_test::framework::register_observer(observer); } ~TestWrap() { observer.close_current_testcase(); boost::unit_test::framework::deregister_observer(observer); } }; /////////////////////////////////////////////////////////////////////////////// BOOST_GLOBAL_FIXTURE(TestWrap); #endif #endif 3. If it is included by any of your allready existing ’boost::test’ unit test .cpp files (must be done right after the in ’boost::test’ mandatory BOOST_TEST_MODULE definition), It is being instantciated and initialized by the boost::test automatically. 4. The ’boost::test’ then will trigger a __coveragescanner_install() call just in the beginning, and __coveragescanner_clear() and __coveragescanner_testname() calls for each test case section which is being executed. Also it triggers __coveragescanner_teststate() and __coveragescanner_save() on each test case’s end. 5. Additionally here is a small example which shows how to include this into your allready existing ’boost::test’ modules: Squish Coco 5.1.0 - 261 - froglogic GmbH xUnit // define the name for yourBoost::Test module. //( *this must be done BEFORE including BoostTestObserver.hpp) #define BOOST_TEST_MODULE MyBoostTest #include // any of your already existing Boost::Test modules can be made // compatible to Coco by including"BoostTestObserver.hpp" // just after the module’s BOOST_TEST_MODULE definition. #include "BoostTestObserver.hpp" // declarea boost test suite: //(when"BoostTestObserver.hpp" is included, this will // automatically instanciatea BoostTestObserver then.) BOOST_AUTO_TEST_SUITE(BOOST_TEST_MODULE) // declare first test case: BOOST_AUTO_TEST_CASE(testA) { // code for testcaseA } //... declare any other testcases // last testcase: BOOST_AUTO_TEST_CASE(testX) {//... } // in the end of the test, close the suite’s scope //(which at least also will unload our BoostTestObserver) BOOST_AUTO_TEST_SUITE_END() 6. Compile the boost test with CoverageScanner activated. 42.6 xUnit When Squish Coco is used with xUnit,5 it is possible to save the coverage for each single test. To do this, we define collections and hooks that are executed before and after each test. The tests themself need not to be instrumented, but the code that is tested must be built with CoverageScanner.A small adaptation of the xUnit test suite must be made. These changes work even if the library that is tested is not instrumented, but in this case no code coverage information will be generated. Proceed as follows: 1. Add to your xUnit test library the following code: 5https://xunit.github.io Squish Coco 5.1.0 - 262 - froglogic GmbH xUnit It provides a set of functions to handle the coverage information and to save the execution report (the .csexe file). using System; using System.Collections.Generic; using System.Text; using Xunit; using System.Reflection; using Xunit.Sdk; using System.Diagnostics; using Xunit.Abstractions; class Coverage { public enum State { PASSED, FAILED, CHECK_MANUALLY, UNKNOWN }; public static void CoverageCleanup() { AppDomain MyDomain = AppDomain.CurrentDomain; Assembly[] AssembliesLoaded = MyDomain.GetAssemblies(); foreach (Assembly a in AssembliesLoaded) { Type coco = a.GetType("CoverageScanner"); if (coco != null) { // clear all coverage information to only get the code coverage of the current executed unit test coco.InvokeMember("__coveragescanner_clear", BindingFlags. InvokeMethod | BindingFlags.Static | BindingFlags.Public, null , null, null); coco.InvokeMember("__coveragescanner_clear_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags .Public, null, null, null); } } } public static void CoverageTextLog(String text, bool bold, bool italic) { String style_begin = ""; String style_end = ""; if (italic) { style_begin += ""; style_end += ""; } if (bold) { style_begin += ""; style_end += ""; } Squish Coco 5.1.0 - 263 - froglogic GmbH xUnit String comment = " public static void CoverageHtmlLog(String comment) { AppDomain MyDomain = AppDomain.CurrentDomain; Assembly[] AssembliesLoaded = MyDomain.GetAssemblies(); foreach (Assembly a in AssembliesLoaded) { Type coco = a.GetType("CoverageScanner"); if (coco != null) { coco.InvokeMember("__coveragescanner_add_html_comment", BindingFlags.InvokeMethod | BindingFlags.Static | BindingFlags .Public, null, null, new object[] { comment }); } } } public static void CoverageRecord(String csexe_filename, String testName, State result) { if (csexe_filename == "") return; AppDomain MyDomain = AppDomain.CurrentDomain; Assembly[] AssembliesLoaded = MyDomain.GetAssemblies(); foreach (Assembly a in AssembliesLoaded) { Type coco = a.GetType("CoverageScanner"); if (coco != null) { String name = testName; String state = ""; switch (result) { case State.FAILED: Squish Coco 5.1.0 - 264 - froglogic GmbH xUnit state = ("FAILED"); break; case State.CHECK_MANUALLY: state = ("CHECK_MANUALLY"); break; case State.PASSED: state = ("PASSED"); break; } name = name.Replace(’.’, ’/’); // set the execution state: PASSES, FAILED or CHECK_MANUALLY coco.InvokeMember("__coveragescanner_teststate", BindingFlags. InvokeMethod | BindingFlags.Static | BindingFlags.Public, null , null, new object[] { state }); if (name.Length > 0) // Test name: Namespace/Class/Testfunction coco.InvokeMember("__coveragescanner_testname", BindingFlags. InvokeMethod | BindingFlags.Static | BindingFlags.Public, null, null, new object[] { name }); // File name is 2. Add to your xUnit test library the following code: It contains a definition of a xUnit collection which saves the coverage after each unit test has been executed. If a collection is already present in the code, it is necessary to adapt the existing one with this file as guideline. using System; using System.Collections.Generic; using System.Text; using Xunit; using System.Reflection; Squish Coco 5.1.0 - 265 - froglogic GmbH xUnit using Xunit.Sdk; using System.Diagnostics; using Xunit.Abstractions; [AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, AllowMultiple = false, Inherited = true)] public class SquishCocoHooks : BeforeAfterTestAttribute { public override void Before(MethodInfo methodUnderTest) { Coverage.CoverageCleanup(); } public override void After(MethodInfo methodUnderTest) { string base_name = "xUnit"; string method_name = methodUnderTest.Name; string class_name = methodUnderTest.ReflectedType.FullName; Coverage.CoverageRecord("coverage.csexe", base_name + "/" + class_name + " /" + method_name, Coverage.State.UNKNOWN); } } public class CustomFixture : IDisposable { public CustomFixture() { // Some initialization } public void Dispose() { // Some cleanup } } [CollectionDefinition("SquishCoco")] [SquishCocoHooks] public class SquishCocoDefinition : ICollectionFixture 3. Activate the collection for each test class. To do it, just add “[Collection("SquishCoco")]” to the definition of each test class. For example: [Collection("SquishCoco")] public class Class2Tests { [Fact()] public void dummyTest() { Assert.True(false, "Dummy test"); Squish Coco 5.1.0 - 266 - froglogic GmbH NUnit } } After running the tests, a file coverage.csexe will be generated; it can then be imported into the corresponding coverage database (.csmes file). With CoverageBrowser, it is then possible to analyze the coverage of each test separately. 42.7 NUnit Squish Coco provides an addin for NUnit6 version 2.4.4 and above as sample. To install it proceed as followings: 1. Build NUnitSquishCoco.dll using the Microsoft® Visual Studio® project NUnitSquishCoco.vsproj pro- vided in the sample directory. 2. Copy NUnitSquishCoco.dll to the addins folder located in the bin folder where NUnit.exe can be found. 3. Start NUnit.exe and verify that the addin “NUnit Squish Coco” is loaded. Once installed, as soon as NUnit’s test driver is executing a C# or C++ managed unit test test.dll, it generates automatically a code coverage execution report test.dll.csexe automatically if test.dll is instrumented with Squish Coco. The code coverage information is organized into a tree containing the coverage and the execution status for each single unit test. The execution report can be then imported into the application’s instrumentation database with CoverageBrowser or cmcsexeimport. 42.8 Catch2 Catch27 is a unit test framework for C++ which can be easily adapted to get the code coverage from each unit test section. Complete code example: #define CATCH_CONFIG_RUNNER #include "catch.hpp" #include class CoverageScannerListener : public Catch::TestEventListenerBase { public: using TestEventListenerBase::TestEventListenerBase; virtual void sectionStarting( Catch::SectionInfo const& sectionInfo ) override { m_testNameHierarchy.push_back( sectionInfo.name ); 6http://nunit.org/ 7Project page of Catch2: https://github.com/catchorg/Catch2 Squish Coco 5.1.0 - 267 - froglogic GmbH Catch2 #ifdef __COVERAGESCANNER__ // Adjusting the name of the test for the CoverageBrowser std::string testname = implodeTestNameHierarchy(); // Reset the code coverage data to get only the code coverage // of the actual unit test. __coveragescanner_clear(); __coveragescanner_testname( testname.c_str() ); #endif } virtual void sectionEnded( Catch::SectionStats const& sectionStats ) override { m_testNameHierarchy.pop_back(); #ifdef __COVERAGESCANNER__ // Recording the execution state in the coverage report if ( sectionStats.assertions.allPassed() ) __coveragescanner_teststate( "PASSED" ); else __coveragescanner_teststate( "FAILED" ); // Saving the code coverage report of the unit test __coveragescanner_save(); __coveragescanner_testname( "" ); #endif } private: std::deque /* Helper method which generatesa hierarchical name string usable by the coveragebrowser. */ std::string implodeTestNameHierarchy() { std::string fullTestName ( "Catch2" ); std::deque // Register the coveragescanner listener with Catch2 CATCH_REGISTER_LISTENER( CoverageScannerListener ) int main( int argc, char* argv[] ) { #ifdef __COVERAGESCANNER__ __coveragescanner_install( argv[0] ); #endif return Catch::Session().run( argc, argv ); } Squish Coco 5.1.0 - 268 - froglogic GmbH Squish Follow these steps: 1. Call __coveragescanner_install() and Catch::Session.run() in the main() function. Catch2 pro- vides a predefined main(), which needs to be disregarded by defining only CATCH_CONFIG_RUNNER before including the Catch2 header: #define CATCH_CONFIG_RUNNER #include "catch.hpp" 2. Create a Catch2 listener class - inheriting Catch::TestEventListenerBase - which records the code cover- age of each section after it is completed. (Note: Testcases also count as sections in Catch2. There is no need to listen to testcase events specifically.) In the created listener class (CoverageScannerListener) we use the following member functions: sectionStarting(): This function is called before each testcase and section begins. In it, we compute a test name with the information provided by Catch2 and pass it to the Squish Coco library with __coveragescanner_testname(). We also call the function __coveragescanner_clear(): It empties the internal database and so makes sure that the coverage of the code that was executed before this test is ignored. sectionEnded(): This function is called after a testcase or section has ended. It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with __coveragescanner_save(). 3. We add this listener by adding the following Catch2 macro: CATCH_REGISTER_LISTENER( CoverageScannerListener ) i Remember to exclude any test framework sources from instrumentation (see Chapter 3.3, page 18)! 42.9 Squish It is easily possible to run the GUI testing tool Squish together with Squish Coco in order to get the C/C++coverage of a Squish test suite. A more in-depth analysis is then possible, correlating each test case (and its results) with the respective coverage information. This is especially true since Squish Coco features the comparison of individual executions, including the calculation of an optimal order. 42.9.1 General Approach The approach depicted below is based on the possibility to apply information about the name, the result and a free-form comment to each execution (stored in .csexe files). See Chapter 34.1, page 189 for the details. As an example we’ll use Squish for Qt’s addressbook on a Unix-based system and a JavaScript test script. 1. First we’ll initialize the execution data with the name of the Squish test case that is being run. Squish Coco 5.1.0 - 269 - froglogic GmbH Squish function main() { var currentAUT = currentApplicationContext(); var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe" var testCase = squishinfo.testCase; var testExecutionName = testCase.substr(testCase.lastIndexOf(’/’) + 1); var file = File.open(execution, "a"); file.write("*" + testExecutionName + "\n"); file.close(); var ctx = startApplication("addressbook"); ... 2. Insert the main test script at this point 3. After the main test script we’ll log the result of the test for the coverage tool: ... // wait until AUT shutdown while (ctx.isRunning) { snooze(1); // increase time if not enough to dump coverage data } // test result summary and status var positive = test.resultCount("passes"); var negative = test.resultCount("fails") + test.resultCount("errors") + test. resultCount("fatals"); var msg = "TEST RESULTS - Passed: " + positive + " | " + "Failed/Errored/ Fatal: " + negative; var status = negative == 0 ? "PASSED" : "FAILED"; var file = File.open(execution, "a"); file.write(" When you execute the scripts containing these steps, the Squish CocoExecution Report loads with the test case name, status and execution summary in the execution details and comments. 42.9.2 Simplified for Reuse 1. Create a file called squishCocoLogging.js in Test Suite Resources with the following functions: function getExecutionPath() { var currentAUT = currentApplicationContext(); var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe" return execution; } function logTestNameToCocoReport(currentTestCase, execution) { var testExecutionName = currentTestCase.substr(currentTestCase.lastIndexOf (’\\’) + 1); Squish Coco 5.1.0 - 270 - froglogic GmbH Squish var file = File.open(execution, "a"); file.write("*" + testExecutionName + "\n"); file.close(); } function logTestResultsToCocoReport(testInfo, execution){ var currentAUT = currentApplicationContext(); // wait until AUT shuts down while (currentAUT.isRunning) snooze(5); // collect test result summary and status var positive = testInfo.resultCount("passes"); var negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo.resultCount("fatals"); var msg = "TEST RESULTS - Passed: " + positive + " | " + "Failed/Errored/ Fatal: " + negative; var status = negative == 0 ? "PASSED" : "FAILED"; // output results and status to Coco execution report file var file = File.open(execution, "a"); file.write(" A Python version of this code is: import re def getExecutionPath(): currentAUT = currentApplicationContext() execution = "%(currAUTPath)s\\%(currAUTName)s.exe.csexe" % {"currAUTPath" : currentAUT.cwd, "currAUTName" : currentAUT.name} return execution def logTestNameToCocoReport(currentTestCase, execution): testExecutionName = re.search(r’[^\\]\w*$’, currentTestCase) testExecutionName = testExecutionName.group(0) file = open(execution, "a") file.write("*" + testExecutionName + "\n") file.close() def logTestResultsToCocoReport(testInfo, execution): currentAUT = currentApplicationContext() # wait until AUT shuts down while (currentAUT.isRunning): snooze(5) # collect test result summary and status Squish Coco 5.1.0 - 271 - froglogic GmbH Squish positive = testInfo.resultCount("passes") negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo.resultCount("fatals") msg = "TEST RESULTS - Passed: %(positive)s | Failed/Errored/Fatal: %( negative)s" % {’positive’: positive, ’negative’: negative} if negative == 0: status = "PASSED" else: status = "FAILED" # output results and status to Coco execution report file file = open(execution, "a") file.write("
") .Replace("\r", "") + style_end + "" ; CoverageHtmlLog(comment); }
2. Add the following function calls after startApplication() in the main test script:
execution = getExecutionPath();
logTestNameToCocoReport(squishinfo.testCase);
In Python:
execution = getExecutionPath()
logTestNameToCocoReport(squishinfo.testCase)
3. At the end of your script, after closing the AUT (for example with steps clicking File > Exit), call the following function:
logTestResultsToCocoReport(test);
In Python:
logTestResultsToCocoReport(test)
4. In the event your AUT closes unexpectedly, or a script error occurs, incorporating a try, catch, finally ensures your results still output to the Coco report file.
Your main test script should be similar to the following:
source(findFile("scripts","squishCocoLogging.JS"))
function main() { startApplication("addressbook"); execution = getExecutionPath(); logTestNameToCocoReport(squishinfo.testCase, execution);
try { // body of script }
Squish Coco 5.1.0 - 272 - froglogic GmbH Squish
catch(e) { test.fail(’An unexpected error occurred’, e.message) } finally { logTestResultsToCocoReport(test, execution) } }
Python version:
source(findFile("scripts","squishCocoLogging.py"))
def main(): startApplication("addressbook") execution = getExecutionPath() logTestNameToCocoReport(squishinfo.testCase, execution)
try: try: # body of script except Exception, e: test.fail("test failed: ", e) finally: logTestResultsToCocoReport(test,execution)
Squish Coco 5.1.0 - 273 - froglogic GmbH Installing Coco in Docker
Chapter Installing Coco in Docker 43
In this chapter we give an example on how to run Squish Coco, terminal only interaction, in Docker.
43.1 Prerequisites
1. A Linux machine, in this example Debian, 2. Docker (latest version, but any stable version should work fine).
43.2 Installation of Docker
Before installing Squish Coco we need docker to be installed. To check if docker is running on your machine, simply type
$ docker --version
Skip the following step if docker is already installed on your machine.
43.2.1 Installation from the Debian repository
A standard Debian distribution contains already a Docker package. It is usually called docker or docker.io. To install it, run
$ sudo apt-get install docker.io
Squish Coco 5.1.0 - 274 - froglogic GmbH Installation of Docker
43.2.2 Installation from the Docker repository
If you need the newest version of Docker, you can use the Docker repository instead. The following procedure to install Docker is mostly the same for all Linux based machines. Please look into docker docs for more info, if necessary.
1. Update the apt package index:
$ sudo apt-get update
2. Install packages to allow apt to use a repository over HTTPS:
$ sudo apt-get install apt-transport-https ca-certificates curl gnupg2 software- properties-common
3. Add Docker’s official GPG key:
$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
4. Use the following command to set up the stable repository:
$ sudo add-apt-respostiory \ "deb [arch=amd64] https://download.docker.com/linux/debian \ $(lsb_release -cs) \ stable"
5. Update the apt package index:
$ sudo apt-get update
6. Install the latest version of Docker CE and containerd:
$ sudo apt-get install docker-ce docker-ce-cli containerd.io
43.2.3 Verification of the installation
Now verify that your user account has the right to run docker containers. On most UNIX® systems, a user must be in the docker group. To verify this, one can use the “id” command, which prints the current groups a user is in. If one is not member of docker, one can run
$ sudo adduser hown accouti docker
But afterwards it is usually necessary to log in again in order to make the group change visible.
To verify that Docker CE is installed correctly, run the hello-world image:
$ docker run hello-world
Note: If you want to remove the hello-world image, first run
Squish Coco 5.1.0 - 275 - froglogic GmbH Installation of Coco in a Docker container
$ docker ps -a
The output should be a list of containers, with name, status, and more. It should contain one header line and one line for the hello-world image. In the first column, there is the container ID of the hello-world image. Run
$ docker rm hcontainer_idi
43.3 Installation of Coco in a Docker container
1. Create an empty directory. We will refer to it as the work directory.
2. Download a Coco package to the work directory. It must be a file that ends in “.run”, like “SquishCocoSetup_5.0.0_Linux_x86_64.run”.
3. In the work directory, create a file named Dockerfile, with the following content:
# Select base image FROM debian
# Install packages for your project RUN apt-get update && apt-get upgrade -y && apt-get install -y libgtk2.0-dev
# Set"secret" as root password RUN echo "root:secret" | chpasswd
# Definea user RUN useradd -ms /bin/bash myapp USER myapp WORKDIR /home/myapp
# Copy the Coco installer to the work directory COPY SquishCocoSetup*.run .
For an actual project, the content of the file must be adapted. In this example, it does the following: (a) Loading an image from the Docker Hub website. In this case it is the most recent Debian image, but any Linux image should work. (b) Install some packages. In a real use case, these would be the prerequisites of the software project for which code coverage should be measured. The installation commands that are given here obviously only work on a Debian or Debian-like system, like Ubuntu. (c) Setting a root password. This is later needed by the Squish Coco installer, which runs from a user account. (d) Create a user account. (e) Copy the Squish Coco installer into the container. It will there be located in the home directory of the user myapp. The wildcard pattern “SquishCocoSetup*.run” matches the name of any Squish Coco installer package, independent of the version number.
Squish Coco 5.1.0 - 276 - froglogic GmbH Installation of Coco in a Docker container
4. In the work directory, run the command
$ docker build -t squishcoco .
Docker will then create an image with the name squishcoco. 5. To run the image, execute the command
$ docker run -it --name runner squishcoco
Then a container named runner will be created and started. 6. You will now see the command prompt of the container, no longer that of your operating system. You are now in the home directory of the user myapp. This directory also contains a copy of the Coco installer package. You can run it with a command like
myapp@a6e8af:~$ bash SquishCocoSetup_5.0.0_Linux_x86_64.run
The installer will then let you choose between different installation modes. Choose the first, “Installation on a the local machine”. The installer will then ask for the root password, which is of course “secret”. At the end of the installation, the installer asks, “Do you want to start Squish Coco License Manager in order to register this product?”. Answer with no, because the installer uses a graphical interface, which is usually not possible in a Docker container. 7. Instead, the non-GUI version of the license manager mut be used. Run one of the following commands: • If you got an activation code, make sure that the Docker container has internet access enabled, then run
myapp@a6e8af:~$/opt/SquishCoco/bin/cocolic --fetch-license-key=hactivation codei
The license key will then be fetched from the license server. Afterwards, the internet access can be disabled again. • If you got a license key, run
myapp@a6e8af:~$/opt/SquishCoco/bin/cocolic --license-key=hlicense keyi
• If you use a license server, run
myapp@a6e8af:~$/opt/SquishCoco/bin/cocolic --license-server=hserveri:hporti
In this case, the Docker container needs to have network access enabled during compilation and whenever a Squish Coco tool runs. When the instrumented program runs, no network access is needed. If an error message is displayed, contact froglogic support. Alternatively, you can check if the license is activated or not by running the command below
myapp@a6e8af:~$/opt/SquishCoco/bin/cocolic --check
8. Now Squish Coco is installed in Docker. You can exit the container with the “exit” command. 9. To enter the container again, run
$ docker start -ai runner
Squish Coco 5.1.0 - 277 - froglogic GmbH Atlassian Bamboo integration
Chapter Atlassian Bamboo integration 44
44.1 Introduction
Atlassian Bamboo is a continuous integration server used to build, test and release software. The Squish Coco plug-in provides seamless integration with Bamboo. Users can add one or more Squish Coco tasks to a Bamboo job. Each Squish Coco task can perform the following actions:
1. Import an execution report into an instrumentation data base. 2. Merge several instrumentations databases. 3. Generate a code coverage report.
The user can choose only one action to be performed, two actions or all three actions. If more than one action is chosen, they are executed in the order mentioned above.
Figure 44.1: Squish Coco task configuration overview
After the actions have ended, files that are created by Squish Coco (like the coverage report), can be stored to allow efficient troubleshooting.
Squish Coco 5.1.0 - 278 - froglogic GmbH Agent configuration
44.2 Agent configuration
An agent is a service that runs Bamboo builds, tests and deployments. The “Agent Capabilities” setting lets users specify which agents are capable of running Squish Coco jobs. In order to execute it, we must first install Squish Coco at the agent host. Then we need to set up an agent-specific capability called ‘coco’ with Squish Coco installation directory as a value. (Go to the configuration entry “Bamboo administration|BUILD RESOURCES| Agents|AgentName|Agent-specific capabilities”.) Later on, during the job configuration, we need to set the existence of a ‘coco’ capability as a job requirement. This way, Squish Coco jobs will be executed only on agents on which Squish Coco in installed.
44.3 Importing an execution report
This action calls cmcsexeimport to import the execution report (.csexe file) into an instrumentation data base (.csmes file). The user needs to provide an absolute or relative path (from the Bamboo build directory) to the .csexe and the .csmes file, and also a title for the execution. When this action is activated, the following command is executed on the agent host:
cmcsexeimport --debug --title=hTitlei -m hCSMesFilei -e hCSExeFilei
44.4 Merging the instrumentation databases
This action calls cmmerge to merge several instrumentation databases (.csmes files). We need to provide the full or relative path to the resulting .csmes output file. If we provide a relative path (or just a filename), then the .csmes output file will be generated in the Bamboo working directory or a subdirectory relative to it. The .csmes input files need to be specified. Multiple files can be used, separated by a comma. When this action is activated, the following command will be executed on the agent host:
cmmerge --verbose -o hCSMesOutputFilei hCSMesInputFile1i ... hCSMesInputFileNi
Additionally, the user can select the option ‘merge only instrumentations and executions present in the reference file’. This option is useful for importing .csmes files with unit tests. When the option is selected, the user must provide the name of a .csmes reference file. This will cause that above cmmerge command will be executed with additional option ‘-i’.
cmmerge --verbose -o hCSMesOutputFilei -i hCSMesRefFilei hCSMesInputFile1i ... hCSMesInputFileNi
Squish Coco 5.1.0 - 279 - froglogic GmbH Generating the coverage report
44.5 Generating the coverage report
The last action calls cmreport to generate an HTML Report. We need to provide .csmes file name (either absolute or relative path). The .csmes file must contain the previously imported execution report (i.e. by using the first action offered by the plugin).
cmreport --title=hBambooJobNamei -m hCSMesFilei --debug --html=coco/report.html
The generated report will be stored in coco subdirectory, therefore we can define Artefact with ‘coco/**’ as the copy pattern. This allows us to view the report directly in Bamboo after job execution. The report title is built using a Bamboo job name.
44.6 Examples
44.6.1 A coverage report from a single execution
We need to activate first the action ‘Import an execution report into an instrumentation data base’ to import the execution report (.csexe file) into an instrumentation data base (.csmes). Finally, we need to activate the third action, ‘Generate code coverage report’. The .csexe file for this action is the same as for first action.
Squish Coco 5.1.0 - 280 - froglogic GmbH Examples
Figure 44.2: Squish Coco Task configuration for example addressbook application
44.6.2 Coverage report for unit tests
In this scenario we would like to generate coverage report for unit tests execution. To achieve this goal in Bamboo, we need a job that consists of two Squish Coco tasks.
After the build of the program which is tested, we have two .csmes files: one for our application and a second for the unit tests. We need to merge those instrumentation databases where the .csmes file for our application will be a reference during the merge. The merge step creates a new .csmes file as its output file. This is done in the following task:
Squish Coco 5.1.0 - 281 - froglogic GmbH Examples
Figure 44.3: Squish Coco Task configuration to merge unit test executions
In the second task, the execution report for the unit tests is imported into .csmes file (which was the .csmes output file of the previous task). The last action in the task, ‘Generate code coverage report’, generates an HTML report.
Squish Coco 5.1.0 - 282 - froglogic GmbH Examples
Figure 44.4: Squish Coco Task configuration for importing execution and generating report
Squish Coco 5.1.0 - 283 - froglogic GmbH Jenkins CI integration
Chapter Jenkins CI integration 45
Jenkins CI is a popular continuous integration tool. Squish Coco can be run under Jenkins, and the changes of the coverage over time displayed by a plugin. This chapter describes the necessary setup.
45.1 Prerequisites
45.1.1 Getting a license
The Jenkins process runs under an own account, usually called “jenkins”. When therefore Squish Coco runs under Jenkins CI with a node-locked license, it needs a specific license for its account. How to get this license is described in the following section; the section after that one describes the use of a license server.
Installing a node-locked license In order to generate the license, the Squish Coco license tools need to be run from the “jenkins” account. There are two ways to do it, depending on whether the Jenkins server has an Internet connection.
• With an Internet connection: You need to get an activation code from froglogic. With this activation code, create a new Jenkins project that only runs (under Microsoft® Windows) the command
"hWindows Cocoi\cocolic" --fetch-license-key=hactivation codei
Under Linux™ and macOS, this line becomes1
hcocobini/cocolic --fetch-license-key=hactivation codei
If the Jenkins server is connected to the Internet through a proxy, additional options are needed to specify the proxy. See Chapter 27, page 172 for the details.
1hcocobini stands for the directory with the Squish Coco binaries. Under Linux™, this is usually /opt/SquishCoco/bin, while under macOS it is /Applications/SquishCoco.
Squish Coco 5.1.0 - 284 - froglogic GmbH Adaption of the project to Jenkins
Now build this project. In the console output window page of Jenkins you can then see whether cocolic was successful. • Without an Internet connection: Here we create a machine ID offline, and from that ID froglogic creates a license key. This key is then installed on the server in a second step. To create a machine ID, we create a Jenkins project that executes (under Windows) the following command:
"hWindows Cocoi\cocolic" --machine-identifier
Under Linux™ and macOS, this becomes
hcocobini/cocolic --machine-identifier
When this project is built, the console output of the build contains the machine ID. Send the machine ID to froglogic. We will send you a license key. Now create a Jenkins project that only contains the following command, and build it:
"hWindows Cocoi\cocolic" --license-key=hyour keyi
Under Linux™ and macOS, this becomes
hcocobini/cocolic --license-key=hyour keyi
Using a license server For the use of Squish Coco with a license server, one has to create a new Jenkins CI project, which only runs (under Microsoft® Windows) the following command:
"hWindows Cocoi\cocolic" --license-server=hhosti:hporti where hhosti and hporti refer to the license server (see Chapter 27, page 172). The colon and the port can be omitted if the default port is used. Under Linux™ and macOS, this becomes
hcocobini/cocolic --license-server=hhosti:hporti
Then the license is installed.
45.1.2 Getting an EMMA plugin
Coco uses the EMMA-XML format to communicate the coverage results to Jenkins. Jenkins therefore needs a plugin to display EMMA data. We use here the plugin at https://wiki.jenkins-ci.org/display/JENKINS/Emma+Plugin. Install it in Jenkins with the Jenkins plugin manager.
45.2 Adaption of the project to Jenkins
We will assume here that you already compile and run your project under Jenkins and that you can also generate coverage information, but not yet under Jenkins.
Squish Coco 5.1.0 - 285 - froglogic GmbH Adaption of the project to Jenkins
With the license installed, code coverage should now work under Jenkins.
The build process must now be configured in such a way that a single .csmes file is generated that contains all the execution data. (This is done with cmcsexeimport.)
Now extend your build process such that at the end a report im EMMA-XML format is generated. This is done by calling
"hWindows Cocoi\cmreport" -m project.csmes --emma=report.xml where project.csmes must be replaced with the name of your results file.
We then use the plugin to display the data that were written to the file report.xml. To do this, select the configuration section of your Jenkins project and add a post-build action. Since the plugin is installed, the "Add post-build action" menu contains an entry "Record Emma coverage report". Select it, and in the field "Folders or files containing Emma XML reports" enter "report.xml". Then after the next build, a coverage report will be generated.
Figure 45.1: Generating a coverage report from report.html with Jenkins CI.
Squish Coco 5.1.0 - 286 - froglogic GmbH Part XI Coco Internals
Squish Coco 5.1.0 - 287 - froglogic GmbH File system and registry
Chapter File system and registry 46
This chapter is about the usage Squish Coco makes of the file system to store permantent settings.
46.1 Location of the installed files
The location of the installed files varies according to the platform.
46.1.1 Microsoft® Windows
The location of the programs can be chosen at installation time. The installer creates and environment vari- able SQUISHCOCO that contains the path to the installation directory. By default it is either C:\Program Files\ squishcoco or C:\Program Files (x86)\squishcoco, depending on the platform. In this documentation we use the expression hWindows Cocoi to refer to it. This directory contains the following files and directories: • The standard Squish Coco executables, like CoverageBrowser, cmcsexeimport, etc. • The documentation. • The prefix versions of the wrapper programs and their configuration files. The prefix version of a compiler wrapper is a file that begins with “cs”, like cscl.exe as wrapper for cl.exe. The configuration file for this program is then cl.cspro.
• Directories with specific versions of the compiler wrappers. An example are the directories visualstudio and visualstudio_x64 which contain the wrappers for Microsoft® Visual Studio®. These directories often contain compiler wrappers without a cs prefix: There is e.g. a program cl.exe in the directory visualstudio that wraps the program cl.exe of Microsoft® Visual Studio®.
• Directories with tutorial and example programs, like parser, textedit and tutorial.
Squish Coco 5.1.0 - 288 - froglogic GmbH Location of the license
46.1.2 Linux™
The location can be chosen at installation time; its default value is /opt/SquishCoco/. The directory has the following subdirectories: bin/, lib/: The Squish Coco binaries, including the prefix versions of the compiler wrappers and their profile files. wrapper/bin/: The compiler wrappers as files without the cs prefix. doc/: Documentation files. samples/: Example programs and files for the tutorials.
46.1.3 macOS
The programs are installed at /Applications/SquishCoco. This directory contains the Squish Coco binaries, including the prefixed versions of the compiler wrappers and their profile files. It also contains the documentation. It has the following subdirectories: wrapper/: The compiler wrappers as files without the cs prefix. samples/: Example programs and files for the tutorials.
46.2 Location of the license
The professional and the non-commercial edition of Squish Coco use different naming schemes to locate the files and registry keys that contain information about the license. In the following description, the professional edition is primarily described, with the values that change in the noncommercial edition following in parentheses.
46.2.1 Node-locked licenses
When searching for a license, Squish Coco first tries to find a node-locked license.
1. If the environment variable SQUISHCOCO_LICENSEKEY_DIR is set, Squish Coco treats the content as a directory name and tries to find there the license file. The license file is named .squishcoco-3-license. 2. It then searches for a license file in the home directory (described below).
3. If Squish Coco runs under Microsoft® Windows, it searches instead in the registry under HKEY_CURRENT_USER \Software\squishcoco\LicenseKey.
It is possible that a license is present but that it has been deactivated. If Squish Coco finds such a license, it stops searching for another node-locked license and tries to find a license server instead.
Squish Coco 5.1.0 - 289 - froglogic GmbH Location of the temporary files
Setting SQUISHCOCO_LICENSEKEY_DIR globally for all users is not recommended. It would force ! all users to use the same license, but each license key is specific to user and host.
46.2.2 Address of the license server
If no node-locked license is found, Squish Coco will try to find the location of a license server.
1. First it tries to read the specification of a license server from the environment variable SQUISHCOCO_LICENSE_SERVER, if such a variable is present and its value is not the empty string.
2. If Squish Coco runs under Microsoft® Windows, then it searches in the registry under HKEY_CURRENT_USER\ Software\squishcoco\LicenseServer for a specification.
3. It then tries to read the specification from a file .squishcoco-3-licserver, which is located in the home directory (see below).
4. It then tries to read the specification from the system file /etc/squishcoco-3-licserver on Unix or %windir%\system32\drivers\etc\squishcoco-3-licserver on Windows.
The specification of the license server consists of a host name or an IP address, possibly followed by a colon and a port number, like ‘myserver.com:49344’.
The home directory When searching for the home directory, Squish Coco evaluates the following environment variables and takes the first one which is set.
1. The content of the variable HOMEPATH.
2. Under Microsoft® Windows, if HOMEPATH is set, either: • If HOMEDRIVE is set, the value of HOMEDRIVE + HOMEPATH. • Otherwise the content of HOMEPATH.
3. The content of HOME.
4. Under Microsoft® Windows, the content of USERPROFILE.
46.3 Location of the temporary files
During compilation, the CoverageScanner creates some temporary files, which it usually deletes automatically after use.
By default, the temporary files are created in the system temporary files directory. This is /tmp on UNIX® systems, and the directory given by the environment variable %TEMP% on Microsoft® Windows. It is possible to change this location by setting the environment variable SQUISHCOCO_TEMP_DIR to the path of another directory. This directory must already exist when the files are written, it is not created automatically. The automatic deletion of temporary files can be switched off with the command line option --cs-keep-instrumentation-files (see Chapter 9.2.5, page 79).
Squish Coco 5.1.0 - 290 - froglogic GmbH Location of the program settings
46.4 Location of the program settings
If a program has other permanent settings that must be stored in a file, that file is located in the following directory:
On Windows: %APPDATA%\squishcoco
On Linux: $HOME/.config/squishcoco
46.5 Location of the installation log file
The Windows installer produces a log file at %TEMP%\SquishCoco_logfile\vsaddin.log. It can be consulted after installation problems.
Squish Coco 5.1.0 - 291 - froglogic GmbH Supported compilers
Chapter Supported compilers 47
The following compilers are currently supported by CoverageScanner:
47.1 C# compilers
The command line C# compiler of Microsoft® Visual Studio® .NET and Mono C# compiler are supported.
Native Command CoverageScanner Command ’mcs’ ’csmcs’ ’gmcs’ ’csgmcs’ ’dmcs’ ’csdmcs’ ’csc’ ’cscsc’
47.2 Microsoft® Visual C++
The command line compiler and linker of Microsoft® Visual C++ and Microsoft® Visual C++ Toolkit 2003 are supported.
Native Command CoverageScanner Command ’cl’ ’cscl’ ’lib’ ’cslib’ ’link’ ’cslink’
Squish Coco 5.1.0 - 292 - froglogic GmbH Intel C++ Compiler
47.3 Intel® C++ Compiler
The C and C++ command line compiler from Intel® is supported.
Native Command CoverageScanner Command ’icl’ ’csicl’ ’icc’ ’csicc’ ’icpc’ ’csicpc’
47.4 GNU gcc
Only the g++ and gcc command line compilers are directly supported.
Native Command CoverageScanner Command ’gcc’ ’csgcc’ ’g++’ ’csg++’ ’ar’ ’csar’
Squish Coco 5.1.0 - 293 - froglogic GmbH Code insertion
Chapter Code insertion 48
(This chapter is a continuation of the example in Chapter5.) CoverageScanner inserts the instrumentation code as illustrated in the following examples. (The inserted code is displayed in bold blue.)
Sequential Statement: The statements are instrumented before their execution. The instrumentation consists in allocating a Boolean variable which detects if the code was executed or not. Example:
a=foo(); a++; break;
will be changed into:
a=foo(); a++; { inst[0]=1; break; }
inst[0] is set to 1 if the ‘break’ statement is executed.
Conditional Statements and Boolean Expressions (full instrumentation): For Boolean expressions the same principle applies except that in addition to recording the execution itself, the state (true or false) is also recorded. Example:
if ( a
will be changed into:
if ( (a inst[0] is set to 1 if the Boolean expression a Squish Coco 5.1.0 - 294 - froglogic GmbH Code insertion Conditional Statements and Boolean Expressions (partial instrumentation): In some cases, recording the value of a Boolean expression is unnecessary for sequential statement instrumenta- tion. For example, the statement if (b)return 0; else return 1; is completely covered by a statement coverage—after all, recording whether b becomes true or false does not provide any extra information. Simi- larly, for the statement if (b)return 0; it is only necessary to check if b was false. By default, Squish Coco suppresses the generation of redundant instrumentation in order to minimize the instrumented code’s size and to maximize execution speed. Example: if ( a will be changed into: if ( a inst[0] is set to 1 if the Boolean expression a The generated instrumentation code that needs to be inserted to provide statement coverage for the earlier foo() function example means that the function’s code is transformed to the code shown here: 1 char inst[5]; 2 void foo() 3 { 4 bool found=false; 5 for (int i=0; (i<100) && (!found); ++i) 6 { 7 if (i==50 ) { inst[0]=1;break;} 8 if (i==20 ) { inst[1]=1;found=true;} 9 if (i==30 ) { inst[2]=1;found=true;} 10 inst[3]=1; } 11 printf("foo\n"); 12 inst[4]=1; } Figure 48.1: Code coverage instrumentation at statement block level If we insert the instrumentation code necessary to support decision coverage into this example, the resulting code will look like this: Squish Coco 5.1.0 - 295 - froglogic GmbH Code insertion 1 char inst[13]; 2 void foo() 3 { 4 bool found=false; 5 for (int i=0; ((i<100 && !found)?inst[0]=1:inst[1]=1,0); ++i) 6 { 7 if ((i==50?inst[2]=1:inst[3]=1,0)) {inst[4]=1; break;} 8 if ((i==20?inst[5]=1:inst[6]=1,0)) {inst[7]=1; found=true;} 9 if ((i==30?inst[8]=1:inst[9]=1,0)) {inst[10]=1; found=true;} 10 inst[11]=1; } 11 printf("foo\n"); 12 inst[12]=1; } Figure 48.2: Code coverage instrumentation at decision level If we insert the instrumentation code necessary to support condition coverage into this example, the resulting code will look like this (except that we have wrapped one line to fit better on the page): 1 char inst[15]; 2 void foo() 3 { 4 bool found=false; 5 for (int i=0;((i<100)?inst[0]=1:inst[1]=1,0) && 6 ((!found)?inst[2]=1:inst[3]=1,0); ++i) { 7 if ((i==50?inst[4]=1:inst[5]=1,0)) {inst[6]=1; break;} 8 if ((i==20?inst[7]=1:inst[8]=1,0)) {inst[9]=1; found=true;} 9 if ((i==30?inst[10]=1:inst[11]=1,0)) {inst[12]=1; found=true;} 10 inst[13]=1; } 11 printf("foo\n"); 12 inst[14]=1; } Figure 48.3: Full code coverage instrumentation at condition level Here is what the code would look like if we inserted the partial instrumentation code for condition coverage (again, with one line wrapped): Squish Coco 5.1.0 - 296 - froglogic GmbH Code insertion 1 char inst[12]; 2 void foo() 3 { 4 bool found=false; 5 for (int i=0; ((i<100)?inst[0]=1:inst[1]=1,0) && 6 ((!found)?inst[2]=1:inst[3]=1,0); ++i) { 7 if (i==50 ) {inst[4]=1; break;} else inst[5]=1; 8 if (i==20 ) {inst[6]=1; found=true;} else inst[7]=1; 9 if (i==30 ) {inst[8]=1; found=true;} else inst[9]=1; 10 inst[10]=1; } 11 printf("foo\n"); 12 inst[11]=1; } Figure 48.4: Partial code coverage instrumentation at condition level Squish Coco 5.1.0 - 297 - froglogic GmbH Code Coverage Benchmarks Chapter Code Coverage Benchmarks 49 49.1 Test Algorithm The sorting algorithm used for the tests is quicksort. Source code: #define SIZE 10000000 #define NB_TESTS 4 #include void sort (int array[],int size); main( ) { int i,t ; long duration[NB_TESTS]; long duration_val; clock_t starttime,endtime ; int *array; for (t=0;t Squish Coco 5.1.0 - 298 - froglogic GmbH Benchmarks for (t=1;t static void quicksort ( int array[], int low, int high ) { int pos ; if ( low < high ) { int item, i, j, t ; item = array[low] ; i = low ; j = high ; while ( i < j ) { while ( array[j] > item ) j = j - 1 ; while ( array[i] <= item && i < j ) i = i + 1 ; if ( i < j ) { t = array[i] ; array[i] = array[j] ; array[j] = t ; } } pos = j ; t = array[low] ; array[low] = array[pos] ; array[pos] = t ; quicksort ( array, pos + 1, high ) ; quicksort ( array, low, pos - 1 ) ; } } void sort (int array[],int size) { quicksort ( array, 0, size-1 ) ; } 49.2 Benchmarks Squish Coco 5.1.0 - 299 - froglogic GmbH Benchmarks Compiler Normal Execution Execution Execution Execution (time) Branch Coverage Decision Coverage Condition Cover- or Function Cover- or age age Line Coverage (time) (time) (time) GCC without optimization 6840ms 8030ms (+17.3%) 9400ms (+37.4%) 9830ms (+43.7%) GCC with optimization -Os 4160ms 4430ms (+6.4%) 6180ms (+48.5%) 6360ms (+52.8%) GCC with optimization -O1 3530ms 4250ms (+20.3%) 5420ms (+53.5%) 5970ms (+69.1%) GCC with optimization -O2 3950ms 4040ms (+2.2%) 5080ms (+28.6%) 5320ms (+34.6%) GCC with optimization -O3 3860ms 4030ms (+4.4%) 5060ms (+31%) 5230ms (+35.4%) Table 49.1: Benchmark (sorting algorithm) Compiler Native Compila- Statement Block Decision Coverage Condition Cover- tion Coverage or age (bytes) or Line Coverage (bytes) Function Cover- (bytes) age (bytes) GCC without optimization 2128 3452 (+1324) 3996 (+1868) 4052 (+1924) GCC with optimization -Os 1940 3196 (+1256) 3716 (+1776) 3760 (+1820) GCC with optimization -O1 1968 3172 (+1204) 3736 (+1768) 3792 (+1824) GCC with optimization -O2 2100 3452 (+1352) 4096 (+1996) 4160 (+2060) GCC with optimization -O3 3544 3548 (+4) 4364 (+820) 4428 (+884) Table 49.2: Sorting algorithm code size of the object Squish Coco 5.1.0 - 300 - froglogic GmbH CoverageScanner Adaptation to a Tool Suite Chapter CoverageScanner Adaptation to a Tool Suite 50 If Squish Coco should be used with a new kind of compiler or tool chain, a profile must be written to adapt the CoverageScanner to the compiler. This chapter describes the content of profiles; for the use of profiles with Squish Coco, see Chapter 8.1, page 70.1 The profile lets CoverageScanner: • interpret the command line of the compiler (see Chapter 50.5, page 306). • interpret the command line of the linker (see Chapter 50.4, page 303). • customize the output of the coverage analysis2 (see Chapter 50.9, page 311). The profile is a text file with a name of the form ’htool namei.cspro’. It must be located in the Squish Coco installation directory. A profile may contain: Comments: Lines that start with a # character are treated as comments. Global parameters: A profile parameter can be assigned a value with the following syntax: hparameteri=hvaluei Only one definition per line is allowed. Parameters for a specific architecture: It is also possible to have a specific version of a parameter that is used only with certain architectures. For this one puts the name of the architecture in brackets behind the parameter, in the following way: hparameteri[harchitecturei]=hvaluei Such a definition must be placed after the global parameter definition. The architecture is determined automatically, but it can also be specified with --cs-architecture. 1Note that some other commands that are not compilers, like ld or ar, are also wrapped with the help of a profile. This is because they, too, are involved in the compilation process. 2This can be necessary on embedded systems which do not have access to a file system. Squish Coco 5.1.0 - 301 - froglogic GmbH Profile Parameters for Architecture Settings 50.1 Profile Parameters for Architecture Settings ARCHITECTURE=hstringi: List of command line arguments which specify the architecture used during the compilation. This parameter should be placed in the first lines of the profile. ARCHITECTURE_APPEND=hYES/NO/ALLOWi: If set, the argument of the command line option, which defines the architecture, is directly appended (no space characters are placed between the option and the arguments). 50.2 Profile Parameters for Instrumentation Settings COVERAGESCANNER_DUMP_ON_EVENT=hregular expressioni: Regular expression which activate the Windows event handler listening the global event Global\COVERAGE. COVERAGESCANNER_RUNTIME_LOG=hregular expressioni: Regular expression which activate the verbose mode of the CoverageScanner API. COVERAGESCANNER_COVERAGE_ON=hregular expressioni: Regular expression which activate the code coverage analysis (equivalent to --cs-on) if this expression match on a command line argument. SOURCE_IGNORE_INSTRUMENTATION=hlisti: List of source file which are not instrumented. The list is a list of wildcard expressions which matches absolute file paths. Environment variables can be access by placing the name between two dollars. (ex: $HOME$ to access to HOME environment variable) 50.3 Profile Parameters for Preprocessor Settings PREPROCESSOR_KEEP_DEFINES=hstringi: List of preprocessor defines that are passed through the preprocessor and the compiler command. Only preprocessor defines that are present in the command line argument of CoverageScanner are concerned. PREPROCESSOR_HIDE_OPTION_NO_ARG=hstringi: List of single options (without arguments) which should not be transmitted to the native preprocessor. PREPROCESSOR_HIDE_OPTION_ONE_ARG=hstringi: List of options with one argument which should not be transmitted to the native preprocessor. PREPROCESSOR_HIDE_OPTION_ONE_ARG_APPEND=hYES/NO/ALLOWi: If set, the argument of the options to hide is directly appended (no space characters are placed between the option and the arguments). PREPROCESSOR_KEEP_OPTION_ONE_ARG=hstringi: List of options with one argument which must be transmitted to the native preprocessor. Squish Coco 5.1.0 - 302 - froglogic GmbH Profile Parameters for Linker Settings PREPROCESSOR_KEEP_OPTION_ONE_ARG_APPEND=hYES/NO/ALLOWi: If set, the argument of the options which must be transmitted is directly appended (no space characters are placed between the option and the arguments). PREPROCESSOR_DEFINE_OPTION=hstringi: Preprocessor command line option which lets you define C or C++ symbols. (ex: -D) PREPROCESSOR_DEFINE_OPTION_APPEND=hYES/NO/ALLOWi: Allows appending the preprocessor symbols just after its command line option string. (without spaces, ex: -DNDEBUG) PREPROCESSOR_INCLUDE_OPTION=hstringi: Preprocessor command line option which lets you include directly some source files. (ex: /FI) PREPROCESSOR_INCLUDE_OPTION_APPEND=hYES/NO/ALLOWi: Allows appending the include file name just after its command line option string. (without spaces, ex: /FImy- header.h) PREPROCESSOR_DEFINE_SEPARATOR_OPTION=hstringi: Separator of the list of preprocessor symbols. For example, if set to ,, CoverageScanner will detect the symbols DEF1 and DEF2 when parsing the string DEF1,DEF2. PREPROCESSOR_CMD=hstringi: Preprocessor command line. CoverageScanner does only analyse preprocessed files. The command generated the C or C++ file used for the code coverage analysis. The following variables are available: $TMP1$. . . $TMP9$: Temporary files names. The temporary files are removed on exit. $OPTIONS$: Preprocessing options. The preprocessing option are extracted from the command line arguments and contains the list of includes, preprocessor symbols, etc. . . $TOOL$: Compiler command (TOOL variable). $ARCHITECTURE$: Architecture detected or specified by --cs-architecture. $PROFILE_PATH$: Path of the profile file. $SOURCE$: Source file name. $OUTPUT$: Preprocessor output file name. If this variable is present in the command line, it contains the name of a temporary filename used as output for the preprocessor. If not present, it is assumed that the preprocessor generate its output to stdout. The preprocessor output is stdout. If this parameter is omitted in the configuration, the preprocessor step is skipped. i The preprocessor command must define the symbol __COVERAGESCANNER__ to 1. (ex: $TOOL$ $OPTIONS$ -D__COVERAGESCANNER__=1 -E $SOURCE$) 50.4 Profile Parameters for Linker Settings LINKER_HIDE_OPTION_NO_ARG=hstringi: List of single options (without arguments) which should not be transmitted to the native linker. LINKER_HIDE_OPTION_ONE_ARG=hstringi: List of options with one argument which should not be transmitted to the native linker. Squish Coco 5.1.0 - 303 - froglogic GmbH Profile Parameters for Linker Settings LINKER_HIDE_OPTION_ONE_ARG_APPEND=hYES/NO/ALLOWi: If set, the argument of the options to hide is directly appended (no space characters are placed between the option and the arguments). OBJ_EXT=hstringi: List of extension for object files separated with a semicolon. (ex: .obj) DYN_LIB_EXT=hstringi: List of extensions for dynamic libraries separated with a semicolon. (ex: .dll) STATIC_LIB_EXT=hstringi: List of extensions for static libraries separated with a semicolon. (ex: .lib) UNIX_LIBRARY_NAME=hYES/NOi: The library name follows the Unix standard. LINK_LIBRARY_OPTION=hstringi: Link option which selects libraries (ex:-l). LINK_LIBRARY_OPTION_APPEND=hYES/NO/ALLOWi: Append the library directly after the link option string (ex: -lfoo) if LINK_LIBRARY_OPTION_APPEND is set. LINK_OUTPUT_OPTION=hstringi: Linker output option (ex:-o) LINK_OUTPUT_OPTION_APPEND=hYES/NO/ALLOWi: Append the linker output file directly after the option string (without spaces, ex: -ofoo.o) if LINK_OUTPUT_OPTION_APPEND is set. LINK_OUTPUT_DEFAULT=hstringi: Linker output default filename. The following variable is available: $TMP1$. . . $TMP9$: Temporary files names. The temporary files are removed on exit. $BASENAME_SRC$: Base name of the first source file. The base name is the filename without extension. (ex: $BASENAME_SRC$.exe) LINK_ADDITIONAL_ARGUMENTS=hlisti: Additional arguments passed to the linker when instrumenting the source code. DEFAULT_LIB_PATH=hstringi: List of directories which contains the shared libraries (ex: /usr/library;/usr/share/library). DLL_OPTION=hstringi: Linker command line option which generates a shared library or a DLL (ex: /DLL). DLL_OUTPUT_STATIC_LIB=hstringi: Static library (associated with the DLL) output file name (ex: /IMPLIB:foo.lib). DLL_OUTPUT_STATI_LIB_APPEND=hYES/NO/ALLOWi: Append the static library (associated with the DLL) output file directly after the option string (without spaces, ex: /IMPLIB:foo.lib) if DLL_OUTPUT_STATIC_LIB_APPEND is set. STATIC_LIB_AS_DEFAULT=hYES/NOi: Instrumentation database name takes as base name the static library name associated with a DLL. LIBRARY_PATH_OPTION=hstringi: Link option which adds additional directories to the library search path (ex:-L) Squish Coco 5.1.0 - 304 - froglogic GmbH Profile Parameters for Linker Settings LIBRARY_PATH_OPTION_APPEND=hYES/NO/ALLOWi: Append the directory name directly after the option string (ex: -L/foo) if LIBRARY_PATH_OPTION_APPEND is set. AR_COMMAND_FORMAT_OPTION=hYES/NOi: The GNU librarian (ar) has a specific command line syntax which is supported when this option is set. STDIN_MRI_SCRIPT_OPTION=hlisti: This option indicates that an MRI script is used to generate a library. The script is provided by the standard input. GENERATE_COVERAGESCANNER_LIBRARY=hYES/NOi: If set, the CoverageScanner library is generated during the linking operation. This should be disabled when generating static libraries. SKIP_GENERATE_COVERAGESCANNER_LIBRARY_OPTION=hstringi: skips the generation of the CoverageScanner library. LIBGEN_AUTODETECT_ARGS_OPTION_NO_ARG=hstringi: List of single compilation options (without arguments) which should be used for the compilation of the CoverageScanner library. LIBGEN_AUTODETECT_ARGS_OPTION_ONE_ARG=hstringi: List of compilation options with one argument which should be used for the compilation of the CoverageScanner library. LIBGEN_AUTODETECT_ARGS_OPTION_ONE_ARG_APPEND=hYES/NO/ALLOWi: If set, the argument of the options is directly appended (no space characters are placed between the option and the arguments). AUTODETECT_MS_RUNTIME_OPTION=hlisti: List of command line option which selects the runtime library (for Microsoft® Visual Studio®: /MT;/MD;/ML;/MLd;/MTd;/MDd) INSTRUMENTATION_TABLES_LINKED_DURING_RUNTIME=hYES/NOi: Instead of linking all instrumentation tables when calling the linker, link it at application runtime. INSTRUMENTATION_TABLES_LINKED_DURING_RUNTIME_SUPPORT=hYES/NOi: Specify if linking all instrumentation tables at application runtime is supported or not. PLUGIN_REGISTRATION_API=hYES/NOi: If set, __coveragescanner_register_library() and __coveragescanner_unregister_library() are provided bythe CoverageScanner API to register instrumented plugins during the execution. FILE_FORMAT_SPECIFIER=hYES/NOi: If set, file format specifier (see command line option --cs-output) are supported at the runtime. INSTRUMENTATION_TABLES_OF_DLL_LINKED_DURING_RUNTIME=hYES/NOi: Instead of linking all instrumentation tables of DLL when calling the linker, link it at application runtime. COVERAGESCANNER_REGISTRATION_UID=hYES/NOi: Generate an unique symbol in each static and dynamic library to retrieve the code coverage counters. INJECT_COVERAGESCANNER_LIBRARY_AT_END=hYES/NOi: If YES, the CoverageScanner library is always inserted at the end of the linker command line arguments. COSMIC_LINKER_SCRIPT=hYES/NOi: Support of COSMIC linker scripts. CoverageScanner will then try to recognize a script by trying to parse it. Squish Coco 5.1.0 - 305 - froglogic GmbH Profile Parameters for Compiler Settings COSMIC_LINKER_SCRIPT_EXT=hstringi: File extension of the COSMIC linker file. GNU_LINKER_SCRIPT=hYES/NOi: Support of GNU linker scripts. CoverageScanner will then try to recognize a script by trying to parse it. GNU_LINKER_SCRIPT_OPTION=hstringi: Command line option which specifies a GNU linker script. GNU_LINKER_SCRIPT_OPTION_APPEND=hYES/NO/ALLOWi: Append the gnu linker script directly after the linker script command line option string (ex: -Tfoo.script) if GNU_LINKER_SCRIPT_OPTION_APPEND is set. VS_DEF_FILE_OPTION=hstringi: Command line option for specifying the module definition file (.def file). (ex: VS_DEF_FILE_OPTION=/DEF:) VS_DEF_FILE_OPTION_APPEND=hYES/NO/ALLOWi: If set, the command line option for specifying the module definition file is directly appended to the file name (no space characters are placed between the option and the arguments). 50.5 Profile Parameters for Compiler Settings COMPILER_ONLY=hYES/NOi: If true, the native tool is only a compiler which has no linker capabilities. COMPILER_HIDE_OPTION_ONE_ARG=hstringi: List of options with one argument which should not be transmitted to the native compiler. COMPILER_HIDE_OPTION_ONE_ARG_APPEND=hYES/NO/ALLOWi: If set, the argument of the options to hide is directly appended (no space characters are placed between the option and the arguments). BYTECODE_EMULATOR=hstringi: If set, the call of the native toolchain is made by calling a byte code intepreter. Example: BYTECODE_EMULATOR=/math/to/mono.exe permits for force the call of mcs.exe in a specific Mono emulator. TOOL=hstringi: Native compiler/linker command (example: gcc). The following variables are available: $PROFILE$: Profile name without extension. REMOVE_EXT=hstringi: List of file extensions, separated with a semicolon, of files to remove from the command line arguments. IGNORE_EXT=hstringi: List of file extensions, separated with a semicolon, of files to ignore. Ignored files are not instrumented. C#_EXT=hstringi: List of extensions for C# files separated with a semicolon. (ex: .cs) C_EXT=hstringi: List of extensions for C files separated with a semicolon. (ex: .c) Squish Coco 5.1.0 - 306 - froglogic GmbH Profile Parameters for Compiler Settings CPP_EXT=hstringi: List of extensions for C++ files separated with a semicolon. (ex: .cpp;.cxx) CPP_LANGUAGE_OPTION=hstringi: List of command line options passed to the compiler in C++ mode. C_LANGUAGE_OPTION=hstringi: List of command line options passed to the compiler in C mode. LANGUAGE_SELECTION_OPTION=hstringi: List of command line options which select the computing language (C, or C++). LANGUAGE_SELECTION_OPTION_APPEND=hYES/NO/ALLOWi: Append the selected language directly after the option string. LANGUAGE_SELECTION_OPTION_C_KEYWORD=hstringi: List of arguments of the language selection command line option which force the switching in C mode. LANGUAGE_SELECTION_OPTION_CPP_KEYWORD=hstringi: List of arguments of the language selection command line option which force the switching in C++ mode. COMPILER_CMD=hstringi: Compiler command line. The compiler command line is used to generate the CoverageScanner library during the linking phase. The following variables are available: $TMP1$. . . $TMP9$: Temporary files names. The temporary files are removed on exit. $COMPILER$: Compiler command (COMPILER variable). $SOURCE$: Source file name. $DESTINATION$: Destination file name. $LIBGEN$: Contents of the command line option --cs-libgen. (ex: $COMPILER$ -c $SOURCE$ -o $DESTINATION$) COMPILER_CMD_OPTION_NO_ARG=hstringi: List of single options (without arguments) which are extracted from the command line and which are used for compiling the CoverageScanner library. COMPILER_CMD_OPTION_ONE_ARG=hstringi: List of options with one argument which are extracted from the command line and which are used for compiling the CoverageScanner library. COMPILER_CMD_OPTION_ONE_ARG_APPEND=hYES/NO/ALLOWi: If set, the argument of the options to use for the compilation is directly appended (no space characters are placed between the option and the arguments). COMPILER_ADDITIONAL_ARGUMENTS=hstringi: Additional arguments passed to the compiler when instrumenting the source code. COMPILE_OPTION=hstringi: Compile command line option. (ex: -c) COMPILER_OUTPUT_OPTION=hstringi: Compiler output option. (ex: -o) COMPILER_OUTPUT_OPTION_AT_END=hYES/NOi: If no output is specified in the command line, add it explicitly at the end (if YES) or at the beginning of the command line arguments. The default is YES. Squish Coco 5.1.0 - 307 - froglogic GmbH Profile Parameters for Compiler Settings COMPILER_OUTPUT_OPTION_APPEND=hYES/NO/ALLOWi: Append the compiler output directly after the option string (without spaces, ex: -ofoo.o) if COMPILER_OUTPUT_OPTION_APPEND is true. COMPILER_OUTPUT_DEFAULT=hstringi: Compiler output default filename. The following variable is available: $TMP1$. . . $TMP9$: Temporary files names. The temporary files are removed on exit. $BASENAME_SRC$: Base name of the first source file. The base name is the filename without extension. (ex: $BASENAME_SRC$.obj) CALLING_CONVENTION=hstringi: CALLING_CONVENTION sets the calling convention (__stdcall, __cdecl or __fastcall) on Windows plat- form. FUNCTION_ATTRIBUTE=hstringi: FUNCTION_ATTRIBUTE sets additional compiler attributes to each function of the CoverageScanner library. DESTRUCTOR_ATTRIBUTE=hstringi: DESTRUCTOR_ATTRIBUTE sets an additional compiler attribute which causes the functions, which reset the code coverage information when the application gets shutdown or when a dynamic library get unloaded, to be automatically called. This attribute is is only necessary when the instrumentation tables are registered at runtime. CONSTRUCTOR_ATTRIBUTE=hstringi: CONSTRUCTOR_ATTRIBUTE sets an additional compiler attribute which causes the functions, which initialize the code coverage information when the application starts, to be automatically called. This attribute is is only necessary when the instrumentation tables are registered at runtime. DESTRUCTOR_PRAGMA=hstringi: DESTRUCTOR_PRAGMA sets additional compiler pragma attribute which cause the functions, which reset the code coverage information when the application get shutdown or when a dynamic library get unloaded, to be automatically called. The variable $SYMBOL$ is parsed and is replaced through the symbol on which the pragma is applyed. This attribute is is only necessary when the instrumentation tables are registered at runtime. CONSTRUCTOR_PRAGMA=hstringi: CONSTRUCTOR_PRAGMA sets additional compiler attribute which cause the functions, which initialize the code coverage information when the application starts, to be automatically called. The variable $SYMBOL$ is parsed and is replaced through the symbol on which the pragma is applyed. This attribute is is only necessary when the instrumentation tables are registered at runtime. DLL_EXPORT=hstringi: DLL_EXPORT contains the compiler attribute which exports a symbol from a DLL/shared library. DLL_IMPORT=hstringi: DLL_IMPORT contains the compiler attribute which imports a symbol from a DLL/shared library. FORCE_DLL_EXPORT=hYES/NOi: FORCE_DLL_EXPORT forces the usage of DLL_EXPORT attribute during the linking phase instead of DLL_IMPORT attribute. FORCE_DLL_EXPORT_OPTION=hstringi: FORCE_DLL_EXPORT_OPTION forces the usage of DLL_EXPORT attribute when it appears in the command line. PARALLEL_COMPILATION_OPTION_APPEND=hYES/NO/ALLOWi: Indicates if the number of processors are directly appended to the command which enables the parallel build. Squish Coco 5.1.0 - 308 - froglogic GmbH Profile Parameters for Compiler Settings PARALLEL_COMPILATION_OPTION=hlisti: Command line option which enables the parallel build. If this command line argument is followed by an integer, this value indicates the maximum number of parallel builds to execute. SYSTEM_INCLUDES=hlisti: List of system include directories. (ex: /usr/include) INCLUDE_PATH_OPTION=hstringi: Compiler option which adds additional include directories (ex:-I) INCLUDE_PATH_OPTION_APPEND=hYES/NO/ALLOWi: Append the include directory name directly after the option string (ex: -I/foo) if INCLUDE_PATH_OPTION_APPEND is set. CUSTOM_MALLOC_INCLUDE=hstringi: Additional include for custom malloc()/free() functions (ex: CUSTOM_MALLOC_FUNCTION=hstringi: Custom body of a malloc() function. The body should return a void* and the parameter int size indicates the number of bytes to be allocated. (ex: return malloc((size_t)size);) CUSTOM_FREE_FUNCTION=hstringi: Custom body of a free() function. The parameter void* ptr points to the memory to be freed. (ex: free(ptr);) LOCALE_SYSTEM_FILE_ENCODING=hYES/NOi: If YES, the locale system configuration of text encoding is used for compiling a source file. OBJECT_DIR_OPTION=hstringi: Compiler option which specifies the directory in which the objects are stored. (ex: -object-dir=debug/) OBJECT_DIR_OPTION_APPEND=hYES/NO/ALLOWi: Append the directory directly after the option string (without spaces, ex: -object-dir=foo) WARNINGS_TO_DISABLE=hlisti: List of compiler warnings to disable. Each item of the list will be used to generate a compiler pragma which enables/disables a compilation warning. WARNING_DISABLE_PRAGMA=hstringi: Preprocessor directive which disables a compiler warning. The variable $WARNING$ will be parsed and contains the warning number which should be disabled and \n is expanded as a cariage return. (ex: #pragma warning disable $WARNING$) WARNING_ENABLE_PRAGMA=hstringi: Preprocessor directive which enables/restores a compiler warning. The variable $WARNING$ will be parsed and contains the warning number which should be enabled and \n is expanded as a cariage return. (ex: #pragma warning restore $WARNING$) COVERAGESCANNER_CSHARP_DYNAMIC=hstringi: Regular expression which enables the support of the C# dynamic type. (ex: ^/define:.*\ COVERAGESCANNER_NO_CSHARP_DYNAMIC=hstringi: Regular expression which disables the support of the C# dynamic type. (ex: ^/define:.*\ ACTIVATE_CSHARP_DYNAMIC=hstringi: Command line argument which sets the define __COVERAGESCANNER_CSHARP_DYNAMIC__ to enable the C# dynamic support. (ex: /define:__COVERAGESCANNER_CSHARP_DYNAMIC__) Squish Coco 5.1.0 - 309 - froglogic GmbH Profile Parameters for Precompiled Headers Support 50.6 Profile Parameters for Precompiled Headers Support PCH_EXT=hstringi: List of extension for precompiled header files separated with a semicolon. (ex: .pch) CREATE_PCH_OPTION_APPEND=hYES/NO/ALLOWi: Indicates if the PCH header generated is directly appended to the command line option. CREATE_PCH_OPTION=hlisti: List of command line options which are generating precompiled headers. (ex: /Yc) USE_PCH_OPTION_APPEND=hYES/NO/ALLOWi: Indicates if the PCH header used is directly appended to the command line option. USE_PCH_OPTION=hlisti: List of command line options which select a precompiled header. (ex: /Yu) PCH_OUTPUT_OPTION=hstringi: Command line option to specify the precompiled header output. (ex: /Fp) PCH_OUTPUT_OPTION_APPEND=hYES/NO/ALLOWi: Append the precompiled header output directly after the option string (without spaces, ex: -Fpfoo.pch) if PCH_OUTPUT_OPTION_APPEND is true. PCH_OUTPUT_DEFAULT=hstringi: Default precompiled header output file name. The following variable is available: $TMP1$. . . $TMP9$: Temporary files names. The temporary files are removed on exit. $BASENAME_SRC$: Base name of the first source file. The base name is the filename without extension. (ex: $BASENAME_SRC$.pdb) 50.7 Profile Parameters for Execution Time Measurement PERF_CLOCK_HEADER=hlisti: Additional C header used to access to the performance counters. PERF_CLOCK_TYPE=hstringi: Type name for the performance counters (ex: clock_t). The size of the counter must be 64 bits. PERF_CLOCK_READ=hstringi: Function which reads the performance counters, The variable $OUT$ is the output parameter of this call. PERF_CLOCK_ADD=hstringi: Function which adds two performance counters. The variable $A$ and $B$ are the input parameters of this call. The variable $O$ stores the output result. PERF_CLOCK_SUB=hstringi: Function which subtracts two performance counters. The variable $A$ and $B$ are the input parameters of this call. The variable $O$ stores the output result. Squish Coco 5.1.0 - 310 - froglogic GmbH Profile Parameters for Profiling PERF_CLOCK_CONVERT=hstringi: Function which converts a performance counter to a string. The variable $I$ and $O$ are the input and output parameters of this call. The output is a fixed point floating point number which correspond to the time in seconds. 50.8 Profile Parameters for Profiling PROF_CLOCK_HEADER=hlisti: Additional C header used to access to the performance counters. PROF_CLOCK_ADDITIONAL_ARG=hlisti: Additional linker argument for building the code with a profiler support. PROF_CLOCK_TYPE=hstringi: Type name for the performance counters (ex: clock_t). The size of the counter must be 64 bits. PROF_CLOCK_READ=hstringi: Function which reads the performance counters, The variable $OUT$ is the output parameter of this call. PROF_CLOCK_ADD_ATOMIC=hstringi: Function which adds a value to performance counters. The variable $I$ is the value to add. The variable $O$ is modified and stores the output result. This operation needs to be atomic because it is used in a multithreading context, PROF_CLOCK_SUB=hstringi: Function which subtracts two performance counters. The variable $A$ and $B$ are the input parameters of this call. The variable $O$ stores the output result. PROF_CLOCK_CONVERT=hstringi: Function which converts a performance counter to a string. The variable $I$ and $O$ are the input and output parameters of this call. The output is a fixed point floating point number which correspond to the time in second. 50.9 Profile Parameters for Custom IO CSEXE_FOPEN=hstringi: ’fopen function used by the generation of the execution report file. CSEXE_FCLOSE=hstringi: ’fclose function used by the generation of the execution report file. CSEXE_FPUTS=hstringi: fputs function used by the generation of the execution report file. CUSTOM_SETUP=hNONE/POSIX/MS/GNU/MSCE/C++i: NONE: if set to NONE the function __coveragescanner_install() will not be provided by CoverageScannerThis˙ parameter should be set to NONE on embedded systems which do not provide the functions signal() and atexit(). The code coverage information is saved only when __coveragescanner_save() is called. Squish Coco 5.1.0 - 311 - froglogic GmbH Miscellaneous Profile Parameters POSIX: CoverageScanner provides a function __coveragescanner_install(), which installs a handler which saves the execution report when the application terminates. The handler is installed using POSIX functions (signal() and atexit()). If __coveragescanner_install() is not called, the code coverage information is only saved when __coveragescanner_save() is called. GNU: This is similar to the POSIX setting except that, if __coveragescanner_install() is not called, a default handler is used which saves the code coverage data in the file coverage.csmes on a normal application exit. If __coveragescanner_install() is not called, the code coverage information is not saved on abnormal exit (crash, software interruption, . . . ). MS: Same as GNU but for Microsoft® Visual Studio® compiler. C++: Install a handler written in C++ which saves the code coverage data in the file coverage.csmes upon normal application exit. CoverageScanner provides a function __coveragescanner_install(), which only changes the name of the destination file. MSCE: Same as C++ but for Microsoft® eMbedded Visual C++® compiler. 50.10 Miscellaneous Profile Parameters PCH_HIDE_OPTION_NO_ARG=hstringi: List of single options (without arguments) which should not be transmitted to the native tool (compiler or linker) when precompiled headers are disabled using --cs-no-pch. (ex: -Yd;/Yd for disabling the precompiled headers on Microsoft® Visual Studio®) PCH_HIDE_OPTION_ONE_ARG=hstringi: List of options with one argument which should not be transmitted to the native tool (compiler or linker) when precompiled headers are disabled using --cs-no-pch. (ex: -Yl;-Yc;-Yu;-YX;/Yl;/Yc;/Yu;/YX for disabling the precompiled headers on Microsoft® Visual Studio®) PCH_HIDE_OPTION_ONE_ARG_APPEND=hYES/NO/ALLOWi: If set, the argument of the options to hide is directly appended (no space characters are placed between the option and the arguments) when precompiled headers are disabled using --cs-no-pch. HIDE_OPTION_NO_ARG=hstringi: List of single options (without arguments) which should not be transmitted to the native tool (compiler or linker). (ex: -Yd;/Yd for disabling the precompiled headers on Microsoft® Visual Studio®) HIDE_OPTION_ONE_ARG=hstringi: List of options with one argument which should not be transmitted to the native tool (compiler or linker). (ex: -Yl;-Yc;-Yu;-YX;/Yl;/Yc;/Yu;/YX for disabling the precompiled headers on Microsoft® Visual Studio®) HIDE_OPTION_ONE_ARG_APPEND=hYES/NO/ALLOWi: If set, the argument of the options to hide is directly appended (no space characters are placed between the option and the arguments). PDB_EXT=hstringi: List of extension for debug information files separated with a semicolon. (ex: .pdb) PDB_OUTPUT_OPTION=hstringi: Command line option for specifying the debug information file. (ex: PDB_OUTPUT_OPTION=/Fd) Squish Coco 5.1.0 - 312 - froglogic GmbH Miscellaneous Profile Parameters PDB_OUTPUT_OPTION_APPEND=hYES/NO/ALLOWi: If set, the command line option for specifying the debug information file is directly appended to the file name (no space characters are placed between the option and the arguments). SBR_EXT=hstringi: List of extension for symbol browser files separated with a semicolon. (ex: .sbr) SBR_OUTPUT_OPTION=hstringi: Command line option for specifying the symbol browser information file. (ex: SBR_OUTPUT_OPTION=/Fr;/FR) SBR_OUTPUT_OPTION_APPEND=hYES/NO/ALLOWi: If set, the command line option for specifying the symbol browser information file is directly appended to the file name (no space characters are placed between the option and the arguments). USE_RESPONSE_FILE=hYES/NO/ALLOWi: Pack all compiler/linker command line option into a response file. This is generally necessary on Microsoft® Windows due to the command line limitation length. The option ALLOW consists of using a response file only if a response file is used in the command line arguments of the native compiler. A response file is only used if the if the command line size reach a limit defined by MINIMUM_COMMAND_LINE_SIZE_FOR_RESPONSE_FILE. MINIMUM_COMMAND_LINE_SIZE_FOR_RESPONSE_FILE=hintegeri: If the command line arguments are lower that the specified size, no response file is used. MAXIMUM_COMMAND_LINE_SIZE_FOR_RESPONSE_FILE=hintegeri: Ensures that, if a command line size does overflow this value, a response file is used. RESPONSE_FILE_EXT=hstringi: List of file extensions for the response file. RESPONSE_FILE_OPTION=hlisti: Command line option for specifying the response file. (ex: RESPONSE_FILE_OPTION=@) RESPONSE_FILE_OPTION_APPEND=hYES/NO/ALLOWi: If set, the command line option for specifying the response file is directly appended to the file name (no space characters are placed between the option and the arguments). STRIP_CPP_COMMENTS_IN_RESPONSE_FILE=hYES/NOi: If set, the C/C++ comments are removed in the command line options. CODE_STYLE=hC++/C#i: Language supported. EXIT_FUNCTIONS=hstringi: List of functions which causes an application exit. (ex: EXIT_FUNCTIONS=abort;exit) ENABLE_COVERAGESCANNER_PER_DEFAULT=hYES/NOi: Enable code coverage analysis per default. This is equivalent to implicitly set the command line option --cs-on. This option has only an impact if the native executable has the same name as CoverageScanner wrapper. DEACTIVATE_COVERAGESCANNER=hYES/NOi: Disable CoverageScanner globally if set to YES. DEACTIVATE_COVERAGESCANNER_OPTION_NO_ARG=hstringi: List of command line option which deactivates CoverageScanner. (ex: DEACTIVATE_COVERAGESCANNER_OPTION_NO_ARG=-M;-MM;-MD;-MMD) Squish Coco 5.1.0 - 313 - froglogic GmbH Miscellaneous Profile Parameters DEACTIVATE_COVERAGESCANNER_OPTION_ONE_ARG=hstringi: List of command line option with one argument which deactivates CoverageScanner. DEACTIVATE_COVERAGESCANNER_OPTION_ONE_ARG_APPEND=hYES/NO/ALLOWi: If set, the command line option which disables CoverageScanner is directly followed by an argument (no space characters are placed between the option and the arguments). COVERAGESCANNER_LIBRARY_OBJECT=hstringi: Name of the CoverageScanner object containing its library code. On Unix, its default value is __cs_library.o. On Microsoft® Windows, its default value is __cs_library.obj. EXIT_VALUE_ON_SUCCESS=hlisti: List of exit value that are return by the native linker or compiler and which means that the command was sucessfull. ERROR_TO_STDOUT=hYES/NOi: Print the error messages to the standard output and not to the error output. ERROR_FORMAT=hstringi: Format of the errors. The following items are parsed: %m: Error message. %f: File name. %e: File name, C-escaped. %l: Line. %c: Column. %e: Error type (warning, info or error) %n: New line. ERROR_FORMAT_LINE=hstringi: Same as ERROR_FORMAT but used when no column information are available. ERROR_FORMAT_TEXT=hstringi: Same as ERROR_FORMAT but used when no column and line information are available. ERROR_FORMAT_BEGIN=hstringi: Start banner of the CoverageScanner error messages. ERROR_FORMAT_END=hstringi: End banner of the CoverageScanner error messages. ADDITIONAL_ARGUMENT_SEPARATOR=hstringi: Additional character used as separator for the command line arguments. (ex: ADDITIONAL_ARGUMENT_SEPARATOR=, for VisualDSP++ compiler) CORRECT_FILENAME_IN_COMPILER_MESSAGE=hYES/NOi: If enabled, CoverageScanner replace in the compiler/linker messages the file name used for the instrumentation through the real source filename of the project. SUPPRESS_PATH_ENTRY=hYES/NOi: If set to YES, CoverageScanner alter the PATH variable and removes its own location when calling the native compiler. ATEXIT=hstringi: Command line used to register the application exit handler which saves the execution report upon the application exit. Squish Coco 5.1.0 - 314 - froglogic GmbH Instrumentation Parameters FORCE_ESCAPING_RULES_RESPONSE_FILE=hNO/MS/POSIXi: Rules for escaping command line arguments in response files: POSIX: Escaping rules for UNIX® platforms MS: Escaping rules for Microsoft® Windows. NO: Native escaping rules for the current platform. FILESYSTEM_8.3=hYES/NOi: If set, the extension of the execution report file name is .cse. If the application file name is application.exe the generated report will be application.cse in this case and not application.exe.csexe OUTPUT_BUFFER_SIZE=hintegeri: Size of the internal buffer used for generating an execution report. A higher value permits a better performance because it minimize the I/O system calls. For embedded system, 64 should be a good value and for other platforms 32768 is recommanded. 50.11 Instrumentation Parameters RETURN_INSTRUMENTED_IN_EXPRESSION=hYES/NOi: If true, return statement are instrumented into the returned expression (ex:return inst[0]++, x;). Squish Coco 5.1.0 - 315 - froglogic GmbH Release Notes Appendix Release Notes A Squish Coco 5.1.0 - 316 - froglogic GmbH Squish Coco v5.1.0 A.1 Squish Coco v5.1.0 Changes between Squish Coco v5.0.3 and Squish Coco v5.1.0: Microsoft® Visual Studio® Add-In: • New Feature: Replaced the old Microsoft® Visual Studio® Add-In with a stand-alone program to edit the project configuration files; it also works with Microsoft® MSBuild and similar build systems. Build Environment Selection: • New Feature: The program has now command line parameters that can be used for scripted installations. CoverageBrowser: • Bug Fix: Diff functionality did not work correctly with code that was split over two lines. • Bug Fix: Accelerator keys were displayed incorrectly in the titles of docking windows. cmreport: • Bug Fix: Issues with Cobertura reports for Azure DevOps corrected. • Bug Fix: Manual validations are now optionally reported in Cobertura reports. CoverageScanner: • Bug Fix: The C++20 threeway comparison operator <=> is now supported. • Bug Fix: If a function name was surrounded by braces (like, “int (foo)() { return 0 ; }”), the wrong name was displayed in the reports. • Bug Fix: Build issues for Linux Kernel modules. • Bug Fix: C++11 handling of final fixed. • Bug Fix: C++11 handling of const_expr fixed. • Bug Fix: Issue with static initializer of C++ structures fixed. • New Feature: A command line flag to specify an additional source file which contains custom code that can be used in an instrumented program (--cs-custom-library-source=hfilei). • New Feature: Command line flags to specify a trigger functions (--cs-trigger-function= htriggerfunctioni) and a custom function which saves the coverage data (--cs-coverage-save-function=hsavefunctioni). • Bug Fix: Support of linker input files for csar (ar switch @). • New Feature: The memory pool provides now a custom exception to handle an out-of-memory exception. (--cs-memory-alloc-failure-function=hstringi) • Bug Fix: Unclear error messages about licensing are reworded. • New Feature: .NET Core support on Windows. • New Feature: Guess typical --cs-libgen parameters for instrumentation of kernel modules and cross- compilations. CocoQML: • New Feature: Instrumentation is now possible with cocoqmlscanner alone and a new “tracker object” that is compiled by the customer. Squish Coco 5.1.0 - 317 - froglogic GmbH Squish Coco v5.0.3 A.2 Squish Coco v5.0.3 Changes between Squish Coco v5.0.2 and Squish Coco v5.0.3: CoverageScanner: • Bug Fix: C++ ISO 646 keywords support of ’and’, ’not’ and ’or’ for MC/DC and condition coverage. • Bug Fix: Removing comments in the preprocessed output that had been enabled in version v5.0.2. A.3 Squish Coco v5.0.2 Changes between Squish Coco v5.0.1 and Squish Coco v5.0.2: CoverageScanner: • Bug Fix: Compilation issues with C++ member function pointer in templates. • New Feature: Q_FOREACH support when compiling with C++17. • New Feature: Null-coalescing assignment support (https://docs.microsoft.com/en-us/dotnet/ csharp/whats-new/csharp-8#null-coalescing-assignment). • Object relocation support for GNU ld. Microsoft® Windows installer: • Bug Fix: Silent installation no longer calls the license wizard. Microsoft® Windows: • Bug Fix: Crash of the command line tools on Microsoft® Windows 10. A.4 Squish Coco v5.0.1 Changes between Squish Coco v5.0.0 and Squish Coco v5.0.1: CoverageScanner: • Bug Fix: Compilation issues when profiling is used on Microsoft® Visual Studio® 2008. • New Feature: Instrumentation of boolean expressions whose arguments are class instances with a custom conversion to boolean or integer types. • Bug Fix: Compilation issues fixed when using boolean expressions in template parameters. • Bug Fix: Line coverage issues fixed. • Bug Fix: Compilation issue fixed when performing boolean operations on function pointers. • Bug Fix: Compilation issue fixed with precompiled headers when compiling with Microsoft® Visual Studio®. cmreport: • Bug Fix: Issue with HTML reports when the sources are spread over several Windows drives (C:, D:, . . . ) Microsoft® Windows installer: • Bug Fix: Missing DLL installed. Squish Coco 5.1.0 - 318 - froglogic GmbH Squish Coco v5.0.0 A.5 Squish Coco v5.0.0 Changes between Squish Coco v4.3.3 and Squish Coco v5.0.0: CoverageBrowser: • New Feature: Function profiler. • New Feature: Metric difference between two releases. • New Feature: Possibility to exclude all uninstrumented files from patch analysis. • Bug Fix: Usability issue when displaying metrics fixed. CoverageScanner: • Bug Fix: Issue with execution counter of 64 bits fixed. • New Feature: Support for ARM DS-5. • New Feature: Function profiler. • New Feature: Automatic integration with Squish for Qt. • Bug Fix: Compilation issues of C# and C++ code fixed. (Ternary operator, noexcept keyword, lambda functions,. . . ) • Bug Fix: Annotations support for C#. • Bug Fix: dotnet exit issue on Linux. cmedit: • New Feature: Possibility to rename functions. (Experimental) cmreport: • Bug Fix: Verification of the coherence of the coverage settings selected through the command line. • New Feature: Possibility to exclude from the patch analysis the file which are not instrumented. CocoQML: • Bug Fix: Multi-threaded QML applications can now be instrumented. • Bug Fix: Various other bug fixes: Instrumentation of case statements, complex ternary operators, other problems with complex syntax. License Server: • New Feature: System wide configuration file (on /etc/) for an easier deployment. Squish: • New Feature: Better integration of Squish and Squish Coco. A.6 Squish Coco v4.3.3 Changes between Squish Coco v4.3.2 and Squish Coco v4.3.3: Microsoft® Visual Studio® Add-In: • New Feature: Microsoft® Visual Studio® 2019 support. • Bug Fix: Installation issues with Microsoft® Visual Studio® 2017 add-in. Squish Coco 5.1.0 - 319 - froglogic GmbH Squish Coco v4.3.2 CoverageScanner: • Bug Fix: C# compilation issues fixed. (Issues with string literals, multi-arrays, out parameters and MCC instrumentations issues) • New Feature: .NET Core support on Linux™. • Bug Fix: Issue with C++ fallthrough attribute fixed. • Bug Fix: Error message compliant with Microsoft® MSBuild. • Bug Fix: Compilation issues fixed with C++ code instrumented at MCC level • Bug Fix: The .csexe extension is now added in the same way for C++ and C# when using COVERAGESCANNER_ARGS environment variable. • Bug Fix: Precompiled header issues fixed. A.7 Squish Coco v4.3.2 Changes between Squish Coco v4.3.1 and Squish Coco v4.3.2: cmmerge: • Bug Fix: C++ files in which a namespace is set by a preprocessor symbol are now handled correctly. CoverageScanner: • Bug Fix: Line coverage issues in nested blocks fixed. • Bug Fix: Compilation issues with GNU’s statement expression extension corrected. • Bug Fix: Better handling of path case in Microsoft® Windows. • Bug Fix: Fixed “undefined symbol” linker issue with static libraries. • New Feature: CoverageScanner can now be disabled with the preprocessor symbol COVERAGESCANNER_COVERAGE_OFF. • New Feature: PureCoverage annotations can now be disabled. • Bug Fix: C# compilation issues fixed. (Issues with new operator, tertiary operators, nullables and bodied expressions.) • Bug Fix: Dead code detection after a throw statement. • Bug Fix: C++-14 and C++-11 compilation issues fixed. (Issues with “if constexpr” statements, noexcept, template ellipsis, lambda expressions and function tables.) A.8 Squish Coco v4.3.1 Changes between Squish Coco v4.3.0 and Squish Coco v4.3.1: cmreport: • Bug Fix: Line coverage fixed for nested blocks with only sequential instructions inside. • Bug Fix: Coverage after merging differently preprocessed versions of a source is now correct. cmmerge: Squish Coco 5.1.0 - 320 - froglogic GmbH Squish Coco v4.3.0 • Bug Fix: Annotations from differently preprocessed source files are now imported correctly. CoverageScanner: • Bug Fix: A C# parsing issue concerning the null-coalescing operator (‘??’) is fixed. • Bug Fix: Some C++ parsing issues are resolved. • Bug Fix: The annotation of some lines was ignored in some cases. • Bug Fix: Extremly long C# compiler command lines are now handled correctly. A.9 Squish Coco v4.3.0 Changes between Squish Coco v4.2.2 and Squish Coco v4.3.0: CoverageScanner: • New Feature: Computing the execution time of tests. • Bug Fix: Compilation issue fixed for Qt5 code. Microsoft® Visual Studio® Add-In: • Bug Fix: Repair of broken executable paths for Microsoft® Visual Studio® 2017. CocoQML: • New Feature: Release of cocoqmlscanner, a program to instruments all QML and JavaScript files of a project in one step. • New Feature: New environment variable COCOQML_DEFAULT_BLACKLIST to exclude or include Qt QML system files from coverage. A.10 Squish Coco v4.2.2 Changes between Squish Coco v4.2.1 and Squish Coco v4.2.2: CoverageScanner: • New Feature: Instrumentation of boolean expressions in return statements A.11 Squish Coco v4.2.1 Changes between Squish Coco v4.2.0 and Squish Coco v4.2.1: CoverageScanner: • Bug Fix: Instrumentation of the ternary operator in C cast expressions. • Bug Fix: Compilation issue fixed for C++11 keyword noexecept. • New Feature: Qt5 support for QStringLiteral. • New Feature: Support for Windows Embedded Compact 2013. Squish Coco 5.1.0 - 321 - froglogic GmbH Squish Coco v4.2.0 A.12 Squish Coco v4.2.0 Changes between Squish Coco v4.1.2 and Squish Coco v4.2.0: Squish Coco: • New Feature: Cyclomatic complexity (McCabe) metric. • New Feature: Source file renaming and merging of test results from different platforms. CoverageScanner: • Bug Fix: Faster generation of execution reports. • On Windows, the absolute file names are no more converted to the lower case. cmmerge: • Bug Fix: Merging issue of manual validations corrected when MCC or MC/DC metric is used. cmreport: • Bug Fix: List of manual validations does no longer include uninstrumented lines. • New Feature: The text report now exports all manual validations. A.13 Squish Coco v4.1.2 Changes between Squish Coco v4.1.1 and Squish Coco v4.1.2: CocoQML: • New Feature: The add-on has now a cache in which it stores the instrumented files. The cache is by default disabled. A.14 Squish Coco v4.1.1 Changes between Squish Coco v4.1.0 and Squish Coco v4.1.1: Command line tools: • New Feature: Most command line tools now have a command line option @hfilenamei that lets you read options from a file. Microsoft® Visual Studio® Add-In: • New Feature: Microsoft® Visual Studio® 2017 support. A.15 Squish Coco v4.1.0 Changes between Squish Coco v4.0.4 and Squish Coco v4.1.0: Squish Coco 5.1.0 - 322 - froglogic GmbH Squish Coco v4.0.4 CoverageBrowser: • New Feature: Streamlined the UI design of the Execution Report import dialog. CoverageScanner: • New Feature: Support of sc_uint template of SystemC. • Bug Fix: Speed and memory usage improvement of instrumented applications. • New Feature: Support of C++ source files compiled with different preprocessor directives. • New Feature: Better support of lambda functions. CocoQML: • Bug Fix: Problem with .pragma fixed. Microsoft® Windows Installer: • Bug Fix: Fixed file permission issues. A.16 Squish Coco v4.0.4 Changes between Squish Coco v4.0.3 and Squish Coco v4.0.4: CoverageScanner: • New Feature: Expansion of environment variables at runtime with command line options of the form --cs-output= A.17 Squish Coco v4.0.3 Changes between Squish Coco v4.0.2 and Squish Coco v4.0.3: CocoQML: • New Feature: An experimental add-on for coverage of QML code is now available. CoverageScanner: • New Feature: The option --cs-no-line-directive now also works for C#. • Speed improvement of the execution report generation. Squish Coco: • Bug Fix: ARM® Keil® µVision v5.23 support. (The installation is now performed by the "Build Environ- ment Selection" tool) A.18 Squish Coco v4.0.2 Changes between Squish Coco v4.0.1 and Squish Coco v4.0.2: Squish Coco 5.1.0 - 323 - froglogic GmbH Squish Coco v4.0.1 CoverageScanner: • Bug Fix: Compilation issues fixed for projects with precompiled headers and MCC or MC/DC instrumen- tation. A.19 Squish Coco v4.0.1 Changes between Squish Coco v4.0.0 and Squish Coco v4.0.1: Microsoft® Visual Studio® Add-In: • Bug Fix: Coverage settings of C# projects are now applied to all platforms of a specific configuration. CoverageScanner: • Bug Fix: Compilation issues fixed for C# projects instrumented with MCC and MC/DC metric. A.20 Squish Coco v4.0.0 Changes between Squish Coco v3.4.1 and Squish Coco v4.0.0: Squish Coco: • New Feature: New coverage metrics MCC (Multiple Condition Coverage) and MC/DC (Modified Condition/Decision Coverage) CoverageBrowser: • New Feature: Difference analysis between two software versions will highlight changes in instructions only and ignore changes in source code comments. • Bug Fix: Minor GUI usability improvements. cmreport: • New Feature: The EMMA-XML report format has been extended to support Decision, MC/DC and MCC coverage. A new dedicated Jenkins EMMA-XML plugin is available. • New Feature: Difference analysis between two software versions will highlight changes in instructions only and ignore changes in source code comments. CoverageScanner: • New Feature: To avoid potential overflows the execution counters are now 64 bits wide. The size can be changed with --cs-counter-size. • New Feature: --cs-mcc and --cs-mcdc switches activate instrumentation for all code coverage metrics up to MCC. A.21 Squish Coco v3.4.1 Changes between Squish Coco v3.4.0 and Squish Coco v3.4.1: Squish Coco 5.1.0 - 324 - froglogic GmbH Squish Coco v3.4.0 CoverageScanner: • New Feature: The CoverageScanner API will propagate the test name, status and comments from an executable or library to all linked DLLs and subordinate DLLs. cmcsexeimport: • New Feature: Upon import a warning will be emitted if test meta-information (name, status and comments) is found to be incoherent. License Server: • Bug Fix: Crash fixed with new license key formats. A.22 Squish Coco v3.4.0 Changes between Squish Coco v3.3.3 and Squish Coco v3.4.0: cmreport: • New Feature: HTML Tree view of source and functions. CoverageScanner: • New Feature: Coverage for the cases in a switch statement is now counted separately even if the case con- tains no statements. The old (less precise) behavior can be restored with --cs-combine-switch-cases. • New Feature: The Decision Coverage level is now available as a distinct choice without the need to turn off Condition Coverage through --cs-decision. • To align naming of coverage levels with other vendors, --cs-branch is now called --cs-basic-blocks. Similarly, the C# pragma and region cov-branch have been renamed to cov-basic-block. The old names are still recognized for backward compatibility. If coverage analysis of ‘empty’ branches is desired, use Decision Coverage. CoverageBrowser: • New Feature: Reverse patch analysis. With this feature, a patch file that is already part of the code can be used to find out which tests cover the patched code. • New Feature: Bug location: computation of the list of source code lines most likely responsible for a failure in the test suite. (see Chapter 20.5, page 144) • New Feature: The computation of the optimized execution order now occurs in the background. • New Feature: The dialog to import an execution report has now an integrated preview. • New Feature: Simplified report generation dialogs. A.23 Squish Coco v3.3.3 Changes between Squish Coco v3.3.2 and Squish Coco v3.3.3: Build Environment Selection: • New Feature: Generation of compiler wrappers for CygWin and MinGW’s compilers. Squish Coco 5.1.0 - 325 - froglogic GmbH Squish Coco v3.3.2 cmreport: • New Feature: Splitting long lists over several pages in the HTML report. Microsoft® Visual Studio® Add-In: • Bug Fix: Installation issue on Microsoft® Visual Studio® 2005 fixed. A.24 Squish Coco v3.3.2 Changes between Squish Coco v3.3.1 and Squish Coco v3.3.2: cmcsexeimport: • New Feature: Wildcards are supported to specify the list of execution reports. Example: cmcsexeimport -m project.csmes --title="execution" *.csexe cmmerge: • New Feature: Wildcards are supported to specify the list of input databases. Example: cmmerge -o output.csmes inputs_*.csmes CoverageScanner: • Bug Fix: Support of C++11 shared_ptr template in expressions. • Bug Fix: C# 6.0 auto-property initializer supported. Microsoft® Visual Studio® Add-In: • New Feature: Microsoft® Visual Studio® 2015 support. A.25 Squish Coco v3.3.1 Changes between Squish Coco v3.3.0 and Squish Coco v3.3.1: Licenses: • Bug Fix: Simplified handling of node-locked licenses under UNIX® and macOS. A Coco program with a node-locked license now expects that the license had been generated on the same account as that one under which it runs. Previously, the account to which a license was bound was that one under which the user that started it had initially logged in. Changing the account with su or sudo had no influence on the license. This means that under some circumstances a license was generated for an account different from that under which Squish Coco ran, but it worked nevertheless. In these cases, the license must now be renewed. CoverageBrowser: • Bug Fix: Patch analysis: Issue related to parsing of date formats from different time zones resolved. Squish Coco 5.1.0 - 326 - froglogic GmbH Squish Coco v3.3.0 A.26 Squish Coco v3.3.0 Changes between Squish Coco v3.2.3 and Squish Coco v3.3.0: CoverageBrowser: • Speed improvement in the detection of duplicate executions. • Speed improvements in the merge of instrumentation databases, the generation of HTML reports, the computation of the list of executions that execute a source line, and in the computation of the code coverage metrics. • New Feature: Import of comments and manual validations from projects which are checked out in different directories. • New Feature: Patch analysis. CoverageScanner: • Bug Fix: Full instrumentation of the C/C++ “ternary if” operator: In an expression a ? b : c, all boolean operands in the expressions a, b or c are instrumented too. Nested ternary operators are therefore instrumented completely. • New Feature: Automatic exclusion of the coverage analysis of all headers which are inconsistent (for example, built with different preprocessor options). A linker warning lists all files which are excluded. cmcsexeimport: • New Feature: Speed improvement of the detection of duplicate executions. cmreport: • New Feature: Patch analysis. • New Feature: In the HTML report, a list of the tests which execute a source code line is displayed in the tooltip. • New Feature: A list of the executions that cover an instrumented statement is displayed in the tooltip for a source code line. cmmerge: • General speed improvements. • New Feature: Importing comments and manual validations from projects which are checked out in different directories. A.27 Squish Coco v3.2.3 Changes between Squish Coco v3.2.2 and Squish Coco v3.2.3: CoverageScanner: • Bug Fix: Memory usage reduced. This fixes an "out-of-memory" error message during the compilation. • New Feature: Explicit initialization of the CoverageScanner API is no longer necessary for C# DLLs. (The program InjectModuleInitializer.exe is no longer needed) Squish Coco 5.1.0 - 327 - froglogic GmbH Squish Coco v3.2.2 A.28 Squish Coco v3.2.2 Changes between Squish Coco v3.2.1 and Squish Coco v3.2.2: CoverageScanner: • Bug Fix: More meaningful instrumentation of the C# keywords using, lock, unsafe and fixed. • New Feature: If the debugger is connected, CoverageScanner runtime information is displayed in the Microsoft® Visual Studio® console. • New Feature: In C# files, the “dump on event” feature can be enabled through a macro definition. Microsoft® Visual Studio® Add-In: • New Feature: Configuration of the debug output of Squish Coco. • New Feature: Configuration of the Windows event handler. A.29 Squish Coco v3.2.1 Changes between Squish Coco v3.2.0 and Squish Coco v3.2.1: CoverageScanner: • New Feature: Support of Shift-JIS text encoding. • New Feature: Execution report of Silverlight C# binaries is stored on an Isolated Storage when storing on the local file system is not allowed. • Bug Fix: Support of strong-named C# assemblies. CoverageBrowser: • New Feature: The Preferences dialog now lets you switch off syntax highlighting. A.30 Squish Coco v3.2.0 Changes between Squish Coco v3.1.0 and Squish Coco v3.2.0: CoverageScanner: • New Feature: Precise error message when a merge failure occurs. CoverageBrowser: • New Feature: Thresholds for the overall statistics. • New Feature: Statistics of manual validated instrumentation (for each function, source code and for the complete project). cmreport: • New Feature: Statistics of manual validated instrumentation (for each function, source code and for the complete project) for the HTML and CSV report. cmmerge: Squish Coco 5.1.0 - 328 - froglogic GmbH Squish Coco v3.1.0 • Speed improvements. • New Feature: --delete deletes all input files after a successful merge. • New Feature: --blackbox generates an instrumentation database for black-box testing. A.31 Squish Coco v3.1.0 Changes between Squish Coco v3.0.2 and Squish Coco v3.1.0: Squish Coco: • New Feature: NUnit addin. Build Environment Selection: • Bug Fix: Detection of CygWin installations for x64 systems corrected. CoverageScanner: • VisualDSP® support. • Bug Fix: Support of UTF16 source files. • New Feature: For embedded systems, CoverageScanner is able to generate execution reports in 8.3 format. In this case the report extension is .cse and not .csexe. • New Feature: New command line switch --cs-no-abort-on-error: if used, instrumentation errors raised while processing C/C++ source code no longer abort the build. cmreport: • New Feature: Refactoring of the HTML report: – New Feature: Single HTML file and full HTML report have now the same look and feel. – New Feature: Source directories statistics in a clickable tree view. – New Feature: Class directories statistics in a clickable tree view. – New Feature: Statistic tables are interactively sortable. – New Feature: Tool-tip for a detailed information on each table item. • New Feature: CSV table export for all informations that can be exported through a HTML report. • New Feature: The source, execution and function dialog can now be interactively switched between a list and tree view. • New Feature: CoverageBrowser reacts automatically when the instrumentation database file is regenerated and reload it automatically. CoverageBrowser: • New Feature: The source, execution and function dialog can now be interactively switched between a list and tree view. • New Feature: CoverageBrowser reacts automatically when the instrumentation database file is regenerated and reload it automatically. A.32 Squish Coco v3.0.2 Changes between Squish Coco v3.0.1 and Squish Coco v3.0.2: Squish Coco 5.1.0 - 329 - froglogic GmbH Squish Coco v3.0.1 Squish Coco: • New Feature: Solaris support. CoverageBrowser: • Bug Fix: EMMA-XML Report generation corrected when excluding source files. • New Feature: Drag and drop of .csmes and .csexe files supported. CoverageScanner: • New Feature: Format specifier %c added to --cs-output. • Bug Fix: Issues with strings in __declspec extension fixed. • Bug Fix: Deal with non-instrumentable OpenMP code. • Bug Fix: C++11 string literal support. (example: R"delim(")delim") • Bug Fix: UTF-8 source file can now be compiled. • Bug Fix: C++ Line comments ending a source file are no more generating a parsing issue. • Bug Fix: C# linq expressions are now instrumented. • New Feature: New C# API function CoverageScanner.__coveragescanner_init(). • New Feature: Dead code detection in constant boolean expressions: – Dead code detection for if ( false ) ... or for if ( true ) ... branches. – Expression do statement while ( false ) is treated if the statement was not included into a do ... while ( ... ) construct. – Dead code detection of while ( false ) .... loops. – Statements ’while ( true ) ...’ and ’do ... while ( true );’ are now treated as the infi- nite loop ’for(;;) ...’. The conditions are no more instrumented to avoid false negative coverage instrumentations. • Bug Fix: New command line switch for C++ mixed mode: --cs-architecture=CLR • Bug Fix: Several parsing issue for C# and C++ code fixed. • Bug Fix: Support of multiple header files included with/FI when generating precompiled headers. • Bug Fix: C# support for Microsoft® Visual Studio® 2013. Build Environment Selection: • New Feature: generation of all wrappers for all installed version of GCC compilers. A.33 Squish Coco v3.0.1 Changes between Squish Coco v3.0.0 and Squish Coco v3.0.1: CoverageBrowser: • New Feature: Validating manually an instrumentation validates also all depending instrumentations. For example, when marking a return statement as validated, all sequential lines before are also automatically validated. CoverageScanner: • New Feature: Source code annotation though code comments. • Bug Fix: clang compiler support on macOS. Microsoft® Visual Studio® Add-In: • Bug Fix: Excluding source directories from the instrumentation is working again. Squish Coco 5.1.0 - 330 - froglogic GmbH Squish Coco v3.0.0 A.34 Squish Coco v3.0.0 Changes between Squish Coco v3.0.0-pre1 and Squish Coco v3.0.0: CoverageScanner: • Bug Fix: Line coverage information of nested compound statements are now computed correctly. • Change: --cs-include-path/--cs-exclude-path works now recursively on all subdirectories. • New Feature: --cs-output supports now format specifiers (%f, %P,...) • Bug Fix: lib.exe wrapper support now Module-Definition (.def) files • Bug Fix: Compilation of expression in the form CLASS c = a && b are now supported even if the operator || or && are not logical operators. CoverageBrowser: • New Feature: Disabling the difference view of a source file is possible. • New Feature: Importing reviewer comments from a previous instrumentation database. cmmerge: • New Feature: Importing reviewer comments (–reviews-only command line option) A.35 Squish Coco v3.0.0-pre1 Changes between Squish Coco v2.1.8 and Squish Coco v3.0.0-pre1: CoverageBrowser: • New Feature: Usability enhancement of the folding functionality in the source window: – possibility to define a number of lines before and after the folded line part of the context and which are not folded. – folded lines are chosen the instrumentation filter dialog. • New Feature: For the code coverage at branch or decision/condition level, the executed (resp. not executed) lines which are not instrumented are displayed in pastel green (resp. pastel red). • New Feature: Line coverage information is now also available in the preprocessed view of the source code. • New Feature: Automatic copying to the clipboard of a selected source code fragment. • New Feature: Generation of Cobertura reports. CoverageScanner: • New Feature: simple support of GNU linker script for MinGW. • New Feature: clang support on macOS. • New Feature: Configuration of instrumentation warnings and errors. • Bug Fix: throw statement are now instrumented in expressions. (example: int a = b ? 0 : throw 0 ;) • Bug Fix: /Yc and /Yu can now be used simultaneously for the Microsoft® Visual Studio® compiler. • New FeatureBETA: Tcl support. • New FeatureBETA: C# support. cmreport: • New Feature: Generation of Cobertura reports. Squish Coco 5.1.0 - 331 - froglogic GmbH Squish Coco v2.1.8 A.36 Squish Coco v2.1.8 Changes between Squish Coco v2.1.7 and Squish Coco v2.1.8: CoverageBrowser: • New Feature: Configuration of the color scheme. • New Feature: Excluding source files interactively from the instrumentation. • New Feature: Configuration of the maximum number of threads which can be used for a computation. • New Feature: Progress information during the generation of a code coverage report. CoverageScanner: • New Feature: Better detection of dead code after a switch/case statement. • New Feature: Optimized handling of C++ headers which reduces the size of the instrumentation database and the time for generating reports. cmreport: • New Feature: Excluding source files from the report generation. • New Feature: Configuration of the maximum number of threads which can be used for a computation. • Change: Function parameter are now appended to each function in the EMMA-XML report. cmcsexeimport: • New Feature: cmcsexeimport is now able to import more than one execution report file. A.37 Squish Coco v2.1.7 Changes between Squish Coco v2.1.6 and Squish Coco v2.1.7: CoverageBrowser: • Change: Navigation buttons are now grouped into one toolbar. • Speed improvement of the source code viewer. • Speed improvement of the computing of the list of tests which are executing a source line. • Speed improvement of the of the text search in the source code function. • New Feature: Source code viewer: – New Feature: Sources are displayed in a tabbed view. – New Feature: Preview of the instrumentation of a whole file into the scroll bar. – New Feature: Tooltip which displays the detail of an explanation of a source line. – New Feature: More intuitive text search function (search when typing into the source code window) and interactive highlighting of the matched patterns. – New Feature: Dock/undock line number column and statistic column. CoverageScanner: • Bug Fix: CoverageScanner is now able to handle "Whole Program Optimization" (option /GL) and "Use Link Time Code Generation" (option /LTCG) of Microsoft® Visual Studio®. • New Feature: Command line option --cs-vs2010-lambda which lets you instrument lambda functions in Microsoft® Visual Studio® 2010 Squish Coco 5.1.0 - 332 - froglogic GmbH Squish Coco v2.1.6 • New Feature: Support for "for each" C++ syntax extension of Microsoft® Visual Studio®. • Bug Fix: Support of ccache wrapper. • Bug Fix: CoverageScanner detects now dead code after a try/catch block. • Bug Fix: Precompiled header issues. • Bug Fix: Handling source code lines which are ending with a backslash. cmreport: • Speed improvement of the generation of EMMA-XML reports. • Speed improvement of the generation of HTML reports. • New Feature: HTML report displays the detailed instrumentation report in a tooltip. • Change: HTML report is generated in a directory whose name ends with _html. This avoids a name clash under Unix where the executable has in most of the cases no extension. A.38 Squish Coco v2.1.6 Changes between Squish Coco v2.1.5 and Squish Coco v2.1.6: CoverageBrowser: • New Feature: Possibility to choose the source code type used for comparing software releases (original source code or preprocessed source code) CoverageScanner: • New Feature: New command line option which lets you specify other command line arguments from a text file (--cs-option-file). • New Feature: New command line option which specifies directly the CoverageScanner profile (--cs-profile). cmreport: • New Feature: possibility to customize the EMMA-XML report A.39 Squish Coco v2.1.5 Changes between Squish Coco v2.1.4 and Squish Coco v2.1.5: CoverageScanner: • New Feature: Support for precompiled headers for Microsoft® Visual Studio® compiler. • New Feature: Xoreax IncrediBuild support (http://www.incredibuild.com/ incredibuild-product-overview.html) • Bug Fix: Wildcard expressions on relative paths are computed on the canonical path. (example: --cs-include-file-wildcard=include/* matches include/foo.h and module/../include/bar.h) • New Feature: New command line options to specify which extensions are used for C++ (--cs-cpp-ext) and C (--cs-c-ext) files. Squish Coco 5.1.0 - 333 - froglogic GmbH Squish Coco v2.1.4 • New Feature: New command line option which sets the tool chain path (--cs-native-toolchain). • Change: Verbose command line option (--cs-verbose) lets you select if the build, the instrumentation or the CoverageScanner API is instrumented. The verbose output of the instrumented file is sent to a log file (name of the execution report with the extension.cslog). • Bug Fix: -m32 and -m64 command line support for GCC. • Bug Fix: Compilation issue on Microsoft® Visual Studio® for instrumented C++0x lambda function corrected. • Bug Fix: Support of __stdcall as default calling convention when compiling with Microsoft® Visual Studio®. • New Feature: New command line option for UNIX® platform which generates a coverage report on the reception of a signal (--cs-dump-on-signal). • New Feature: New command line option for Microsoft® Windows platform which generates a coverage report on the reception of an event (--cs-dump-on-event). • Change: Instrumentation tables of libraries are chained together at runtime when compiling with Microsoft® Visual Studio®. The old behavior can be restored with --cs-link-instrumentation-tables. cmreport: • Change: cmreport generate now per default a code coverage report at decision/condition level with line coverage for uncovered lines. Microsoft® Visual Studio® Add-In: • Change: Microsoft® Visual Studio® Add-In does not disable the precompiled headers anymore. A.40 Squish Coco v2.1.4 Changes between Squish Coco v2.1.3 and Squish Coco v2.1.4: CoverageBrowser: • New Feature: Import summary display the imported execution comment. • New Feature: New instrumentation mode available which displays the computed line coverage of unin- strumented lines. This enables the readability of the source code but has no impact on the coverage statistics. CoverageScanner: • Bug Fix: Support for /clr:safe and clr:pure command line option. • Optimization: CoverageScanner calls now directly the native compiler without calling a second instance of it. • Bug Fix: The line coverage was wrongly calculated before some switch/case statements. License Manager: • New Feature: The license manager is now a wizard dialog. • New Feature: Command line tool to activate a license on Windows and Unix Squish Coco 5.1.0 - 334 - froglogic GmbH Squish Coco v2.1.3 A.41 Squish Coco v2.1.3 Changes between Squish Coco v2.1.2 and Squish Coco v2.1.3: CoverageScanner: • New Feature: /MP command line (parallel build) supported for Microsoft® Visual Studio® compiler. • Bug Fix: Parsing issues of C++11 templates corrected. • New Feature: The Build Environment Selection tool generate CoverageScanner wrapper for Cyg- Win. Microsoft® Visual Studio® Add-In: • New Feature: Since CoverageScanner instruments all non system folders, the selection dialog for a list of directories to include is removed. A dialog which permits selection of the list of directories to exclude replaces it. Linux™ installer: • New Feature: Possibility to specify a custom installation directory. cocolic: • New Feature: Command line interface on Unix. • Bug Fix: More verbose error message in the case of network failures. A.42 Squish Coco v2.1.2 Changes between Squish Coco v2.1.1 and Squish Coco v2.1.2: CoverageScanner: • Bug Fix: Multi line #pragma supported. • Bug Fix: ::new keyword supported. Microsoft® Visual Studio® Add-In: • Bug Fix: Support of solutions with folder. • Bug Fix: Support of solutions which mix C++ and C# projects. A.43 Squish Coco v2.1.1 Changes between Squish Coco v2.1.0 and Squish Coco v2.1.1: CoverageScanner: • Change: CoverageScanner instruments now per defaults all sources files except those which are in the system directory (/usr, /opt or C:\Windows) or the application installation directory (C:\Program Files). Squish Coco 5.1.0 - 335 - froglogic GmbH Squish Coco v2.1.0 • New Feature: Automatic detection of the runtime library necessary when compiling using Microsoft® Visual Studio®. Adding /MD or /MT to the build command argument of the CoverageScanner library is no more necessary. • Bug Fix: Crash when using wildcard expression on file name on UNIX®. • New Feature: The instrumentation tables are now generated at the runtime. This lets us handle stubs, which overwrite objects from libraries. • New Feature: Constructors which use member initialization lists are instrumented even if they have an empty body. • New Feature: C++ CLR language extension of Microsoft® Visual Studio® supported. CoverageBrowser: • New Feature: Navigation buttons are disables if they are irrelevant. After reaching the end of search, the navigation buttons start from the beginning again. • Bug Fix: Clicking on a function in the method list, show it entirely in the source window. • Bug Fix: Statistics count was wrongly computed when the instrumentation is computed with line coverage support and the test count mode was activated. • New Feature: Importing an execution report does not generate an error message any more. If some executions cannot be imported, the information window is automatically shown. A list of imported executions and errors are displayed on it. • New Feature: Windows x64 package available. cmreport: • New Feature: EMMA-XML report generated now condition coverage statistics. This requires Jenkins EMMA-XML plugin version 1.29 or above. • New Feature: JUnit report generation. A.44 Squish Coco v2.1.0 Changes between Squish Coco v2.0.3 and Squish Coco v2.1.0: Squish Coco: • New Feature: Recognition of dead code inside a function. • New Feature: Function coverage. • New Feature: Line coverage. CoverageScanner: • Code coverage count is enabled per default. cmreport: • Bug Fix: EMMA-XML report corrected: the line field correspond now to the line coverage. A.45 Squish Coco v2.0.3 Changes between Squish Coco v2.0.2 and Squish Coco v2.0.3: Squish Coco 5.1.0 - 336 - froglogic GmbH Squish Coco v2.0.2 Squish Coco: • Bug Fix: License activation issues. A.46 Squish Coco v2.0.2 Changes between Squish Coco v2.0.1 and Squish Coco v2.0.2: cmreport: • Improving the speed of the production of HTML reports through parallelization of the generation of the source code view. CoverageScanner: • Bug Fix: __try and __finally keyword now handled on Windows for C files. • Bug Fix: g++ compiler wrapper treats files with .c extension as C++ files. • Bug Fix: On Windows, command line containing directory name which ends with a backslash are now correctly escaped. • Bug Fix: ar and ld wrapper are now working, and do not report that the profile cannot be found. • New Feature: The code behind the macro BOOST_FOREACH is no more analyzed, unless the command line --cs-no-boost is used. • New Feature: New command line option --cs-output-abs. • Change: The support for Qt3, Qt4 and Boost is per default activated. Disabling it can be performed using command line switches --cs-no-qt3, --cs-no-qt4 and --cs-no-boost. • Change: The full instrumentation at decision and condition level is now the default setting for the instrumentation. A.47 Squish Coco v2.0.1 Changes between Squish Coco v2.0.0 and Squish Coco v2.0.1: CoverageScanner: • Bug Fix: Support setting define which contains escaped quotes per command line (ex: "-DVERSION=\"version 1.1\"") • Bug Fix: Compilation issue of TextEdit sample when spaces are in the path corrected. License Dialog: • Bug Fix: Retrieving a license key is now interruptible and displays a progress bar. A.48 Squish Coco v2.0.0 Changes between TestCocoon v1.6 and Squish Coco v2.0.0: Squish Coco 5.1.0 - 337 - froglogic GmbH Squish Coco v2.0.0 Microsoft® Visual Studio® Add-In: • New Feature: Support for Microsoft® Visual Studio® 2010. • Bug Fix: x64 platform are now recognized and handled. CoverageBrowser: • New Feature: Supporting linking several times the same object into an application. • New Feature: Possibility to filter out execution which does not cover any code. • New Feature: Possibility to filter out source files and functions which are not instrumented. • Bug Fix: Calculation of statistics is able to saturate all available CPU. • Bug Fix: After opening a new instrumentation database, no executions become selected. • New Feature: Possibility to choose between the native file dialog and the dialog provided by Qt. • New Feature: Displaying the execution of a reference release into the list of execution. Executions of the reference release are strikeout, new executions are underlined. • New Feature: Generation of browsable HTML output. CoverageScanner: • Empty functions are no more instrumented. • New Feature: C++0x support. • New Feature: Verbose output of the CoverageScanner output when using command line option --cs-verbose. • New Feature: Saving the contents of a tree view waits until all statistics are computed. • New Feature: Q_FOREACH macro es now handled to avoid false negative code coverage statistics. Once a look is executed once, the Q_FOREACH macro is considered as 100% covered. cmcsexeimport: • New Feature: Execution comments in HTML form are parsed from the execution report. This lets us add extra log information to the execution report. cmreport: • Bug Fix: Correction of an EMMA-XML issue: global statistics are now ordered in EMMA-XML report as following: Line, Method, Class and then Block • New Feature: Generation of browsable HTML output. Squish Coco 5.1.0 - 338 - froglogic GmbH Part XII License Information Squish Coco 5.1.0 - 339 - froglogic GmbH License Agreement Appendix License Agreement B Squish Coco License Agreement Agreement version 2.0 This froglogic Squish Coco Commercial License Agreement ("Agreement") is made by and between froglogic GmbH ("froglogic"), and you (either an individual or a legal entity) ("Licensee") (froglogic and Licensee together as "Parties"). IMPORTANT - READ CAREFULLY: 1. froglogic developed and originated Squish Coco including source code, documentation and example programs ("Licensed Software"). The Licensed Software is protected by copyright laws and international copyright treaties, as well as other intellectual property laws and treaties. froglogic retains all rights not expressly granted. 2. By installing, copying, or otherwise using the Licensed Software, Licensee agrees to be bound by the terms of this Agreement. If Licensee does not agree to the terms of this Agreement, Licensee must not install, copy, or otherwise use the Licensed Software. By installing, copying, or otherwise using any updates or other components of the Licensed Software that Licensee receives separately as part of the Licensed Software ("Updates"), Licensee agrees to be bound by any additional license terms that accompany such Updates. If Licensee does not agree to the additional license terms that accompany such Updates, Licensee may not install, copy, or otherwise use such Updates. 3. Upon Licensee’s acceptance of the terms and conditions of this Agreement, froglogic grants Licensee the right to use the Licensed Software during the agreed validity period as set forth in the respective License Certificate ("License Term"), solely for Licensee’s internal business purposes in the manner provided below. 4. Options (a) Evaluation License Licensee may use the Licensed Software for a limited time (evaluation period) to be determined and stated by froglogic, Squish Coco 5.1.0 - 340 - froglogic GmbH License Agreement solely for the purpose of determining whether the Licensed Software meets Licensee’s requirements. After the evaluation period has finished, Licensee must either: (i) discontinue use of the Licensed Software, (ii) acquire a commercial license (see section (b)), Licensee may not make commercial use of a derived work of the Licensed Software, under the scope of the evaluation license. (b) User Node-Locked Subscription Licensee may install and use the Licensed Software locally on one identified computer (Node), with the Node executing the Licensed Software for only one Authorized User. During the term of a paid-up and unexpired support agreement, Licensee may transfer a license certificate to another individual and Node after notification to and approval from froglogic, but not more often than once every 4 weeks, provided the future Licensee is employed or contracted by the same individual or legal entity as the initial Licensee. froglogic protects the personally identifiable information of the Authorized User under European data protection law. (c) Floating Subscription Licensee may install the Licensed Software on an unlimited number of its computers. All computers using the Licensed Software must have the ability to communicate with a license server. The number of Floating Users that may use a component of the Licensed Software concurrently at any time is limited by the number of Floating User Licenses purchased for such use, regardless of whether such use is by persons or computer systems. 5. Licensee may modify the Licensed Software except for altering or removing any details of ownership, copyright, trademark or other property rights connected with the Licensed Software. 6. Licensee may not distribute the Licensed Software, modified or unmodified, separately or as part of any software package or other product or service. 7. Upon expiry of the initially agreed License Term, the respective License Terms shall be automatically extended to one or more extensions of previous License Term as agreed between the Parties ("Renewal Term"), unless and until either Party notifies the other Party in writing, or any other method acceptable to froglogic, that it does not wish to continue the License Term, such notification to be provided to the other Party no less than thirty (30) days before expiry of the respective License Term. Unless otherwise agreed between the Parties, Renewal Term shall be of equal length with the initial License Term. Any such Renewal Term shall be subject to License Fees agreed between the Parties or, if no advance agreement exists, subject to froglogic standard pricing applicable at the commencement date of any such Renewal Term. Squish Coco 5.1.0 - 341 - froglogic GmbH License Agreement Any price or other term specified for a Renewal Term shall be valid only for the specified time. PRIVACY 8. Licensee acknowledges and agrees that for the purpose of this agreement, froglogic may collect, use, transfer and disclose personal data pertaining to Users as well as any other employees and directors of the Licensee and its contractors relevant for carrying out the intent of this agreement. Such personal data may be collected from the Licensee or directly from the relevant individuals. The parties acknowledge that with regard to such personal data processed hereunder, froglogic shall be regarded as the Data Controller under the applicable data protection legislation. froglogic shall process any such personal data in accordance with its privacy policies and practices, which will comply with all applicable requirements of the General Data Protection Regulation (GDPR) and any national implementing laws and regulations. WARRANTY DISCLAIMER 9. The Licensed Software is licensed to Licensee "as is". To the maximum extent permitted by applicable law, froglogic on behalf of itself and its suppliers, disclaims all warranties and conditions, either expressed or implied, including, but not limited to, implied warranties of merchantability, fitness for a particular purpose, title and non-infringement with regard to the Licensed Software. LIMITATION OF LIABILITY 10. If, froglogic’s warranty disclaimer notwithstanding, froglogic is held liable to Licensee, whether in contract, tort or any other legal theory, based on the Licensed Software, froglogic’s entire liability to Licensee and Licensee’s exclusive remedy shall be, at froglogic’s option, either (A) return of the price Licensee paid for the Licensed Software, or (B) repair or replacement of the Licensed Software, provided Licensee returns to froglogic all copies of the Licensed Software as originally delivered to Licensee. froglogic shall not under any circumstances be liable to Licensee based on failure of the Licensed Software if the failure resulted, wholly or in part, from accident, abuse or misapplication, nor shall froglogic under any circumstances be liable for special damages, punitive or exemplary damages, damages for loss of profits or interruption of business or for loss or corruption of data. Any award of damages from froglogic to Licensee shall not exceed the total amount Licensee has paid to froglogic in connection with this Agreement. SUPPORT AND UPDATES 11. Licensee will be eligible to receive email based software support and access to Updates to the Licensed Software for the License Term, in accordance with froglogic’s then current policies and procedures, if any. Such policies and procedures may be changed from time to time. Unless the Licensee and froglogic renew or extend the support Squish Coco 5.1.0 - 342 - froglogic GmbH License Agreement and update agreement, the expiration of the original support and update term has the following effect: froglogic will not provide any support and update services; froglogic will not provide email support; loss of the Licensed Software is entirely at Licensee’s risk, whether resulting from failure of hardware, media or backups or caused by other acts or omissions on Licensee’s part; froglogic will not assist with, and will not approve and activate, reassignments of users or computers; and froglogic will not assist with the recovery of lost downloads of software or with late license activations. These consequences are mere illustrations; Licensee acknowledges that none of froglogic’s obligations for support and updates continue after the expiration of a support and update agreement. GENERAL PROVISIONS 12. This Agreement may only be modified in writing signed by authorized representatives of Licensee and froglogic. In case of a conflict between this Agreement and the terms of any purchase order or other ordering document, this Agreement shall prevail. If any provision of this Agreement is found void or unenforceable, the remainder will remain valid and enforceable according to its terms. If any remedy provided is determined to have failed for its essential purpose, all limitations of liability and exclusions of damages set forth in this Agreement shall remain in effect. 13. This Agreement shall be construed, interpreted and governed by the jurisdiction and venue for the resolution of disputes resulting from, or in any way related to, this Agreement to be Hamburg, Germany. froglogic reserves all rights not specifically granted in this Agreement. Squish Coco 5.1.0 - 343 - froglogic GmbH Copyright Acknowledgements Appendix Copyright Acknowledgements C • Some of the icons in the CoverageBrowser are from the Tango Icon Library, which is in public domain. They are icons derived from Tango-Materia by Marco Sordi and Unofficial Tango by Jones Lee. • CoverageScannerTcl uses the Win32 LD_PRELOAD library. It has the following BSD 2.0 license: Copyright © 2001 Steven Engelhardt All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions, and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions, and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE AUTHOR ‘‘AS IS’’ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. • The Unix installer uses the makeself script. Its license is GPL v2. • The example program in the tutorial (see Chapter3, page 11) uses the CppUnit testing framework. Its license is LGPL V2.1. • The parser itself is the C++ expression parser by Jos de Jong. It has an Apache license v2.0. • The FindQWT script is used in the Squish Coco build process. It has the following simplified BSD license: Squish Coco 5.1.0 - 344 - froglogic GmbH Copyright Acknowledgements Copyright 2010-2013, Julien Schueller All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of the FreeBSD Project. • CoverageScannerTcl uses mach_override by Wolf Rentzsch. It has a MIT license. • CoverageScannerTcl uses the libudis86 disassembler library by Vivek Thampi. It has the following Simplified BSD License: Copyright (c) 2002-2012, Vivek Thampi Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Squish Coco 5.1.0 - 345 - froglogic GmbH Copyright Acknowledgements • CoverageScannerTcl uses Tcl/Tk, which uses the Tcl/TK Licensing Terms. • The CoverageScanner uses the library ms_dirent library by Kevlin Henney. It has a MIT-style license: Copyright Kevlin Henney, 1997, 2003. All rights reserved. Permission to use, copy, modify, and distribute this software and its documentation for any purpose is hereby granted without fee, provided that this copyright and permissions notice appear in all copies and derivatives. This software is supplied "as is" without express or implied warranty. Squish Coco 5.1.0 - 346 - froglogic GmbH Appendix Index C --cs-annotation-warnings, 79 --cs-function, 74 --cs-architecture, 83, 232, 301, 303 --cs-function-profiler, 26, 75 --cs-basic-blocks, 325 --cs-function-profiler=all, 84 --cs-boost, 75 --cs-function-profiler=option, 151 --cs-branch, 325 --cs-function-profiler=skip-trivial, 84 --cs-c-ext, 83, 333 --cs-hit, 73, 84, 137, 143 --cs-combine-switch-cases, 75, 84, 325 --cs-include, 72 --cs-compilation-warnings, 79 --cs-include-code-regexp, 115 --cs-compiler, 71, 72, 83 --cs-include-code-wildcard, 115 --cs-compiler-option, 81 --cs-include-file-abs-regex, 77 --cs-condition, 73, 74, 84 --cs-include-file-abs-wildcard, 77 --cs-count, 32, 73, 84, 137, 143 --cs-include-file-regex, 78 --cs-counter-size, 76, 324 --cs-include-file-wildcard, 77 --cs-coverage-save-function, 81, 82, 317 --cs-include-function-regex, 78 --cs-cpp-ext, 83, 333 --cs-include-function-wildcard, 78 --cs-csharp-dynamic, 81, 84 --cs-include-init-scripts, 115 --cs-custom-library-source, 81, 82, 317 --cs-include-path, 37, 77, 115, 331 --cs-decision, 73, 74, 84, 143, 325 --cs-include-unnamed, 115 --cs-disable-coveragescanner-library-abs-wildcard, --cs-keep-instrumentation-files, 79, 84, 290 81 --cs-libgen, 81, 232, 234, 307, 317 --cs-dotnet-exit-handler, 82, 84 --cs-library-after, 81 --cs-dump-on-event, 80, 84, 194, 195, 334 --cs-library-at-end, 81 --cs-dump-on-signal, 80, 194, 334 --cs-library-at-start, 81 --cs-enable-coveragescanner-library-abs-wildcard, --cs-line, 74, 148 81 --cs-link-instrumentation-tables, 82, 334 --cs-exclude, 72 --cs-lock-csexe, 80, 84 --cs-exclude-code, 120 --cs-mcc, 73, 74, 84, 324 --cs-exclude-code-regexp, 115, 116 --cs-mcc-max, 76 --cs-exclude-code-wildcard, 115 --cs-mcdc, 74, 324 --cs-exclude-file-abs-regex, 77 --cs-mcdc-max, 76 --cs-exclude-file-abs-regexp, 115 --cs-memory-alloc-failure-function, 82, 317 --cs-exclude-file-abs-wildcard, 32, 77, 115 --cs-memory-pool, 82 --cs-exclude-file-regex, 78, 185 --cs-memory-pool-code, 82 --cs-exclude-file-wildcard, 19, 77 --cs-minimal-api, 82, 84 --cs-exclude-function-regex, 78 --cs-native-toolchain, 83, 334 --cs-exclude-function-wildcard, 78 --cs-no-annotations, 76 --cs-exclude-path, 19, 77, 115, 331 --cs-no-assignments, 74, 84 --cs-execution-time, 74 --cs-no-boost, 75, 337 --cs-exit-function, 76 --cs-no-csharp-dynamic, 54, 81, 84 --cs-file-name-format-specifier, 80 --cs-no-cspch, 83 --cs-file-name-lower-case, 83 --cs-no-exceptions, 74, 84 --cs-full-instrumentation, 32, 74, 75, 84 --cs-no-execution-time, 74 347 INDEX --cs-no-exit-handler-installation, 82 __coveragescanner_set_custom_io(), 91 --cs-no-file-name-format-specifier, 80 __coveragescanner_testname(), 87, 189, 250, 252, 254, --cs-no-function, 74 261, 269 --cs-no-function-profiler, 74 __coveragescanner_teststate(), 87, 189, 250, 252, 255, --cs-no-line, 74 261, 269 --cs-no-line-directive, 76, 84, 323 __coveragescanner_unregister_library(), 90, 185, 186, --cs-no-pch, 83, 312 305 --cs-no-purecov-annotations, 76 __pragma, 98 --cs-no-qt3, 76, 337 Condition Coverage, 53 --cs-no-qt4, 32, 75, 337 --cs-no-returns, 75 add(), 29 --cs-no-squish, 83 Apple Xcode, 238, 239 --cs-nolock-csexe, 80, 84 ar, 237, 293, 301 --cs-off, 73, 84 Arm DS, 222, 224, 226 --cs-on, 14, 18, 63, 71, 73, 84, 183, 213, 227, 230, 231, 234, ARM Keil uVision, 220, 323 236, 238, 240, 242, 243, 245, 302, 313 arm-linux-gcc, 191 --cs-option-file, 83, 333 --cs-output, 32, 79, 84, 115, 116, 120, 305, 323, 330, 331 bash,9, 12, 13 --cs-output-abs, 337 benchmark, 300 --cs-parallel-build, 83 bin, 102 --cs-partial-instrumentation, 74, 75, 84, 143 Branch Coverage, 52 --cs-pointer-type, 81 Build Environment Selection, 212, 220, 222, 226, 317, 325, 329, --cs-profile, 72, 83, 333 330 --cs-profiler, 23 build_cpp, 195 --cs-qt3, 75, 218 build_cs, 195 --cs-qt4, 75, 218 Button.qml, 110 --cs-record-empty-functions, 78 --cs-register-instrumentation-tables, 82 C,3–5, 32, 55, 71, 82, 86, 96, 98, 134, 136, 149, 177, 193, 212, --cs-squish, 83 217, 230, 231, 234–236, 238, 269, 293, 303 --cs-statement-block, 73, 74, 84, 143, 148 C++,3–5, 11, 12, 15, 32, 51, 66, 67, 71, 75, 76, 82, 86, 96, 98, --cs-trigger-function, 80, 81, 317 134, 136, 149, 160, 177, 195, 212, 217, 230, 231, --cs-verbose, 79, 334, 338 234–236, 238, 267, 269, 293, 303, 319–321, 323, --cs-verbose-file, 79 330 --cs-verbose-source-lines, 79 C#,3–5, 10, 54, 66, 67, 71, 73, 75, 76, 81, 82, 92, 95, 99, --cs-verbose=api, 84 177, 180, 195, 230, 267, 292, 306, 309, 319–321, --cs-verbose=build, 84 323–328, 330, 331 --cs-vs2010-lambda, 76, 332 Cache, 156 --cs-warnings, 79 cl, 292 .NET Core, 82, 177, 317, 320 cl.exe, 70, 288 #endregion, 99 clang, 86, 87 #pragma, 98 CMake, 213 #region, 99 CMakeLists.txt, 213, 214 _Pragma, 98 cmcsexeimport, 122, 216, 286 __COVERAGESCANNER__, 86, 92, 303 cmmerge, 160 __coveragescanner_add_html_comment(), 88, 93, 189 Cobertura, 154, 170, 171, 331 __coveragescanner_clear(), 88, 89, 250, 252, 254, 261, cocolic, 285 269 COCOQML, 109, 111 __coveragescanner_clear_html_comment(), 88 cocoqml.txt, 110 _ __coveragescanner_filename(), 89, 186, 187 COCOQML BLACKLIST, 110, 111 _ __coveragescanner_get_filename(), 89 COCOQML CACHE, 111 _ _ __coveragescanner_install(), 82, 86, 87, 90, 94, 131, 250, COCOQML DEFAULT BLACKLIST, 112 252, 254, 261, 269, 311, 312 COCOQML_DUMPLOC, 112 _ __coveragescanner_memory_pool_stat(), 93 COCOQML LOGFILE, 105, 112 _ __coveragescanner_register_library(), 89, 90, 185, COCOQML OUTPUT, 111 186, 305 COCOQML_QUIET, 105, 112 _ __coveragescanner_register_squish(), 90 COCOQML VERBOSE, 105, 109, 111, 112 __coveragescanner_save(), 86–88, 91, 186, 187, 250, 252, cocoqmlscanner, 102–104, 107, 108, 112 255, 261, 269, 311, 312 cocoqmlscanner.exe, 108 Squish Coco 5.1.0 - 348 - froglogic GmbH INDEX cocoqmlscanner_result.csmes, 103 Rename Sources. . . , 25 Code coverage count, 63, 137, 143 File Code coverage hit, 63, 143 Execution Comparison Analysis, 27 Color Generate Black-Box Configuration. . . , 44, 127 Blue, 138 Import Reviewer Comments. . . , 151 Gray, 138 Import Unit Tests. . . , 40, 150, 190 Magenta, 138 Load Execution Report. . . , 34, 131, 162, 181, 195 Orange, 137 Merge with. . . , 45, 150, 160 Red, 137, 138 Open. . . , 33 Comments, 138, 139, 155 Save, 13, 17, 25 cov-assignment-off Instrumentation pragma, 99 Add/Set Comment, 138 region, 100 Clear Comment, 139 cov-assignment-on Coverage Method, 143 pragma, 98 Level:x, 143 region, 100 Reports cov-condition Export JUnit Report. . . , 154 pragma, 98 Export Statistics in Cobertura-XML Format. . . , 154 region, 99 Export Statistics in EMMA-XML Format. . . , 153 cov-count Generate HTML/CSV Report. . . , 153 pragma, 98 Generate Text Report. . . , 154 region, 99 Tools cov-decision Analysis of Modified Methods, 43, 150 pragma, 98 Analysis on Identical Methods, 150 region, 99 Compare with. . . , 25, 43, 150 cov-full-instrumentation Execution Comparison Analysis, 24, 42 pragma, 98 Patch File Analysis. . . , 20, 148 region, 99 Switch databases, 150 cov-hit Test Coverage Count Mode, 143 pragma, 98 View, 22 region, 99 Bug Location. . . , 148 cov-off Function Profiler, 23 pragma, 98 Functions, 135 region, 99 New Source Window, 136 cov-on Optimized Execution Order. . . , 144 pragma, 98 Sources, 134 region, 99 coveragescanner, 191, 218 cov-partial-instrumentation CoverageScanner (C# class) pragma, 98 __coveragescanner_add_html_comment, 93 region, 99 __coveragescanner_clear_html_comment, 93 cov-return-off __coveragescanner_clear, 94 pragma, 99 __coveragescanner_filename, 94 cov-return-on __coveragescanner_init, 92, 330 pragma, 99 __coveragescanner_memory_pool_stat, 93, 208 cov-statement-block __coveragescanner_save, 93–95 pragma, 98 __coveragescanner_set_custom_io, 94, 95 region, 99 __coveragescanner_testname, 92 cov-switch-case-off __coveragescanner_teststate, 92 pragma, 99 CoverageScanner() directive, 98–100 region, 100 COVERAGESCANNER_ARGS, 14, 18, 70–72, 183 cov-switch-case-on COVERAGESCANNER_COVERAGE_BASIC_BLOCK, 84 pragma, 99 COVERAGESCANNER_COVERAGE_COMBINE_SWITCH_CASES, 84 region, 100 COVERAGESCANNER_COVERAGE_CONDITION, 84 coveragebrowser, 12 COVERAGESCANNER_COVERAGE_COUNT, 84 CoverageBrowser Menu COVERAGESCANNER_COVERAGE_DECISION, 84 Context Menu COVERAGESCANNER_COVERAGE_FULL_INSTRUMENTATION, 84 Add/Set Comment, 138 COVERAGESCANNER_COVERAGE_HIT, 84 Clear Comments, 139 COVERAGESCANNER_COVERAGE_LOCK_CSEXE, 84 Squish Coco 5.1.0 - 349 - froglogic GmbH INDEX COVERAGESCANNER_COVERAGE_MCC, 84 g++, 70, 293 COVERAGESCANNER_COVERAGE_NO_ASSIGNMENTS, 84 gcc, 86, 87, 191, 293 COVERAGESCANNER_COVERAGE_NO_EXCEPTIONS, 84 get_id(), 28 COVERAGESCANNER_COVERAGE_NOLOCK_CSEXE, 84 git, 209, 210 COVERAGESCANNER_COVERAGE_OFF, 84 gmcs, 292 COVERAGESCANNER_COVERAGE_ON, 84, 180, 181, 230 Green Hills MULTI, 226 COVERAGESCANNER_COVERAGE_PARTIAL_INSTRUMENTATION, 84 HTML,5, 42, 46, 57, 88, 93, 118, 120, 123, 148, 149, 153, 164, COVERAGESCANNER_CSHARP_DYNAMIC, 84 167, 181, 182, 189, 190, 280, 282, 328, 329, 333, COVERAGESCANNER_DOTNET_EXIT_HANDLER, 84 337, 338 COVERAGESCANNER_DUMP_ON_EVENT, 84 COVERAGESCANNER_KEEP_INSTRUMENTATION_FILES, 84 IAR, 241 COVERAGESCANNER_MINIMAL_API, 84 icc, 293 COVERAGESCANNER_NO_CSHARP_DYNAMIC, 84 icl, 293 COVERAGESCANNER_NO_LINE_DIRECTIVE, 84 Icons COVERAGESCANNER_OUTPUT_ABS, 84 ..., 130, 140 COVERAGESCANNER_PROFILER_ALL, 84 , 139 COVERAGESCANNER_PROFILER_SKIP_TRIVIAL, 84 , 139 COVERAGESCANNER_RUNTIME_LOG, 84 , 139 COVERAGESCANNER_VERBOSE, 84 , 139 coveragescannertcl, 118 , 139 csar, 237, 293 , 137, 139 csc, 292 , 139 cscl, 292 , 139 cscl.exe, 288 , 137, 138 cscsc, 292 , 139 csdmcs, 292 , 137, 139 csg++, 293 , 131 csgcc, 293 , 130 csgmcs, 292 ! , 141 .* csicc, 293 , 141 ?* csicl, 293 , 141 csicpc, 293 , 129, 135, 136 cslib, 292 , 137 cslink, 292 , 137 csmcs, 292 , 137 CSV, 148, 153, 164, 167, 169, 328, 329 , 137 cvs, 210 , 137 CygWin, 212, 329, 335 , 136, 137 , 135, 137 Decision Coverage, 52 , 137, 139 diff, 20, 43, 209 ok , 137 CPP dmcs, 292 C++ , 136, 137 Dockerfile, 276 , 137 dump_on_event, 195, 196 , 137 dump_on_event.cpp, 195 , 137 dump_on_event_cs, 195 , 137 , 136, 137 Eclipse, 236, 237 , 135, 137 eLOC, 66 , 129 EMMA, 153, 170, 285, 286, 324, 330, 332, 333, 336, 338 1.2.3. , 143 - event_sample_cpp, 195 + , 130, 133 event_sample_cpp.cpp, 195 ok , 137 event_sample_cpp.exe.csexe, 195 Open, 20 event_sample_cpp.exe.csmes, 195 icpc, 293 Execution Report, 131 exit(), 76 Jenkins, 153, 154, 284–286 JUnit, 154, 170 Squish Coco 5.1.0 - 350 - froglogic GmbH INDEX kill, 194 Ctrl+Shift+B, 137 Ctrl+Shift+D, 137 ld, 301 Ctrl+Shift+F, 135, 137 lib, 292 Ctrl+Shift+I, 137 link, 292 Ctrl+Shift+J, 137 Linux,3,9, 31, 33, 107, 118, 148, 175, 176, 192, 198, 200, 208, Ctrl+Shift+K, 137 216, 239, 284, 285, 289, 320, 335 Ctrl+Shift+T, 137 Ctrl+Shift+U, 137 macOS,3,9, 11, 31, 33, 108, 109, 175, 176, 183, 198, 208, 216, Ctrl+Shift+V, 137 284, 285, 289, 326, 330, 331 Ctrl+Shift+Z, 137, 139 main(), 19 Ctrl+T, 137 make, 14, 18, 71, 212, 217 Ctrl+U, 137, 139 malloc(), 82 Ctrl+V, 137 McCabe, 66–68 Ctrl+Wheel, 137 mcs, 292 Ctrl+Z, 137, 139 Meta-Object Compiler (moc),4, 32, 46, 75, 193, 218 Shift+Wheel, 137 Microsoft Visual Studio,3, 10, 15, 33, 54, 76, 81, 86, 87, 177, Wheel, 137 178, 180, 181, 184, 187, 213, 214, 219, 230–232, SonarQube, 154, 171 267, 288, 292, 305, 318, 319, 321, 322, 326, 328, SQUISHCOCO_TEMP_DIR, 73, 290 330–336, 338 Statement Block Coverage, 51 Microsoft Visual Studio .NET, 230, 292 Status Microsoft Visual Studio Add-In, 317, 319, 321, 322, 324, 326, Dead-Code, 138 328, 330, 334, 335, 338 Executed, 137 Microsoft Windows,3,9, 11, 12, 15, 18, 20, 31, 33, 70, 77, 80, Execution count too low, 138 83, 89, 90, 103, 108, 109, 159, 176, 177, 190–192, Hidden, 138 194, 196, 198, 200, 208, 209, 216, 217, 284, 285, Manually Set To Be Executed, 138 288–290, 313–315, 318, 320, 323, 334 Never Executed, 138 Microsoft Windows CE, 232 Partially Executed, 137 Minimum Comment Size, 155 Unknown, 138 moc,4, 32, 46, 75, 193, 218 su, 326 Mono C#, 184, 292 sudo, 326 MSBuild, 177, 184, 317, 320 SystemC,3, 323 nmake, 33, 217 TASKING, 243, 246 Tcl,3–5, 117, 118, 331 pop tclsh, 117, 118 pragma, 98, 99 tclsh8.5, 118 Test coverage count, 63, 137, 143 qbs, 219 Thresholds, 155 qmake, 31–33, 37–39, 102, 107, 108, 193, 217, 218 toupper(), 28, 29 QmlJsCoverage/, 103 trackerplugin, 102 qrc, 32, 46 Qt Build Suite (qbs), 219 uic, 32, 46 Qt Framework,4, 102, 106, 216, 218 UNIX, 11, 14, 15, 20, 70, 89, 90, 102, 108, 109, 118, 175, 183, 194, 209, 217, 237, 275, 290, 315, 326, 334, 336 Reload, 155 Variablelist::add(), 28 SCons, 219 _ Shortcut Variablelist::get id(), 28 Ctrl+B, 137, 139 Visual Studio Coco Wizard,3, 177, 180 Ctrl+D, 137 VisualDSP, 228, 329 Ctrl+F, 135, 137 VxWorks, 239 Ctrl+I, 137, 139 Wind River, 239, 240 Ctrl+J, 137, 139 Ctrl+K, 137 XBuild, 184 Ctrl+L, 139 XML, 120, 153, 154, 164, 170, 171, 285, 286 Ctrl+M, 139 xUnit, 262, 265 Ctrl+R, 139 Ctrl+S, 139 Squish Coco 5.1.0 - 351 - froglogic GmbH