Squish Coco 3.3.2 - Copyright ©2015 froglogic GmbH CONTENTS
Contents
1 Introduction 1 1.1 Squish Coco - Code Coverage Tool for Tcl, C# and C/C++ ...... 1 1.2 CoverageScanner—Instrumentation during the Generation ...... 2 1.3 CoverageBrowser—View, Analyse, and Manage, Code Coverage Results ...... 2
I Quick Start and Tutorials 4
2 Synopsis 5
3 Using Squish Coco 6
4 Creating an instrumented project 7 4.1 Installing Squish Coco ...... 7 4.2 C++ on Microsoft Visual Studio using the Microsoft Visual Studio Add-in ...... 7 4.3 C# on Microsoft Visual Studio ...... 8 4.4 Tcl...... 9 4.4.1 Using more than one Tcl version on one system ...... 10 4.5 Command Line Tools ...... 10
5 Generating Instrumentations Without Modifying Projects 12 5.1 GNU Make ...... 12 5.2 Microsoft NMake ...... 12 5.3 Microsoft Visual Studio ...... 13 5.4 Microsoft MSBuild ...... 13 5.5 Mono C# XBuild ...... 13
6 Instrumenting a simple project 14 6.1 UNIX and Apple Mac OS X setup ...... 14 6.1.1 Setup ...... 14
6.1.2 Structure of the parser directories ...... 15 6.1.3 Compiling and testing ...... 15 6.1.4 Instrumentation ...... 15
- i - froglogic GmbH CONTENTS
6.1.5 How the project is instrumented ...... 16 6.1.6 Additional changes ...... 17 6.2 Microsoft Windows setup ...... 17 6.2.1 Setup ...... 17
6.2.2 Structure of the parser directories ...... 17 6.2.3 Compiling and testing ...... 18 6.2.4 Instrumentation ...... 18 6.2.5 How the project is instrumented ...... 19 6.2.6 Additional changes ...... 20 6.3 Beyond the minimal instrumentation ...... 20 6.3.1 Excluding code from instrumentation ...... 20 6.3.2 Making the test names visible ...... 21 6.3.3 Patch file analysis ...... 21 6.3.4 The patch analysis report ...... 22
7 Getting started with Qt 23 7.1 Compiling the example application ...... 23 7.1.1 The First Code Coverage Results ...... 25 7.1.2 Interactive testing ...... 26 7.1.3 Writing unit tests ...... 28 7.2 Working with code coverage data ...... 32 7.2.1 Post mortem analysis ...... 33 7.2.2 Evaluating the impact of a hot fix ...... 34 7.2.3 Black-box testing/distributed testing ...... 35 7.2.4 Verifying if a bug fix is correctly tested ...... 36 7.3 Conclusion ...... 37
II Code Coverage Overview 38
8 Code Coverage Overview 39 8.1 Code instrumentation ...... 39 8.1.1 Detection ...... 39 8.1.2 Result of coverage analysis ...... 42 8.2 Performance ...... 42 8.3 Statistics ...... 43 8.4 Other Metrics ...... 45 8.4.1 Function Coverage ...... 45
- ii - froglogic GmbH CONTENTS
8.4.2 Line Coverage ...... 45
9 Testing Methodologies 47 9.1 Hit vs Count ...... 47 9.2 Strategies ...... 47 9.2.1 Manual tests ...... 48 9.2.2 Black box tests ...... 48 9.2.3 Unit Tests ...... 48 9.2.4 Automatic Tests ...... 48
III CoverageBrowser Reference Manual 50
10 Introduction 51 10.1 Command Line Arguments ...... 51
11 Black box and white box testing 52
12 The windows of CoverageBrowser 53 12.1 The Executions Window ...... 53 12.1.1 Principles ...... 53 12.1.2 Loading an Execution Report ...... 55 12.1.3 The Execution Comparison Analysis Mode ...... 56 12.2 The Source Browser Window ...... 57 12.3 The Function Browser Window ...... 58 12.4 The Source Viewer Window ...... 58 12.4.1 Source Display ...... 59 12.4.2 Color Convention ...... 59 12.4.3 Comments ...... 60 12.4.3.1 Editing Comments ...... 60 12.4.3.2 Removing Comments ...... 61 12.5 The Explanation Window ...... 61 12.6 The Statistics Window ...... 62
13 Working with CoverageBrowser 63 13.1 Filtering ...... 63 13.1.1 Wildcard Expressions ...... 63 13.1.2 Regular Expression ...... 63 13.1.2.1 Pattern matching ...... 64
- iii - froglogic GmbH CONTENTS
13.1.2.2 String substitution ...... 64 13.2 Code Coverage Level ...... 65 13.3 Code Coverage Algorithm ...... 65 13.4 Optimized Execution Order ...... 65 13.5 Comparing Releases ...... 66 13.5.1 Reference Database ...... 66 13.5.2 Coverage analysis of modified/unmodified source code ...... 67 13.6 Changing the Instrumentation Database ...... 67 13.6.1 Merging Instrumentations ...... 67 13.6.2 Importing Unit Tests ...... 67 13.6.3 Importing Reviewer Comments ...... 67
14 Generation of Reports 68 14.1 EMMA-XML Report ...... 68 14.2 Cobertura-XML Report ...... 68 14.3 JUnit Report ...... 69 14.4 Statistics per Source File ...... 69 14.5 Statistics per Method ...... 70 14.6 HTML Report ...... 70 14.7 Text Report ...... 70 14.8 Patch Analysis ...... 70
15 Preferences 72 15.1 Save/Load Project ...... 72 15.2 Comments ...... 72 15.3 Thresholds ...... 72 15.4 Cache ...... 73
IV CoverageScanner Reference Manual 74
16 Introduction 75 16.1 Invoking CoverageScanner ...... 75
17 Command Line Arguments 77 17.1 Environment variables ...... 77 17.2 List of options ...... 78 17.2.1 File and path inclusion options ...... 78 17.2.2 Function inclusion options ...... 79
- iv - froglogic GmbH CONTENTS
17.2.3 Instrumentation options ...... 80 17.2.4 Debugging options ...... 81 17.2.5 Execution report options ...... 81 17.2.6 Miscellaneous options ...... 82 17.3 Instrumenting using preprocessor defines ...... 84 17.4 Regular Expressions ...... 84
18 Library calls 86 18.1 C and C++ Library ...... 86
18.1.1 __coveragescanner_install() ...... 86
18.1.2 __coveragescanner_testname() ...... 87
18.1.3 __coveragescanner_teststate() ...... 87
18.1.4 __coveragescanner_add_html_comment() ...... 87
18.1.5 __coveragescanner_clear_html_comment() ...... 88
18.1.6 __coveragescanner_save() ...... 88
18.1.7 __coveragescanner_clear() ...... 88
18.1.8 __coveragescanner_filename() ...... 88
18.1.9 __coveragescanner_register_library() ...... 89
18.1.10 __coveragescanner_unregister_library() ...... 89
18.1.11 __coveragescanner_set_custom_io() ...... 90 18.2 C# Library ...... 91
18.2.1 CoverageScanner.__coveragescanner_init() ...... 91
18.2.2 CoverageScanner.__coveragescanner_testname() ...... 91
18.2.3 CoverageScanner.__coveragescanner_teststate() ...... 91
18.2.4 CoverageScanner.__coveragescanner_add_html_comment() ...... 92
18.2.5 CoverageScanner.__coveragescanner_clear_html_comment() ...... 92
18.2.6 CoverageScanner.__coveragescanner_save() ...... 92
18.2.7 CoverageScanner.__coveragescanner_clear() ...... 92
18.2.8 CoverageScanner.__coveragescanner_filename() ...... 92
18.2.9 CoverageScanner.__coveragescanner_set_custom_io() ...... 92
19 Controlling the instrumentation during the compilation 94 19.1 Source Code Annotations ...... 94 19.1.1 Comments ...... 94 19.1.2 Manual Validation ...... 95 19.2 C and C++ Pragma ...... 95 19.3 C# regions ...... 96
- v - froglogic GmbH CONTENTS
20 Support for IDEs and toolkits 98 20.1 GNU Makefiles ...... 98 20.2 CygWin ...... 98 20.3 Scratchbox ...... 98 20.4 Kitware CMake ...... 99 20.4.1 Adding a new build type ...... 99 20.4.2 Microsoft Visual Studio ...... 99 20.4.3 Microsoft NMake ...... 100 20.4.4 GNU GCC ...... 100 20.5Qt...... 101 20.5.1 Qt Creator ...... 101
20.5.2 qmake ...... 102 20.5.3 moc ...... 103
20.6 SCons ...... 104 20.7 ARM Keil ...... 104 20.8 Green Hills ...... 106 20.8.1 Installation ...... 106 20.8.2 Command Line Tools ...... 106 20.8.3 CoverageScanner Library ...... 106 20.9 Visual DSP ...... 107 20.10Microsoft Visual Studio ...... 109 20.10.1 Microsoft Visual Studio .NET C# Compiler ...... 109 20.10.2 Microsoft Visual Studio .NET C++ Compiler ...... 109 20.10.3 Squish Coco Microsoft Visual Studio Add-In ...... 112 20.10.4 Microsoft Visual C++ Express ...... 113 20.10.5 Microsoft Visual Studio 6.0 ...... 114 20.11eMbedded Visual C++ ...... 115 20.12Eclipse ...... 116 20.13Xcode ...... 117
21 Code Coverage of Libraries 119 21.1 Code Coverage of Static/Shared Libraries and DLL ...... 119 21.2 Code Coverage of Plugins/Manually Loaded Shared Libraries ...... 119 21.2.1 Generating Code Coverage Information directly from the Main Application ...... 119 21.2.2 Generating Code Coverage Information directly from the Plugin ...... 120 21.2.2.1 Code Coverage of Plugins Generated with Microsoft Visual Studio ...... 120 21.2.2.2 Code Coverage of Plugins Generated with GNU gcc ...... 121
- vi - froglogic GmbH CONTENTS
22 Test suites and Squish Coco 123 22.1 Execution comment, name and status ...... 123 22.2 Controlling the generation of the execution report ...... 124 22.2.1 Generation of an execution report using UNIX signals ...... 124 22.2.2 Generation of an execution report using Microsoft Windows event ...... 124 22.2.3 Unit Testing ...... 125 22.3 Support for specific test frameworks ...... 126 22.3.1 CppUnit ...... 126 22.3.2 QTestLib ...... 128 22.3.3 GoogleTest ...... 130 22.3.4 CxxTest ...... 132 22.3.5 NUnit ...... 135 22.3.6 Squish ...... 136 22.3.6.1 General Approach ...... 136 22.3.6.2 Simplified for Reuse ...... 137
V CoverageScannerTcl Reference Manual 140
23 Command Line Arguments 141
24 Mixing Tcl with C++ 142 24.1 Analysis of Tcl script coverage ...... 142 24.1.1 Demo application build ...... 142 24.1.2 Ad-hoc Tcl coverage run ...... 142 24.1.3 Result analysis ...... 143 24.1.4 Filtered Tcl coverage run ...... 143 24.1.5 Report Generation ...... 143 24.2 Analysis of C/C++ code coverage ...... 144 24.2.1 Instrumentation ...... 144 24.2.2 Execution ...... 144 24.2.3 Result analysis ...... 144 24.2.4 Report Generation ...... 144 24.2.5 Analysis of mixed Tcl and C/C++ code coverage ...... 145 24.2.6 Combined coverage run ...... 145 24.2.7 Merging measurement data ...... 145 24.2.8 Report Generation ...... 145
- vii - froglogic GmbH CONTENTS
VI Command Line Tools 146
25 cmreport - Code coverage report generation 147 25.1 File selection options ...... 148 25.2 Execution selection options ...... 148 25.3 Options for HTML or CSV reports ...... 149 25.3.1 Obsolete options for CSV reports ...... 151 25.4 Options for text reports ...... 151
25.5 Options for EMMA-XML reports ...... 152 25.6 Options for JUnit reports ...... 152
25.7 Options for Cobertura reports ...... 152
26 cmcsexeimport 154
27 cmmerge - Merging Utility 155
28 cocolic - License activation 156 28.1 Location of the license ...... 157 28.1.1 Node-locked licenses ...... 157 28.1.2 Address of the license server ...... 157 28.1.3 The default file locations ...... 157
29 cocolicserver 158
VII Frequently Asked Questions 159
30 Installation 160 30.1 Is there a way to deploy Squish Coco on Windows without any user interactions? ...... 160 30.2 Microsoft Visual Studio Add-In is Not Installed ...... 160 30.3 Microsoft Visual Studio Add-In is Crashing when "My Documents" is on a Network Drive ...... 161
31 Compiling 163
31.1 Using CoverageScanner with ccache (Unix only) ...... 163 31.2 Compiling issue when using the Boost library ...... 163 31.3 Microsoft MSBuild Does Not Call CoverageScanner on x86 Platforms ...... 164
32 Instrumentation 165 32.1 Compiling Issues of Lambda Expressions with Microsoft VisualStudio 2010? ...... 165 32.2 How to exclude a source file from the code coverage analysis? ...... 165 32.3 My source code contains inline functions, are they instrumented? ...... 166
- viii - froglogic GmbH CONTENTS
32.4 Is it normal that the .csmes file contains a copy of the source code? ...... 166
33 CoverageBrowser 167 33.1 Is there a way to minimize the CPU usage? ...... 167 33.2 When I’m exporting the statistics per methods I can only see the 65536 first entries ...... 167
34 Unit Tests 168 34.1 How to get the code coverage of a library from my unit tests? ...... 168
35 Reports 170 35.1 Is is possible to compute a code coverage report of selection of files? ...... 170
36 Miscellaneous 171 36.1 The generation of an execution report through Windows events does not work when logged on an other account?171 36.2 Is it possible to debug CoverageScanner API? ...... 171 36.3 How to report an issue concerning CoverageScanner?...... 172
VIII Continuous Integration Server 173
37 Atlassian Bamboo integration 174 37.1 Introduction ...... 174 37.2 Agent configuration ...... 175 37.3 Importing an execution report ...... 175 37.4 Merging instrumentation databases ...... 175 37.5 Generating coverage report ...... 175 37.6 Use Case Examples ...... 176 37.6.1 Coverage Report from single execution ...... 176 37.6.2 Coverage Report for Unit Tests ...... 176
38 Jenkins CI integration 179 38.1 Prerequisites ...... 179 38.1.1 Getting the License ...... 179 38.1.2 Getting an EMMA plugin ...... 180 38.2 Adaption of the project to Jenkins ...... 180
IX Appendix 181
A Release Notes 182 A.1 Squish Coco v3.3.2 ...... 182
- ix - froglogic GmbH CONTENTS
A.2 Squish Coco v3.3.1 ...... 182 A.3 Squish Coco v3.3.0 ...... 183 A.4 Squish Coco v3.2.3 ...... 183 A.5 Squish Coco v3.2.2 ...... 183 A.6 Squish Coco v3.2.1 ...... 184 A.7 Squish Coco v3.2.0 ...... 184 A.8 Squish Coco v3.1.0 ...... 185 A.9 Squish Coco v3.0.2 ...... 185 A.10 Squish Coco v3.0.1 ...... 186 A.11 Squish Coco v3.0.0 ...... 186 A.12 Squish Coco v3.0.0-pre1 ...... 187 A.13 Squish Coco v2.1.8 ...... 187 A.14 Squish Coco v2.1.7 ...... 188 A.15 Squish Coco v2.1.6 ...... 189 A.16 Squish Coco v2.1.5 ...... 189 A.17 Squish Coco v2.1.4 ...... 190 A.18 Squish Coco v2.1.3 ...... 190 A.19 Squish Coco v2.1.2 ...... 190 A.20 Squish Coco v2.1.1 ...... 191 A.21 Squish Coco v2.1.0 ...... 191 A.22 Squish Coco v2.0.3 ...... 192 A.23 Squish Coco v2.0.2 ...... 192 A.24 Squish Coco v2.0.1 ...... 192 A.25 Squish Coco v2.0.0 ...... 192
B File system structure 194 B.1 Location of the license ...... 194 B.2 Where are the settings stored? ...... 194
C Code insertion 195
D Code Coverage Benchmarks 198 D.1 Test Algorithm ...... 198 D.2 Benchmarks ...... 199
E License Agreement 201
- x - froglogic GmbH CONTENTS
F Customizing I/O of CoverageScanner library 204 F.1 Custom I/O using C file access ...... 204 F.2 Custom I/O using SFTP protocol ...... 206
G Supported compilers 213 G.1 C# compilers ...... 213 G.2 Microsoft Visual C++ ...... 213 G.3 Intel C++ Compiler ...... 214 G.4 GNU gcc ...... 214
H CoverageScanner Adaptation to a Tool Suite 215 H.1 Profile Parameters for Architecture Settings ...... 216 H.2 Profile Parameters for Instrumentation Settings ...... 216 H.3 Profile Parameters for Preprocessor Settings ...... 216 H.4 Profile Parameters for Linker Settings ...... 217 H.5 Profile Parameters for Compiler Settings ...... 219 H.6 Profile Parameters for Precompiled Headers Support ...... 222 H.7 Profile Parameters for Custom IO ...... 223 H.8 Miscellaneous Profile Parameters ...... 223 H.9 Instrumentation Parameters ...... 225
Index 225
- xi - froglogic GmbH Introduction
Chapter Introduction 1
1.1 Squish Coco - Code Coverage Tool for Tcl, C# and C/C++
Squish Coco is a complete code coverage tool chain for Tcl, C# and C/C++ programs that runs on Apple® Mac OS X, Linux™ and Microsoft® Windows. It analyzes the performance of an application being exercised (e.g., by a test suite), to reveal important information that can improve the testing of Tcl, C# and C/C++ applications. In particular, Squish Coco can:
• Find untested code sections. • Find redundant tests which can then be eliminated. Squish Coco makes it possible to identify which portions of the source code are covered by a test, and to detect whether a new test covers source code lines that the existing tests don’t cover.
• Find dead code by displaying code that is never executed. • Calculate the optimum test execution order so as to maximize test coverage for each run. This is particularly useful for manual testing. • Analyse two separate versions of an application and compare the differences. This feature makes it possible to see which tests are affected by source code modifications and also to get some measure of the test coverage of a patch or hot fix.
Squish Coco can be used at every stage of testing and for all testing methodologies (unit tests, automatic tests, manual white box tests, black box tests, etc. . . ), and makes it possible to merge multiple execution reports into a single unified report. The Squish Coco package provides two separate tools and an add-in: 1. CoverageScanner—this analyzes, instruments and generates the Tcl, C# and C/C++ application. 2. CoverageBrowser—this displays and manages the results of the coverage analysis.
3. Microsoft® Visual Studio® Add-In—this is an optional add-in which makes it possible to generate code coverage configurations for every Tcl, C# and C/C++ project created by Microsoft® Visual Studio®.
- 1 - froglogic GmbH CoverageScanner—Instrumentation during the Generation
CoverageScanner CoverageBrowser Instrumentation Source Code C/C++ Preprocessing (.csmes file) (.cpp file)
Executions Instrumentation (.csexe file)
Application Compiling (.exe file)
Figure 1.1: Overview of Code Coverage Toolchain for Tcl, C# and C/C++
1.2 CoverageScanner—Instrumentation during the Generation
CoverageScanner instruments the source code seen by the compiler (leaving the original code untouched) and generates an instrumented executable, shared library, or plugin. The instrumentation keeps track of the code that is executed when the executable is run or when the library or plugin are used (e.g., how many times each line is executed, or simply whether a line was executed) and produces an execution report (.csexe file) upon program termination. In order to get the best quality of code coverage metrics, CoverageScanner instruments not only at the function and statement level, but also records statement coverage and decision coverage information. Optionally, it is also possible to insert the test name and its execution status (passed or failed) into the report using a batch script or directly in the application itself. This makes it possible to integrate CoverageScanner into a test framework (e.g., CppUnit or CxxTest), with the generation of the code coverage information for each test item. CoverageScanner is a command line tool which substitutes itself for the the compiler that is normally used to build the executable, library, or plugin. Its main function is to insert instrumentation instructions into the preprocessed source code and then to compile this modified code in place of the original code using the normal compiler. CoverageScanner works with almost any standard compiler, including GNU gcc, GNU g++, Microsoft® Visual Studio® 6.0, .NET, Express, and Embedded C++, Intel® C++, etc. A database (.csmes file), which contains the list of instrumentations and a copy of the instrumented source code, is generated at the same time for later analysis. CoverageScanner’s ability to work with most compilers is because it can be configured to suit most compilers’ needs.
After the code generation, the instrumented executable can be run as usual (or the instrumented library or plugin used via the program that accesses them)—this will result in an execution report being produced (in a .csexe file) when the program terminates. CoverageScanner has built-in support for Qt Library which makes it possible to prevent the instrumentation Info of code generated by the moc.
1.3 CoverageBrowser—View, Analyse, and Manage, Code Coverage Results
The execution report (in the .csexe file) can be analyzed and managed using the CoverageBrowser tool. This tool has a GUI (graphical user interface) through which the user can browse and manage their execution reports. This makes it straightforward to locate untested code, dead code, and inefficient tests. CoverageBrowser’s highlights:
- 2 - froglogic GmbH CoverageBrowser—View, Analyse, and Manage, Code Coverage Results
• Commenting instrumented source code lines. • Displaying the list of executions in a tree view. • Managing portions of code which cannot be tested by identifying them as “Manually Validated”. • Browsing through the instrumented code.
• Switching between code coverage condition and code coverage branch analysis. • Displaying a detailed explanation of the state of the instrumentation. This includes: – User comments. – The state (executed, not executed, partially executed). – The code coverage count. – The list of tests executing the instrumentation. • Retrieving the execution status from an automated test suite. • Code coverage analysis of unit tests.
• Exporting statistics to spreadsheets. • A test benefit analysis mode which makes it possible to see what additional code coverage a set of tests adds. • An API which makes it possible to control CoverageBrowser using an external Tcl, C# and C/C++application.
• A source code and method browser which displays code coverage statistics per file, per class, per namespace, and per Tcl, C# and C/C++ function. • Generation of a report in HTML format, which includes statistics per source file, methods, executions, and a list of unexecuted code lines. • Black box testing support.
• The ability to comparare coverage data for different versions of the software.
Navigation
Class, Namespace, and Method Browser
Source Navigation and Statistics
Detailed Explanation Manual Validation
Executions Management User Comments
Figure 1.2: CoverageBrowser: Code Coverage Screenshot
- 3 - froglogic GmbH Quick Start and Part I Tutorials
- 4 - froglogic GmbH Synopsis
Chapter Synopsis 2
Code coverage is commonly used in the development of safety critical applications in order to obtain a measure of how much of the code is tested. This approach is also used more widely, especially when the application quality is important. To be able to analyze the code coverage that a test suite achieves it is necessary to compile a version of the the application in which statements are inserted that monitor the execution of the source code. The generation of such a modified version of the program is called instrumentation. Several kinds of instrumentation are possible. The most common are:
Line coverage: Instrumenting the execution of every executable source code line.
Branch coverage: Instrumenting the execution of each branch’s block (e.g., the body of an if statement).
Decision coverage: Instrumenting each Boolean decision for loop and selection statements (e.g., record both the Boolean expression itself—true or false—and the body of the while, for or if statement).
Condition coverage: Instrumenting of each sub-expression of Boolean expressions (i.e., like decision coverage but for multi-part Boolean expressions recording every part).
- 5 - froglogic GmbH Using Squish Coco
Chapter Using Squish Coco 3
Squish Coco is a coverage tool for C and C++ code. In principle it does the same job as GNU gcov, but has some additional and different functionality. For example, Squish Coco: • works on release and debug builds. • uses generic code and is compiler independent.
• instruments branches, decisions, and conditions, whereas gcov is limited to line coverage. • provides a graphical code browser. • records an instrumentation report for each test run—these reports can be compared (e.g., to see if the coverage has improved), or merged.
• supports adding code coverage reports from unit tests into the main application’s instrumentation. • makes it possible to compare two versions of the application to see what effect the code changes have on code coverage. • supports black box testing.
• provides an extension to instrument Qt source code that omits the unnecessary Qt infrastructure. Squish Coco consists of two main tools:
CoverageScanner: This is a compiler wrapper which feeds instrumented source code to the native compiler. Using it is easy—simply tell the build tool to use CoverageScanner as the compiler.
CoverageBrowser: This is a GUI tool for visualizing and analysing code coverage based on the reports generated by running the test suite on an instrumented version of the application.
During the compilation, an instrumentation database is generated for each object, library, and executable. This database contains a copy of the source code, the instrumentation points, and once executed, all the test reports.
- 6 - froglogic GmbH Creating an instrumented project
Chapter Creating an instrumented project 4
This chapter contains recipes how a software project can be set up for instrumentation. The methods depend on the languages and the development environments. Here we show how to create a new software project so that it is prepared for instrumentation. This is also a simple way to get acquainted with Squish Coco. A description of a larger project can be found in Chapter 7, page 23.
4.1 Installing Squish Coco
Installing Squish Coco is straightforward: just download the installer via the links at http://www.froglogic.com/squish/ coco and execute it.
4.2 C++ on Microsoft® Visual Studio® using the Microsoft® Visual Studio® Add-In
There is an add-in for Microsoft® Visual Studio® (see Chapter 20.10.3, page 112) that supports the work with Squish Coco. The following description shows the setup of a project that uses this add-in. Start Microsoft® Visual Studio® and create a new C++ application:
1. Click on "File¡New¡Project..." to pop up the new project wizard.
2. Choose a project type of "Visual C++¡Win32" and the "Win32 Console Application" template.
3. Enter a project name of squishcoco_sample, then click the "OK" button.
4. When the wizard’s second page appears, click the "Finish" button. At this stage the application is not yet instrumented. So now we will create a copy of the build and activate the instrumentation:
1. Open the configuration manager by clicking "Build¡Configuration Manager...".
2. In the "Configuration" column, select "New..." in the combobox
3. In the "New Project Configuration" dialog: (a) Enter Code Coverage in the "Name" field,
- 7 - froglogic GmbH C# on Microsoft Visual Studio
(b) Select Release or Debug in the "Copy settings from" selection dialog. (c) Click the "OK" button. To activate the instrumentation, use the Microsoft® Visual Studio® Add-In:
1. Click "Tools¡Code Coverage Build Mode..." to pop up the Squish Coco wizard.
2. In the "Project:" selection dialog, select squishcoco_sample.
3. In the selection dialog "Configuration:", select Code Coverage.
4. In the Configuration section at the bottom, select the radio button "Modify", and then click on the button, "Enable code coverage for C++ projects".
The Code Coverage configuration has now been modified to generate code coverage information. The "SquishCoco" output window summarizes all the modifications that have been made:
... Modifying configuration ’Code Coverage’ for the project ’squishcoco_test’ for the platform ’Code Coverage|Win32’ Compiler Configuration Additional command line arguments ’--cs-on --cs-condition --cs-count --cs-partial-instrumentation’ are appended Linker Configuration Additional command line arguments ’--cs-on --cs-condition --cs-count --cs-partial-instrumentation --cs-libgen=/MD’ are appended ...
Build the squishcoco_sample project. This will cause the executable squishcoco_sample.exe to be built and the code coverage instrumentation file squishcoco_sample.exe.csmes to be generated. Double click on squishcoco_sample.exe. csmes to inspect this file in CoverageBrowser. Right now there is no code coverage statistics visible in CoverageBrowser: This is because the application has not yet been executed. Click on squishcoco_sample.cpp in the source list to display the main function. All the instrumented lines are shown grayed out, to indicate that nothing has been executed.
Now execute squishcoco_sample.exe by double clicking it. This will result in a file called squishcoco_sample.exe. csexe being generated. The file contains a code coverage snapshot which can be imported into CoverageBrowser:
1. Click "File¡Load Execution Report...".
2. Select the "File" item and enter the path of the squishcoco_sample.exe.csexe file.
3. Click on the "Import" button. This will cause the code coverage statistics to be updated. Now, in the source code window, the main function’s return statement will be colored green to indicate that this line has been executed.
4.3 C# on Microsoft® Visual Studio®
Start Microsoft® Visual Studio® and create a new C# application:
1. Click on "File¡New¡Project..." to pop up the new project wizard.
2. Choose a project type of "Visual C#¡Windows" and the "Console Application" template.
3. Enter a project name of squishcoco_sample, then click the "OK" button.
- 8 - froglogic GmbH Tcl
4. When the wizard’s second page appears, click the "Finish" button. To activate the instrumentation, use the project properties:
1. In the "Solution Explorer", double click "squishcoco_sample¡Properties" to open the property dialog.
2. Click on the "Build" tab.
3. Select "Conditional compilation symbols" and enter COVERAGESCANNER_COVERAGE_ON.
The code coverage analysis is enabled when the symbol COVERAGESCANNER_COVERAGE_ON is defined from the command line.
Build the squish_coco project. This will cause the executable squishcoco_sample.exe to be built and the code coverage instrumentation file squishcoco_sample.exe.csmes to be generated. Double click on squishcoco_sample.exe.csmes to inspect this file in CoverageBrowser. Right now there are no code coverage statistics to be seen in CoverageBrowser: this is because the application has not yet been executed. Click on Program.cs in the source list to display the main function. All the instrumented lines are shown grayed out to indicate that nothing has been executed.
Now execute squishcoco_sample.exe by double clicking it. This will result in a file called squishcoco_sample.exe. csexe being generated. This file contains a code coverage snapshot which can be imported into CoverageBrowser:
1. Click "File¡Load Execution Report...".
2. Select the "File" item and enter the path of the squishcoco_sample.exe.csexe file.
3. Click on the "Import" button. This will cause the code coverage statistics to be updated. Now, in the source code window, the main function’s return statement will be colored green to indicate that this line has been executed.
4.4 Tcl i At the moment, this example only works with Tcl v8.5 or earlier. See Section 4.4.1, page 10 below for details.
Create a file hello.tcl with the following content:
if { $argc == 0 } { puts "Hello World!" } else { for {set i 0} {$i<$argc} {incr i} { if { $i == 0 } { set hello "Hello [lindex $argv $i]" } elseif { $i == $argc - 1 } { set hello "$hello and [lindex $argv $i]" } else { set hello "$hello, [lindex $argv $i]" } } puts $hello }
Execute hello.tcl using the tclsh interpreter:
$ coveragescannertcl tclsh hello.tcl Jim John James
- 9 - froglogic GmbH Command Line Tools
Two files are generated: tclsh.csmes: The instrumentation database of hello.tcl tclsh.csexe: The execution report. To import the execution report into its instrumentation database, execute:
$ cmcsexeimport -m tclsh.csmes --title="Jim John James" tclsh.csexe
After the import, it is possible to analyze the code coverage data directly with CoverageBrowser or generate a HTML report:
$ cmreport --title="Hello application" -m tclsh.csmes --html=tclsh.html
4.4.1 Using more than one Tcl version on one system
Squish Coco v3.0 does not yet support the use of tclsh under Tcl v8.6. If you want to use coveragescannertcl with tclsh, Tcl v8.5 or earlier is necessary. In some UNIX®systems (like Debian Linux™), more than one Tcl version can be present at the same time. The executables are then distinguished by their names. On such a system, the tclsh executable of Tcl v8.5 may be called tclsh8.5.
We can then run CoverageScannerTcl such that it calls tclsh8.5. On its own, this would however change the names of the generated files into tclsh8.5.csmes and tclsh8.5.csexe. We can correct this with another commandline option, --cs-output=tclsh. The first command in the example above now becomes:
$ coveragescannertcl --cs-output=tclsh tclsh8.5 hello.tcl Jim John James
Afterwards, the rest of the example can be done as before.
4.5 Command Line Tools
Open a console window (MS-DOS Prompt or Command Prompt window on Windows) and make sure that the Microsoft® Visual Studio® compiler (cl.exe) or GCC is installed on your system.
Create a simple hello.c source file that contains the following code:
#include
int main(int argc, char *argv[]) { printf("hello\n"); return 0; }
Compile this program using the CoverageScanner compiler wrapper instead of the native compiler. This is done by prepending cs to the command line compiler’s name: With Microsoft® Visual Studio® With GCC $ cscl.exe hello.c /Fehello.exe $ csgcc hello.c -o hello.exe
- 10 - froglogic GmbH Command Line Tools
Then the executable hello.exe and the file hello.exe.csmes, which contains the code coverage instrumentation, will be generated.
Execute hello.exe; this will cause the file hello.exe.csexe to be generated. It contains a code coverage snapshot which can be imported into hello.exe.csmes using cmcsexeimport:
$ cmcsexeimport --title="Hello execution" -m hello.exe.csmes -e hello.exe.csexe
Once imported, the hello.exe.csexe file is no longer needed and can be deleted. Now hello.exe.csmes contains a single execution record. We can generate an HTML report with cmreport:
$ cmreport --title="Hello application" -m hello.exe.csmes --html=hello.html
Here is a summary of the command line options that were used in this command:
--title="Hello application": Set a title for the report. It appears on all generated HTML pages.
-m hello.exe.csmes: Select the instrumentation database.
–html=hello.html: Specify the name of the HTML file that the report should be written to.
- 11 - froglogic GmbH Generating Instrumentations Without Modifying Projects
Chapter Generating Instrumentations Without Modifying 5 Projects
Squish Coco can also generate code coverage information for a project without the need to modify it. The principle is to prepend to the PATH variable the path of the CoverageScanner compiler wrappers and to set the instrumentation parame- ters with the COVERAGESCANNER_ARGS environment variable. To activate the instrumentation, --cs-on must be present in COVERAGESCANNER_ARGS. If this is not the case, CoverageScanner is completely deactivated.
The variable COVERAGESCANNER_ARGS should only be set locally, e.g. in a script or on the command line. If it is ! set globally, it will influence every build.
5.1 GNU Make
Proceed as follows to instrument a project which can be generated using GNU Make:
export PATH=/opt/SquishCoco/wrapper/bin:$PATH export COVERAGESCANNER_ARGS=--cs-on make clean make
5.2 Microsoft® NMake
Proceed as follows to instrument a project which can be generated using NMake:
set PATH=%SQUISHCOCO%\visualstudio;%PATH% set COVERAGESCANNER_ARGS=--cs-on nmake clean nmake
- 12 - froglogic GmbH Microsoft Visual Studio
5.3 Microsoft® Visual Studio®
Proceed as follows to instrument a project which can be generated using Microsoft® Visual Studio®:
set PATH=%SQUISHCOCO%\visualstudio;%PATH% set COVERAGESCANNER_ARGS=--cs-on devenv /useenv myproject.sln /Rebuild
5.4 Microsoft® MSBuild
Proceed as follows to instrument a project which can be generated using Microsoft® MSBuild
set PATH=%SQUISHCOCO%\visualstudio;%PATH% set COVERAGESCANNER_ARGS=--cs-on msbuild /p:UseEnv=true myproject.sln /t:ReBuild
5.5 Mono C# XBuild
Proceed as follows to instrument a project which can be generated using Mono C# XBuild
export COVERAGESCANNER_ARGS=--cs-on msbuild \ /p:UseEnv=true \ /p:UseHostCompilerIfAvailable=true \ /p:CscToolPath="${SQUISHCOCO}/wrapper" \ myproject.sln /t:ReBuild
The environment variable SQUISHCOCO points to the location of the CoverageScanner executable.
- 13 - froglogic GmbH Instrumenting a simple project
Chapter Instrumenting a simple project 6
In this section we will show a small project with unit tests and show how it can be instrumented. The project is a simple expression parser, and it has not many requirements besides a C++ compiler. The project replicates in miniature an existing project that has been extend with unit tests and for which we will now use Squish Coco to find out how good the test coverage is. Instrumentation should therefore be non-intrusive and should not change the project very much. The procedures to set up a project for instrumentation under UNIX® and Apple® Mac OS X differ from that for i Microsoft® Windows. The setup section has therefore two versions, which appear in the following sections.
6.1 UNIX® and Apple® Mac OS X setup
6.1.1 Setup
The parser example can be found in Squish Coco’s installation directory. Under UNIX®, this is the directory /opt/SquishCoco/ or, if you have installed Squish Coco locally, the subdirectory SquishCoco/ of your home directory.1 Under Apple® Mac OS X, the installation directory is /Applications/SquishCoco/.
We will refer to it as the SquishCoco/ directory, wherever it is located. The Squish Coco examples, together with their supporting programs, are in SquishCoco/samples/, and the parser is in SquishCoco/samples/parser/. This directory contains three versions of the program, in the directories parser_v1/ to parser_v3/. They represent the parser in different stages of its development.
The example uses CppUnit as its unit test framework. There is a version of CppUnit in the SquishCoco/samples/ directory, and the parser example is prepared to use it.
Copy therefore now the content of the whole samples/ directory to your workspace and make parser_v1/ your working directory. If Squish Coco is installed in /opt/SquishCoco/, this is done in the following way:
$ cp -a /opt/SquishCoco/samples . $ cd samples/parser/parser_v1
Make also sure that the Squish Coco tools are in your search path. If they are not so already, you can now do it by writing
$ . cocopath.sh
1If you have chosen a custom directory to install Squish Coco, the example will not work without modifications.
- 14 - froglogic GmbH UNIX and Apple Mac OS X setup
(Don’t forget the dot at the beginning!) Now programs like coveragebrowser can be called from the command line.
6.1.2 Structure of the parser directories
We we will use samples/parser/parser_v1/ as our working directory. It contains C++ source files and header files, together with an unit test file, unittests.cpp. The makefile has been prepared for unit tests, but not for instrumentation. The instrumentation is done with the help of the bash script instrumented. (There are also some files that are needed in the Microsoft® Windows version. We will ignore them here.) The makefile is called gmake.mak to distinguish it from the Microsoft® Windows makefile, nmake.mak. Therefore i one has to write make -f gmake.mak in places where otherwise a make would be enough.
6.1.3 Compiling and testing
Run “make -f gmake.mak” to compile the program. It is a simple expression parser and calculator.
$ ./parser Enter an expression an press Enter to calculate the result. Enter an empty expression to quit.
> 2 + 2 Ans = 4 > Pi Ans = 3.14159 > sin(Pi) Ans = 1.22465e-16 > sinn(90) Error: Unknown function sinn (col 9) > sin(90) Ans = 0.893997 > cos(pi) Ans = -1 > $
We have added some unit tests for the main class, Parser. Look into the file unittests.cpp to see the tests that have been included. Execute it with make -f gmake.mak tests. You will see that eight tests have been executed.
6.1.4 Instrumentation
We have kept the instrumentation separate from the main project. The core of the instrumentation is a short shell script, instrumented. It is a simple wrapper, and calling “instrumented
$ make -f gmake.mak clean $ ./instrumented make -f gmake.mak tests
The first command removes all object files, since we need everything to be recompiled. The second command then compiles the program with instrumentation and runs the tests. That’s all!
We now have a look at what the script has done and how it has done it. List the contents of your parser directory:
- 15 - froglogic GmbH UNIX and Apple Mac OS X setup
$ ls constants.h functions.o.csmes parser.cpp unittests.o error.cpp instrumented parser.h unittests.o.csmes error.h LICENSE parser.o variablelist.cpp error.o main.cpp parser.o.csmes variablelist.h error.o.csmes main.o unittests variablelist.o functions.cpp main.o.csmes unittests.cpp variablelist.o.csmes functions.h Makefile unittests.csexe functions.o NOTICE unittests.csmes
You see two kinds of files that do not appear as result of normal compilation. The .csmes files contain the information that is needed for coverage measurement, and the .csexe files contain the results of code execution. The files that end in .o.csmes are temporary files and are only used during compilation.
This time, the only program that was actually executed was unittests, and therefore the only .csexe file is unittests.csexe. To see the coverage results, you can therefore start the CoverageBrowser with the command
$ coveragebrowser -m unittests.csmes -e unittests.csexe
Then the CoverageBrowser will start with a modal window, "Load Execution Result". Click on the "Import" button to load the data. By default, CoverageBrowser automatically deletes the .csexe file after it loads it. You can switch this behavior off by unselecting the "Delete after loading" checkbox. Or, if you don’t, select the "File¡Save" menu item ! to save the execution report in the unittests.csexe file.
For the use of CoverageBrowser, see Part III, page 50. We will now rather describe how the instrumentation is done.
6.1.5 How the project is instrumented
The file instrumented is a short bash script:
#!/bin/bash
. getcoco.sh # Get Coco variables
export PATH=$COCO_WRAPPER_DIR:$PATH export COVERAGESCANNER_ARGS=’--cs-on’
"$@"
At its beginning, the shell script getcoco.sh sets the shell variable COCO_WRAPPER_DIR. It contains the name of the directory in which Squish Coco is installed. Then there are two export statements, and the final cryptic statement executes the command line parameters of instrumented. So if you call ./instrumented make -f gmake.mak tests, the command make -f gmake.mak tests is executed by the script, but in a different environment than normally.
The important part of it are therefore the two export statements. In the first one, the search part is manipulated so that the programs in /opt/SquishCoco/wrapper/bin/ are searched first. This directory contains a lot of files with the same names as the compilers2 that are supported by Squish Coco:
$ ls /opt/SquishCoco/wrapper/bin ar g++-4.9 x86_64-linux-gnu-ar c89-gcc gcc x86_64-linux-gnu-g++
2There is also ar, which is not a compiler but takes part in the compilation process. This is the output under UNIX®; under Mac OS X, the directory /Applications/SquishCoco/wrapper/ contains a much smaller list of files.
- 16 - froglogic GmbH Microsoft Windows setup
c99-gcc gcc-4.6 x86_64-linux-gnu-g++-4.6 ...
These programs are the compiler wrappers. With the new PATH, they are executed instead of the real compilers. The compiler wrappers are actually symbolic links to a single program, coveragescanner (see Part IV, page 74). When executed to compile a source file, they create an instrumented version of the source and then run the original compiler to compile it.
In the second export statement, additional flags for the compiler wrappers (see Chapter 17, page 77) are set. Here we set only one option, --cs-on. If it is not present, the compiler wrappers are inactive and just call the compilers they represent.
6.1.6 Additional changes
It is also convenient to add make targets to handle the files generated by CoverageScanner. In the parser directory, the Makefile has been changed in the following way:
clean: testclean ... -$(DEL_FILE) *.o.csmes # (added)
distclean: clean ... -$(DEL_FILE) *.csmes *.csexe # (added)
Since the .o.csmes files are needed only for compilation, they can be deleted whenever the .o files are deleted (which is that what make clean does). The .csmes and .csexe files are more precious and should only be deleted when all generated files are removed. Therefore we have added their deletion statements to the distclean target.
6.2 Microsoft® Windows setup
6.2.1 Setup
The parser example can be found in the directory C:\Program Files\squishcoco\parser. This directory contains three versions of the program, in the subdirectories parser_v1 to parser_v3. They represent the parser in different stages of its development.
The example uses CppUnit as its unit test framework. There is a version of CppUnit in the C:\Program Files\squishcoco directory, and the parser example is prepared to use it. Since these directories are write-protected, you need to create your own working copies. Copy therefore the two directories C:\Program Files\squishcoco\parser and C:\Program Files\squishcoco\cppunit-1.12.1 to a directory of your choice. Then remove the write protection of the directories and all the files contained in them.
6.2.2 Structure of the parser directories
We will use parser\parser_v1 as our working directory. It contains C++ source files and header files, together with an unit test file, unittests.cpp. The Makefile has been prepared for unit tests, but not for instrumentation. The instrumentation is done with the help of the batch file instrumented.bat. (There are also some files that are needed in the UNIX®version. We will ignore them here.) The makefile is called nmake.mak to distinguish it from the UNIX® makefile, gmake.mak. Therefore one has to i write nmake /F gmake.mak in places where otherwise a nmake would be enough.
- 17 - froglogic GmbH Microsoft Windows setup
6.2.3 Compiling and testing
We will do the compilation of the example on the command line. To get a command window, execute the batch file CocoCmd. bat that is located in the parser_v1 directory. In this window, the Microsoft® Visual Studio® command line tools (like nmake) are accessible, and also the main Squish Coco programs (like CoverageBrowser).
Run “nmake /F nmake.mak” to compile the program. It is a simple expression parser and calculator.
$ C:\code\parser\parser_v1>parser.exe Enter an expression an press Enter to calculate the result. Enter an empty expression to quit.
> 2 + 2 Ans = 4 > Pi Ans = 3.14159 > sin(Pi) Ans = 1.22465e-16 > sinn(90) Error: Unknown function sinn (col 9) > sin(90) Ans = 0.893997 > cos(pi) Ans = -1 > C:\code\parser\parser_v1>
We have added some unit tests for the main class, Parser. Look into the file unittests.cpp to see the tests that have been included. Execute it with nmake /F nmake.mak tests. You will see that eight tests have been executed.
6.2.4 Instrumentation
We have kept the instrumentation separate from the main project. The core of the instrumentation is a short shell script, instrumented.bat. It is a simple wrapper, and calling “instrumented.bat
C:\code\parser\parser_v1>nmake /F nmake.mak clean C:\code\parser\parser_v1>instrumented.bat nmake /F nmake.mak tests
The first command removes all object files, since we need everything to be recompiled. The second command then compiles the program with instrumentation and runs the tests. That’s all!
We now have a look at what the script has done and how it has done it. List the contents of your parser directory:
C:\code\parser\parser_v1>dir /D
Directory of C:\code\parser\parser_v1
[.] LICENSE unittests.exe [..] main.cpp unittests.exe.csexe constants.h main.obj unittests.exe.csmes error.cpp main.obj.csmes unittests.exp error.h Makefile unittests.lib error.obj nmake.mak unittests.obj error.obj.csmes NOTICE unittests.obj.csmes functions.cpp parser.cpp variablelist.cpp
- 18 - froglogic GmbH Microsoft Windows setup
functions.h parser.h variablelist.h functions.obj parser.obj variablelist.obj functions.obj.csmes parser.obj.csmes variablelist.obj.csmes instrumented README.squishcoco instrumented.bat unittests.cpp 35 File(s) 1,079,919 bytes 2 Dir(s) 35,159,457,792 bytes free
You see two kinds of files that do not appear as result of normal compilation. The .csmes files contain the information that is needed for coverage measurement, and the .csexe files contain the results of code execution. The files that end in .obj.csmes are temporary files and are only used during compilation.
This time, the only program that was actually executed was unittests, and therefore the only .csexe file is unittests.exe. csexe. To see the coverage results, you can therefore start the CoverageBrowser with the command
C:\code\parser\parser_v1>coveragebrowser -m unittests.exe.csmes -e unittests.exe.csexe
Then the CoverageBrowser will start with a modal window, "Load Execution Result". Click on the "Import" button to load the data. By default, CoverageBrowser automatically deletes the .csmes file after it loads it. You can switch this behavior off by unselecting the "Delete after loading" checkbox. Or, if you don’t, select the "File¡Save" menu item ! to save the execution report in the unittests.csexe file.
For the use of CoverageBrowser, see Part III, page 50. We will now rather describe how the instrumentation is done.
6.2.5 How the project is instrumented
The file instrumented.bat is a short batch file:
@echo off setlocal
set PATH=%SQUISHCOCO%\visualstudio;%PATH% set COVERAGESCANNER_ARGS=--cs-on
call %*
endlocal
The variable SQUISHCOCO contains the name of the directory in where Squish Coco is installed. It is set by Squish Coco during installation.
At the beginning, the setlocal command ensures that the following commands change the environment variables only temporarily. Then there are two set statements, and the final call statement executes the command line parameters of instrumented. So if you call “instrumented nmake /F nmake.mak tests”, the command “nmake /F nmake.mak tests” is executed by the batch file, but in a different environment than normally. At the end, endlocal undoes the changes in the environment variables.
The important part of the script are therefore the two set statements. In the first one, the search part is manipulated so that the programs in C:\Program Files\squishcoco\visualstudio are searched first.3 This directory contains files with the same names as the compilers and the linker
C:\code\parser\parser_v1>dir /d "\Program Files\squishcoco\visualstudio"
3This is the version for the 32 bit compiler. There is also a directory C:\Program Files\squishcoco\visualstudio_x64 for 64 bit compilation.
- 19 - froglogic GmbH Beyond the minimal instrumentation
Directory of C:\Program Files\squishcoco\visualstudio
[.] cl.exe link.cspro msvcr100.dll [..] lib.cspro link.exe cl.cspro lib.exe msvcp100.dll 8 File(s) 5,662,299 bytes 2 Dir(s) 35,053,502,464 bytes free
The .exe files in this directory are the compiler wrappers. With the new PATH, they are executed instead of the real compiler. The compiler wrappers are actually copies of a single program, coveragescanner.exe (see Part IV, page 74). When executed to compile a source file, they create an instrumented version of the source and then run the original compiler to compile it.
In the second set statement, additional flags for the compiler wrappers (see Chapter 17, page 77) are set. Here we set only one option, --cs-on. If it is not present, the compiler wrappers are inactive and just call the compilers they represent. The resulting script should work without changes for many simple projects. If more customization is needed, it can often be achieved by adding more options to COVERAGESCANNER_ARGS.
6.2.6 Additional changes
It is also convenient to add make targets to handle the files generated by CoverageScanner. In the parser_v1 directory, the file nmake.mak has been changed in the following way:
clean: testclean ... -$(DEL_FILE) *.obj.csmes # (added)
distclean: clean ... -$(DEL_FILE) *.csmes *.csexe # (added)
Since the .obj.csmes files are needed only for compilation, they can be deleted whenever the .obj files are deleted (which is that what make clean does). The .csmes and .csexe files are more precious and should only be deleted when all generated files are removed. Therefore we have added their deletion statements to the distclean target.
6.3 Beyond the minimal instrumentation
In the following sections we will show additional abilities of Squish Coco. They will require small changes in the code of the project.
6.3.1 Excluding code from instrumentation
The coverage information generated so far has a problem: It covers too many files. The problematic files are those that belong to the testing framework and not to the tested program. Including them would create artificially low coverage rates.
With Squish Coco, one can exclude files from coverage by additional command line options. In parser_v2, this has been done. Look into parser_v2/instrumented (or parser_v2\instrumented.bat under Microsoft® Windows). In it, three additional command line options have been set, which we will now explain:
- 20 - froglogic GmbH Beyond the minimal instrumentation
• The option --cs-exclude-path=../../cppunit-1.12.1 excludes the source files of a directory and all its subdirec- tories. Here we use it to exclude all the files of the CppUnit framework. You can use slashes or backslashes with this option—Squish Coco normalizes them internally.
• The options --cs-exclude-file-wildcard=unittests.cpp and --cs-exclude-file-wildcard=CppUnitListener.cpp exclude specific files. We use them to exclude the files unittests.cpp and CppUnitListener.cpp. (The second file is described below.)
6.3.2 Making the test names visible
For the next modification, we want to change the project such that we know not only whether a line of code is covered by tests, but also by which tests it is covered. For this we will add calls of the CoverageScanner C/C++ library (see Chapter 18.1, page 86) to the code, to tell Squish Coco the names of the tests and where they begin and end.
An updated version of the project can be found in the directory parser_v2. The greatest difference to the version in parser_v1 is that the file CppUnitListener.cpp has been added. It is copied almost verbatim from Chapter 22.3.1. The file contains a class CppUnitListener and a new main() function. The main() function in unittests.cpp has been removed, but the file is otherwise unchanged.
CppUnitListener.cpp provides a unit test listener which allows to hook into the framework before and after the execution of each test. One can thus record additional test information, like the name and the result of a test, in the code coverage data without modifying the test code itself. (For a listing of CppUnitListener.cpp and an explanation how it works, see Chapter 22.3.1) Now you can execute this program with the same way as its previous version. View the results in the CoverageBrowser and see the code coverage for each single test item.
6.3.3 Patch file analysis
Now consider the follwing scenario: In a large project, a last-minute patch has to be evaluated. It is not enough time to run the full test suite, but some risk assessment needs to be done. For situations like this, Squish Coco provides the feature of patch analysis. With it, one can specifically display the code coverage for the changed lines of code, and find the tests in a large suite that cover them. One can now see how risky the changes are. We will simulate this situation in our example. In a new version of the parser, the character classification functions in the code, isWhiteSpace(), isAlpha(), etc., have been changed and use the standard C classification functions, like isspace(), instead of strchr(). The new version of the parser can be found in the directory parser_v3.
We will now compare it with the version in parser_v2, but neither run the tests nor even compile it. Instead we need the following two pieces of information:
1. The coverage data from parser_v2, as generated in the previous section.
2. A patch file showing the differences between the two directories. There is already a diff file in the parser directory that you can use, parser.diff. The diff file must be in the “unified” difference format. This is the standard output format of the diff functionality of many version control systems, e.g. of git diff. Under UNIX®-like systems, the patch file can also be generated by the diff utility. It would be invoked from the parser directory in the following way:
$ diff -u parser_v2 parser_v3 > parser.diff
There is also a Microsoft® Windows version of GNU diff in the parser directory, therefore the same command works in a Windows command shell too.
Start the CoverageBrowser. Then load the instrumentation database parser_v2/unittests.csmes via the menu entry "File¡Open...", and the measurements file parser_v2/unittests.csexe with "File¡Load Execution Report...".
- 21 - froglogic GmbH Beyond the minimal instrumentation
Now select the menu entry "Reports¡Patch File Analysis...". When the "Patch File Analysis" dialog appears:
• Enter a title in the "Title" box.
• Enter the path to the patch file in the "Patch File" box.
• Enter a path (including the file name) for the report in the "Type" box.
• Set the "Tooltip Maximal Size" to 5 (or any value greater than zero).
• Make any other option adjustments.
Then click "Open" to view the report in the browser.
6.3.4 The patch analysis report
The report consists of three tables that summarize the influence that the patch has on code coverage, and then an annotated version of the patch file. The two tables in the section “Overview” contain statistics about the number and kind of the lines that were influenced by the patch. The first table groups the patched lines in the code by the results of the tests that executed them. One can see here how much influence the patch has on tests that have passed (and now could fail) or failed (and could now succeed). There are also entries for manually checked tests and for those whose status is unknown. In our example, we did not register the test results and all our tests are counted as “Unknown”. The second table shows the kind of changes to expect in the test coverage after the patch has been applied. It consists of three columns, containing the statistics about removed and inserted lines and their sum. From the first two columns one can see whether the test coverage for the patched code grows or falls. (In the parser example, it stays the same.) The last line in the table is also important: It shows the number of lines which Squish Coco could not classify as inserted or removed. Patch analysis is a heuristics, after all. The section “List of tests influenced by the modifications” is a list with the names of the tests that executed the patched code, togehter with their result. It is helpful for a qualitative analysis of the patch. In our example, we can see that all tests execute code that is affected by the patch.
Figure 6.1: Coverage of the patched lines by the tests
More details can be found in the “Patch File” section of the report. It is an annotated version of the original patch file, with the old version of the text in red and the new version in green. Lines that did not change are shown in gray. The most important column is “Tests”, which shows for each code line the the number of tests that executed it (if it is removed) or will probably execute it after the patch is applied. A tooltip shows the names of these tests.
- 22 - froglogic GmbH Getting started with Qt
Chapter Getting started with Qt 7
The following example is more complex. We will take Qt’s TextEdit example and use it to illustrate how Squish Coco can be used at different stages of the development process. To cover the whole coding cycle, we will first show how an instrumented application is created, perform manual tests and analyze their results. Then we will create an instrumented unit test. In a second step (see Chapter 7.2, page 32), we will cover those aspects which are more interesting to product managers: analysing the impact of code changes (e.g., bug fixes)—in particular, tracking their test progress, externalizing testing, and collecting the code coverage analysis of a complete testing team. i The Qt framework can directly be downloaded from http://qt-project.org.
The modified TextEdit sample is available in the doc directory of the Squish Coco installation. Copy the directory doc/ textedit to your working directory.
7.1 Compiling the example application
In this section, we will work with the files in the directory textedit/textedit_v1. We want to be able to build our application both normally and with generated test coverage instrumentation code, without having to change our source code. This can be achieved by making a small change to the application’s project (.pro) file. We can then use a command line option for qmake to generate an instrumented build instead of a normal one. To make the project file suitable both for normal and for instrumented builds, we create a set of definitions that can be activated by a command line switch; in qmake’s terminology this is called a scope. The following listing (see Figure 7.1, page 24) shows a minimal scope for code instrumentation. The following must be done:
• We must ensure that precompiled headers are disabled when the code is instrumented. qmake allows to do this by setting the PRECOMPILED_HEADER variable to an empty value.
• It is also necessary to increase the value of QMAKE_LINK_OBJECT_MAX in order to disable the usage of the linker script. We set it here to 10000.
• Finally, qmake must be instructed to use CoverageScanner’s wrappers for compilation. This is done by prefixing the names of the compilation tools with cs.
- 23 - froglogic GmbH Compiling the example application
CodeCoverage { PRECOMPILED_HEADERS= QMAKE_LINK_OBJECT_MAX=10000
QMAKE_CC=cs$$QMAKE_CC QMAKE_CXX=cs$$QMAKE_CXX QMAKE_LINK=cs$$QMAKE_LINK QMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIB QMAKE_AR=cs$$QMAKE_AR QMAKE_LIB=cs$$QMAKE_LIB }
Figure 7.1: Minimal qmake configuration
These modification are sufficient for most standard C and C++ applications. For Qt applications we must use additional settings in order to ensure that CoverageScanner does not instrument the source code that is generated by Qt’s tools (e.g., by uic, qrc and the moc).
To exclude qrc resource files from instrumentation, we must tell CoverageScanner not to instrument any file with a name that begins with qrc_. This can be done with the command line option --cs-exclude-file-regex=qrc_.*. Since we don’t want to have to enter this option manually, we will put it in the .pro file. Similarly, to let CoverageScanner ignore the files generated by uic we can use the same command line option, only this time with a different file matching regular expression: --cs-exclude-file-regex=ui_.*. Squish Coco also provides a command line option that is specific to applications built using the Qt 4 toolkit: (--cs-qt4). This option ensures that CoverageScanner:
• does not instrument the Q_OBJECT and Q_DECLARE_PLUGIN macros.
• does not instrument code generated by the moc, except that signal emissions and slot receives are instrumented, since they are vital to a Qt program’s logic. We can also exercise some control over the level of instrumentation and what information is reported. For example, we can switch on the counting of code executions with the --cs-count command line option, or we can enable full instrumentation at decision/condition level with the --cs-full-instrumentation option. With the --cs-output option we can specify the file the execution report is written to when the application terminates. (By default the output is written to the file
So, for a Qt 4-based application, the final Squish Coco scope in the application’s .pro file will look something like this.
- 24 - froglogic GmbH Compiling the example application
CodeCoverage { COVERAGE_OPTIONS = --cs-count --cs-full-instrumentation COVERAGE_OPTIONS += --cs-qt4 COVERAGE_OPTIONS += --cs-output=textedit.exe COVERAGE_OPTIONS += --cs-exclude-file-regex=qrc_.*
QMAKE_CFLAGS += $$COVERAGE_OPTIONS QMAKE_CXXFLAGS += $$COVERAGE_OPTIONS QMAKE_LFLAGS += $$COVERAGE_OPTIONS
QMAKE_CC=cs$$QMAKE_CC QMAKE_CXX=cs$$QMAKE_CXX QMAKE_LINK=cs$$QMAKE_LINK QMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIB QMAKE_AR=cs$$QMAKE_AR QMAKE_LIB=cs$$QMAKE_LIB }
Figure 7.2: Final qmake configuration
In this form, the scope has been added to the project file textedit_v1/textedit_v1.pro. When we now run qmake without options, a Makefile for a normal build is generated; but we can also build an instrumented version of the program in the following way: Linux™/Apple® Mac OS X Microsoft® Windows
qmake CONFIG+=CodeCoverage qmake CONFIG+=CodeCoverage make nmake
In either case we still end up with a textedit.exe executable, but with an instrumented build we will also get an additional file, textedit.exe.csmes. It contains the instrumentation database.
7.1.1 The First Code Coverage Results
For our very first exercise we will simply execute TextEdit and then quit the application straight away. This will cause a file called textedit.exe.csexe to be generated. The file is in a binary format, so it is not human readable. It is an instrumentation database that contains a snapshot of the most recent execution that we have just done.
To see the results we must run the CoverageBrowser tool and load the textedit.exe.csmes instrumentation database (Menu: "File¡Open..."). After the file has been opened, no coverage information is available because no execution reports have been imported. The instrumented code lines are shown grayed-out and no coverage statistics has been computed (see Figure 7.3, page 26).
- 25 - froglogic GmbH Compiling the example application
Figure 7.3: CoverageBrowser after loading the TextEdit’s instrumentation database
In order to see an execution report, click "File¡Load Execution Report...", which invokes the import dialog. As a minimum you should enter the filename (including the full path) of the textedit.exe.csexe file (field: ‘File Name’), and give the test a name (Field: ‘Name’), e.g. “Start and Quit”. Switch the ‘Delete after loading’ option on because the report is no longer needed after our import. It is also helpful to set the ‘When file becomes modified’ option to “Open this dialog”, because then the file import dialog is automatically opened after each run and the new .csexe file that it created can be added to the database. After the import has finished, the code coverage information is visible: • Coverage statistics for the functions and methods of all source files of the application is shown. • The source window is now colored and shows executed code on a green background and unexecuted code on a red background.
• The execution list now contains one selected item called “Start and Quit”, which is the only test execution report we have so far created.
7.1.2 Interactive testing
CoverageBrowser correctly reveals, for example, that the TextEdit::fileSave() function is not executed. We will try to validate this function interactively, guided by the code coverage analysis. In the source window, all unexecuted source code lines are shown with a red background. (see Listing 7.4, page 27)
- 26 - froglogic GmbH Compiling the example application
bool TextEdit::fileSave() { if (fileName.isEmpty()) { QMessageBox::warning(this, tr("No file name specified"), tr("Save first your document using ’Save As...’ from the menu"), QMessageBox::Ok); return false; }
QTextDocumentWriter writer(fileName); bool success = writer.write(textEdit->document()); if (success) textEdit->document()->setModified(false); return success; }
Figure 7.4: CoverageBrowser source view of the function TextEdit::fileSave()
To test this function we must perform the following steps:
1. Start the TextEdit application.
2. Click on the ‘Save’ button: TextEdit should display the error message "Save first your document using ‘Save As. . . ’ from the menu". 3. Quit the application.
After these steps have been done and the coverage report imported, CoverageBrowser shows that the return false; line just after the call to QMessageBox::warning() has been executed (as indicated by the green background). However, the line if (fileName.isEmpty()) is shown as partially executed (indicated by an orange background); (see Listing 7.5, page 27).
bool TextEdit::fileSave() { if (fileName.isEmpty()) { QMessageBox::warning(this, tr("No file name specified"), tr("Save first your document using ’Save As...’ from the menu"), QMessageBox::Ok); return false; }
QTextDocumentWriter writer(fileName); bool success = writer.write(textEdit->document()); if (success) textEdit->document()->setModified(false); return success; }
Figure 7.5: CoverageBrowser source view after clicking TextEdit’s ‘Save’ button.
The explanation window (see Listing 7.6, page 28) tells us that the value of the expression fileName.isEmpty() was true during one execution but was never false (hence, it is considered only partially executed). In order to fully test this expression we must click on the ‘Save As. . . ’ button, then choose a filename, and finally click on the ‘Save’ button.
- 27 - froglogic GmbH Compiling the example application partially executed: fileName.isEmpty()
TRUE FALSE
yes no Execution Count: 1 Execution Count: 0 Executed by: - Save Clicked
Figure 7.6: CoverageBrowser explanation window after clicking on the ’Save’ button of TextEdit.
After rerunning the application and doing a “Save as”, the new execution report now has only one source code line that is partially untested (see Listing 7.7, page 28). In this case, CoverageBrowser reveals that the Boolean variable success was never false, which means that saving the document has never failed. We could force a write failure, and this would ensure that we had 100% code coverage for this function. But we will use a different test strategy to get complete code coverage: we will use a unit test and import the execution result into the TextEdit instrumentation database.
bool TextEdit::fileSave() { if (fileName.isEmpty()) { QMessageBox::warning(this,tr("No file name specified"), tr("Save first your document using ’Save As...’ from the menu"), QMessageBox::Ok ); return false; }
QTextDocumentWriter writer(fileName); bool success = writer.write(textEdit->document()); if (success) textEdit->document()->setModified(false); return success; }
Figure 7.7: CoverageBrowser source view after clicking TextEdit’s ‘Save As. . . ’ button and then the ‘Save’ button.
7.1.3 Writing unit tests
The unit test infrastructure can be found in the directory textedit_v1_tests/. It contains just one test, which sets an illegal filename and then tries to execute the TextEdit’s fileSave() function. To do this we use the QTestLib unit testing library that is supplied with Qt. The test is contained in the file textedit_v1_tests/tst_textedit.cpp (see Listing 7.8, page 29).
- 28 - froglogic GmbH Compiling the example application
#include "tst_textedit.h"
void TestTextEdit::tst_saveFile() { TextEdit textEdit; textEdit.fileName="/"; QVERIFY( !textEdit.fileSave() ); }
QTEST_MAIN(TestTextEdit);
Figure 7.8: Unit test for the TextEdit application, tst_textedit.cpp
To import this test’s instrumentation result into TextEdit’s instrumentation database, the following infrastructure is necessary:
1.A qmake project file with code coverage configured identically with that of the TextEdit project. 2. A post-build rule which automatically executes the test and collects the coverage information.
3. A unit test listener which saves the code coverage data (and the test status—passed or failed) for every executed unit into the unit test’s own instrumentation database.
4. A way to import the code coverage report into the TextEdit’s instrumentation database.
The unit test will recompile along with textedit_v1/textedit.cpp. To make its results importable into the TextEdit instrumentation database, it is necessary that both executables (TextEdit and the unit test) are instrumented in exactly the same way. So, for this example, we must use the instrumentation options --cs-count, --cs-full-instrumentation and --cs-qt4. Unfortunately, these command line options alone are insufficient, because Squish Coco’s default behavior is only to instrument header and source files in the current directory. But here we need to instrument the TextEdit application’s sources in addition to the unit test, so we must use another command line option to specify an additional path for files to instrument: --cs-include-path.
As before, we don’t want to have to remember these command line arguments every time, so we set them in the qmake project file, textedit_v1_tests.pro (see Figure 7.9, page 30). With these lines in the unit test’s project file, the result will be that the qmake-generated Makefile creates the tst_textedit.exe executable which, when run, produces the tst_textedit.exe. csexe execution report. We then can use the CoverageBrowser to import this report into the file tst_textedit.exe.csmes.
- 29 - froglogic GmbH Compiling the example application
HEADERS = ../textedit_v1/textedit.h tst_textedit.h SOURCES = ../textedit_v1/textedit.cpp tst_textedit.cpp
CodeCoverage { COVERAGE_OPTIONS = --cs-count --cs-full-instrumentation COVERAGE_OPTIONS += --cs-qt4 COVERAGE_OPTIONS += --cs-output=tst_textedit.exe COVERAGE_OPTIONS += --cs-include-path=../textedit_v1 COVERAGE_OPTIONS += --cs-exclude-file-regex=qrc_.*
QMAKE_CXXFLAGS += $$COVERAGE_OPTIONS QMAKE_CCFLAGS += $$COVERAGE_OPTIONS QMAKE_LFLAGS += $$COVERAGE_OPTIONS
QMAKE_CC=cs$$QMAKE_CC QMAKE_LINK=cs$$QMAKE_LINK QMAKE_CXX=cs$$QMAKE_CXX }
Figure 7.9: An extract from the unit test qmake project file
We can also execute the unit test automatically and import the execution report with a post-build rule. Squish Coco provides an extra command line tool, cmcsexeimport (see Chapter 26, page 154), which imports an execution report into an instrumentation database. The post-build rule first deletes any previous execution report, then executes the test itself, and finally imports the results into the application’s execution database, tst_textedit.exe.csexe (see Listing 7.10, page 30).
CodeCoverage { win32: MAINDIR=$$replace(PWD,"/","\\") !win32:MAINDIR=$$PWD
unix { QMAKE_POST_LINK = rm $$MAINDIR/tst_textedit_v1.exe.csexe ; QMAKE_POST_LINK += $$MAINDIR/tst_textedit_v1.exe ; QMAKE_POST_LINK += cmcsexeimport -m $$MAINDIR/tst_textedit_v1.exe.csmes \ -e $$MAINDIR/tst_textedit_v1.exe.csexe -t UnitTest } win32 { QMAKE_POST_LINK = del /F $$MAINDIR\\tst_textedit_v1.csexe & QMAKE_POST_LINK += $$MAINDIR\\tst_textedit_v1.exe & QMAKE_POST_LINK += cmcsexeimport -m $$MAINDIR\\tst_textedit_v1.exe.csmes \ -e $$MAINDIR\\tst_textedit_v1.csexe -t UnitTest } }
Figure 7.10: Post-build rules for the import of the execution report into the unit test’s instrumentation database
By default, the coverage data is imported without any information about the executed tests: instead, an execution report called ‘UnitTest’ is created, which does not describe which test was executed or if its execution was successful. To provide the missing information, we must use the CoverageScanner API and generate an execution report for each test that is executed. An example that shows how this is done is available in the chapter 22.3.2. In the example, the API is used in the following way:
• Two Squish Coco source files are added to the qmake project file (see Listing 7.11, page 31).
• The unit test class, TestTextEdit, inherits from TestCoverageObject instead of directly from QObject (see List-
- 30 - froglogic GmbH Compiling the example application
ing 7.12, page 31).
HEADERS += testcoverageobject.h SOURCES += testcoverageobject.cpp
Figure 7.11: Including the CoverageScanner listener in textedit_v1_tests.pro
#include "testcoverageobject.h" #include "../textedit_v1/textedit.h" #include
class TestTextEdit : public TestCoverageObject { Q_OBJECT private slots: void tst_saveFile(); };
Figure 7.12: The TextEdit unit test header file, tst_textedit.h
The testcoverageobject.cpp file’s source code (see Listing 7.13, page 31) is simple to understand. The file adds a single cleanup() function to QTestLib, which is executed after each unit test item. The code between #ifdef __COVERAGESCANNER__ and #endif is only compiled when CoverageScanner is invoked1. This extra code creates a test name by combining the test class’s object name and the test function’s name. In the the example, an execution item called unittest/ TestTextEdit/tst_saveFile is generated. The slash is used to support the organization of tests in a tree view. The current test status (“PASSED” or “FAILED”) is also recorded and added to the execution report by the __coveragescanner_save() function.
... void TestCoverageObject::cleanup() { cleanupTest(); #ifdef __COVERAGESCANNER__ QString test_name="unittest/"; test_name += metaObject()->className(); test_name += "/"; test_name += QTest::currentTestFunction(); __coveragescanner_testname(test_name.toLatin1()); if (QTest::currentTestFailed()) __coveragescanner_teststate("FAILED"); else __coveragescanner_teststate("PASSED") ; __coveragescanner_save(); #endif }
Figure 7.13: An extract from the TestCoverageObject’s source code
At this stage we could start CoverageBrowser, load the TextEdit instrumentation database, and import the unit test’s instrumentation database by clicking "File¡Import Unit Tests...". An even more convenient alternative is to use the
1The symbol __COVERAGESCANNER__ is defined automatically by CoverageScanner and so does not need to be defined manually.
- 31 - froglogic GmbH Working with code coverage data cmmerge tool to automate this step. The cmmerge program is designed to import one instrumentation database’s execution report into another instrumentation database. This means that we can extend our post-build rules to use the cmmerge program to import the coverage information automatically from the unit test into the TextEdit program’s instrumentation database (see Listing 7.14, page 32).
CodeCoverage { # Merge coverage database into TextEdit database unix { QMAKE_POST_LINK += ; QMAKE_POST_LINK += cmmerge -o $$MAINDIR/../textedit_v1/textedit.tmp \ -i $$MAINDIR/../textedit_v1/textedit.exe.csmes \ $$MAINDIR/./tst_textedit_v1.exe.csmes && QMAKE_POST_LINK += rm $$MAINDIR/../textedit_v1/textedit.exe.csmes && QMAKE_POST_LINK += mv $$MAINDIR/../textedit_v1/textedit.tmp \ $$MAINDIR/../textedit_v1/textedit.exe.csmes } win32 { QMAKE_POST_LINK += & QMAKE_POST_LINK += echo Merging unit test result into the main application & QMAKE_POST_LINK += cmmerge -o $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes \ -i $$MAINDIR\\..\\textedit_v1\\textedit.exe.csmes \ $$MAINDIR\\tst_textedit_v1.exe.csmes & QMAKE_POST_LINK += COPY /Y $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes \ $$MAINDIR\\..\\textedit_v1\\textedit.exe.csmes & QMAKE_POST_LINK += DEL /F $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes } }
Figure 7.14: Post build rules: merging instrumentation results into the TextEdit instrumentation database
With all these changes to the .pro file in place, we can once again build and run the unit test. Now the CoverageBrowser shows the fileSave() function to be 100% covered, with the execution list containing our three original manual tests and the single unit test (see Figure 7.15, page 32).
Figure 7.15: The execution list after all the tests have been executed
7.2 Working with code coverage data
The most common ways in which code coverage is used are for developers to use it to find untested code, and for managers to use it to produce test status reports (e.g., as diagrams). In addition to fully supporting the common use cases, Squish Coco also provides additional features which make it possible to go beyond these fundamentals and extend what can be achieved with code coverage. This will be discussed in the current subsection.
- 32 - froglogic GmbH Working with code coverage data
7.2.1 Post mortem analysis
Recording each test’s coverage data makes it possible to compare their data to answer the question, “What does this test cover that the others don’t?” This is particularly useful if just one test fails, since it can help us to identify which part of the code is involved.
To see how this works in practice, let us return to the TextEdit example. If we click ‘Save’, we will get an error message that no filename is defined. This is not very convenient for users—we should have designed TextEdit to handle this particular case by opening the ‘Save As. . . ’ dialog rather than by producing an error.
To identify where in the code this problem arises, we simply compare the ‘Save Clicked’ execution with all other executions that involve the ‘Save’ button. To do this we must first switch to the “Execution Comparison Analysis” mode (Menu: "Tools¡Execution Comparison Analysis"). Select the checkboxes in the “Reference” column for the “tst_saveFile” and “SaveAs clicked before Save clicked” tests. This will make the execution comparison symbol appears in front of the affected names (see Figure 7.16, page 33). In the “Executions” column, click on “Save clicked” checkbox.
Figure 7.16: The execution list being used to compare different executions
In this mode, the coverage analysis is based only on source code lines which are not executed by “tst_saveFile” and “SaveAs clicked before Save clicked”. This is why the overall coverage decreases to 1.29%: It means that “Save clicked” executes 1.29% more code than the selected tests. Using cmreport it is possible to generate a HTML report which displays the same information:
cmreport --csmes=textedit_v1/textedit.exe.csmes \ --html=textedit.html \ --section=execution \ --select-reference=".*tst_saveFile" \ --select-reference="SaveAs clicked before Save clicked"
If we now look at the source code itself, we will see that only two lines of the TextEdit::fileSave() function (see Listing 7.17, page 34) are not grayed: the lines which pop up the error message. These are the lines that must be modified to change the “Save” button’s behavior.
- 33 - froglogic GmbH Working with code coverage data
bool TextEdit::fileSave() { if (fileName.isEmpty()) { QMessageBox::warning(this,tr("No file name specified"), tr("Save first your document using ’Save As...’ from the menu"), QMessageBox::Ok ); return false; }
QTextDocumentWriter writer(fileName); bool success = writer.write(textEdit->document()); if (success) textEdit->document()->setModified(false); return success; }
Figure 7.17: CoverageBrowser source view of the comparison of the “Save clicked” execution.
In this case, changing the fileSave() function is easy—we simply replace the QMessageBox::warning() call with a call to the fileSaveAs() method. (see Listing 7.18, page 34)
bool TextEdit::fileSave() { if (fileName.isEmpty()) return fileSaveAs();
QTextDocumentWriter writer(fileName); bool success = writer.write(textEdit->document()); if (success) textEdit->document()->setModified(false); return success; }
Figure 7.18: The TextEdit’s improved fileSave() function.
7.2.2 Evaluating the impact of a hot fix
Before committing a change or starting to test a hot fix, it is possible to estimate the impact of the code modification. CoverageBrowser is able to perform an analysis on the difference between two source sets and can list the tests that will be affected (and those which won’t).
Start CoverageBrowser and load the modified TextEdit example’s instrumentation database. Now click on "Tools¡Compare with..." and select the original version of the TextEdit instrumentation database. CoverageBrowser will now display the source code like a text comparison application does (e.g., diff).
Click on "Tools¡Analysis of Modified Methods" to exclude all unmodified functions from the coverage analysis. In the TextEdit case, doing this will mean that only one function, TextEdit::fileSave(), will be treated as being instrumented since that is the only method we have changed (see Figure 7.19, page 35). This also affects the statistic calculations since execution coverage statistics will now be limited to just this function. The test executions whose coverage statistic is not zero are the ones that are affected by the code modifications we have made. In our case we have:
- 34 - froglogic GmbH Working with code coverage data
• “Save clicked”, • “SaveAs clicked before Save clicked” and • “tst_saveFile” (our unit test). The “Start and Exit” case has a coverage of 0% and so does not execute our modified code. It is for this reason no longer visible in the execution list. All entries in the ‘Execution’ column are struck through to inform us that these tests are not executed in the newest version and only present in the reference database. In other words, only the two manual tests and the unit test listed above must be re-executed to ensure that no regressions have been introduced by our code changes.
Figure 7.19: List of tests affected by a code modification
7.2.3 Black-box testing/distributed testing
Up to now we have done white-box testing, i.e. testing where we have access to the source code and which makes use of our knowledge of the code. It is also possible to use Squish Coco for black-box testing. In other words, we can still do code coverage analysis without having access to or even knowledge of the source code. If we use this approach, the generated instrumentation database will, of course, contain no source code.
To use black-box testing we must create a suitable instrumentation database by clicking "File¡Generate Black-Box Configuration...". This database, along with the TextEdit executable, can be given to the test team which can then use them with a simplified version of CoverageBrowser (see Figure 7.20, page 36). This version of CoverageBrowser only supports the importing and managing of execution reports since it does not have access to the application’s source code.
Once all the tests are finished, the black-box database can be merged into the original TextEdit instrumentation database using CoverageBrowser’s merge facilities (Menu: "File¡Merge with...").
- 35 - froglogic GmbH Working with code coverage data
Figure 7.20: Black-box testing results as shown by CoverageBrowser
7.2.4 Verifying if a bug fix is correctly tested
Often, when a small bug fix is made, the effects are very localized and leave most of the source code unchanged. In view of this, it is often unnecessary to retest the entire application with the whole test suite. Squish Coco makes it possible to avoid unnecessary testing. We can tell it to restrict itself to the source code that has changed between the original and fixed version of the application (see Chapter 7.2.2, page 34). This allows us to focus purely on the analysis of the fix. To achieve this, simply load the fixed application’s freshly generated instrumentation database (e.g., for the modified TextEdit application), and compare it with the earlier database for the unfixed version, using Squish Coco’s facility for analyzing modified functions.
Figure 7.21: Coverage of the patched function
We have done just such a comparison and the results are shown in in 7.21: The two tests, “Save clicked” and “Start and Exit”, cover 85% of the TextEdit::fileSave() function, the only method that was modified for our fix. From this we know exactly what additional testing is necessary to achieve 100% code coverage for our tests for the fixed version of the application. CoverageBrowser continues to display the list of missing tests (which are only executed using the first version of TextEdit) in strikeout style. This gives a hint of what remains for testing effort.
- 36 - froglogic GmbH Conclusion
Using cmreport it is possible to generate a HTML report which displays the same information:
cmreport --csmes=textedit_v2/textedit.exe.csmes \ --csmes-reference=textedit_v1/textedit.exe.csmes \ --html=textedit.html \ --section=execution
7.3 Conclusion
Squish Coco provides code coverage analysis which can be applied to all the usual testing techniques: unit, manual, and black-box testing. Squish Coco can easily be told to ignore the generated code produced by the Qt library’s tools (moc, qrc, and uic), so that only the code written by developers is instrumented. Test results can be collected into a database and can be used to evaluate how much code coverage our tests achieve and to show which statements are not currently tested. With this information we can target our testing efforts towards 100% test code coverage. In addition, Squish Coco makes it possible to see what effect a code modification would have in terms of test code coverage without having to test the entire application. Overall, Squish Coco can help us target our tests to ensure that our applications have as much test coverage as possible, while avoiding or minimizing test coverage duplication. Furthermore, Squish Coco can help us see what effects changes to our code have on test coverage, so that we can adapt our test suites accordingly.
- 37 - froglogic GmbH Code Coverage Part II Overview
- 38 - froglogic GmbH Code Coverage Overview
Chapter Overview of Squish Coco’s Code Coverage Analysis 8
CoverageScanner is a C++ program that—in effect—replaces the usual compiler. It preprocesses the source code using the native preprocessor program, in the course of which it inserts instrumentation code, and at the end it compiles and links the project file, just the same as for a normal build. CoverageScanner’s instrumentation consists of: 1. declaring a static table that contains the of instrumentation results.
2. generating a library which produces an execution report upon program termination, a UNIX® signal or a Microsoft® Windows event. 3. adding the instrumentation code itself for each relevant C and C++ statement or Boolean expression. The compiler, linker and preprocessor used are from the native environments (gcc, Visual Studio, etc.). These tools are used transparently and the developer has only to prepend ’cs’ to the name of the compiler executable to activate the code coverage1. For example, using ’csgcc’ instruments the source code and uses the ’gcc’ compiler to generate objects. Another code coverage technique involves adding measurement points according to debug information. Since CoverageScanner produces instrumented code during the generation phase, it is possible to analyze the optimized code and achieve a more fine-grained analysis than some other tools achieve. (Many compilers produce unusable debug information on optimized code, but this doesn’t affect CoverageScanner.) CoverageScanner’s principle of code coverage analysis is not to highlight executed source lines, but rather to place marks on execution paths. This makes the analysis independent of the source code’s formatting and groups all sequential instructions together in a single measurement point. This also minimizes the extra memory and processing overhead that the instrumented code incurs.
8.1 Code instrumentation
This section describes how CoverageScanner instruments the code, with each aspect described in its own subsection.
8.1.1 Detection
CoverageScanner parses all C++ language constructs and detects: 1. Executed functions.
1 For Microsoft® Visual Studio®, an additional wrapper is provided which directly uses the Microsoft® ’cl’ and ’link’ (see Chapter 20.10, page 109)
- 39 - froglogic GmbH Code instrumentation
2. Executed source code lines. 3. Execution paths.
4. Boolean expressions which produce different execution paths (if, for, while, switch, . . . ). To put this in concrete terms we present the following very simple example function:
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i<100) && (!found); ++i) {
5 if (i==50) break;
6 if (i==20) found=true;
7 if (i==30) found=true;
8 }
9 printf("foo\n");
10 }
Figure 8.1: Source code sample
CoverageScanner first analyses the sequential statements to record each execution path. It determines which instructions need to be instrumented and places a flag to record their execution. This kind of instrumentation is usually called Statement Coverage. To avoid performance problems, CoverageScanner groups all sequential instructions together and records the execution of the whole group only once. In the following listing, the instrumented statements are underlined:
1 void foo()
2 {
3 bool found=false; // [3] not instrumented
4 for (int i=0; (i<100) && (!found); ++i) {
5 if (i==50) break;
6 if (i==20) found=true;
7 if (i==30) found=true;
8 }
9 printf("foo\n"); // [9] not instrumented
10 } // [10]
Figure 8.2: Code coverage at branch level
Lines 3 and 9 are not monitored since lines 3, 9 and 10 are sequential operations. This has no impact on coverage because if line 10 is executed, then lines 3 and 9 must have be executed before it.
CoverageScanner’s analysis of conditions determines which expressions are used in conditional statements (if, while, for, . . . ). These statements separate the execution path into two or more branches. Code coverage monitors execution by recording if the condition was true or false during the application’s execution. This kind of code coverage instrumentation is usually called Decision Coverage. In the following listing, the conditions instrumented for Decision Coverage are underlined. The instrumented statements are displayed with a gray background:
- 40 - froglogic GmbH Code instrumentation
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i<100) && (!found) ; ++i) {
5 if (i==50 ) break;
6 if (i==20 ) found=true;
7 if (i==30 ) found=true;
8 }
9 printf("foo\n");
10 }
Figure 8.3: Code coverage at decision level
The analysis of Boolean expressions is used to determine which combinations are used in a conditional statements’ (if, while, for, . . . ) Boolean operations (and, or, . . . ). This makes it possible to have a more fine grained analysis why a particular code segment was executed. This kind of code coverage instrumentation is usually called Condition Coverage.
For example, the statement ‘return 0;’ of ‘if (a || b) return 0;’ can be executed if a or b is true. Analysis of the Boolean expressions will record whether variables a and b were true or false. This tells us if the statement ‘return 0;’ was executed, which it will be if either or both of the two expressions was true during the test. In the next example, the instrumented Boolean expressions are underlined:
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i<100) && (!found) ; ++i) {
5 if (i==50) break;
6 if (i==20) found=true;
7 if (i==30) found=true;
8 }
9 printf("foo\n");
10 }
Figure 8.4: Code coverage at condition level
Squish Coco’s analysis is not limited to control statements (such as do... while or if... then constructs). For example, CoverageScanner also instruments the assignment of Boolean expressions. Constant or static expressions are not instrumented since their values are computed at compile time or once during program initialization. In the following example, the instrumented Boolean expressions are underlined:
void foo() { static bool a = x && y; bool b = x && y; bool c = b; int d = b ? 0 : 1; }
Figure 8.5: Code coverage of assignments
- 41 - froglogic GmbH Performance
The statement static bool a = x && y is not instrumented because it is static. And the statement bool c = b is not instrumented because the expression is a value (albeit a Boolean value), not a Boolean expression. After the detection phase, CoverageScanner inserts the instrumentation statements into the code. Code insertion is described in detail in Section C.
8.1.2 Result of coverage analysis
The CoverageBrowser program features a graphical user interface which it uses to display the analyzed results of the instrumentation. CoverageBrowser uses color coding to indicate the status of the statements. For this manual we have used the same colors as the program, but in addition we have applied text effects to aid those who read a monochrome print out.
• bold green indicates statements which are executed.
• shaded bold red indicates statements which are not executed.
• italic orange indicates statements which are partially executed. These are Boolean expressions or conditional statements which were always true or always false. The output for the example would be:
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i<100) && (!found); ++i)
5 {
6 if (i==50) break;
7 if (i==20) found=true;
8 if (i==30) found=true;
9 }
10 printf("foo\n");
11 }
Figure 8.6: Code coverage output
The expression i<100 is partially executed because is was always true. The expressions i==50 and i==30 are always false, yet they are shown as being fully executed. Since these expressions are incorporated in an if ... then statement, it is not necessary to check whether they are true. (The non-execution of the break; and found=true; statements provide the same information).
8.2 Performance
The insertion of instrumentation increases the code size and also affects the instrumented application’s performance—it will use more memory and run slower. For non-conditional expressions the instrumentation code involves a simple write to a fixed memory location. However, for conditional expressions more detailed analysis is needed and this is more computationally expensive. Overall, an instrumented application will be from 60% to 90% larger and will run 10% to 30% slower. i Detailed measurements are available in the Appendix (see Chapter D, page 198).
- 42 - froglogic GmbH Statistics
8.3 Statistics
Some developers write more lines of code than others simply by using a particular coding style—for example by putting opening braces on a line of their own rather than on the same line as, say, an if statement. In view of this, Squish Coco computes its statistics using a more subtle measure that isn’t susceptible to the the vagaries of coding style: its calculations are based on the number of executed instrumented instructions compared with the total number of instrumented instructions.
Every instrumented simple instruction (return, break, the last instruction of a function, etc.) is recorded by one single instrumentation. A fully instrumented condition in an IF...THEN...ENDIF statement uses two instrumentations: one for the true case and one for the false case. If the code is only partially instrumented, only one condition is recorded (either the false case or the true case). The statistics themselves depend of the type of instrumentation. It is not generally possible to compare code coverage at branch level with that at decision level: having 80% coverage at decision level does not tell us anything about the coverage at branch level, which could be bigger or smaller. The only thing we can be sure of is that reaching 100% coverage is more difficult with condition coverage than with decision coverage or branch coverage. In our example, the coverage is between 60% and 75%. The following table shows the details for each instrumentation.
Code coverage type Instrumentations Executed Coverage Branch (see Figure 8.7, page 43) 5 3 60% Decision partial (see Figure 8.9, page 44) 9 7 77% Decision full (see Figure 8.8, page 44) 13 9 69% Condition partial (see Figure 8.11, page 45) 12 9 75% Condition full (see Figure 8.10, page 44) 15 10 66%
In the examples below, the details of the calculations are displayed using subscripts. The first number shows how many instrumentations were executed; the second is the number of instrumentations in total.
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i<100) && (!found); ++i)
5 { 6 if (i==50) break; 0/1[not executed] 7 if (i==20) found=true;1/1[executed] 8 if (i==30) found=true; 0/1[not executed] 9 }1/1[executed] 10 printf("foo\n"); 11 }1/1[executed]
Figure 8.7: Code coverage statistics for full branch level instrumentation
- 43 - froglogic GmbH Statistics
1 void foo()
2 {
3 bool found=false; 4 for (int i=0; (i<100) && (!found)2/2[was f alse and true]; ++i) 5 { 6 if (i==501/2[was f alse but not true]) break; 0/1[not executed] 7 if (i==202/2[was f alse and true]) found=true;1/1[executed] 8 if (i==301/2[was f alse but not true]) found=true; 0/1[not executed] }1/1[executed] 9 printf("foo\n"); 10 }1/1[executed]
Figure 8.8: Code coverage statistics for full decision level instrumentation
1 void foo()
2 {
3 bool found=false; 4 for (int i=0; (i<100) && (!found)1/1[was f alse]; ++i) 5 { 6 if (i==501/1[was f alse]) break; 0/1[not executed] 7 if (i==201/1[was f alse]) found=true;1/1[executed] 8 if (i==301/1[was f alse]) found=true; 0/1[not executed] 9 }1/1[executed] 10 printf("foo\n"); 11 }1/1[executed]
Figure 8.9: Code coverage statistics for partial decision level instrumentation
1 void foo()
2 {
3 bool found=false; 4 for (int i=0; (i<100)1/2[was true but not f alse] && (!found)2/2[was f alse and true]; ++i) 5 { 6 if (i==501/2[was f alse but not true]) break; 0/1[not executed] 7 if (i==202/2[was f alse and true]) found=true;1/1[executed] 8 if (i==301/2[was f alse but not true]) found=true; 0/1[not executed] 9 }1/1[executed] 10 printf("foo\n"); 11 }1/1[executed]
Figure 8.10: Code coverage statistics for full condition level instrumentation
- 44 - froglogic GmbH Other Metrics
1 void foo()
2 {
3 bool found=false; 4 for (int i=0; (i<100)1/2[was true but not f alse] && (!found)2/2[was f alse and true]; ++i) 5 { 6 if (i==501/1[was f alse]) break; 0/1[not executed] 7 if (i==201/1[was f alse]) found=true;1/1[executed] 8 if (i==301/1[was f alse]) found=true; 0/1[not executed] 9 }1/1[executed] 10 printf("foo\n"); 11 }1/1[executed]
Figure 8.11: Code coverage statistics for partial condition level instrumentation
8.4 Other Metrics
8.4.1 Function Coverage
Squish Coco computes the function coverage according the information provided by the code coverage at branch level. This metric consists of the number of times that a function is called.
8.4.2 Line Coverage
Squish Coco supports also the code coverage metric at line level. For this metric, all source code lines that are executed are recorded. To compute it, it is necessary to compile the source code at least at the decision level with a full instrumentation. The line coverage is a natural metric which permits easily to see which lines of code are executed. But it is more inaccurate than the instrumentation at branch level and its results rely on the developer’s coding style. The following example illustrates this problem:
int main() { if (true) return 1; foo(); return 0; }
Executing it would produce the following result:
int main() { if (true) return 1; // Executed foo(); // Not executed return 0; // Not executed }
This execution correspond to a coverage of 33%.
Since the first line of main contains two executed statements, splitting it in two would increases the number of executed lines, and so the test coverage. So we reformat the main function as follow:
- 45 - froglogic GmbH Other Metrics
int main() { if (true) return 1; foo(); return 0; }
The execution produces a code coverage of 66%:
int main() { if (true) // Executed return 1; // Executed foo(); // Not executed return 0; // Not executed }
Another way to increase the coverage by reformatting is to hide an uncovered statement behind an executed one. To do it it is only necessary to write the whole code of this main function in one line:
int main() { if (true) return 1; foo(); return 0; }
The execution produces a line coverage of 100%:
int main() { if (true) return 1; foo(); return 0; // Executed }
This small example illustrate how the result depends on how the source code is formated. Therefore, Squish Coco provides the line coverage as additional measurement to the decision and the condition coverage and so does not allow that a source code is only instrumented at line level.
- 46 - froglogic GmbH Testing Methodologies
Chapter Testing Methodologies 9
This chapter discusses the various testing strategies that can be carried out so as to make the best use of the Squish Coco tools.
9.1 Coverage Hits vs. Counts
If we want to set very stringent standards for an application to ensure that it is of the highest possible quality, executing just portions of its code, or executing code just once, is insufficient. To address this issue the Squish Coco tools provide two possible approaches:
Code Coverage Count: We can require that each portion of instrumented code is executed a minimum number of times. This can be achieved by adding to the instrumented code so that it counts each execution.1 Naturally, setting such minimums, and the extra bookkeeping itself, can make the application consume more memory and run a bit slower. An important disadvantage of this approach is that loops in the source code end up having a high count with only a single execution, so counting this way is not always useful.
Test Coverage Count: We can require that each portion of instrumented code is executed by a minimum number of test items. This avoids the code coverage count disadvantage regarding loops, since by setting a minimum number of test executions for each portion of instrumented code, we give equal value to loops, recursions, and to instructions that are executed only once.
9.2 Testing Strategies
Squish Coco can be adapted to fit in with many different testing strategies, from unit tests during early development to complete product validation (e.g., acceptance tests). CoverageBrowser supports the merging of code coverage results at each stage of testing and so provides a precise overview of the software’s overall quality.
1This can be done using CoverageScanner’s --cs-on command line option.
- 47 - froglogic GmbH Strategies
9.2.1 Manual White Box Tests
White box testing (a.k.a. clear box testing, glass box testing or structural testing) uses an internal perspective of the system to design test cases based on internal structure. It requires programming skills to identify all paths through the software. The tester chooses test case inputs to exercise paths through the code and determines the appropriate outputs.
Definition from Wikipedia
Interactive white box tests are easy to do using Squish Coco: once the application has been compiled with CoverageScanner, the test engineer can execute the it, and after each execution they can import the execution report into CoverageBrowserfor analysis.
9.2.2 Manual Black Box Tests
Black box testing takes an external perspective of the test object to derive test cases. These tests can be functional or non-functional, though usually functional. The test designer selects valid and invalid input and determines the correct output. There is no knowledge of the test object’s internal structure.
Definition from Wikipedia
Black box tests assume that test engineers have no knowledge of the application’s internals—they work purely in terms of inputs and the corresponding expected outputs (whether of data or of error reports for invalid input). For this kind of testing it is possible to follow the same pattern as for white box testing, but without providing the test engineer with the instrumentation database (.csmes file) generated by CoverageScanner. The application itself will generate an execution report which can be analyzed after the testing cycle. However, this approach has the disadvantage that test engineers cannot manage their own execution reports (e.g., to add comments). Squish Coco has a facility which makes black box testing possible without disadvantaging test engineers. The CoverageBrowser tool can generate an instrumentation database (.csmes file) which does not contain any source code or source browsing information. This allows test engineers to manage the list of tests and to view the statistics, but it does not provide any source level coverage information.
9.2.3 Unit Tests
Unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit could be an entire module but is more commonly an individual function or procedure. In object-oriented programming a unit is often an entire interface, such as a class, but could be an individual method.
Definition from Wikipedia
CoverageScanner supports the saving of the code coverage data generated by each unit test using its library. (see Chapter 22.2.3, page 125) As long as all objects are well compiled2, it is possible to merge the unit tests instrumentation database into those of the real application. And so see the code coverage of the unit tests on the main application.
9.2.4 Automatic Tests
Squish Coco allows us to annotate the execution report with the execution name and state. This does not require any specific library—the information can be inserted directly by editing the execution report before and after the execution of a test item.
2Well compiled means that a source file is not be compiled twice or at least it is not be compiled with different preprocessor options.
- 48 - froglogic GmbH Strategies
This makes it possible to integrate the report into an automatic testing framework or for use with a test executing script. (see Chapter 22.1, page 123)
- 49 - froglogic GmbH CoverageBrowser Part III Reference Manual
- 50 - froglogic GmbH Introduction
Chapter Introduction 10
CoverageBrowser is a graphical user interface program which enables the user to analyze the test coverage. CoverageBrowser is typically used in the following way:
1. Load an instrumentation database (a .csmes file) that was generated by CoverageScanner.
2. Load a corresponding execution report (a .csexe file). There may be several reports to choose from: CoverageBrowser displays them in a tree view where they can be selected or deselected individually for coverage analysis. 3. Search for untested code segments. 4. Mark dead code or code which cannot be tested as “manually validated”.
5. Add comments to the instrumented code.
CoverageBrowser saves all these data (execution reports, comments, etc.) to the instrumentation database.
10.1 Command Line Arguments
Syntax:
coveragebrowser -m
Where:
-m
-e
- 51 - froglogic GmbH Black box and white box testing
Chapter Black box and white box testing 11
CoverageBrowser can be used both for white box testing and black box testing: If no source code information is available in the instrumentation database (i.e., in the .csmes file), CoverageBrowser will switch to black box testing mode. In this mode, CoverageBrowser has a simpler user interface that does not provide the functionality that is possible only with access to the source code. Nevertheless, even with this reduced functionality, it is still possible to import and manage executions.
A black box instrumentation database can be generated by clicking on "File¡Generate Black-Box Configuration...". It is possible to merge such a database into a white box configuration at a later stage.
Figure 11.1: CoverageBrowser’s simplified user interface for black box tests
- 52 - froglogic GmbH The windows of CoverageBrowser
Chapter The windows of CoverageBrowser 12
12.1 The Executions Window
This window is present in both black box and white box testing mode. It shows all application executions that have been done, including details of their code coverage.
12.1.1 Principles
Executions of the instrumented application are displayed in a tree view in the "Executions" window. CoverageBrowser uses a slash ‘/’ as a separator for grouping measurements together. For example, the tests shown in Figure 12.1, page 53 have the following names:
• SubTest/test1 • SubTest/test2 (12) • SubTest/test2 (2)
Code Coverage Execution State Statistics
Selection using Selection Wildcard Expressions
or Regular Expressions
Figure 12.1: The Executions View
The checkbox next to each item can be used to select executions. The "Sources", "Functions" and "Source Viewer" windows only display the code coverage status of the selected executions.
The code coverage percentage of the tests is visualized by gray horizontal bars in the "Coverage" column. The input field allows to filter the output with regular expressions (see Chapter 13.1, page 63). Click the button to select all the visible executions (i.e., all those that have not been filtered out). Or click the button to unselect all the executions.
For a more finely controlled filter, click the "..." button: This will pop up a dialog by which it is possible to set filtering depending on the execution state and the comments.
- 53 - froglogic GmbH The Executions Window
Note that the text which is filtered is a test execution’s full name—for example, SubTest/test1.
- Click the + button to switch to the execution comparison analysis mode (see Chapter 12.1.3, page 56).
The user can set the state of the executed test by clicking into the "State" field of a test. The new state can be any of the following:
"Unknown": Default state.
" Passed ": This state is used to mark the test as passed. The background colour of the "State" field is then green.
" Failed ": This state is used to mark the test as failed. The background colour of the "State" field is then red.
" Need manual check ": This state is used to indicate that the test must be done manually. The background colour of the "State" field is then orange. i The name of the test item and its state can also be defined by an external test suite. (see Chapter 22, page 123)
It is possible to rename, delete or merge executions, or add comments to them through the use of CoverageBrowser’s context menus and dock windows1. One can use regular expressions to identify the executions to which these modifications are applied. (The regular expression syntax is described in Chapter 13.1, page 63.) Before regular expression-driven actions are carried out, CoverageBrowser shows a preview of what effects the change would have.
To delete executions, right-click into the "Executions" window and select "Delete multiple executions..." from the context menu. A window appears into which the name of the executions can be entered. Here are some examples: • To delete all execution using the wildcard syntax: Execution Name: *
• To delete all executions in TESTS using the wildcard syntax: Execution Name: TESTS/*
• To delete all executions in TESTS using the regular expression syntax: Execution Name: =TESTS/.* To rename executions, select "Rename multiple executions..." from the context menu. A window appears (see Fig- ure 12.2, page 55) into which expressions for the old and new names of the executions can be entered. Here are some examples:
• To move all executions to the directory TESTS, set: Actual Execution Name: =.* New Execution Name: TESTS/&
• To move all executions in the directory TESTS to the directory OLD, set: Actual Execution Name: =TESTS/(.*) New Execution Name: OLD/\1
• To rename all executions in all directories in testname [directory], set: Actual Execution Name: =([^/]*)/([^/]*) New Execution Name: \2 [\1]
1After merging, the executions used as the source of the new merged execution are deleted.
- 54 - froglogic GmbH The Executions Window
Regular Expression
Executions to Rename
New Execution Name
Preview
Error
Error Message
Figure 12.2: Renaming with regular expressions
12.1.2 Loading an Execution Report
The execution report is produced when an instrumented application finishes execution. Its name is defined by the initialization function __coveragescanner_install() of the CoverageScanner library (see Chapter 18.1.1, page 86) and always ends with ".csexe". The report contains the list of all executed code segments for each application that was run. The execution report is never overwritten; execution data are always appended.
To load an execution report, click on "File¡Load Execution Report..." or the icon on the toolbar. A dialog like that in Figure 12.3, page 55 will appear.
Execution result file name
File watch Script file
Default execution name Script arguments
Default execution status
Import policy
Figure 12.3: Loading Execution Report Dialog
The file can be loaded directly or by a script. To load it directly, click on "File" and enter the path of the ".csexe" file to load into the free form input box, or use the browse button. If the box at "Delete after loading" is selected, the execution report file will be deleted after it has been loaded. Select the option "Open this dialog" for the field "When file becomes modified" to reopen the dialog automatically when the execution report has been modified.
The "Name" field allows the user to specify the name of the imported instrumentation if this is not already specified in the execution report (see Chapter 18.1.2, page 87). It is also possible to set the status (“Passed”, “Failed” or “To be checked manually”) of all imported executions. If more than one instrumentation is imported, an index numer is appended to its default name to make it unique.
The option "Import Preprocessing" allows to select the behaviour in the case of conflicts or redundant executions:
- 55 - froglogic GmbH The Executions Window
"Ignore Duplicate Executions": Executions which have executed the same code as an execution which are already imported are ignored.
"Import Duplicate Executions": Executions are imported if at least one instrumented source code line is executed.
"Import Duplicate And Empty Executions": All executions are imported.
"Merge all executions with the same name together": All executions with the same name are merged. The execution counts of all instrumentations are added.
Invalid executions are not imported. If "Display import summary" option is selected, a summary is shown after the operation is completed (see figure 12.4, page 56).
Figure 12.4: Loading Executions - Summary
If the execution report is not accessible through the file system, a script can be used. The script must print the content of the execution report to the standard output (stdout). The standard error output (stderr) is displayed on the screen and can be used for debugging. On success, the script must return the value 0.
12.1.3 The Execution Comparison Analysis Mode
- The Execution Comparison Analysis mode is activated by clicking the button + . In this mode, one or more executions must be selected as references and others as those to analyze. CoverageBrowser then shows only those lines as covered that were executed in the analyzed executions but not in the reference executions. Similarly, the coverage statistics displayed in the "Sources" list contain only the percentage of instrumented statements that were executed only by the executions to analyze. With Execution Comparison Analysis mode it is therefore possible to find changes in the coverage between different runs of a program.
To select the reference executions, use the checkboxes in the "Reference" column at the right of the "Executions" window. They must be selected first. Then select the executions to analyze as before in the "Execution" column. If the execution to analyse is present in the list of reference executions, it is implicitly removed from the list. So i if execution A is compared to executions A and B, CoverageBrowser actually compares execution A only with B, since comparing A with itself will provide no useful information.
- 56 - froglogic GmbH The Source Browser Window
Activation
Reference List
Coverage Statistic of Execution of lines which Analysed are only coveraged by the selected execution
Instrumented lines which are only coveraged by the selected execution are displayed
Figure 12.5: Execution Comparison Analysis Mode
12.2 The Source Browser Window i This feature is not available for black box testing.
The "Sources" window can be displayed by clicking on "View¡Sources".
Source
Header and Statistics Source Files per File Compiled
Source excluded from the instrumentation
Figure 12.6: Source Browser Window
Each line represents a source file. For C and C++, there are sub-entries for included header files which have been instrumented. When an item is clicked, the "Source Window" is displayed.
The "Coverage" column displays a rudimentary code coverage statistics for each source code file. There is an underlying horizontal bar in each field whose length represents the fraction of the code that is covered. The color of the main part of the bar is selected according to code coverage statistics for each file and the value of the thresholds (see Chapter 15.3, page 72).
If parts of the code have been manually validated, then there is a second, gray, horizontal bar at the left of the "Coverage" column; its width represents the number of tests that have been only manually validated. The full length of the column still represents the fraction of all tests that have been validated. When the window however also contains a "Manual Validations" column, the gray horizontal bar is only displayed there.
- 57 - froglogic GmbH The Function Browser Window
The checkbox in front of each source file allows to exclude or include the source file to the statistics computation. If excluded, the file is treated as if it and its functions had not been instrumented. The input field allows to filter the content of the window with regular expressions (see Chapter 13.1, page 63). The filter expression refers to the full path of the source file. (Example: c:\directory\file.cpp)
Icon Shortcut Description Ctrl+Shift+F Previous source file Ctrl+F Next source file
Table 12.1: Source Browser - Shortcuts
12.3 The Function Browser Window i This feature is not available for black box testing.
The "Functions" window can be opened by clicking on "View¡Functions".
Prototype
Namespace or Class
Statistics Method
Figure 12.7: Function Browser Window
It displays the code coverage statistics for all functions, classes and namespaces. A click on an item in the window shows the code for the corresponding object in the "Source Viewer" window and highlights it.
The "Coverage" column displays a rudimentary code coverage statistics for each function. There is an underlying horizontal bar in each field whose length represents the fraction of the code that is covered. The color of the main part of the bar is selected according to code coverage statistics for each file and the value of the thresholds (see Chapter 15.3, page 72).
If parts of the code have been manually validated, then there is a second, gray, horizontal bar at the left of the "Coverage" column; its width represents the number of tests that have been only manually validated. The full length of the column still represents the fraction of all tests that have been validated. When the window however also contains a "Manual Validations" column, the gray horizontal bar is only displayed there. The input field allows to filter the output with regular expressions (see Chapter 13.1, page 63). The filter expression refers to the full names of the items, including the class name and the namespace. (Example: MyNamespace::MyClass::MyProc)
12.4 The Source Viewer Window i This feature is not available for black box testing.
- 58 - froglogic GmbH The Source Viewer Window
12.4.1 Source Display
The "Source Viewer" window can be displayed by clicking on "View¡New Source Window".
Selected Folding Instrumentation Filter
Unselected
Comment Instrumentation Mark
Test and Code Coverage Count
Figure 12.8: Source Window
CPP The "Source Viewer" window displays the source file or its C or C++ preprocessed view. Clicking on C++ enables the user to toggle between the 2 different views. The source code is colored with code coverage instrumentations. The colors used are described in section 12.4.2, page 59. By selecting an area with the mouse, corresponding instrumentations are highlighted and a detailed description of them is displayed in the "Explanation" window (see Chapter 12.5, page 61). It is possible to navigate between instrumentations using the navigation buttons and . Navigation buttons in yellow, blue, red and green permit to jump to the next or previous comments, manually validated instrumentations, non-executed code parts or executed code parts. Clicking on the source code selects the nearest instrumentation. If a comment is entered for an instrumentation, the icon is displayed in the margin. On the right side, CoverageBrowser displays the test coverage count2 or the code coverage count3 for each line. If a source code line contains more than one instrumentation, CoverageBrowser display the range of their counts.
Mouse Wheel Description Wheel Scroll up/down Ctrl+Wheel Zoom in/out Shift+Wheel Next/previous instrumentation
Table 12.2: Source Display - Mouse Wheel
12.4.2 Color Convention
Instrumentations are displayed in a source window using different colors:
Green - "Executed": An instrumentation is displayed in green when the code has been executed.
Orange - "Partially Executed": An instrumentation is marked as "Partially Executed" when it is not completely executed. This occurs when a Boolean expression was only true or false for example. In the case of a source code line which contains more than
2The test coverage count is obtained by generating the instrumentation with CoverageScanner using the option --cs-hit 3The code coverage count is obtained by generating the instrumentation with CoverageScanner using the option --cs-count
- 59 - froglogic GmbH The Source Viewer Window
Icon Shortcut Description Ctrl+Shift+B Previous comment Ctrl+B Next comment Ctrl+Shift+U Previous unexecuted code Ctrl+U Next unexecuted code Ctrl+Shift+T Previous executed code Ctrl+T Next executed code Ctrl+Shift+V Previous manually validated instrumentation Ctrl+V Next manually validated instrumentation Ctrl+Shift+I Previous instrumentation Ctrl+I Next instrumentation Ctrl+Shift+F Previous source file Ctrl+F Next source file Ctrl+J Open a new source window
CPP C++ Ctrl+Shift+J Switch between the preprocessor view and the original source Ctrl+K Add/Edit Comments Ctrl+Shift+K Remove Comments ok Ctrl+W Mark as Validated ok Ctrl+Shift+W Clear Validation Flag Ctrl+Z Undo Ctrl+Shift+Z Redo
Table 12.3: Source Display - Shortcuts
one instrumentation, the line is marked as "Partially Executed" when one of its instrumentations has not been "Executed". A detailed information is displayed in the "Explanation" window (see Chapter 12.5, page 61).
Red - "Never Executed" or "Execution count too low": An instrumentation is displayed in red when the code is never executed or when the execution count is lower that than the execution count requested.
Magenta - "Dead-Code": An instrumentation is displayed in magenta when the code cannot be executed.
Blue - "Manually Set To Be Executed": The user has the possibility to mark an instrumentation as ’Manually Validated’. This is usually to exclude dead code or code which cannot be tested for code coverage statistics. This state is only relevant if executions are in a "Never Executed" or "Partially Executed" state.
Gray - "Unknown" or "Hidden": Gray is used when no information about instrumentation is available. This occurs when no executions are selected or when comparing executions of tests (see Chapter 12.1.3, page 56).
12.4.3 Comments i This feature is not available for black box testing.
12.4.3.1 Editing Comments
It is possible to add a comment by selecting an instrumentation and clicking on the context menu entry "Add/Set Comment", the main menu entry "Instrumentation¡Add/Set Comment" or the icon on the toolbar.
The "Comment" Window 12.9, page 61 appears and allows a comment to be edited. The most recently entered comments can be retrieved by clicking on the "Last Comments" selection field. Basic text formatting is possible using the integrated toolbar buttons (see 12.4, page 61).
- 60 - froglogic GmbH The Explanation Window
Edit Field Text Formating
Last Comment
Comments Length
Figure 12.9: Comment Editing
If a minimal length for a comment is set, the comment can only be entered if this is reached (see Chapter 15.2, i page 72).
The comment is printed in the explanation in a yellow box and the icon ( ) is displayed in the source window near the line number.
Icon Shortcut Description Ctrl+S Strikeout Ctrl+B Bold Ctrl+I Italic Ctrl+U Underline Ctrl+J Justify Ctrl+R Align Right Ctrl+L Align Left Ctrl+M Center Ctrl+Z Undo Ctrl+Shift+Z Redo
Table 12.4: Comments - Shortcuts
12.4.3.2 Removing Comments
It is possible to remove a comment by selecting an instrumentation and clicking on the context menu entry "Clear Comments", the main menu entry "Instrumentation¡Clear Comment" or the icon on the toolbar.
12.5 The Explanation Window i This feature is not available for black box testing.
The "Explanation" Window 12.10, page 62 is a docking window which is automatically updated with a detailed description of the selected instrumentations of the source window. For each instrumentation, the following information is displayed: 1. A short description of the instrumentation state (see Chapter 12.4.2, page 59). 2. The preprocessed source code which is concerned by the instrumentation. 3. For Boolean expressions, the truth-table which shows executed and unexecuted states. 4. The list of executions which are executing the portion of code.
- 61 - froglogic GmbH The Statistics Window
5. User comments.
Truth-Table
Preprocessed
State Source
Comment
Figure 12.10: Explanation Window
CoverageBrowser displays the truth-table in the case of a Boolean expression which is partially executed. The truth-table indicates which value the expression has or has not reached during execution. Example: the truth-table 12.5, page 62 indicates that the expression was false but not true.
TRUE FALSE no yes
Table 12.5: Truth-Table Example
12.6 The Statistics Window
The "Statistics" Window 12.11, page 62 is a docking window which is automatically updated with the code coverage statistic for the whole project.
Figure 12.11: Statistics Window
If parts of the code are manually validated, then their percentage is also displayed in the coverage statistics. The bar chart has then two regions: the percentage of the manually validated code at the left and then the percentage of the code that is covered by the automatic tests. The numbers in the bar chart refer to all validated code, that which was manually validated together with that covered by tests.
By clicking the "..." button, the two kinds of validation are split into two separate bars.
Figure 12.12: Statistics Window with Manual Validation, Split
- 62 - froglogic GmbH Working with CoverageBrowser
Chapter Working with CoverageBrowser 13
13.1 Filtering with wildcards or regular expressions
CoverageBrowser provides a generic filtering mechanism of rows using wildcard or regular expressions. Wildcard expressions are activated per default and regular expressions are selected when the expression starts with an equal sign (’=’). Clicking on the filter icon converts the expression from wildcard into regular form as far as this is possible and vice versa.
Icon Description .* The filter uses regular expression syntax. ?* The filter uses wildcard syntax. ! Syntax error. More information are displayed in the status bar.
Table 13.1: Filter States
13.1.1 Wildcard Expressions
Element Meaning * any characters (0 or more) ? any character [...] set of character
Example: foo*bar match any tests containing the string foo followed by bar.
13.1.2 Regular Expression
The first character must be ’=’ to activate the regular expressions.
- 63 - froglogic GmbH Filtering
13.1.2.1 Pattern matching
Element Meaning c Any character represents itself unless it has a special regexp meaning. Thus c matches the character c. \c A character that follows a backslash matches the character itself except where mentioned below. For example if you wished to match a literal caret at the beginning of a string you would write \^. \a This matches the ASCII bell character (BEL, 0x07). \f This matches the ASCII form feed character (FF, 0x0C). \n This matches the ASCII line feed character (LF, 0x0A, Unix newline). \r This matches the ASCII carriage return character (CR, 0x0D). \t This matches the ASCII horizontal tab character (HT, 0x09). \v This matches the ASCII vertical tab character (VT, 0x0B). \xhhhh This matches the Unicode character corresponding to the hexadecimal number hhhh (between 0x0000 and 0xFFFF). \0ooo (i.e., zero ooo) matches the ASCII/Latin1 character corresponding to the octal number ooo (between 0 and 0377). . (dot) This matches any character (including newline). \d This matches a digit. \D This matches a non-digit. \s This matches a whitespace. \S This matches a non-whitespace. \w This matches a word character. \W This matches a non-word character. ^ The caret negates the character set if it occurs as the first character, i.e. immediately after the opening square bracket. For example, [abc] matches ’a’ or ’b’ or ’c’, but [^abc] matches anything except ’a’ or ’b’ or ’c’. - The dash is used to indicate a range of characters, for example [W-Z] matches ’W’ or ’X’ or ’Y’ or ’Z’. E? Matches zero or one occurrence of E. This quantifier means "the previous expression is optional" since it will match whether or not the expression occurs in the string. It is the same as E{0,1}. For example dents? will match ’dent’ and ’dents’. E+ Matches one or more occurrences of E. This is the same as E{1,}. For example, 0+ will match ’0’, ’00’, ’000’, etc. . . E* Matches zero or more occurrences of E. This is the same as E{0,}. The * quantifier is often used by a mistake. Since it matches zero or more occurrences it will match no occurrences at all. For example if we want to match strings that end in whitespace and use the regexp \s *$ we would get a match on every string. This is because we have said find zero or more whitespace followed by the end of string, so even strings that don’t end in whitespace will match. The regexp we want in this case is \s +$ to match strings that have at least one whitespace at the end. E{n} Matches exactly n occurrences of the expression. This is the same as repeating the expression n times. For example, x{5} is the same as xxxxx. It is also the same as E{n,n}, e.g. x{5,5}. E{n,} Matches at least n occurrences of the expression. E{,m} Matches at most m occurrences of the expression. This is the same as E{0,m}. E{n,m} Matches at least n occurrences of the expression and at most m occurrences of the expression. () Permits to group expressions into sub-expressions. | Alternative. Example:"aaa|bbb" matches the string "aaa" or "bbb".
13.1.2.2 String substitution
Element Meaning & Matched expression \n sub-expression number n. Example: the regular expression is "(.*):([0-9]*)" matches the string "joe:18". The replacement string "\1 is \2" will produce the result: "joe is 18"
- 64 - froglogic GmbH Code Coverage Level
13.2 Code/Test Coverage Level
The menu entry "Instrumentation¡Level:x" permits to set the targeted code coverage count or, if the compiled with instrumentation hit support1, the targeted test coverage count. The level is corresponding of the number of code/test coverage count necessary to consider that an instrumented code is executed. Example: Setting the level to 10, will made necessary to execute 10 times the each line of the source code if compiled with code coverage count. If compiled with code coverage hit, 10 execution runs need to execute each lines of the source code. The menu entry "Tools¡Test Coverage Count Mode" and the button 1.2.3. permits to switch between code coverage count and test coverage count analysis. This simulates the behaviour of the compilation with code coverage hit support2 when the project is compiled with code coverage count support3.
13.3 Code Coverage Algorithm
CoverageBrowser displays the code coverage analysis (branch, decision or condition) generated be CoverageScanner. But "Instrumentation¡Coverage Method¡Branch only" permits to reduce the analysis to the code coverage of branches. This produces the same result as compiling with the --cs-branch of CoverageScanner."Instrumentation¡Coverage Method¡Decision, Condition and Branches" permits to show the code coverage analysis at the level defined at the compilation. Here a short overview of the command line options necessary for each code coverage analysis method:
Coverage analysis CoverageScanner command line option Branch --cs-branch Decision with full instrumentation --cs-decision --cs-full-instrumentation Decision with partial instrumentation --cs-decision Condition with full instrumentation --cs-full-instrumentation Condition with partial instrumentation (default)
13.4 Optimized Execution Order
CoverageBrowser is able to calculate an optimized order of the executions (i.e.: the order of tests which permits to maximize the code coverage after each execution). This order is specially adapted to manual testing: following this order permits to execute first the tests which are giving a high code coverage and so detecting rapidly errors in the first test executions. To calculate the execution order proceed as follows:
1. Select a set of executions in the "Executions" window.
2. Click on "Tools¡Optimized Execution Order...". The window (see Figure 13.1, page 66) will de display.
1Command line option --cs-hit from CoverageScanner. To compile with code coverage count support, the option --cs-count must be added in the command line. 2CoverageScanner command line option --cs-hit 3CoverageScanner command line option --cs-count
- 65 - froglogic GmbH Comparing Releases
Execution name
Execution Order
Coverage of the 'n'
first executions
Figure 13.1: Optimized Execution Order
13.5 Comparing Code Coverage of Two Software Releases
CoverageBrowser is able to instrumentation database together in order to: 1. check is the modified/unmodified code is correctly tested. 2. find which tests are impacted by a source code modification. This feature is particularly adapted to compare two releases together which contains small modifications (bug fixes only) and to limit the tests of the modified code only. In this mode CoverageBrowser uses the following typographic rules:
Rule Source Window Method List Source List Execution List Normal font Identical4 source part Identical4 methods Identical4 files Executions available in both releases Bold Modified methods Modified files Bold+Underline New text inserted New methods New files New executions Bold+Strike Deleted text Deleted methods Deleted files Missing executions
CoverageBrowser comparison and difference algorithm is particularly designed for C and C++ source code and ignore white spaces and modifications in comments.
13.5.1 Reference Database
The reference database is the base instrumentation database which are used for the comparison. To select it click on "Tools¡Compare with..." and select a .csmes database. Switching the working database with the reference database can be performed by clicking on "Tools¡Switch databases".
Once the reference file loaded, additional filter possibilities are available in the "Executions", the "Sources" and the "Methods" window. This filters permits to show/hide, modified, new, deleted or identical procedures and source files.
The "Executions" window displays a mix between the executions of the reference and the current release: • stroked executions are only available in the reference release. The statistic computed is corresponding to the coverage for the reference release. This value can be interpreted as an expected code coverage when this tests get re-executed on the current release. • underlined executions are new tests. • executions available in both version are not highlighted.
- 66 - froglogic GmbH Changing the Instrumentation Database
13.5.2 Coverage analysis of modified/unmodified source code
CoverageBrowser is able to limit the code coverage analysis to the modified (resp. unmodified) functions. When selecting the coverage analysis on the modified (resp. unmodified) functions only, CoverageBrowser treat all unmodified (resp. modified) functions as if they are not instrumented. Limiting the code coverage analysis to modified functions can be a practical way to verify that the new features are tested and to identify the list of tests which have are impacted by a modification. To limit the code coverage to modified function (resp. unmodified functions) click on "Tools¡Analysis of Modified Methods" (resp. "Tools¡Analysis on Identical Methods").
13.6 Changing the Instrumentation Database
13.6.1 Merging Instrumentations
Clicking on the menu entry "File¡Merge with..." permits to import the executions, the source code and the instrumentations from other .csmes files. Comments and code mark as validated are merged together.
13.6.2 Importing Unit Tests
Clicking on the menu entry "File¡Import Unit Tests..." permits to import the execution report of unit tests into the current application. Only execution reports of source file present into the main application are imported, instrumentation of other source files (for example test code) are ignored.
13.6.3 Importing Reviewer Comments
Clicking on the menu entry "File¡Import Reviewer Comments..." permits to import comments and manual validations of a previous version of the current instrumentation database. Comments and manual validations of unmodified functions will be imported even if if the source code is modified.
- 67 - froglogic GmbH Generation of Reports
Chapter Generation of Reports 14
i This feature is not available for black box testing.
14.1 EMMA-XML Report
Clicking on the menu entry "Reports¡Export Statistics in EMMA-XML Format..." permits to export code coverage statistics of in EMMA-XML format. This output format generate only global statistics in a format that is compatible with EMMA. This format allows to use Squish Coco in tools that provide support for EMMA, notably giving an easy way to use Squish Coco with continuous integration servers like Hudson.
EMMA defines only four categories for coverage: classes, methods, blocks, and lines. Due to the fact that blocks does not make sense with Squish Coco, this cathegories have the following meaning:
EMMA category Meaning classes A class is considered executed if one of its method is called. Code which are not in a class are located in the class "" (empty). methods A method is covered if it was called. blocks Code coverage at branch level. lines Line code coverage. (if compiled with line coverage support) conditions Decision/Condition line coverage. (if compiled with Decision/Condition coverage)
14.2 Cobertura-XML Report
Clicking on the menu entry "Reports¡Export Statistics in Cobertura-XML Format..." permits to export code cov- erage statistics of in Cobertura-XML format. This output format generate only global statistics in a format that is compatible with Cobertura. This format allows to use Squish Coco in tools that provide support for Cobertura, notably giving an easy way to use Squish Coco with continuous integration servers like Hudson and SonarQube.
The statistics in the Cobertura report are computed a little bit differently as usual due to the limitations of the report format. The report is in fact a mis between line and condition coverage.
- 68 - froglogic GmbH JUnit Report
For this reason, we record as Cobertura-condition every branch, decision and condition instrumentation. So for example, the end of a function is marked as one condition to fulfill. Also, the report imposes that each line with a condition is also instrumented at line coverage level. This is not always the case (for example for empty functions) and so in this case we add artificially some instrumentations. And finally, if an instrumented statement is coded in more than one line, it is necessary to generate one instrumentation for each of this line. All this have an impact on the computation of the statistics for classes, method and sources. But the values obtained are comparable to the all other reports.
14.3 JUnit Report
Clicking on the menu entry "Reports¡Export JUnit Report..." permits to export the test result as JUnit report. This report does not contain coverage data and only list the test execution result (passed or failed) for each test item.
14.4 Statistics per Source File
Clicking on the menu entry "Reports¡Export Statistics per Source File" permits to export code coverage statistics of each source file in CSV format. The file contains the coverage of all combination of source files and executions selected and the execution status (see Figure 14.1, page 69). Comma and semi-colon can be selected as separator in the CSV file using the "File Type" input field.
Coverage per Executions
Coverage per Source and Executions
Coverage per Execution Status Source File
Global Coverage
Figure 14.1: Statistics Format - CSV table
- 69 - froglogic GmbH Statistics per Method
14.5 Statistics per Method
Clicking on the menu entry "Reports¡Export Statistics per Method" permits to export code coverage statistics of each function and procedure in CSV format.
14.6 HTML Report
Clicking on the menu entry "Reports¡Generate HTML Report..." permits to export code coverage statistics (per methods, source files, executions, . . . ) of the selected executions in HTML format. It permits also to list the manually validated and unexecuted code parts.
14.7 Text Report
Clicking on the menu entry "Reports¡Generate Text Report..." allows to generate a small text report in the form of one line per executed/unexecuted item. A distinct line format can be specified for executed or unexecuted lines. The following keywords are recognized:
%f: Source code file name
%l: Line number
%c: Column number
%m: Explanation
Example: Setting the field "Unexecuted Code Fragments" to "%f:%l: %m" will create a text file which contains all unexecuted code parts. Each line will look like as follows:
foo.cpp:55: Unexecuted: ’return;’
14.8 Patch Analysis
With the menu entry "Reports¡Patch File Analysis..." one can generate a report about the influence of a patch on the test coverage of a project, without running the test suite for the patched version.
Prerequisites are a project for which .csmes and .csexe files exist and are loaded into CoverageBrowser, and a diff file. Patch analysis works best with programs that have automatic tests and which are instrumented in such a way that the names of the tests are known to Squish Coco (see Chapter 22.1, page 123). Line coverage (--cs-line) and branch coverage (--cs-branch) should not be disabled. (They are on by default.)
The diff file must be in the “unified” format. It is generated by the Linux™ diff utility with the option -u, and is also the default output format of several version control systems.
Clicking the menu entry "Reports¡Patch File Analysis..." lets the dialog "Patch File Analysis" appear. It has the following entries:
- 70 - froglogic GmbH Patch Analysis
Title: The title of the report, both for HTML and CSV.
Patch File: Path of the patch file that contains the changes to the project.
Output: To select the output file and its type: Type: Either HTML or the Excel CSV format. The field to the right of the "Type" field contains the name and path of the report file that is generated.
Source Code Parameter: For HTML reports, to select the display of the annotated source code: Tooltip Maximal Size: The annotated patch file in the HTML report has a tool tip which displays the tests that have executed a certain line of code. This parameter sets the maximal number of tests that can appear in a tooltip. With a value of 0, no tooltips appear.
CSV Parameter: To select the format of the CSV report: Column Separator: The column separator symbol can be either a comma or a semicolon.
Sections: To choose the content of the report: Execution Statistics: Create a table that groups the tests by their results. It shows how many of the tests have passed, failed, need manual testing, and whose execution status is unknown. Executions: Create a list of the tests that execute code which is affected by the patch. For each test, the name and the execution result is shown. Source Code Statistics: Create a table that shows the influence of the patch on the test coverage. It shows how many lines in the patch are covered by a test, how many are not, and for how many lines Squish Coco could not find out whether they are covered. These numbers are shown for the lines that were removed, or added, and for all lines of the patch. Annotated Patch Source: Create an annotated version of the patch file. Each line of code in the patch is shown in red if it is removed, green if it is added, or otherwise gray. Also shown are the line numbers of the code lines, both before and after the patch is applied, and the number of tests that cover a line. This last field also has a tooltip that shows which tests cover the specific line. (The tooltip is only visible if "Tooltip Maximal Size" is set to a non-zero value.)
To generate the report, one clicks either the button "OK" or "Show": The second button also opens a browser window to show the generated report. Clicking "Apply" saves the values of the dialog entries without generating a report, while "Cancel" closes the dialog without saving anything.
- 71 - froglogic GmbH Preferences
Chapter Preferences 15
15.1 Save/Load Project
"Save/Restore window position": If this option is selected, the position of all windows and toolbars will be restored upon application restart.
"Reload automatically the last project": If this option is selected, the last project opened will automatically be reloaded upon application restart.
"Saves project automatically on exit": Saves the project file automatically without asking on application exit.
15.2 Comments
"Minimum Comment Size": The minimum comment size, is the minimum length requested for a comment.
"Do not request a comment when setting an item to the ’manually validated’ state": This option is used to allow the user to manually modify the state of an instrumentation without entering a comment.
Enabling this option should be avoided because modifying the state of an instrumentation should be i performed with a valid reason which should be recorded as a comment.
15.3 Thresholds
Thresholds are trigger values that control the background color of:
• the instrumented source files in in the "Sources" window.
• the instrumented classes or namespaces in the "Methods" window.
• the instrumented functions, methods or procedures in the "Methods" window.
- 72 - froglogic GmbH Cache
Description:
"Medium/High Coverage Level": If the statistic is above this value, the background color is set to green. Otherwise, the color is orange.
"Low/Medium Coverage Level": If the statistic is below this value, the background color is set to red. Otherwise, the color is orange.
15.4 Cache
Description:
"Execution": Maximum number of executions loaded into the RAM.
"Source": Maximum number of source files loaded into the RAM.
- 73 - froglogic GmbH CoverageScanner Part IV Reference Manual
- 74 - froglogic GmbH Introduction
Chapter Introduction 16
CoverageScanner is the application which inserts the instrumentation into the source code. It performs the following operations: 1. Runs the C preprocessor. 2. Instruments the preprocessed source code. 3. Compiles the source code using the native C or C++ compiler. Mainly, during normal usage, it replaces the native compiler. CoverageScanner does only support precompiled headers with Microsoft® Visual Studio® .NET tool chain. i CoverageScanner automatically deactivates this feature during compilation.
16.1 Invoking CoverageScanner
CoverageScanner works like a pre-processor which calls after instrumenting a source file the native compiler (Microsoft® cl.exe, GNU g++, . . . ). Several invocation method are possible: 1. Using the dedicated CoverageScanner executable for a specific compiler: CoverageScanner executable for a specific compiler is called to cs+’native compiler name’ or alternatively ’native compiler name’+-cs. (example: cl.exe is the Microsoft® C++ compiler and the corresponding CoverageScanner executable is cscl.exe or cl-cs.exe). CoverageScanner aims to work then as the native compiler with the difference that the code get instrumented. Example:
csg++ source.cpp -o application.exe
2. Using CoverageScanner compiler wrapper and activating the instrumentation using --cs-on in the command line arguments: Squish Coco provides also a replacement of the native compiler called wrapper which extends its command line argument to support the code coverage feature. To use it it is necessary to add at the first position in the PATH variable the path of CoverageScanner wrapper. The instrumentation get activated, when adding --cs-on to the command line argument. Example:
- 75 - froglogic GmbH Invoking CoverageScanner
export PATH=/opt/SquishCoco/wrapper/bin:$PATH g++ --cs-on source.cpp -o application.exe
Or, without modifying globally the PATH variable:
PATH=/opt/SquishCoco/wrapper/bin:$PATH g++ --cs-on source.cpp -o application.exe
3. Using CoverageScanner compiler wrapper and activating the instrumentation through COVERAGESCANNER_ARGS environ- ment variable: It is also possible to set the command line arguments of the wrapper using the COVERAGESCANNER_ARGS environment variable (see Chapter 17.1, page 77). In this case it is no longer necessary to modify the command line arguments. Example:
export PATH=/opt/SquishCoco/wrapper/bin:$PATH export COVERAGESCANNER_ARGS=’--cs-on’ g++ source.cpp -o application.exe
Or, to instrument a complete project built using GNU make:
COVERAGESCANNER_ARGS=’--cs-on’ PATH=/opt/SquishCoco/wrapper/bin:$PATH make all
4. Using --cs-compiler command line option of CoverageScanner: --cs-compiler permits to select which native compiler should be used. All other command line arguments, except those specific to CoverageScanner, are passed through the native compiler. Example:
coveragescanner --cs-compiler=g++ source.cpp -o application.exe
The native compiler path needs to appear in the PATH variable. This is necessary for finding the location of the i native compiler after insertion of the instrumentation code.
Compiler support is done with the help of a profile. It contains declarations which make it possible to adapt CoverageScanner to C, C++ and C# preprocessors, compilers and linkers. For a description of the compilers that are currently supported see Chapter G, page 213. For a description of the profile syntax see Chapter H, page 215
- 76 - froglogic GmbH Command Line Arguments
Chapter Command Line Arguments 17
Syntax:
cs
For the use of --cs-compiler and --cs-profile see Chapter 17.2.6, page 82. The command line arguments are prioritized by their order in the command line: the latest entered option has a higher priority than the first one. i Example: cscl --cs-include=foo.h --cs-exclude=foo.h will exclude the header file foo.h from the coverage analysis.
17.1 Environment variables
Apart from the command line, CoverageBrowser also reads options from the environment variable COVERAGESCANNER_ARGS. If more than one option is present, the options must be separated by spaces or tab characters (‘ ’ or ‘\t’). If both command line arguments are present and COVERAGESCANNER_ARGS is set, the command line options take precedence over the options in the environment variable. Therefore this variable should only be set temporarily, e.g. in a script, because otherwise it would influence all builds.
- 77 - froglogic GmbH List of options
17.2 List of options
17.2.1 File and path inclusion options
The following options allow to specify which files are included in the code coverage. The options apply to files on the command line and to those that are specified on #include statements.
One can use wildcards and regular expressions. In contrast to the common usage, the ‘*’ in wildcard expressions matches also the path separation characters. The syntax of regular expressions is described in Section 17.4, page 84. Options accumulate. If more than one of the file inclusion/exclusion option is present, the last one that matches a given file determines whether it is included or excluded. In the next options, name of the file in question is first converted to an absolute path.
--cs-include-file-abs-wildcard=
--cs-exclude-file-abs-wildcard=
Example: --cs-exclude-file-abs-wildcard=*.h excludes all files with the extension .h.
--cs-include-file-abs-regex=
--cs-exclude-file-abs-regex=
Example: --cs-exclude-file-abs-regex=^.*\.h$ excludes all files with the extension .h.
--cs-include-path=
--cs-exclude-path=
Examples for complex inclusion patterns:
• --cs-exclude-file-abs-wildcard=* --cs-include-file-abs-wildcard=*.cpp instruments only .cpp files.
• --cs-exclude-file-abs-wildcard=* --cs-include-file-abs-wildcard=*.cpp --cs-include-file-abs-wildcard=*.cxx instruments only .cpp and .cxx files.
• --cs-exclude-file-abs-wildcard=*/3rdparty/* does not instrument any files from the directory 3rdparty.
• --cs-exclude-file-abs-wildcard=*/3rdparty/* --cs-include-file-abs-wildcard=* instruments all files: The first filter rule has no effect.
Relative paths The following options refer to the name under which a file occurs in an #include statement or on the command line.
File paths and wildcard expression are converted to a canonical form which does not contain ‘.’, ‘..’, or a duplicated path separation character. (A ‘..’ at the beginning is however not removed.) On Microsoft® Windows, characters are also converted to lower case and every slash (‘/’) is replaced with a backslash (‘\’). Regular expressions are untouched. It is e.g. necessary to ensure that they do not contain duplicated path separators.
- 78 - froglogic GmbH List of options
--cs-include-file-wildcard=
Example: --cs-include-file-wildcard=../include/*.h includes all files with the extension .h located in the directory ../include/ and its subdirectories.
--cs-exclude-file-wildcard=
--cs-include-file-regex=
Example: --cs-include-file-regex=^\.\./include/[^/]*\.h$ includes all files with the extension .h located in ../include/.
--cs-exclude-file-regex=
Relative file names are computed relatively to the current working directory. For example, depending on the working directory one has to use different parameters in order to inhibit the instrumentation of a file src/test.c: If one is outside the directory src, one may use
$ csgcc src/test.c --cs-exclude-file-wildcard=src/test.c but when switching to src, the parameters must change:
$ cd src $ csgcc test.c --cs-exclude-file-wildcard=test.c
Setting inclusion/exclusion expressions on absolute paths should be preferred on complex build systems, since modern build system often change the current working directory. Working with absolute paths make it easier to ! set global rules for a project.
17.2.2 Function inclusion options
--cs-include-function-wildcard=
Example: --cs-include-function-wildcard=C::* includes all members of the class C.
--cs-exclude-function-wildcard=
Example: --cs-exclude-function-wildcard=*::Get* excludes all members of any classes which starts with Get.
--cs-include-function-regex=
--cs-exclude-function-regex=
If more than one inclusion/exclusion option is present, the last one that matches a given function determines whether it is included or excluded.
- 79 - froglogic GmbH List of options
17.2.3 Instrumentation options
--cs-off: Disables the code coverage analysis of CoverageScanner. This is equivalent to calling the native compiler or linker.
--cs-on: Activates the code coverage analysis of CoverageScanner. This option is set by default.
--cs-hit: Generation of code coverage hit instrumentation.
--cs-count: Generation of code coverage count instrumentation. (default)
--cs-branch: Generation of code coverage instrumentation of branches.
--cs-decision: Generation of code coverage instrumentation of branches and decisions.
--cs-condition: Generation of code coverage instrumentation of branches, decisions and conditions. (default)
--cs-function: Generation of function code coverage instrumentation. (default)
--cs-no-function: Disable the function code coverage instrumentation.
--cs-line: Generation of line code coverage instrumentation. (default) This option requires a full instrumentation and an instrumen- tation at decision or condition level.
--cs-no-line: Disable the line code coverage instrumentation.
--cs-partial-instrumentation: Suppression of redundant conditions.
--cs-full-instrumentation: No suppression of redundant conditions. (default)
--cs-no-exceptions: Do not instrument the catch block of a try... catch statement.
--cs-no-assignments: Do not instrument Boolean expressions in assignments.
--cs-boost: The code behind the macro BOOST_FOREACH is not instrumented. (this option is enabled per default)
--cs-no-boost: No specific handling for the Boost library.
--cs-qt4: Function filter which suppresses the instrumentation of class members (Q_OBJECT, qt_metacall, qt_metacast,...) generated by mocfor Qt4 library. The code behind the macro Q_FOREACH is also not instrumented. (this option is enabled per default)
--cs-no-qt4: No specific handling for the Qt4 library.
- 80 - froglogic GmbH List of options
--cs-qt3: Function filter which suppresses the instrumentation of class members (Q_OBJECT, className, tr, . . . ) generated by mocfor Qt3 library. (this option is enabled per default)
--cs-no-qt3: No specific handling for the Qt3 library.
--cs-no-annotations: No extraction of annotations (comments and manual validations) from the source files.
17.2.4 Debugging options
--cs-verbose-source-lines=
--cs-verbose=
--cs-verbose-file=
--cs-keep-instrumentation-files: Do not delete temporary instrumentation files. They will be left under random names in /tmp on Unix and in %TEMP% on Windows.
--cs-warnings=
17.2.5 Execution report options
--cs-output=
- 81 - froglogic GmbH List of options
%c: Current working directory. The current working directory is computed before executing the main() function. Even if the current working directory change, this path remains identical during the whole execution. %p: Process identifier. If for any reason it cannot be computed, %p will be replaced through an empty string. %t: Current time. If for any reason it cannot be computed, %t will be replaced through an empty string. %d: Current date. If for any reason it cannot be computed, %d will be replaced through an empty string.
--cs-lock-csexe: Create a lock file which lock the access of the .csexe file during its generation.
--cs-nolock-csexe: Inverse option to --cs-lock-csexe. The preprocessor output is transmitted to CoverageScannerthrough a temporary file.
--cs-dump-on-signal=int: Generate an execution report when a specific signal was received. The signal can be specified by a number (1 to 63) or using the conventional signal same (SIGINT, SIGTERM, . . . )
--cs-dump-on-event=string: Generate an execution report when a specific Microsoft® Windows event was received.
--cs-minimum-api: The CoverageScanner API generated using this command line option has less dependencies to external libraries as usual. This may be necessary for embedded systems or when compiling C# application with the command line switch /noconfig.
17.2.6 Miscellaneous options
--cs-compiler=
--cs-profile=
--cs-architecture=
--cs-libgen=
--cs-link-instrumentation-tables: Chain all instrumentation tables together during the link time and not during the run time.
--cs-library-after=
--cs-library-at-end=
--cs-parallel-build=
- 82 - froglogic GmbH List of options
--cs-cpp-ext=: List of file extensions corresponding to C++ files (ex: .cpp,.cxx)
--cs-c-ext=: List of file extensions corresponding to C files (ex: .c)
--cs-native-toolchain=
--cs-option-file=
--cs-vs2010-lambda: Instrument directly the expression of a return statement of a lambda function and permits a better support of lambda expression on Microsoft® Visual Studio® 2010. This is only valid for C++11 compiler or Microsoft® Visual Studio® 2010. For C or older C++ compiler, this option has no effect.
--cs-no-cspch: If set, the files with the extension .cspch are not generated. This files are only used to enhance the compilation speed when using precompiled headers.
--cs-architecture: Internal option of CoverageScanner which permits to select alternate settings.
--cs-no-abort-on-error: By default, any errors raised while instrumenting C/C++ source code will stop the compilation. This behaviour can be overridden with the option --cs-no-abort-on-error. If it is set, the compiler wrapper will display an error message if a file cannot be instrumented, but then compile it again without instrumentation.
--cs-no-exit-handler-installation: Disable the automatic installation of the handler which saves the code coverage report. If this handler is not installed, no executions report will be generated when the application exits or when a custom signal or Windows event is received. Calling __coveragescanner_install() permits to install it manually.
--cs-enable-coveragescanner-library-abs-wildcard=
--cs-disable-coveragescanner-library-abs-wildcard=
--cs-memory-pool=
--cs-no-line-directive: Remove all #line directives from the code generated by the C/C++ preprocessor. The preprocessor is called by the CoverageScanner before instrumenting a file; afterwards the original compiler is called. On some systems, compiling a file this way does cause problems that do not occur when the compiler is called directly. Removing the #line statements then sometimes helps.
- 83 - froglogic GmbH Instrumenting using preprocessor defines
17.3 Instrumenting using preprocessor defines
CoverageScanner activates also the instrumentation if some defines are present in the command line option. This permits to instrument code with an IDE which does not permits to add custom command line options to the compiler or linker. Example: Defining COVERAGESCANNER_COVERAGE_ON is equivalent of adding --cs-on to the command line. Define Equivalence COVERAGESCANNER_RUNTIME_LOG --cs-verbose=api COVERAGESCANNER_DUMP_ON_EVENT --cs-dump-on-event=Global\COVERAGE COVERAGESCANNER_COVERAGE_ON --cs-on COVERAGESCANNER_COVERAGE_HIT --cs-hit COVERAGESCANNER_COVERAGE_COUNT --cs-count COVERAGESCANNER_COVERAGE_BRANCH --cs-branch COVERAGESCANNER_COVERAGE_DECISION --cs-decision COVERAGESCANNER_COVERAGE_CONDITION --cs-condition COVERAGESCANNER_COVERAGE_LOCK_CSEXE --cs-lock-csexe COVERAGESCANNER_COVERAGE_NOLOCK_CSEXE --cs-nolock-csexe COVERAGESCANNER_COVERAGE_PARTIAL_INSTRUMENTATION --cs-partial-instrumentation COVERAGESCANNER_COVERAGE_FULL_INSTRUMENTATION --cs-full-instrumentation COVERAGESCANNER_COVERAGE_NO_EXCEPTIONS --cs-no-exceptions COVERAGESCANNER_COVERAGE_NO_ASSIGNMENTS --cs-no-assignments COVERAGESCANNER_MINIMUM_API --cs-minimum-api
17.4 Regular Expressions
Regular expressions are composed of items (see table 17.1, page 85 for the full list) and combining operators. These can be:
• [... ]: The brackets are used to define a set of characters. For example, [123] corresponds to the characters 1, 2 or 3. It is possible to define a range using the minus sign: [a-z] corresponds to all characters between a and z.
• [^... ]: The caret defines a set of characters which should not match. For example, [^0-9] corresponds to any character except a digit.
• ?: The question mark after a regular expression item makes it optional. Example: ab? matches the string a or ab.
• +: The plus operator makes it possible to repeat the regular expression item. Example: ab+ matches every strings ab... b.
• *: The star operator after a regular expression item signifies that the item is optional and can be repeated. Example: ab* matches every string ab... b and a.
• |: The pipe operator specifies an alternative. Example: a|b matches the character a or b.
• (... ): The parenthesis groups expressions into sub-expressions. Example: (true)|(false) matches the string true or false. • Regular Expression Items: The regular expression items are ASCII or magic characters (See table 17.1, page 85 for the complete list). If a magic character needs to be used as an ASCII character, it has to be escaped with a backslash (\).
- 84 - froglogic GmbH Regular Expressions
Element Description Equivalence Reverse Element ^ The caret marks the beginning of the string. For example, ^c: will only match file names with an absolute path on the drive C:. If you wish to match a literal ^ you must escape it by writing \^. $ The dollar marks the end of the string. For example \.cpp$ will match every C++ file name. If you wish to match a literal $ you must escape it by writing \$. \< Match the start of a word. \> Match the end of a word. . Any character \ Escape character. Example: \. corresponds to the character ’.’ and not any character \m Alphanumeric character [0-9a-zA-Z] \M \a Alphabetic character [a-zA-Z] \A \b Blank character [ t] \B \c Control character \C \d Digit [0-9] \D \g Printable character except space [] \G \l Lower-case character [a-z] \L \p Printable character \P \n Printable character except space or alphanumeric character \N \s White-space characters (space, form-feed , newline , carriage return , [ fnrtv] \S horizontal tab and vertical tab) \u Uppercase letter [A-Z] \U \x Hexadecimal digits [0-9a-fA-F] \X \w Alphanumeric character or underscore [0-9a-zA-Z_] \W
Table 17.1: Non ASCII Regular Expression Items
- 85 - froglogic GmbH Library calls
Chapter Library calls 18
18.1 C and C++ Library
The CoverageScanner library is generated during instrumentation and enables the user
1. to initialize the measurement system and select the name of the .csexe file, 2. to install a handler which saves the execution report on exit, 3. in a unit test scenario, to pass information about the tests to the execution report, especially the name of the test, comments and the test result, 4. to handle dynamically loaded libraries, and 5. to customize the output system for special needs.
The library functions are accessible from any instrumented file, without the need for an #include directive. When a file is instrumented, a preprocessor symbol __COVERAGESCANNER__ is defined: It can be used to exclude calls to the library functions from the compilation when instrumentation is switched off.
18.1.1 __coveragescanner_install()
Syntax:
void __coveragescanner_install(const char *appname) i This function is not needed for gcc, clang or Microsoft® Visual Studio®.
Initialize the CoverageScanner library. It should be the first function call in the main() function. The parameter appname contains the name of the executable. It is used to set the name of the measurement file to
__coveragescanner_install() registers an exit handler via atexit() in order to save the execution report whenever the application terminates. The handler calls the function __coveragescanner_save() to store the execution traces at the end of the program. It is called when the program terminates regularly and also if the signals SIGABRT, SIGTERM, SIGFPE, SIGILL, SIGINT, SIGSEGV and SIGTERM are received. Suggested usage:
int main(argc,char *argv[]) { #ifdef __COVERAGESCANNER__
- 86 - froglogic GmbH C and C++ Library
__coveragescanner_install(argv[0]); #endif ... }
Platform dependencies With gcc, clang or Microsoft® Visual Studio®, CoverageScanner automatically installs a min- imal handler. It saves the execution in the file
__coveragescanner_install() is not available for a compiler if its profile parameter CUSTOM_SETUP is set to NONE (see Chapter H, page 215).
18.1.2 __coveragescanner_testname()
Syntax:
void __coveragescanner_testname(const char *name)
Set the name of the current test. It will be saved in the execution report and be displayed in the "Executions" window (see Chapter 12.1, page 53) when the .csmes file is loaded into CoverageBrowser.
All code that is executed until the end of the program or the next call of __coveragescanner_save() is considered part of the test name. If tests get the same name, CoverageBrowser disambiguates them by numbers, like “MyTest (2)”.
Test names may be hierarchical, with hierarchy levels separated by slashes: If a program defines two tests, “group/test1” and “group/test2”, CoverageBrowser displays them as two tests, “test1” and “test2” under a common heading, “group”. This and the following functions until __coveragescanner_save() are especially intended for the use with i unit test frameworks (see Chapter 22, page 123).
18.1.3 __coveragescanner_teststate()
Syntax:
void __coveragescanner_teststate(const char *state)
Set the state of the current test. The parameter state may have the following values:
"PASSED": to indicate that the test was successful.
"FAILED": to indicate that the test was not successful.
"CHECK_MANUALLY": to indicate that it was not possible to determine whether the test was successful. If the function is called twice, the first state is overwritten. The test state will be saved in the execution report and be displayed in the "Executions" window (see Chapter 12.1, page 53) when the .csmes file is loaded into CoverageBrowser.
18.1.4 __coveragescanner_add_html_comment()
Syntax:
- 87 - froglogic GmbH C and C++ Library
void __coveragescanner_add_html_comment(const char *comment)
Set am HTML comment to the current execution. There is only one HTML comment active at any time, but it can be put together with several calls of __coveragescanner_add_html_comment(). In this case, the values of the various comment parameters are concatenated.
The full comment must follow the HTML syntax, but only the content of the
tag is used. A minimal commend might therefore be something like “My comment”.18.1.5 __coveragescanner_clear_html_comment()
Syntax: void __coveragescanner_clear_html_comment()
Remove the comment added by calls of the function __coveragescanner_add_html_comment().
18.1.6 __coveragescanner_save()
Syntax: void __coveragescanner_save()
Save the execution report and reset the status of all instrumentations. The coverage counters for each line of code and the execution status of the current test are reset, but the HTML comment stays unchanged.
18.1.7 __coveragescanner_clear()
Syntax: void __coveragescanner_clear()
Reset the coverage counters for all lines of code.
This is useful in a unittest framework. After the framework has started, __coveragescanner_reset() can be called. Then the activity of the framework at startup does not become part of the code coverage.
18.1.8 __coveragescanner_filename()
Syntax:
void __coveragescanner_filename(const char *name)
Set the file name of the execution report. The extension ’.csexe’ is added automatically. If name is NULL, the generation of execution reports is disabled. name may contain the following escape sequences:
%p: The process identifier, if it is provided by the platform, otherwise the empty string.
%w: The working directory. The value is computed when the CoverageBrowser library is initialized; it does not change afterwards.
%t: The current time, or the empty string if it cannot be computed.
%d: The current date, or the empty string if it cannot be computed.
- 88 - froglogic GmbH C and C++ Library
Platform dependencies The escape sequences are only available for a compiler if its profile parameter FILE_FORMAT_SPECIFIER is set to YES (see Chapter H, page 215). Date and time may not be available on some embedded platforms.
18.1.9 __coveragescanner_register_library()
Syntax:
int __coveragescanner_register_library(const char *library_name)
Register a shared library that was loaded on demand. The function must be called after a library has been loaded with dlopen() (UNIX®) or LoadLibrary() (Microsoft® Windows). After it is registered, the code coverage of the library is saved in the same coverage report file as the coverage of the main application.
It is possible to call __coveragescanner_register_library() more than once, but every call must be matched by a corresponding call of __coveragescanner_unregister_library(). For each library there is a use counter to keep track of the registrations. Returned values:
0: Success.
-1: Error: library could not be loaded.
-2: Error: invalid library name.
1: Success, but the library is already registered. The use count of the library has been incremented.
2: Success, but the library is not instrumented.
Platform dependencies __coveragescanner_register_library() and __coveragescanner_unregister_library() are not available for a compiler if its profile parameter PLUGIN_REGISTRATION_API is set to NO (see Chapter H, page 215).
18.1.10 __coveragescanner_unregister_library()
Syntax:
int __coveragescanner_unregister_library(const char *library_name)
De-register a shared library that was loaded on demand. The function must be called before a library is unloaded with dlclose() (UNIX®) or CloseHandle() (Microsoft® Windows). Returned values:
0: Success.
-1: Error: library was not registered.
-2: Error: invalid library name.
1: Success, but the library was not unregistered because its use count was not null.
2: Success, but the library is not instrumented.
- 89 - froglogic GmbH C and C++ Library
18.1.11 __coveragescanner_set_custom_io()
Syntax:
void __coveragescanner_set_custom_io( char *(*csfgets)(char *s, int size, void *stream), int (*csfputs)(const char *s, void *stream), void *(*csfopenappend)(const char *path), void *(*csfopenread)(const char *path), void *(*csfopenwrite)(const char *path), int (*csfclose)(void *fp), int (*csremove)(const char *filename) )
Define your own set of I/O functions. Parameters: csfgets: Read at most size - 1 characters from stream and store them into the buffer pointed to by s. Reading stops after an EOF or a newline is read. If a newline is read, it is stored in the buffer. A ‘\0’ is stored after the last character in the buffer. csfputs: Write the string s to stream, without its trailing ’\0’. csfopenappend: Open the file path for appending text at the end. csfopenread: Open the file path for reading. csfopenwrite: Open the file path for writing. csfclose: Flush and close the stream pointed to by fp. csremove: Remove the file filename from the file system.
By default, CoverageScanner writes the execution report (the .csexe file) to the file systems with help of common C library functions. In some situations, e.g. with embedded systems, it is necessary to use other methods. For this reason, CoverageScanner provides __coveragescanner_set_custom_io(), which allows to replace the I/O functions used by __coveragescanner_save(). Default implementation:
char *csfgets(char *s, int size, void *stream) { return fgets(s,size,(FILE *)stream); } int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); } void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); } void *csfopenread(const char *path) { return (void*)fopen(path,"r"); } void *csfopenwrite(const char *path) { return (void*)fopen(path,"w"); } int csremove(const char *filename) { return remove(filename); } int csfclose(void *fp) { return fclose((FILE*)fp); }
Some examples are available in Appendix F, page 204.
- 90 - froglogic GmbH C# Library
18.2 C# Library
CoverageScanner library is generated during the link process and enables the user to: 1. Generate the execution report. 2. Give a name to the executed tests.
3. Install a signal handler which saved the execution report on exit.
Additionally CoverageScanner sets the preprocessor define __COVERAGESCANNER__ which excludes this function from a normal compilation.
18.2.1 CoverageScanner.__coveragescanner_init()
Syntax:
void CoverageScanner.__coveragescanner_init()
CoverageScanner.__coveragescanner_init() initialize explicitly the CoverageScanner library. Calling this function is only necessary on C# DLL which are not calling the static module initializer automatically.
18.2.2 CoverageScanner.__coveragescanner_testname()
Syntax:
void CoverageScanner.__coveragescanner_testname(string name)
CoverageScanner.__coveragescanner_testname() sets the name of the test which is currently being executed. It will be saved to the execution report and is displayed in the "Executions" window (see Chapter 12.1, page 53) when the loaded in CoverageBrowser.
18.2.3 CoverageScanner.__coveragescanner_teststate()
Syntax:
void CoverageScanner.__coveragescanner_teststate(string state)
CoverageScanner.__coveragescanner_teststate() sets the state of the test which is currently being executed. The string parameter state can have the following values:
"PASSED": indicates that the test was successfully executed.
"FAILED": indicates that the test was not successfully passed.
"CHECK_MANUALLY": indicates that it was not possible to determinate if the test was successfully executed.
It will be saved to the execution report and is displayed in the "Executions" window (see Chapter 12.1, page 53) when the loaded in CoverageBrowser.
- 91 - froglogic GmbH C# Library
18.2.4 CoverageScanner.__coveragescanner_add_html_comment()
Syntax:
void __coveragescanner_add_html_comment(string comment)
__coveragescanner_add_html_comment() append a comment to the current execution. The comments needs to follow the HTML syntax but only the body is parsed. (The contents of the “
” tag is ignored). This function can be called several times, in this case the content is appended.18.2.5 CoverageScanner.__coveragescanner_clear_html_comment()
Syntax:
void __coveragescanner_clear_html_comment()
__coveragescanner_add_html_comment() clear comments added using void __coveragescanner_add_html_comment (string comment).
18.2.6 CoverageScanner.__coveragescanner_save()
Syntax:
void CoverageScanner.__coveragescanner_save()
CoverageScanner.__coveragescanner_save() saves the execution report and resets the status of all instrumentations.
18.2.7 CoverageScanner.__coveragescanner_clear()
Syntax:
void CoverageScanner.__coveragescanner_clear()
CoverageScanner.__coveragescanner_clear() resets the status of all instrumentations.
18.2.8 CoverageScanner.__coveragescanner_filename()
Syntax:
void CoverageScanner.__coveragescanner_filename(string name) this function sets the file name of the execution report. The extension ’.csexe’ is added automatically. Setting the execution report file name is necessary if the initialization function __coveragescanner_install() i is not used.
18.2.9 CoverageScanner.__coveragescanner_set_custom_io()
Syntax:
- 92 - froglogic GmbH C# Library
void CoverageScanner.__coveragescanner_set_custom_io( __cs_fgets_delegate cs_fgets, __cs_fputs_delegate cs_fputs, __cs_fopenappend_delegate cs_fopenappend, __cs_fopenread_delegate cs_fopenread, __cs_fopenwrite_delegate cs_fopenwrite, __cs_fclose_delegate cs_fclose, __cs_remove_delegate cs_remove)
Parameters: csfgets: Type:public delegate string __cs_fgets_delegate(System.IO.Stream stream) csfgets reads in at most one less than size characters from stream and stores them into the buffer pointed to by s. Reading stops after an EOF or a newline. If a newline is read, it is stored into the buffer. A ’\0’ is stored after the last character in the buffer. csfputs: Type:public delegate void __cs_fputs_delegate(string s, System.IO.Stream stream) csfputs writes the string s to stream, without its trailing ’\0’. csfopenappend: Type:public delegate System.IO.Stream __cs_fopenappend_delegate(string path) csfopenappend open the file path for writing at the end. csfopenread: Type:public delegate System.IO.Stream __cs_fopenread_delegate(string path) csfopenread open the file path for reading. csfopenwrite: Type:public delegate System.IO.Stream __cs_fopenwrite_delegate(string path) csfopenwrite open the file path for writing. csfclose: Type:public delegate void __cs_fclose_delegate(System.IO.Stream fp) The csfclose function will flush and close the stream pointed to by fp. csremove: Type:public delegate void __cs_remove_delegate(string n) csremove removes filename from the file system.
CoverageScanner writes the execution report (.csexe file) into file using the common C# file IO functions. In some situations, for embedded systems for example, it is necessary to use an other way for recording it. For this reason, CoverageScanner provides CoverageScanner.__coveragescanner_set_custom_io() which permits to replace the IO functions used by CoverageScanner.__coveragescanner_save().
- 93 - froglogic GmbH Controlling the instrumentation during the compilation
Chapter Controlling the instrumentation during the 19 compilation
CoverageScanner offers the possibility to modify the coverage settings of some source code parts during the compilation. This permits, for example, to exclude some source code from the coverage analysis.
19.1 Source Code Annotations
Source code annotations are specific comments which permits to validate some source code parts. Two kind of annotations are existing:
Comments: comments are simple code comments which can be used for giving advices to the tester. They don’t have any impact on the code coverage
Manual Validations: Manual validations permits to indicate that some lines of code should be considered as validated. Manual validations have the same effect for the code coverage as executing the source lines. Source code annotations can directly be edited with CoverageBrowser or generated by writing specific comments into the source code.
19.1.1 Comments
To generate comments for a single source line it is only necessary to start a C or C++ comment with the keyword “coco:
1 void write_log( const char *text ) 2 { 3 FILE * f = fopen( "log.txt", "a+" ); 4 if ( f ) /* coco: To test the failure, 5 change the current working 6 directory toa read-only directory */ 7 {
8 fputs( text, f ):
- 94 - froglogic GmbH C and C++ Pragma
9 fclose( f );
10 }
11 }
Comments can also be generated for multiples source lines using the keywords “coco begin:
1 void write_log( const char *text ) 2 { 3 /* coco begin: To test the failure, 4 change the current working 5 directory toa read-only directory */ 6 FILE * f = fopen( "log.txt", "a+" ); 7 if ( f )
8 {
9 fputs( text, f ):
10 fclose( f );
11 } 12 /* coco end */ 13 }
19.1.2 Manual Validation
Manual validated instrumentation can be generated using a comment starting with the keyword “coco validated:
1 void write_log( const char *text ) 2 {
3 // coco begin validated: This function does not need to be tested, only used for debugging. 4 FILE * f = fopen( "log.txt", "a+" ); 5 if ( f )
6 {
7 fputs( text, f ):
8 fclose( f );
9 }
10 // coco end
11 }
19.2 C and C++ Pragma
CoverageScanner defines C and C++ pragmas which permit to control during the compilation the generation of the instrumenta- tion. All control commands are handled in a stack. The pragmas have the following syntax:
- 95 - froglogic GmbH C# regions
#pragma CoverageScanner ( string ) or: __pragma ( CoverageScanner ( string )) or: _Pragma ( CoverageScanner ( string ))
List of supported pragmas:
#pragma CoverageScanner(cov-on) : Enables code coverage instrumentation.
#pragma CoverageScanner(cov-off) : Disables code coverage instrumentation.
#pragma CoverageScanner(cov-hit) : Instruments using code coverage hit.
#pragma CoverageScanner(cov-count) : Instruments using code coverage count.
#pragma CoverageScanner(cov-branch) : Select the code coverage branch as instrumentation method.
#pragma CoverageScanner(cov-decision) : Select the code coverage decision as instrumentation method.
#pragma CoverageScanner(cov-condition) : Select the code coverage condition as instrumentation method.
#pragma CoverageScanner(cov-partial-instrumentation) : Partial instrumentation of conditions and decisions.
#pragma CoverageScanner(cov-full-instrumentation) : Full instrumentation of conditions and decisions.
#pragma CoverageScanner(cov-assignment-on) : Instrument Boolean expressions in assignments.
#pragma CoverageScanner(cov-assignment-off) : Stop instrumenting Boolean expressions in assignments.
#pragma CoverageScanner(pop) : Restore the instrumentation option before the call of the last CoverageScanner pragma.
19.3 C# regions
CoverageScanner defines C# extensions of the #region keyword which permits to control during the compilation the generation of the instrumentation. The #region extension have the following syntax: #region CoverageScanner ( string )
The keyword #endregion restores the generation options. List of supported regions:
#region CoverageScanner(cov-on) : Enables code coverage instrumentation.
#region CoverageScanner(cov-off) : Disables code coverage instrumentation.
- 96 - froglogic GmbH C# regions
#region CoverageScanner(cov-hit) : Instruments using code coverage hit.
#region CoverageScanner(cov-count) : Instruments using code coverage count.
#region CoverageScanner(cov-branch) : Select the code coverage branch as instrumentation method.
#region CoverageScanner(cov-decision) : Select the code coverage decision as instrumentation method.
#region CoverageScanner(cov-condition) : Select the code coverage condition as instrumentation method.
#region CoverageScanner(cov-partial-instrumentation) : Partial instrumentation of conditions and decisions.
#region CoverageScanner(cov-full-instrumentation) : Full instrumentation of conditions and decisions.
- 97 - froglogic GmbH Support for IDEs and toolkits
Chapter Support for IDEs and toolkits 20
In this chapter we give examples how Squish Coco can be combined with several IDEs and build environments.
20.1 GNU Makefiles
Mostly, in makefiles, the C and C++ compiler and the linker are defined using the environment variables CC, CXX and LINK. This can be substituted by CoverageScanner by setting CC, CXX and LINK in the command arguments of make.
Example: make LINK=csg++ CXX=csg++ CC=csgcc
20.2 CygWin
To install the CoverageScanner compiler wrapper for GCC and G++ on CygWin, proceed as follows:
1. Open the “Build Environment Selection” application. (%SQUISHCOCO%\toolselector.exe) 2. Select the item “CygWin - www.cygwin.com”. 3. Click on “Install CygWin Support”. The build environment selection dialog then displays the list of generated compiler wrappers.
Open then the CygWin console and compile your application with csgcc instead of gcc (or csg++ instead of g++).
20.3 Scratchbox
If Squish Coco is installed on the root file system1, a compiler wrapper is created for each compiler supported by Scratchbox. To invoke CoverageScanner, prepend cs to the name of the cross-compiler.
1Installation option: "1) Installation on the local machine (need to be root)"
- 98 - froglogic GmbH Kitware CMake
20.4 Kitware CMake
CMake is a platform independent build tool from Kitware which can be downloaded from http://www.cmake.org.
When Squish Coco is used with CMake, the changes are partially dependent on the tool chain that is used for compilation. We will now first describe the addition of a new build type, which is independend from the tool chain, and then the additional changes for Microsoft® Visual Studio® and GNU GCC.
20.4.1 Adding a new build type
Create a new CMake COVERAGE build type by adding to CMakeLists.txt the following lines:
SET(COVERAGE_FLAGS "--cs-on --cs-count") SET( CMAKE_CXX_FLAGS_COVERAGE "${CMAKE_CXX_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING "Flags used by the C++ compiler during coverage builds." FORCE ) SET( CMAKE_C_FLAGS_COVERAGE "${CMAKE_C_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING "Flags used by the C compiler during coverage builds." FORCE ) SET( CMAKE_EXE_LINKER_FLAGS_COVERAGE "${CMAKE_EXE_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING "Flags used for linking binaries during coverage builds." FORCE ) SET( CMAKE_SHARED_LINKER_FLAGS_COVERAGE "${CMAKE_SHARED_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING "Flags used by the shared libraries linker during coverage builds." FORCE ) SET( CMAKE_STATIC_LINKER_FLAGS_COVERAGE "${CMAKE_STATIC_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING "Flags used by the static libraries linker during coverage builds." FORCE ) MARK_AS_ADVANCED( CMAKE_CXX_FLAGS_COVERAGE CMAKE_C_FLAGS_COVERAGE CMAKE_EXE_LINKER_FLAGS_COVERAGE CMAKE_SHARED_LINKER_FLAGS_COVERAGE CMAKE_STATIC_LINKER_FLAGS_COVERAGE COMPILE_DEFINITIONS_COVERAGE )
These commands take the compiler and linker flags from the “Release” build type and add to them the coverage flags. If you want to use instead the flags of another build type, replace the suffix “_RELEASE” in this code with the name of another build type, e.g. with “_DEBUG”.
The variable COVERAGE_FLAGS in the first line specifies the CoverageScanner command line options. Change its value to fit you needs. Only --cs-on should always be present.
20.4.2 Microsoft® Visual Studio®
Under Microsoft® Visual Studio®, we need to make the new Coverage build type visible to the IDE. To do this, add to CMakeLists.txt the lines
- 99 - froglogic GmbH Kitware CMake
if(CMAKE_CONFIGURATION_TYPES) set(CMAKE_CONFIGURATION_TYPES Debug Release MinSizeRel RelWithDebInfo Coverage) set(CMAKE_CONFIGURATION_TYPES "${CMAKE_CONFIGURATION_TYPES}" CACHE STRING "Reset the configurations to what we need" FORCE) endif()
If necessary, customize the list of configuration types according to your needs. To make the changes visible to Microsoft® Visual Studio®, a complex procedure is apparently needed:2
1. Compile CMakeLists.txt with Ctrl-F7. If warnings occur, they can be ignored.
2. Make a trivial change in CMakeLists.txt, like adding a space.
3. Compile again. Then a dialog, "File modification detected", should appear. Click "Reload". Then the list of solution configurations is updated and one can compile in coverage mode.
20.4.3 Microsoft® NMake
Proceed as follows:
1. Create a toolchain definition file cl.cmake which replaces the compiler and linker with their CoverageScanner wrappers. Example:
# this one is important SET(CMAKE_SYSTEM_NAME Windows)
# specify the cross compiler FILE(TO_CMAKE_PATH "$ENV{SQUISHCOCO}/visualstudio" SQUISHCOCO) SET(CMAKE_C_COMPILER ${SQUISHCOCO}/cl.exe CACHE FILEPATH "CoverageScanner wrapper" FORCE) SET(CMAKE_CXX_COMPILER ${SQUISHCOCO}/cl.exe CACHE FILEPATH "CoverageScanner wrapper" FORCE) SET(CMAKE_LINKER ${SQUISHCOCO}/link.exe CACHE FILEPATH "CoverageScanner wrapper" FORCE)
2. Create a Makefile project. Set the toolchain to the CoverageScanner wrapper and the build mode to COVERAGE. Example:
cmake.exe -DCMAKE_TOOLCHAIN_FILE=cl.cmake -DCMAKE_BUILD_TYPE=COVERAGE \ -G "NMake Makefiles"
3. Build the project with nmake.
20.4.4 GNU GCC
1. Create a toolchain definition file gcc.cmake which replaces the compiler and linker with their CoverageScannerwrappers. Example:
find_program(CODE_COVERAGE_GCC csgcc) find_program(CODE_COVERAGE_GXX csg++) find_program(CODE_COVERAGE_AR csar)
2This applies at least to CMake 3.2 and Microsoft® Visual Studio® 2013.
- 100 - froglogic GmbH Qt
# specify the cross compiler SET(CMAKE_C_COMPILER "${CODE_COVERAGE_GCC}" CACHE FILEPATH "CoverageScanner wrapper" FORCE) SET(CMAKE_CXX_COMPILER "${CODE_COVERAGE_GXX}" CACHE FILEPATH "CoverageScanner wrapper" FORCE) SET(CMAKE_LINKER "${CODE_COVERAGE_GXX}" CACHE FILEPATH "CoverageScanner wrapper" FORCE) SET(CMAKE_AR "${CODE_COVERAGE_AR}" CACHE FILEPATH "CoverageScanner wrapper" FORCE)
2. Create a Makefile project. Set the toolchain to the CoverageScanner wrapper and the build mode to COVERAGE. Example:
cmake -DCMAKE_TOOLCHAIN_FILE=gcc.cmake -DCMAKE_BUILD_TYPE=COVERAGE \ -G "Unix Makefiles"
3. Build the project with make.
20.5 Qt Library
20.5.1 Qt Creator
The following steps are needed to add instrumentation to a Qt Creator project.
• Switch to the Projects mode using the corresponding button in the mode pane on the left side of the IDE’s interface.
• Create a custom build mode called CodeCoverage: – Select the Debug or Release field in the Edit build configuration combo box. – Click on Add¡Clone Selected. – Enter CodeCoverage as new configuration name.
• Adjust the Build directory such that the CodeCoverage configuration uses a directory different from that of the other builds.
• In the Build Environment section, expand the System Environment pane and modify the PATH variable by prepending the path of the CoverageScanner binary:
Platform new Path Apple® Mac OS X /Applications/SquishCoco:
Under Linux™, Squish Coco may have been installed locally. If it has been your account name is usr, use the path /home/usr/SquishCoco/bin:
• Select the build step qmake and add to the Additional arguments the text CONFIG+=CodeCoverage. This allows to specify additional code coverage settings by modifying the qmake’s profile file (20.5.2, page 102).
• Select the build step Make and enter in the Additional arguments box the following text:
- 101 - froglogic GmbH Qt
Platform Additional Arguments Apple® Mac OS X LINK=csclang++ AR=csar CXX=csclang++ CC=csclang Linux™ LINK=csg++ AR=csar CXX=csg++ CC=csgcc Microsoft® Windows LINK=cscl LIB=cslib CXX=cscl CC=cscl
With these settings, a new build mode CodeCoverage has been created that creates an instrumented version of your program. If the project has unit tests, it is possible to execute them at every build and also generate a coverage report for them. In order to do this, the following additional steps are needed.
• In the Build Steps section, click on Add Build Step and select Custom Process Step. A new build step is created to execute the unit test the code coverage of which you are interested in. Fill in the Command box the name of the program that runs the unit tests and into the Working directory box the directory in which it should be executed. If necessary, also fill in the command line arguments of the program.
• Add another Custom Process Step. For this step enter the full path of the cmcsexeimport tool in the Command box. In Working directory, enter the same directory as in before for the unit test program. As for the arguments of cmcsexeimport, make sure the csmes and csexe files are included. An entry in the Arguments could therefore look like
--title="Execution" -m unit_test.csmes -P -e unit_test.csexe.
Remember to replace the word unit_test in unit_test.csexe and unit_test.csmes with the actual name of your unit test.
• The final Custom Process Step that needs to be added is the one for the cmreport tool, in case that report generation was wanted. Again set the Working directory as that of the unit test, and set cmreport’s absolute path as the Command. Then set the arguments, e.g.:
--title="Execution" -m unit_test.csmes --html=report
Remember again to replace unit_test with the name of your unit test.
Now, every time you click on Build (not just Run) with this configuration, the project will be built, instrumented and optionally a report will be generated. Note that in case the project has been built before this configuration was performed, it will be required to execute the Clean Build step before building, so that the coverage measurement data will be generated. Also keep in mind that when there was an error and one of the build steps exits with a non-zero exit status code, Qt Creator stops the build and displays the error message of the executable that failed, instead of continuing the build. This means that if a unit test fails, instead of cmcsexeimport running and importing the execution as a failed execution, cmcsexeimport does not run at all until the test case failure was fixed.
20.5.2 qmake qmake is a tool to generate a Makefile in a platform-independent way, using as specification a so-called “project file”. By default, qmake chooses the programs that are used for compilation. We can however use the Squish Coco wrappers by setting some of qmake’s variables to new values. In simple cases, we can do this on the command line. There are two methods.
• The first method works with older versions of Qt. In it, one passes the names of the Squish Coco wrappers as paramters to qmake. Under Windows this looks like
C:\code> qmake QMAKE_LINK=cslink QMAKE_CXX=cscl QMAKE_CC=cscl
Here we redefine the variables QMAKE_CC, QMAKE_CXX and QMAKE_LINK: They contain the name of the C compiler, the C++ compiler and the linker that are used in the Makefiles generated by qmake. For other platforms, the values of these variables must be replaced with the wrappers for the compilers and linker that are used there.
- 102 - froglogic GmbH Qt
• In some newer versions of Qt there is however a bug in qmake that makes the previous approach unusuable. In this case, one can overwrite the compiler versions later, when the Makefile generated by qmake is used for compilation. In every call of the make program, certain Makefile variables must be overwitten. Under Microsoft® Windows, the resulting command line is:
C:\code> nmake LINK=cslink CXX=cscl CC=cscl
(Under UNIX®-like systems, make must be called instead of nmake.)
In more complex cases it is however better to create a new scope inside the project file. (A scope is a region in a qmake project file that can be activated on demand.) The following listing shows a template for such a scope, named CoverageScanner, which should be sufficient for most most projects.
CodeCoverage { COVERAGE_OPTIONS =
QMAKE_CFLAGS += $$COVERAGE_OPTIONS QMAKE_CXXFLAGS += $$COVERAGE_OPTIONS QMAKE_LFLAGS += $$COVERAGE_OPTIONS
QMAKE_CC=cs$$QMAKE_CC QMAKE_CXX=cs$$QMAKE_CXX QMAKE_LINK=cs$$QMAKE_LINK QMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIB QMAKE_AR=cs$$QMAKE_AR QMAKE_LIB=cs$$QMAKE_LIB }
Here we also set the variables QMAKE_LINK_SHLIB and QMAKE_AR, which contain the names of the command to link shared libraries and that to generate archives. Furthermore, can use COVERAGE_OPTIONS to set coveragescanner commandline options (see Chapter 17.2, page 78) to customize the project. An empty value for COVERAGE_OPTIONS will also work and results in a default instrumentation. If we have a project file that contains such a scope, we can create a Makefile that generates instrumented code with the command:
$ qmake CONFIG+=CodeCoverage
20.5.3 moc
The Meta-Object Compiler moc adds automatically new methods to each classes derived from QObject. For example, the translation function tr, the source code for all signals, the cast operator qt_cast, . . . In order to instrument the code using the Qt Library and not the library itself, CoverageScanner provides the command line options --cs-qt4 for Qt4 (resp. --cs-qt3 for Qt3). In this case:
• Q_OBJECT macro is no more instrumented. • All signal are instrumented in order to track their emission.
• The glue code necessary for the signal/slot mechanism is not instrumented.
• The code of Q_FOREACH macro is not instrumented on Qt4.
- 103 - froglogic GmbH SCons
20.6 SCons
To use Squish Coco with SCons, proceed as followings: • Prepend the path of CoverageScanner’s wrapper (csgcc, cscl, . . . ) to the PATH environment variable. The PATH environment variable should be set to be able to execute CoverageScanner wrappers .
• Set CC, AR and LINK variable to CoverageScanner corresponding wrapper. Example: when using Microsoft® Visual Studio® compiler, set CC to cscl.exe, set LINK to cslink.exe and set AR to cslib.exe.
Do no use absolute file paths to the compiler wrapper since that some versions of SCons do not handle ! properly spaces in file names.
• Add additional code coverage settings to the variables CCFLAGS, ARFLAGS and LINKFLAGS to exclude for example files from the instrumentation. Here is a code snippet which can be used for Microsoft® Visual Studio® command line tools:
import os from os.path import pathsep
env = Environment()
# Add the path of Squish Coco compiler wrapper env[ ’ENV’ ][ ’PATH’ ] = os.environ[ ’SQUISHCOCO’ ] + pathsep + env[ ’ENV’ ][ ’PATH’ ] # TEMP variable need to be defined env[ ’ENV’ ][ ’TEMP’ ] = os.environ[ ’TEMP’ ]
# Set the compiler to Squish Coco wrappers env[ ’CC’ ] = ’cs’ + env[ ’CC’ ] ; env[ ’AR’ ] = ’cs’ + env[ ’AR’ ] ; env[ ’LINK’ ] = ’cs’ + env[ ’LINK’ ] ;
# Code coverage settings coverageflags = [ ’--cs-count’ ] env[ ’CCFLAGS’ ] = env[ ’CCFLAGS’ ] + coverageflags ; env[ ’ARFLAGS’ ] = env[ ’ARFLAGS’ ] + coverageflags ; env[ ’LINKFLAGS’ ] = env[ ’LINKFLAGS’ ] + coverageflags ;
20.7 ARM® Keil® µVision
To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapper for ARM and enable the code coverage analysis during the compilation. Processed as followings: 1. Select the Squish Coco tool chain instead of the original ARM toolchain. (a) On the ARM® Keil® µVision IDE click on "Project¡Manage¡Components, Environments, Books...". (b) Click on "Folders/Extensions", and replace .\ARMCC\bin through .\SquishCoco\bin in the field "ARMCC Folder:".
- 104 - froglogic GmbH ARM Keil
Figure 20.1: Installation of CoverageScanner on ARM® Keil® µVision: Setting the path of CoverageScanner.
2. Activate the code coverage analysis during the compilation (a) Click on "Project¡Option for Target..." (b) On the "C/C++" tab, add --cs-on into the "Misc Controls" field.
Figure 20.2: Activating the code coverage analysis for an ARM® compiler
(c) On the "Linker" tab, add --cs-on into the "Misc Controls" field.
- 105 - froglogic GmbH Green Hills
Figure 20.3: Activating the code coverage analysis for an ARM® linker
20.8 Green Hills® Software MULTI Launcher
20.8.1 Installation
To install CoverageScanner compiler wrapper for the Green Hills® Software tool chain, proceed as follows:
1. Open the “Build Environment Selection” application. (%SQUISHCOCO%\toolselector.exe) 2. Select the item “Green Hills”.
3. Select the directory o which the Green Hills® Software compilers are installed 4. Click on “Install Green Hills Support” and wait for the confirmation dialog that the tools are generated.
20.8.2 Command Line Tools
The command line tools for the Green Hills® Software compiler are installed in the folder squishcoco of the installed native tool chain. Example: if the native tool chain is installed under c:\ghs\comp_201426, the Squish Coco compiler wrapper are installed under c:\ghs\comp_201426\squishcoco. The compiler wrappers are replacing completely the Green Hills® Software tool chain. To activate the code coverage analysis, it is necessary to add the parameter --cs-on to the compiler arguments.
20.8.3 CoverageScanner Library
In order to save an execution report, it is necessary to: 1. provide to the CoverageScanner library a list of I/O function which permits to upload the coverage information to the host.
- 106 - froglogic GmbH Visual DSP
2. Save manually the coverage report on a specific trigger. To save a coverage report the following code snippet can be used (an this code in an event handler which should trigger the report generation):
1 #ifdef __COVERAGESCANNER__
2 __coveragescanner_save();
3 __coveragescanner_clear();
4 #endif
To provide the i/O functions it is necessary to call __coveragescanner_set_custom_io() in the main() function of the application. At least three functions need to be provided by __coveragescanner_set_custom_io():
1. An equivalent function of fopen() which opens the execution report file (with the extension .csexe) in append mode.
2. An equivalent function of fclose()
3. An equivalent function of fputs() to transfer the contain. Example: The following code write an execution report to the local file system using the C file API.
1 #ifdef __COVERAGESCANNER__ 2 static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); } 3 static void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); } 4 static int csfclose(void *fp) { return fclose((FILE*)fp); } 5 #endif
6
7 int main()
8 {
9 #ifdef __COVERAGESCANNER__
10 __coveragescanner_set_custom_io( NULL,
11 csfputs,
12 csfopenappend,
13 NULL,
14 NULL,
15 csfclose,
16 NULL);
17 #endif
18 ....
19 }
20.9 VisualDSP®
To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapper and enable the code coverage analysis during the compilation in the VisualDSP® configuration. Processed as followings:
1. Click on "Project¡Project Options..."
2. On the "Compile" tab, add --cs-on into the "Additional options" field.
- 107 - froglogic GmbH Visual DSP
Figure 20.4: Activating the code coverage analysis for the compiler
3. On the "Link" tab, add --cs-on into the "Additional options" field.
Figure 20.5: Activating the code coverage analysis for the linker
4. Rebuild your project. Like for all embedded targets, it is necessary to provide a dedicated I/O since that in most of the cases no file systems are available for storing the code coverage information. Also since that embedded applications are generally not exiting, it is also necessary to implement a event handler which saves the execution report upon the reception of a specific trigger. The simulator emulates the support of a file system. To save the coverage report on the current build directory, it is only necessary to register in the first lines of the main() a custom file I/O which uses the standard C file API. Example:
1 #ifdef __COVERAGESCANNER__ 2 static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); }
- 108 - froglogic GmbH Microsoft Visual Studio
3 static void *csfopenappend(const char *path) { return (void*)fopen(path,"a+"); } 4 static int csfclose(void *fp) { return fclose((FILE*)fp); } 5 #endif
6
7 int main()
8 {
9 #ifdef __COVERAGESCANNER__
10 __coveragescanner_set_custom_io( NULL,
11 csfputs,
12 csfopenappend,
13 NULL,
14 NULL,
15 csfclose,
16 NULL);
17 #endif
18 ....
19 }
To record a coverage report when a specific trigger occurs, add the following source code lines in its handler:
1 #ifdef __COVERAGESCANNER__
2 __coveragescanner_save();
3 __coveragescanner_clear();
4 #endif
20.10 Microsoft® Visual Studio®
Squish Coco provides a wrapper for link.exe and cl.exe located on the %SQUISHCOCO%\visualstudio directory. It behaves exactly like the corresponding Microsoft® wrapper except that the code coverage analysis becomes activated when the option --cs-on is added to the command arguments. These wrappers call the Microsoft® tools for compilation or for linkage.
20.10.1 Microsoft® Visual Studio® .NET C# Compiler
To activate the instrumentation of C# source code, it is only necessary to add the define COVERAGESCANNER_COVERAGE_ON in the properties of the Microsoft® Visual Studio® .NET project. Other defines can be appended to select additional instrumentation options. The full list can be found on the chapter 17.3, page 84.
20.10.2 Microsoft® Visual Studio® .NET C and C++ Compiler
To use Squish Coco with Microsoft® Visual Studio® .NET proceed as follows:
1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories. For Microsoft® Visual Studio® 2005 or 2008: (a) Start Microsoft® Visual Studio® 2005 or 2008. (b) Open the option dialog: click on "Tools¡Preferences...". (c) Select the item "Projects¡VC++ Directories". (d) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Figure 20.6, page 110)
- 109 - froglogic GmbH Microsoft Visual Studio
Add the path of the compiler wrapper for Visual Studio to the first position
Figure 20.6: Installation of CoverageScanner on Microsoft® Visual Studio® 2005 and 2008: Setting the path of CoverageScanner.
For Microsoft® Visual Studio® 2010: (a) Start Microsoft® Visual Studio® 2010. (b) Open a C++ project. (c) Open the project properties using the context menu of the laded project. (d) Select the item "Configuration Properties¡VC++ Directories". (e) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of executable directories. (see Figure 20.7, page 110)
Add the path of the compiler wrapper for Visual Studio to the first position
Figure 20.7: Installation of CoverageScanner on Microsoft® Visual Studio® 2010: Setting the path of CoverageScanner.
2. To activate code coverage analysis: (a) Open a Visual C or C++ project. (b) Edit the project settings (click on "Project¡Properties"). (c) Add to the option --cs-on to the additional command line arguments of the C or C++ compiler and linker. (see Figure 20.8, page 111) (d) In the additional arguments of the linker, add the --cs-libgen which permits to specify which library should be used for the generation of the CoverageScanner library. The table 20.1, page 111 contains the list of recommended settings.
- 110 - froglogic GmbH Microsoft Visual Studio
(e) For Microsoft® Windows CE applications, append to the linker arguments the command line option --cs-architecture which permits to specify the target platform. The table 20.2, page 111 contains the list of available architectures.
Library Library File Command line option Single Threaded LIBC.LIB --cs-libgen=/ML Static MultiThread LIBCMT.LIB --cs-libgen=/MT Dynamic Link (DLL) LIBCRT.LIB --cs-libgen=/MD Debug Single Threaded LIBCD.LIB --cs-libgen=/MLd Debug Static MultiThread LIBCMTD.LIB --cs-libgen=/MTd Debug Dynamic Link (DLL) LIBCRTD.LIB --cs-libgen=/MDd
Table 20.1: CoverageScanner library settings for Microsoft® Visual Studio®
Targeted Architecture Command line option ARM Microprocessor --cs-architecture=ARM ARM Microprocessor (Thumb code) --cs-architecture=THUMB x86 Microprocessor --cs-architecture=IX86 MIPS16 Microprocessor --cs-architecture=MIPS16 MIPS Microprocessor --cs-architecture=MIPS MIPS Microprocessor with FPU --cs-architecture=MIPSFPU SH3 Microprocessor with FPU --cs-architecture=SH3 SH4 Microprocessor with FPU --cs-architecture=SH4
Table 20.2: List of target architectures
Add the command line option '--cs-on' here
Figure 20.8: Activation of the instrumentation under Visual Studio® .NET.
- 111 - froglogic GmbH Microsoft Visual Studio
Add the command line option '--cs-on' here
Figure 20.9: Activation of the instrumentation under Visual Studio® .NET.
20.10.3 Squish Coco Add-In for Microsoft® Visual Studio®
Squish Coco provides for Microsoft® Visual Studio® 2005 up to Microsoft® Visual Studio® 2013 an Add-In which does automatically the steps described in the section 20.10.2, page 109. The Add-In itself is normally not mandatory to build an instrumented application. Two different workflow are supported: • Enabling/disabling the code coverage analysis on an existing C++ or C# configuration. This method permits to quickly enable the code coverage analysis on an existing build mode. Unfortunately, a complete rebuild is then necessary. • Creating a new configuration which provides the coverage analysis. The Add-In duplicate a specified configuration and apply the code coverage settings on it. This method is more convenient for a developer since that it permits to quickly switch from the Debug/Release to the Coverage build mode without rebuilding the whole application. Also once the code coverage settings are applied it is possible to save it permanently in the new configuration. Unfortunately, this method may require some manual adjustments on some projects which are using fixed encoded paths. To use it: 1. Open a C++ or a C# project.
2. Click on “Tools¡Code Coverage Build Mode...”.
3. Select the base configuration and the project which should be compiled with code coverage support. 4. And finally: • Select “Modify” and click on “Enable code coverage” to alter the current build mode and add the code coverage support. “Disable code coverage” permits removes all CoverageScanner command line options. • Select “New” and click on “Create new configuration” to generate a new build mode with code coverage support.
- 112 - froglogic GmbH Microsoft Visual Studio
Project and Configuration used as template
Coverage options
Include paths in which
the inline code is instrumented
Creation of a new configuration
Enabling/disabling the coverage analysis on the current configuration
Figure 20.10: Microsoft® Visual Studio® Add-In
For Microsoft® Visual Studio® 2010, it might be necessary to add manually the path Squish Coco executables to i the search path. See section 20.10.2, page 109 for more the detailed procedure.
20.10.4 Microsoft® Visual C++ Express
To use Squish Coco with Microsoft® Visual C++ Express proceed as follows:
1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories.
(a) Start Microsoft® Visual C++ Express. (b) Open the option dialog: click on "Tools¡Preferences...". (c) Select the item "Projects¡VC++ Directories". (d) Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Figure 20.11, page 113)
Add the path of the compiler wrapper for Visual Studio to the first position
Figure 20.11: Installation of CoverageScanner on Visual C++ Express: Setting the path of CoverageScanner.
2. The activation of the code coverage analysis is similar to Microsoft® Visual Studio® .NET. (see Chapter 20.10.2, page 109)
- 113 - froglogic GmbH Microsoft Visual Studio
20.10.5 Microsoft® Visual Studio® 6.0
To use Squish Coco with Microsoft® Visual Studio® 6.0 proceed as follows: 1. Add the location of the CoverageScanner wrappers to the first position in the executable directories.
(a) Start Microsoft® Visual Studio® 6.0. (b) Open the option dialog: click on "Tools¡Preferences...". (c) Select the item "Directories". (d) Select "Executable files" in the combobox "Show directories for:". (e) Add the path of the directory visualstudio of the Squish Coco installation3 to the first position in the list of directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the path c:\programme\SquishCoco\visualstudio, (see Figure 20.12, page 114))
Add the path of the compiler wrapper for Visual Studio to the first position
Figure 20.12: Installation of CoverageScanner on Visual Studio® 6.0: Setting the path of CoverageScanner.
2. To activate the code coverage analysis: (a) Open a Visual C or C++ project. (b) Edit the project settings (click on "Project¡Properties"). (c) Add the option --cs-on to the additional command line arguments of the C or C++ compiler and linker. (see Figure 20.13, page 114) (d) In the additional arguments of the linker, add the --cs-libgen which permits to specify which library should be used for the generation of the CoverageScanner library. The table 20.1, page 111 contains the list of recommanded settings.
Add the command line option '--cs-on' here
Figure 20.13: Activation of the instrumentation under Visual Studio® 6.0.
3Microsoft® Visual Studio® 6.0 does not handle system variables in the path list. So the %SQUISHCOCO% variable needs to be expanded.
- 114 - froglogic GmbH eMbedded Visual C++
Add the command line option '--cs-on' here
Figure 20.14: Activation of the instrumentation under Visual Studio® 6.0.
20.11 Microsoft® eMbedded Visual C++®
To use Squish Coco with Microsoft® eMbedded Visual C++®proceed as follows: 1. Add the location of the CoverageScanner wrappers to the first position in the executable directories.
(a) Start Microsoft® eMbedded Visual C++®. (b) Open the option dialog: click on "Tools¡Preferences...". (c) Select the item "Directories". (d) Select "Executable files" in the combobox "Show directories for:". (e) Select "Platform" and the targeted "CPUs". (f) Add the path of the directory WinCE of the Squish Coco installation4 to the first position in the list of directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the path c:\programme\SquishCoco\WinCE, (see Figure 20.15, page 115))
Select the Platform and the CPU
Add the path of the compiler wrapper for Windows CE to the first position
Figure 20.15: Installation of CoverageScanner on eMbedded Visual C++®: Setting the path of CoverageScanner.
2. To activate the code coverage analysis: (a) Open a Visual C or C++ project. (b) Edit the project settings (click on "Project¡Properties"). (c) Add the option --cs-on to the additional command line arguments of the C and C++ compiler and linker. (see Figure 20.16, page 116)
4Microsoft® eMbedded Visual C++®does not handle system variables in the path list. So the %SQUISHCOCO% variable needs to be expanded.
- 115 - froglogic GmbH Eclipse
Add the command line option '--cs-on' here
Figure 20.16: Activation of the instrumentation under eMbedded Visual C++®.
Add the command line option '--cs-on' here
Figure 20.17: Activation of the instrumentation under eMbedded Visual C++®.
20.12 Eclipse™ IDE for C/C++
Eclipse™ IDE for C/C++ rely on a GNU-GCC command line compiler. Before trying to compile a project with coverage support, it is necessary to create a compiler wrapper ((see Chapter G.4, page 214)). To compile a project with code coverage support, proceed as follows:
1. Start Eclipse™. 2. Load a C or C++ project.
3. Open the property window (Project¡ Properties) 4. Click on "C/C++ Build/Settings" 5. Create a new configuration by clicking on "Manage Configurations..." and select it.
6. Click on "Tools Settings" tab.
7. Click on "GCC C++ Compiler" and prepend cs to the name of the compiler.
8. Click on "GCC C Compiler" and prepend cs to the name of the compiler.
9. Click on "C++ Linker" and prepend cs to the name of the linker.
- 116 - froglogic GmbH Xcode
Append 'cs' to the compiler/linker executable name
Figure 20.18: Eclipse™ settings
20.13 Apple® Xcode
To use Squish Coco with Apple® Xcode proceed as follows: To activate the code coverage analysis:
1. Open a terminal window and set the CPLUSPLUS, LDPLUSPLUS, LD and CC to CoverageScanner compiler wrapper. The path of native compiler (clang, clang++, gcc or g++) need to be present in the PATH environment variable. Start Xcode using the open command. If GCC is used as compiler:
SQUISHCOCO=/Applications/SquishCoco/wrapper export CC=$SQUISHCOCO/gcc export LD=$SQUISHCOCO/gcc export CPLUSPLUS=$SQUISHCOCO/g++ export LDPLUSPLUS=$SQUISHCOCO/g++
open /Developer/Applications/Xcode.app
If clang is used as compiler:
SQUISHCOCO=/Applications/SquishCoco/wrapper export CC=$SQUISHCOCO/clang export LD=$SQUISHCOCO/clang export CPLUSPLUS=$SQUISHCOCO/clang++ export LDPLUSPLUS=$SQUISHCOCO/clang++ export XCODE_TOOLCHAIN_DIR=/Applications/Xcode.app/Contents/Developer/Toolchains export XCODE_TOOLCHAIN=$XCODE_TOOLCHAIN_DIR/XcodeDefault.xctoolchain/usr/bin/
export PATH=$XCODE_TOOLCHAIN:$PATH
- 117 - froglogic GmbH Xcode
open /Applications/Xcode.app
2. Open a Xcode C or C++ project.
3. Edit the project settings (click on "Project¡Edit Project Settings").
4. Add the option --cs-on to the additional command line arguments of the C and C++ compiler (fields Other C Flags and Other C++ Flags) and linker (field Other Linker Flags). (see Figure 20.19, page 118)
5. Disable the usage of the precomiled header: Open the settings of the active target ("Project¡Edit Active Target") and remove the contains of Prefix Header.
Add the linker command line option '--cs-on' here
Add the compiler command line option '--cs-on' here
Figure 20.19: Activation of the instrumentation under Apple® Xcode.
- 118 - froglogic GmbH Code Coverage of Libraries
Chapter Code Coverage of Libraries 21
21.1 Code Coverage of Static/Shared Libraries and DLL
During the linking operation, CoverageScanner includes all instrumentations of the shared libraries (if these are compiled with CoverageScanner). CoverageBrowser displays the code coverage of the complete application (executable and its libraries) in one view. To get an analysis of the code coverage of a library only, it is necessary to compile the main application and i exclude its sources from the code coverage (by adding in the command line --cs-exclude-file-regex=.* for example.)
21.2 Code Coverage of Plugins/Manually Loaded Shared Libraries
Libraries loaded dynamically can also be instrumented, but it is necessary to handle the generation of the execution report in the main application or in the plugin code itself.
21.2.1 Generating Code Coverage Information directly from the Main Application
Handling the plugins into the main application can be easily performed using the register/unregister mechanism of the CoverageScanner API: It is just necessary to call __coveragescanner_register_library() after loading a library and call __coveragescanner_unregister_library() just before unloading it. Example:
#include
int main(int argc, char **argv) { void *handle; double (*cosine)(double); char *error;
- 119 - froglogic GmbH Code Coverage of Plugins/Manually Loaded Shared Libraries
handle = dlopen("libm.so", RTLD_LAZY); #ifdef __COVERAGESCANNER__ __coveragescanner_register_library("libm.so"); #endif if (!handle) { fprintf(stderr, "%s\n", dlerror()); exit(EXIT_FAILURE); }
dlerror(); /* Clear any existing error */
/* Writing: cosine=(double( *)(double)) dlsym(handle,"cos"); would seem more natural, but the C99 standard leaves casting from"void *" toa function pointer undefined. The assignment used below is the POSIX.1-2003(Technical Corrigendum 1) workaround; see the Rationale for the POSIX specification of dlsym(). */
*(void **) (&cosine) = dlsym(handle, "cos");
if ((error = dlerror()) != NULL) { fprintf(stderr, "%s\n", error); exit(EXIT_FAILURE); }
printf("%f\n", (*cosine)(2.0)); #ifdef __COVERAGESCANNER__ __coveragescanner_unregister_library("libm.so"); #endif dlclose(handle); exit(EXIT_SUCCESS); }
Calling __coveragescanner_register_library() or __coveragescanner_unregister_library() on i an non-instrumented library is allowed.
21.2.2 Generating Code Coverage Information directly from the Plugin
CoverageScanner cannot handle the instrumentation of plugins (i.e. shared libraries loaded manually) during the linking phase. In this case, the library must initialize and store itself the executions. Therefore, the shared library needs to call __coveragescanner_filename() to set the name of the execution file during its initialization and __coveragescanner_save() to save the instrumentations when its becomes unloaded.
21.2.2.1 Code Coverage of Plugins Generated with Microsoft® Visual Studio®
The function DllMain is called on the initialization and the termination of the DLL generated using Microsoft® Visual Studio®. When the ’reason’ field is equal to DLL_PROCESS_ATTACH the function __coveragescanner_filename() should be called. To save the measures on exit the function __coveragescanner_save() shall be called when ’reason’ is DLL_PROCESS_DETACH.
Example:
extern "C"
- 120 - froglogic GmbH Code Coverage of Plugins/Manually Loaded Shared Libraries
BOOL WINAPI DllMain(HINSTANCE hInstance, DWORD dwReason, LPVOID /*lpReserved*/) { switch( dwReason ) { case DLL_PROCESS_ATTACH: #ifdef __COVERAGESCANNER__ /* Initialization of the CoverageScanner library. */ /* Replace"mylib" with your filename without extension */ __coveragescanner_filename("mylib"); #endif ... break; case DLL_PROCESS_DETACH: ... #ifdef __COVERAGESCANNER__ /* Saves the execution report */ __coveragescanner_save(); #endif break; } return TRUE; }
21.2.2.2 Code Coverage of Plugins Generated with GNU gcc
The GNU compiler offers 2 attributes which permits to execute a function when a library becomes loaded or unloaded:
__attribute__ ((constructor)) my_init(void);: This attribute permits to call a function when the library is loaded. Call the function __coveragescanner_filename() in the custom initialization function of the library.
__attribute__ ((destructor)) my_fini(void);: This attribute permits to call a function when the library is unloaded. Call the function __coveragescanner_save() on the termination of the library. Example:
static void plugin_load(void) __attribute__ ((constructor)) ; static void plugin_unload(void) __attribute__ ((destructor)) ;
static void plugin_load(void) { #ifdef __COVERAGESCANNER__ /* Initialization of the CoverageScanner library. */ /* Replace"mylib" with your filename without extension */ __coveragescanner_filename("mylib"); #endif ... }
static void plugin_unload(void) { ... #ifdef __COVERAGESCANNER__ /* Saves the execution report */ __coveragescanner_save();
- 121 - froglogic GmbH Code Coverage of Plugins/Manually Loaded Shared Libraries
#endif }
- 122 - froglogic GmbH Test suites and Squish Coco
Chapter Test suites and Squish Coco 22
22.1 Execution comment, name and status
Test suites are able to provide to CoverageBrowser the name of the test, a comment in HTML format and also the status of their execution (passed or failed). Two methods are possible:
• The functions __coveragescanner_testname(), __coveragescanner_add_html_comment() and __coveragescanner_teststate() can be used to specify the test name, its comment and its status. This is particularly useful for unit tests in which the test framework is compiled into the application to test (see Chapter 18.1.2, page 87).
• One can write these data directly into the .csexe file. (See below.)
The .csexe file is a simple text file on which is it possible to append additional lines to extend the information provided to CoverageBrowser.
To set the name of a test, simply add before executing it the following line: “*
To set the status of a test, simply add after executing it the following line: “!
"PASSED": to indicate that the test was successfully executed,
"FAILED": to indicate that the test was not successfully passed, and
"CHECK_MANUALLY": to indicate that it was not possible to determinate whether the test was successfully executed. To append an execution comment, insert the contents of an HTML document just after the execution of the application.
Example: The following batch file execute the test First Test and set the status of the execution to CHECK_MANUALLY.
echo *First Test >> myapp.csexe myapp echo "
Execution of myapp" >> myapp.csexe echo !CHECK_MANUALLY >> myapp.csexe1 It is also possible to begin the line with a paragraph sign (“§”) instead of a “*”. The “§” must be the in Latin-1 encoding and has the numerical value of 167. Since this may lead to internationalization problems, we strongly dicourage its use. It is only supported for compatibility with older versions of the program.
- 123 - froglogic GmbH Controlling the generation of the execution report
There may be more than one HTML comment, and the comments and the status declaration may occur in any order, but they all must be appended to the .csexe file after the test has been executed.
22.2 Controlling the generation of the execution report
The default behaviour of CoverageScanner library is to produce an execution report upon the instrumented application exits. This might not be enough in the cases of unit test (for this tests, it is preferable to generate a report after the execution of each single test) or for application like daemons which never terminates. For these purposes, CoverageScanner library permits also to: 1. Embedded a C code into the instrumented application to control the report generation (22.2.3, page 125).
2. Generate the coverage report on the reception of a Microsoft® Windows event or a UNIX® signal.
22.2.1 Generation of an execution report using UNIX® signals
To generate the execution using UNIX® signals, add --cs-dump-on-signal=SIG to the linker command line. SIG is the signal number or its common name (example SIGUSR1).
Using the UNIX® kill command it is possible to trigger the report generation. (ex: kill -SIGUSR1
22.2.2 Generation of an execution report using Microsoft® Windows event
To generate the execution using Microsoft® Windows events, add --cs-dump-on-event=EVENT to the linker command line.
Using the Microsoft® function SetEvent(), it is possible to trigger the generation of a coverage report. A sample of code is provided here:
#include
void DisplayError( LPTSTR lpszMessage ) { // Retrieve the system error message for the last-error code
LPSTR lpMsgBuf; DWORD dw = GetLastError();
FormatMessageA( FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS, NULL, dw, MAKELANGID( LANG_NEUTRAL, SUBLANG_DEFAULT ), ( LPSTR ) &lpMsgBuf,
- 124 - froglogic GmbH Controlling the generation of the execution report
0, NULL );
// Display the error message and exit the process std::cerr << lpszMessage << " (error " << dw << "): " << lpMsgBuf << ’\n’;
LocalFree( lpMsgBuf ); }
int main( int argc, char *argv[] ) { if ( argc != 2 ) { std::cerr << "Usage: " << argv[0] << "
const char * eventName = argv[1] ;
HANDLE ghEvent = OpenEventA( EVENT_MODIFY_STATE, FALSE, eventName );
if ( ghEvent == NULL ) { DisplayError( "Opening event failed" ); return 1; }
if ( ! SetEvent( ghEvent ) ) { DisplayError( "Raising event failed" ); CloseHandle( ghEvent ); return 1; }
CloseHandle( ghEvent ); return 0; }
22.2.3 Unit Testing
CoverageBrowser import execution results for each object which are part of an application. This mean that, if an unit test uses the same objects as these for generating the application, CoverageBrowser is able to import the execution result of the unit test and merge it into the code coverage of the compiled application. Example: An application is composed to 3 files: app.cpp: This file contains the main() function of the application. library.cpp: This file contains a set of functions called in app.cpp.
- 125 - froglogic GmbH Support for specific test frameworks testlibrary.cpp: This contains the test code of the functions of the file library.cpp. testlibrary.cpp have its own main() function. To generate the application:
cscl app.cpp /Foapp.obj cscl library.cpp /Folibrary.obj cscl library.obj app.obj /Feapp.exe
To generate the unit test:
cscl testlibrary.cpp /Fotestlibrary.obj cscl library.obj testlibrary.obj /Fetestlibrary.exe
To import the execution report from the unit tests into the instrumentation database of the main application, two methods can be used:
• The execution of testlibrary.exe generates the execution report testlibrary.exe.csexe. This report can be loaded in the measurement database app.exe.csmes.
• The execution of testlibrary.exe generates the execution report testlibrary.exe.csexe. This report can be loaded in the measurement database testlibrary.exe.csmes of the test code. This measurement database can be imported into this one from the main application using "File¡Import Unit Tests..." function of CoverageBrowser. (see Chapter 13.6.2, page 67)
In both cases, only the code coverage analysis of the file library.cpp is loaded, and the execution report of the test code is ignored.
22.3 Support for specific test frameworks
22.3.1 CppUnit
CppUnit2 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test. The following code is an example how this can be done:
#include
class CoverageScannerListener : public CppUnit::TestListener { public: CoverageScannerListener() {}
void startTest( CppUnit::Test *test ) { m_testFailed = false;
2Project page of CppUnit:http://cppunit.sourceforge.net
- 126 - froglogic GmbH Support for specific test frameworks
#ifdef __COVERAGESCANNER__ int pos; // Adjusting the name of the test to display the tests // ina tree view in CoverageBrowser std::string testname = "CppUnit/" + test->getName(); while ( ( pos = testname.find( "::", 0 ) ) != std::string::npos ) testname.replace( pos, 2, "/" );
// Reset the code coverage data to get only the code coverage // of the actual unit test. __coveragescanner_clear(); __coveragescanner_testname( testname.c_str() ) ; #endif }
void addFailure( const CppUnit::TestFailure &failure ) { m_testFailed = true; }
void endTest( CppUnit::Test *test ) { #ifdef __COVERAGESCANNER__ // Recording the execution state in the coverage report if ( m_testFailed ) __coveragescanner_teststate( "FAILED" ); else __coveragescanner_teststate( "PASSED" );
// Saving the code coverage report of the unit test __coveragescanner_save(); __coveragescanner_testname( "" ); #endif }
private: bool m_testFailed; // Prevents the use of the copy constructor and operator. CoverageScannerListener( const CoverageScannerListener © ); void operator =( const CoverageScannerListener © ); }; int main( int argc, char* argv[] ) { #ifdef __COVERAGESCANNER__ __coveragescanner_install( argv[0] ); #endif // Create the event manager and test controller CPPUNIT_NS::TestResult controller;
// Adda listener that colllects test result CPPUNIT_NS::TestResultCollector result; controller.addListener( &result );
// Adda listener that print dots as test run. CPPUNIT_NS::BriefTestProgressListener progress;
- 127 - froglogic GmbH Support for specific test frameworks
controller.addListener( &progress );
// Adda listener that saves the code coverage information CoverageScannerListener coveragescannerlistener; controller.addListener( &coveragescannerlistener );
// Add the top suite to the test runner CPPUNIT_NS::TestRunner runner; runner.addTest( CPPUNIT_NS::TestFactoryRegistry::getRegistry().makeTest() ); runner.run( controller );
return result.wasSuccessful() ? 0 : 1; }
In the example, we have done the following steps: 1. We write a CppUnit listener class which records the code coverage of unit each test after it is completed. We want to be able to run the program with and without Squish Coco. Therefore we use in the code the macro __COVERAGESCANNER__ for conditional compilation. The macro is defined in every file that is instrumented by Squish Coco, without the need to #include anything. In the listener class, CppUnitListener, we use the following menber functions: startTest(): This function is called before each test begins. In it, we compute a test name with the information provided by CppUnit and pass it to the Squish Coco library with __coveragescanner_testname(). We also call the function __coveragescanner_clear(): It empties the internal database and so makes sure that the coverage of the code that was executed before this test is ignored. addFailure(): This function is called after a test fails. It just sets a flag that is used by the other functions. endTest(): This function is called after a test has ended. It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with __coveragescanner_save().
2. We call __coveragescanner_install() in the main() function.
3. We add this listener in the test manager of CppUnit, the class CPPUNIT_NS::TestResult. In the example above, this is done by the following lines:
CoverageScannerListener coveragescannerlistener; controller.addListener( &coveragescannerlistener );
22.3.2 QTestLib
QTestLib is a unit test framework for the Qt Library. This environment can easily be adapted to get the code coverage from each unit test. Simply proceed as follows:
1. Call __coveragebrowser_install() in the main() function.
2. Write a QObject derived class called TestCoverageObject which records the code coverage report upon every unit test completion.
3. Instead of inheriting from QObject, make all your test cases inheriting from TestCoverageObject.
- 128 - froglogic GmbH Support for specific test frameworks
4. TestCoverageObject class provide its own init() and cleanup() slot, which use the CoverageScanner API to save the code coverage report. If these slots are also declared in the test case class, it is necessary to rename it in initTest() and cleanupTest(). 5. Compile your project with code code coverage enabled.
TestCoverageObject header:
#ifndef _TEST_COVERAGE_OBJECT_H #define _TEST_COVERAGE_OBJECT_H #include
TestCoverageObject source:
#include "testcoverageobject.h" #include
void TestCoverageObject::init() { #ifdef __COVERAGESCANNER__ __coveragescanner_clear(); #endif initTest(); }
void TestCoverageObject::cleanup() { cleanupTest(); #ifdef __COVERAGESCANNER__ QString test_name="unittest/"; test_name+=metaObject()->className(); test_name+="/"; test_name+=QTest::currentTestFunction(); __coveragescanner_testname(test_name.toLatin1()); if (QTest::currentTestFailed()) __coveragescanner_teststate("FAILED"); else __coveragescanner_teststate("PASSED") ; __coveragescanner_save(); __coveragescanner_testname(""); #endif }
- 129 - froglogic GmbH Support for specific test frameworks
22.3.3 GoogleTest
GoogleTest3 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test. Simply proceed as follows:
1. Call __coveragebrowser_install() in the main() function. 2. Write a TestEventListenet class which records the code coverage report upon every unit test completion. The listener should set the name (using __coveragescanner_testname()) and clear the instrumentation (using __coveragescanner_clear()) before executing a test item (class member startTest()) to ensure to get only the coverage data of the concerned test. When an test item is executed, the instrumentation and the execution status should be saved (using __coveragescanner_teststate() and __coveragescanner_save()) in the class member endTest(). The class CodeCoverageListener give an implementation example.
3. Add this listener in the Append function of the GoogleTest listener (function ::testing::UnitTest::GetInstance()->listeners().Append()). 4. Compile the unit test using CoverageScanner.
#include
class CodeCoverageListener : public ::testing::TestEventListener { public: // Fired before any test activity starts. virtual void OnTestProgramStart(const ::testing::UnitTest& unit_test) { }
// Fired before each iteration of tests starts. There may be more than // one iteration if GTEST_FLAG(repeat) is set. iteration is the iteration // index, starting from 0. virtual void OnTestIterationStart(const ::testing::UnitTest& unit_test, int iteration) { }
// Fired before environment set-up for each iteration of tests starts. virtual void OnEnvironmentsSetUpStart(const ::testing::UnitTest& unit_test) { }
// Fired after environment set-up for each iteration of tests ends. virtual void OnEnvironmentsSetUpEnd(const ::testing::UnitTest& unit_test) { }
// Fired before the test case starts. virtual void OnTestCaseStart(const ::testing::TestCase& test_case) { }
// Fired before the test starts. virtual void OnTestStart(const ::testing::TestInfo& test_info) {
3Project page of GoogleTest:http://code.google.com/p/googletest/
- 130 - froglogic GmbH Support for specific test frameworks
#ifdef __COVERAGESCANNER__ __coveragescanner_clear(); std::string test_name= std::string(test_info.test_case_name()) + ’/’ + std::string(test_info.name()); __coveragescanner_testname(test_name.c_str()); #endif }
// Fired aftera failed assertion ora SUCCESS(). virtual void OnTestPartResult(const ::testing::TestPartResult& test_part_result) { }
// Fired after the test ends. virtual void OnTestEnd(const ::testing::TestInfo& test_info) { #ifdef __COVERAGESCANNER__ __coveragescanner_teststate("UNKNOWN"); if (test_info.result()) { if (test_info.result()->Passed()) __coveragescanner_teststate("PASSED"); if (test_info.result()->Failed()) __coveragescanner_teststate("FAILED"); } __coveragescanner_save(); #endif }
// Fired after the test case ends. virtual void OnTestCaseEnd(const ::testing::TestCase& test_case) { }
// Fired before environment tear-down for each iteration of tests starts. virtual void OnEnvironmentsTearDownStart(const ::testing::UnitTest& unit_test) { }
// Fired after environment tear-down for each iteration of tests ends. virtual void OnEnvironmentsTearDownEnd(const ::testing::UnitTest& unit_test) { }
// Fired after each iteration of tests finishes. virtual void OnTestIterationEnd(const ::testing::UnitTest& unit_test, int iteration) { }
// Fired after all test activities have ended. virtual void OnTestProgramEnd(const ::testing::UnitTest& unit_test) { } };
- 131 - froglogic GmbH Support for specific test frameworks
int main(int argc, char **argv){
//initialize CoverageScanner library #ifdef __COVERAGESCANNER__ __coveragescanner_install(argv[0]); #endif
::testing::FLAGS_gtest_output = "xml"; ::testing::UnitTest::GetInstance()->listeners().Append(new CodeCoverageListener);
::testing::InitGoogleTest(&argc, argv); //init google test framework
return RUN_ALL_TESTS(); //run all tests which are listed in this project }
22.3.4 CxxTest
CxxTest4 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test. Proceed as follows:
1. Call __coveragebrowser_install() in the main() function.
2. Create a CxxTest TestListener class CoverageScannerListener by subclassing an existing listener. In the example below this is ErrorPrinter. It will record the code coverage report upon every unit test com- pletion. To ensure to get only the coverage data of the concerned test, the listener should set the name with __coveragescanner_testname() and clear the instrumentation with __coveragescanner_clear() before executing a test item (class member enterTest()). When an test item is executed, the instrumentation and the execution status should be saved in the member function leaveTest() with __coveragescanner_teststate() and __coveragescanner_save(). Finally, all test failure members of this class must be reimplemented to record the test failures.
3. In the main() function call the run() function of CoverageScannerListener instead of CxxTest::ErrorPrinter() .run().
4. Compile the unit test with CoverageScanner activated. Example:
#include
class CoverageScannerListener : public CxxTest::ErrorPrinter { public: CoverageScannerListener(std::ostream &o=std::cout, const char *preLine = ":", const char *postLine = "") : CxxTest::ErrorPrinter( o, preLine , postLine ) {}
4Project page of CxxTest:http://cxxtest.com
- 132 - froglogic GmbH Support for specific test frameworks
int run() { return CxxTest::ErrorPrinter::run(); }
void enterTest( const CxxTest::TestDescription & desc) { test_passed=true;
#ifdef __COVERAGESCANNER__ // Adjust the name of the test to display the tests // ina tree view in CoverageBrowser std::string testname="CxxTest/"; testname += desc.suiteName(); testname += "/"; testname += desc.testName();
// Reset the code coverage data to get only the code coverage // of the actual unit test. __coveragescanner_clear(); __coveragescanner_testname(testname.c_str()); #endif return CxxTest::ErrorPrinter::enterTest( desc ); }
void leaveTest( const CxxTest::TestDescription & desc) { #ifdef __COVERAGESCANNER__ // Record the execution state in the coverage report if (test_passed) __coveragescanner_teststate("PASSED"); else __coveragescanner_teststate("FAILED");
// Save the code coverage report of the unit test __coveragescanner_save(); #endif return CxxTest::ErrorPrinter::leaveTest( desc ); }
void failedTest(const char *file, int line, const char *expression) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedTest( file, line, expression ); }
void failedAssert(const char *file, int line, const char *expression) { // Only record that the test fails test_passed = false; return CxxTest::ErrorPrinter::failedAssert( file, line, expression ); }
void failedAssertEquals(const char *file, int line, const char *xStr, const char *yStr, const char *x, const char *y)
- 133 - froglogic GmbH Support for specific test frameworks
{ // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertEquals( file, line, xStr, yStr, x, y ); } void failedAssertSameData(const char *file, int line, const char *xStr, const char *yStr, const char *sizeStr, const void *x, const void *y, unsigned size) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertSameData( file, line, xStr, yStr, sizeStr, x, y, size ); } void failedAssertDelta(const char *file, int line, const char *xStr, const char *yStr, const char *dStr, const char *x, const char *y, const char *d) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertDelta( file, line, xStr, yStr, dStr, x, y, d ); } void failedAssertDiffers(const char *file, int line, const char *xStr, const char *yStr, const char *value) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertDiffers(file, line, xStr, yStr, value ); } void failedAssertLessThan(const char *file, int line, const char *xStr, const char *yStr, const char *x, const char *y) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertLessThan(file, line, xStr, yStr, x, y ); } void failedAssertLessThanEquals(const char *file, int line, const char *xStr, const char *yStr, const char *x, const char *y) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertLessThanEquals( file, line, xStr, yStr, x, y ); } void failedAssertRelation(const char *file, int line, const char *relation,
- 134 - froglogic GmbH Support for specific test frameworks
const char *xStr, const char *yStr, const char *x, const char *y) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertRelation( file, line, relation, xStr, yStr, x, y); }
void failedAssertPredicate(const char *file, int line, const char *predicate, const char *xStr, const char *x ) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertPredicate( file, line, predicate, xStr, x); }
void failedAssertThrows(const char *file, int line, const char *expression, const char *type, bool otherThrown) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertThrows( file, line, expression, type, otherThrown ); }
void failedAssertThrowsNot(const char *file, int line, const char *expression) { // Only record that the test fails test_passed=false; return CxxTest::ErrorPrinter::failedAssertThrowsNot( file, line, expression ); }
private: bool test_passed; };
int main() { #ifdef __COVERAGESCANNER__ __coveragescanner_install(argv[0]); #endif
// Use"CoverageScannerListener().run()" instead of"CxxTest::ErrorPrinter().run()" return CoverageScannerListener().run(); }
22.3.5 NUnit
Squish Coco provides an addin for NUnit5 version 2.4.4 and above as sample. To install it proceed as followings:
1. Build NUnitSquishCoco.dll using the Microsoft® Visual Studio® project NUnitSquishCoco.vsproj provided in the sample directory.
5http://nunit.org/
- 135 - froglogic GmbH Support for specific test frameworks
2. Copy NUnitSquishCoco.dll to the addins folder located in the bin folder where NUnit.exe can be found.
3. Start NUnit.exe and verify that the addin “NUnit Squish Coco” is loaded.
Once installed, as soon as NUnit’s test driver is executing a C# or C++ managed unit test test.dll, it generates automatically a code coverage execution report test.dll.csexe automatically if test.dll is instrumented with Squish Coco. The code coverage information is organized into a tree containing the coverage and the execution status for each single unit test. The execution report can be then imported into the application’s instrumentation database with CoverageBrowser or cmcsexeimport.
22.3.6 Squish
It is easily possible to run the GUI testing tool Squish together with Squish Coco in order to get the C/C++coverage of a Squishtest suite. A more in-depth analysis is then possible, correlating each test case (and its results) with the respective coverage information. This is especially true since Squish Coco features the comparison of individual executions, including the calculation of an optimal order.
22.3.6.1 General Approach
The approach depicted below is based on the possibility to apply information about the name, the result and a free-form comment to each execution (stored in .csexe files). See Chapter 22.1, page 123 for the details.
As an example we’ll use Squish for Qt’s addressbook on a Unix-based system and a JavaScript test script.
1. First we’ll initialize the execution data with the name of the Squish test case that is being run.
function main() { var currentAUT = currentApplicationContext(); var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe" var testCase = squishinfo.testCase; var testExecutionName = testCase.substr(testCase.lastIndexOf(’/’) + 1); var file = File.open(execution, "a"); file.write("*" + testExecutionName + "\n"); file.close(); var ctx = startApplication("addressbook"); ...
2. Insert the main test script at this point 3. After the main test script we’ll log the result of the test for the coverage tool:
... // wait until AUT shutdown while (ctx.isRunning) { snooze(1); // increase time if not enough to dump coverage data }
// test result summary and status var positive = test.resultCount("passes"); var negative = test.resultCount("fails") + test.resultCount("errors") + test. resultCount("fatals"); var msg = "TEST RESULTS - Passed: " + positive + " | " + "Failed/Errored/Fatal: " + negative; var status = negative == 0 ? "PASSED" : "FAILED"; var file = File.open(execution, "a"); file.write("
" + msg + "\n"); file.write("!" + status + "\n")- 136 - froglogic GmbH Support for specific test frameworks
file.close(); }
When you execute the scripts containing these steps, the Squish CocoExecution Report loads with the test case name, status and execution summary in the execution details and comments.
22.3.6.2 Simplified for Reuse
1. Create a file called squishCocoLogging.js in Test Suite Resources with the following functions:
function getExecutionPath() { var currentAUT = currentApplicationContext(); var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe" return execution; }
function logTestNameToCocoReport(currentTestCase, execution) { var testExecutionName = currentTestCase.substr(currentTestCase.lastIndexOf(’\\’) + 1) ; var file = File.open(execution, "a"); file.write("*" + testExecutionName + "\n"); file.close(); }
function logTestResultsToCocoReport(testInfo, execution){
var currentAUT = currentApplicationContext();
// wait until AUT shuts down while (currentAUT.isRunning) snooze(5);
// collect test result summary and status var positive = testInfo.resultCount("passes"); var negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo.resultCount("fatals"); var msg = "TEST RESULTS - Passed: " + positive + " | " + "Failed/Errored/Fatal: " + negative; var status = negative == 0 ? "PASSED" : "FAILED";
// output results and status to Coco execution report file var file = File.open(execution, "a"); file.write("
" + msg + "\n"); file.write("!" + status + "\n") file.close(); }A Python version of this code is:
import re
def getExecutionPath(): currentAUT = currentApplicationContext() execution = "%(currAUTPath)s\\%(currAUTName)s.exe.csexe" % {"currAUTPath" : currentAUT.cwd, "currAUTName" : currentAUT.name} return execution
- 137 - froglogic GmbH Support for specific test frameworks
def logTestNameToCocoReport(currentTestCase, execution): testExecutionName = re.search(r’[^\\]\w*$’, currentTestCase) testExecutionName = testExecutionName.group(0) file = open(execution, "a") file.write("*" + testExecutionName + "\n") file.close()
def logTestResultsToCocoReport(testInfo, execution): currentAUT = currentApplicationContext() # wait until AUT shuts down while (currentAUT.isRunning): snooze(5)
# collect test result summary and status positive = testInfo.resultCount("passes") negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo. resultCount("fatals") msg = "TEST RESULTS - Passed: %(positive)s | Failed/Errored/Fatal: %(negative)s" % {’positive’: positive, ’negative’: negative} if negative == 0: status = "PASSED" else: status = "FAILED"
# output results and status to Coco execution report file file = open(execution, "a") file.write("
" + msg + "\n") file.write("!" + status + "\n") file.close()2. Add the following function calls after startApplication() in the main test script:
execution = getExecutionPath();
logTestNameToCocoReport(squishinfo.testCase);
In Python:
execution = getExecutionPath()
logTestNameToCocoReport(squishinfo.testCase)
3. At the end of your script, after closing the AUT (for example with steps clicking File > Exit), call the following function:
logTestResultsToCocoReport(test);
In Python:
logTestResultsToCocoReport(test)
4. In the event your AUT closes unexpectedly, or a script error occurs, incorporating a try, catch, finally ensures your results still output to the Coco report file.
Your main test script should be similar to the following:
- 138 - froglogic GmbH Support for specific test frameworks
source(findFile("scripts","squishCocoLogging.JS"))
function main() { startApplication("addressbook"); execution = getExecutionPath(); logTestNameToCocoReport(squishinfo.testCase, execution);
try { // body of script } catch(e) { test.fail(’An unexpected error occurred’, e.message) } finally { logTestResultsToCocoReport(test, execution) } }
Python version:
source(findFile("scripts","squishCocoLogging.py"))
def main(): startApplication("addressbook") execution = getExecutionPath() logTestNameToCocoReport(squishinfo.testCase, execution)
try: try: # body of script except Exception, e: test.fail("test failed: ", e) finally: logTestResultsToCocoReport(test,execution)
- 139 - froglogic GmbH CoverageScannerTcl Part V Reference Manual
- 140 - froglogic GmbH Command Line Arguments
Chapter Command Line Arguments 23
Syntax:
coveragescannertcl [
Where:
Once the application exits a pair of .csmes and .csexe files will be written out that contain the code instrumentation information and execution data, respectively.
If the --cs-output switch is omitted the base name of the output files will the same as the
See (see Chapter 24, page 142) for a recommended set of --cs-exclude-code-regexp switches for typical Tcl/Tk applica- tions.
- 141 - froglogic GmbH Mixing Tcl with C++
Chapter Mixed Tcl/C/C++ Code Coverage Analysis 24
24.1 Analysis of Tcl script coverage
24.1.1 Demo application build
Copy the sample to your working diretcory
$ cp -r /opt/SquishCoco/doc/demoapp_tcl ./ $ cd demoapp_tcl
Edit Makefile and adjust the following variables
TCLLIBS=-ltk8.5 -litcl3.4 -ltcl8.5 TCLINCLUDES=-I/usr/include/tcl8.5 to match your system’s library names and include paths. Build the C++ executable with:
$ make
Test the simple Tk GUI with an invocation of the launcher script:
$ ./proto
24.1.2 Ad-hoc Tcl coverage run
Launch the application under test via the Tcl coverage scanning tool :
$ coveragescannertcl --cs-output=tcl ./proto
The GUI will show up as normal while code coverage is monitored. At this point the application might already be shut down via the "File¡Exit" menu entry or the [X] button in the window’s title bar. Observe the two files tcl.csmes and tcl.csexe being generated. The latter contains data about the code covered in the last execution. The former is a measurement database containing information about the instrumented code and possibly imported previous executions. In this case the base name of the output files was set via the --cs-output switch. If omitted it will be deduced from the application name.
- 142 - froglogic GmbH Analysis of Tcl script coverage
24.1.3 Result analysis
The results can be view interactively in the GUI provided by Coco. Run
$ coveragebrowser to launch it. Select "File¡Open..." to open the tcl.csmes file. You’ll see all Tcl files and procedures listed in the Sources and Methods tab, respectively. When selecting either of them a source code viewer will open that shows instrumented code with a gray background color. Note that the coloring is still limited to instrumented, i.e. seen code only. To check which code got actually executed (or not) select "File¡Load Execution Report..." and open tcl.csexe. Now red and green colors denoting positive and negative coverage will show up for files, procedures and the code itself.
24.1.4 Filtered Tcl coverage run
As you might have noticed that above ad-hoc run produced results for several unnamed and internal files. These stem from internal Tcl/Tk files or e.g. mouse event handlers that result in on-the-fly eval calls. Compared to the application-specific Tcl files the coverage information for these internal pieces of code is of little interest.
The easiest way to hide the non-relevant pieces of code is through usage of the various --cs-exclude-code options. A recommended list of switches will be shown in the repetition of the previous run with filtering options applied:
$ coveragescannertcl --cs-output=tcl --cs-exclude-code-regex="^\s*[^\n]*\s*$" --cs-exclude-code-regex="(tcl|tk)Init" --cs-exclude-code-regex="namespace eval ::itcl" --cs-exclude-code-regex="^# This file is generated" --cs-exclude-code-regex="tk::Menu(Motion|Leave)" ./proto
When viewing the new tcl.csmes in CoverageBrowser you’ll notice that the list of files is reduced to the relevant canvas.tclx, input.tclx, mainwindow.tclx, point.itcl and rect.itcl scripts.
24.1.5 Report Generation
All of text, XML and HTML reports can be generated from the CoverageBrowser GUI. With future automation of nightly test runs or Continuous Integration (CI) in mind we’ll create the report from the command line. And for this very purpose we’ll also pretend that the execution data has not been imported into the measurement database, yet. This step can be accomplished like this from the command line:
$ cmcsexeimport --title="First run" -m tcl.csmes -e tcl.csexe
The resulting proto.csmes file now contains everything we need to generated a report:
$ cmreport --title="Coverage Data" -m tcl.csmes --html=report.html
The resulting report.html file (including a report_html/ directory) can now be visited with a Web browser. It includes a listing of executions, source files, functions and metrics on a global level, per file and per functions.
- 143 - froglogic GmbH Analysis of C/C++ code coverage
24.2 Analysis of C/C++ code coverage
24.2.1 Instrumentation
While Tcl code can be instrumented for coverage analysis on-the-fly this step requires a rebuild for C or C++ applications. But this can be accomplished with almost no change to your regular application build. Here are the two possible approaches: • Modify your build system to make use of the code scanning tools csgcc, csg++, etc. that will both instrument the code and later use the original compiler to compile the code as usual.
• Sneak Coco’s compiler wrappers named gcc, g++, etc. into the executable search PATH. They will again instrument the code and then let the real compiler do its job. We’ll make use of approach number 2. First, clean the existing object files:
$ make clean
Then, recompile the application with a changed PATH and a switch that enables the compiler wrappers. For brevity, we’ll do this all in a single line:
$ PATH=~/SquishCoco/wrapper/bin:$PATH COVERAGESCANNER_ARGS="--cs-on" make
As a result we’ll not only get the main executable (proto.exe) but also a corresponding instrumentation file proto.exe.csmes. Information about the instrumented code (gray background color) can optionally be reviewed at this point
$ coveragebrowser -m proto.exe.csmes
24.2.2 Execution
Let’s run the application
$ ./proto and quit it via its GUI right after. The current working directory will now contain a proto.exe.csexe containing information about completed execution.
24.2.3 Result analysis
Same as with Tcl the results can now be reviewed in CoverageBrowser. A short cut invocation that opens both the instrumentation and execution files is shown here:
$ coveragebrowser -m proto.exe.csmes -e proto.exe.csexe
24.2.4 Report Generation
The process of a report generation works analogous to the approach shown for Tcl code above.
- 144 - froglogic GmbH Analysis of C/C++ code coverage
24.2.5 Analysis of mixed Tcl and C/C++ code coverage
Coverage data for the Tcl and C/C++ parts of a hybrid application can be gathered independently as shown above. Either in a single or multiple executions. The only extra step needed is the merging of the data into a single measurement database. Which then can be used to generate a single, combined report taking both languages into account.
24.2.6 Combined coverage run
Assuming the instrumentation of the C/C++ code is still in place we can gather coverage data for both types of languages with a single run:
$ coveragescannertcl --cs-output=tcl ./proto
With the proto.exe.csmes file still in place from the application rebuild we’ll now have two sets of files:
1. tcl.csexe
2. tcl.csmes
3. proto.exe.csexe
4. proto.exe.csmes
At this point the coverage data for Tcl and C/C++ could be analysed independently using cmcsexeimport. But we want to gain an overall picture and will therefore merge the results in the next section.
24.2.7 Merging measurement data
Instead of using the Coverage Browser GUI we’ll again use the command line tools. In the first step we’ll merge the instrumentation files stemming from the two languages into one:
cmmerge -o all.csmes tcl.csmes proto.exe.csmes
To save some disk space the --append option can be used rather than creating the new all.csmes file.
In the second step the two execution files are merged using the merge policy for the 2nd language:
$ cmcsexeimport -m all.csmes -e tcl.csexe --title="Hybrid" --passed $ cmcsexeimport -m all.csmes -e proto.exe.csexe -p merge --title="Hybrid" --passed
For the merge to actually happen the title has to be identical.
24.2.8 Report Generation
An HTML report encompassing coverage for both application parts can be generated with a single command line call:
$ cmreport --title="Hybrid Tcl/C++ Coverage" -m all.csmes --html=report.html
Select "Source Files" to see both Tcl (various files) and C++ code (main.cpp) being listed in a combined report.
- 145 - froglogic GmbH Part VI Command Line Tools
- 146 - froglogic GmbH cmreport - Code coverage report generation
Chapter cmreport - Code coverage report generation 25
cmreport is a utility which permits to generate Text, HTML, XML or CSV reports from an instrumentation data base (a .csmes file). It generates exactly the same reports as these generated by CoverageBrowser (see Chapter 14, page 68). Example: The simplest use case is the generation of an HTML report from a single instrumentation database. If the name of the file is project.csmes, we may write:
$ cmreport --csmes=project.csmes --html=report
This command creates a file report.html and a directory report_html. The file report.html is the main page of the report and can be opened with the browser, while report_html contains all the other files. Most of the other command line options just modify the content of such a report. Syntax:
cmreport -m
Where:
-m
-mr
--release-comparison=
-p
--source-type=
- 147 - froglogic GmbH File selection options
-l
--max-threads=
--stat: Print global coverage statistics to the standard output
25.1 File selection options
The following options allow to specify the source files shown in the report.
--include-file-abs-regex=
--exclude-file-abs-regex=
--include-file-abs-wildcard=
--exclude-file-abs-wildcard=
25.2 Execution selection options
The following options allow to specify the executions that occur in the report.
-s
-d
--deselect-to-check: Deselect all executions with the status “CHECK_MANUALLY”
--select-passed: Select all executions with the status “PASSED”
--deselect-passed: Deselect all executions with the status “PASSED”
--select-failed: Select all executions with the status “FAILED”
--deselect-failed: Deselect all executions with the status “FAILED”
- 148 - froglogic GmbH Options for HTML or CSV reports
--select-unknown: Select all executions with unknown status
--deselect-unknown: Deselect all executions with unknown status
--executions-reference-from-csmes=
-sr
-dd
--deselect-to-check-reference: Deselect all reference executions with the status “CHECK_MANUALLY”
--select-passed-reference: Select all reference executions with the status “PASSED”
--deselect-passed-reference: Deselect all reference executions with the status “PASSED”
--select-failed-reference: Select all reference executions with the status “FAILED”
--deselect-failed-reference: Deselect all reference executions with the status “FAILED”
--select-unknown-reference: Select all reference executions with unknown status
--deselect-unknown-reference: Deselect all reference executions with unknown status
25.3 Options for HTML or CSV reports
Command line arguments:
-h
--html-single=
--csv-excel=
--execution-level=
--css=
--icon=
--title=
- 149 - froglogic GmbH Options for HTML or CSV reports
--section=
--global-threshold-low-medium=
--global-threshold-medium-high=
--source-threshold-low-medium=
--source-threshold-medium-high=
--function-threshold-low-medium=
--function-threshold-medium-high=
-b | --coverage-branch : Code coverage on branch level only
--coverage-condition : Code coverage at decision/condition level
--line-coverage : Line coverage analysis
--function-coverage : Function coverage analysis
-t | --test-coverage : Test count mode
- 150 - froglogic GmbH Options for text reports
-D | --debug : Debug flag
--csv-field-separator=
--csv-comma=
25.3.1 Obsolete options for CSV reports
Command line arguments:
--csv-method=
--csv-source=
--csv-field-separator=
--csv-comma=
25.4 Options for text reports
Command line arguments:
-t
--format-executed=
--format-unexecuted=
--format-dead-code=
%f: Source code file name
%l: Line number
%c: Column number
%m: Explanation
It is allowed to to set the output file to an empty string (i.e.: --text=), in this case the standard output of the console is used.
- 151 - froglogic GmbH Options for EMMA-XML reports
25.5 Options for EMMA-XML reports
The following options are used if a report in the EMMA-XML format is generated:
--emma=
--emma-source-pattern=
--emma-package-pattern=
%f: file name (without path)
%b: file name without path and extension
%e: file extension
%p: file path (without file name)
%P: absolute file path (without file name) For Jenkins-CI users, it is not recommended to use the last two parameters. The default values are chosen to i work with the EMMA-XML plugin.
25.6 Options for JUnit reports
Command line arguments:
-j
25.7 Options for Cobertura reports
Command line arguments:
--cobertura=
--cobertura-source=
--sonarqube-project=
To generate a Cobertura reports two work flows are supported: 1. Specifying directly all necessary arguments to generate the report which are:
- 152 - froglogic GmbH Options for Cobertura reports
(a) the output file name (--cobertura=
2. Indicating the path of a SonarQube project properties file sonar-project.properties (--sonarqube-project=
- 153 - froglogic GmbH cmcsexeimport
Chapter cmcsexeimport - Command Line Import Utility 26
Syntax:
cmcsexeimport -m