Masaryk University
Faculty of Informatics
C++ Unit Testing Frameworks
Diploma Thesis
Brno, January 2010 Miroslav Sedlák i Statement
I declare that this thesis is my original work of authorship which I developed individually. I quote properly full reference to all sources I used while developing.
...…………..……… Miroslav Sedlák
Acknowledgements
I am grateful to RNDr. Jan Bouda, Ph.D. for his inspiration and productive discussion. I would like to thank my family and girlfriend for the encouragement they provided. Last, but not least, I would like to thank my friends Ing.Michal Bella who was very supportive in verifying the architecture of the extension of Unit Test Framework and Mgr. Irena Zigmanová who helped me with proofreading of this thesis.
ii Abstract
The aim of this work is to gather, clearly present and test the use of existing UTFs (CppUnit, CppUnitLite, CppUTest, CxxTest and other) and supporting tools for software project development in the programming language C++. We will compare the advantages and disadvantages of UTFs and theirs tools. Another challenge is to design effective solutions for testing by using one of the UTFs in dependence on the result of analysis.
Keywords
C++, Unit Test (UT), Test Framework (TF), Unit Test Framework (UTF), Test Driven Development (TDD), Refactoring, xUnit, Standard Template Library (STL), CppUnit, CppUnitLite, CppUTest Run-Time Type Information (RTTI), Graphical User Interface (GNU).
iii Contents
1 Introduction ...... 2 2 The Software Testing ...... 5 2.1 Types of Testing ...... 6 2.2 The Test Environment (TE) ...... 8 2.3 The Unit Test (UT) ...... 9 2.4 The Test Framework (TF) ...... 10 2.5 The Unit Test Framework (UTF) ...... 10 2.6 The Test Driven Development (TDD) ...... 13 3 The Comparison of the Unit Test Frameworks ...... 16 3.1 The xUnit ...... 17 3.2 The CppUnit ...... 19 3.3 The CppUnitLite ...... 26 3.4 The CppUTest ...... 28 3.5 The CxxTest ...... 30 3.6 The Conclusion of Analysis ...... 33 4 Programs for UTF ...... 34 4.1 CppUnit Base Programs ...... 34 5 The Unit Test Envinronment (UTE) ...... 38 5.1 The Framework Design ...... 39 5.2 Implementation of Extension CppUnit ...... 44 5.3 Implementation of UTE ...... 47 6 Conclusion ...... 55 7 Bibliography ...... 57 8 Appendix ...... 58 8.1 Abbreviations ...... 58 8.2 Glossary ...... 58 8.3 Source code ...... 60
1 1 Introduction
Today, large amount of software is developed for various purposes. Quality of software source code is measured by various factors, where the most important one is how the product fits customer’s requirements. Ensuring that it is so and that the customer is able to use the final product efficiently can be achieved through continuous testing, debugging and other techniques (see http://en.wikipedia.org/wiki/Software_quality, where different views of software quality are presented), that are out of scope of my thesis. Important phase of detection and localisation of a problem is testing, which increases the quality of developed software. The main purpose of testing is to reduce the risk of an error. There are many possibilities how to use testing during the development phase of an application. The most common is the possibility to create tests to verify functionality when software is developed. If tests have not revealed an error, we may assume that the application has no errors. However, the occurrence of errors should rather be expected, since it is known that no large-scale application is perfect. The process of testing can not verify everything, so we try to verify those parts of the system whose failure would affect the execution of the application. According to Kernigham, the most important rule of testing is to do testing as described in the book: The Practice of Programming [1]. The test can be created on different levels. This thesis is aimed at the first level of testing (Unit Testing), which is performed by developer of an application. The cornerstone for the first level of testing is development of tests and their automation for smallest units.
Issues that affect source the code quality include ease of maintenance, testing, debugging, fixing, modification and portability. This involves usage of various software applications (also known as tools) exploited to assure these requirements. If we want to create an environment containing such a program, we must first choose a program, which might be difficult to combine with other programs. Creating this environment is very challenging because we must determine which of the programs is more difficult to relate with other programs. It is testing framework, because this program is developed in C ++ programming language. There are several ways how to include the functionality of other programs into the testing framework:
Using program independently of the testing framework – for example running a binary test in a debugger Including programs into the testing framework – this is the subject of this thesis Implementing the functionality of a given software as a module/plug-in in the testing framework
2
Today, there are many test frameworks, but there is no environment to interconnect the test framework and programs. It is similar to IDE, which interconnects programs:
a source code editor program a compiler and / or an interpreter program build automation programs a debugger program
One of the goals of this thesis is to implement a software application that provides comprehensive facilities for the aforementioned issues of the software code quality. This environment should be configurable and extensible by developers according to their needs. In order to design environment correctly, first we have to use concepts related to software testing. Then we determine requirements of the test framework and compare it with the existing testing frameworks. On the basis of our analysis we select a test framework that will be used for our testing of developed software. Then a review of the programs that can be used to achieve the source code quality follows. We focus on the existing programs developed for the individual test framework as well. As we review later, there is no environment that allows additional programs to cooperate with the test framework, what forces us to define the requirements and implement a new environment. We also define functional requirements for the extension of the testing framework. Finally, we describe the implementation of the test framework extension and the environment.
This thesis focuses on software testing methodology in the C++ programming language and comparison of existing testing systems (also known as frameworks) from the first-level testing point of view. Today, there are many of software testing methodologies and the choice depends on many factors. To determine whether an environment is usable for testing of our application, it is necessary to define a set of requirements. If these requirements are defined unambiguously, we may begin to analyze the advantages and disadvantages of existing Unit Test Frameworks (UTF) of the C++ programming language. To determine an ideal system it is important not only to specify our requirements, but also to analyze the possibilities of use of various instruments to facilitate testing on the first level. The result of this work would be selecting particular the Unit Test Framework and to specify an extension, if necessary. Then the analysis of existing programs and possibilities of their combinations for testing and debugging follows. The benefits of testing and debugging used in the software development should emerge from this description. The last part of this thesis is the implementation of the testing environment allowing an easy integration of various debugging and testing software.
3 The beginning of the next chapter presents an introduction to software testing, followed by a short description of types of testing; we define single levels for software testing as well. The most important terms of testing area are explained and defined in the same chapter, which are necessary for a full understanding of UTF and their possibilities for testing. The one possible model how to develop an application using Test Driven Development (TDD) is presented at the end of this chapter.
The third chapter describes the comparison of UTFs and programs for projects in the programming language C++. In this chapter we shall present the list of features that are important for ideal framework. A concept closely related to CppUnit is described for better understanding of one of UTFs. At the end of this chapter, the overview of programs for CppUnit framework (User Interfaces) and others which can be used for debugging of developed software is presented.
The fourth chapter describes unit testing environments we would like to include into our new environment. It also describes modules designed for CppUnit’s testing automatization.
In the fifth chapter the requirements for selection of UTF are presented. Rules which should be used to improve the code’s readability and establish more consistency are described in this chapter too. Finally, chapter the rules for UTE summarize the main contributions of the thesis and propose possible directions for future research.
The Appendix contains a list of abbreviations used in the thesis and in the Glossary all the concepts are explained for better understanding of our issue.
4 2 The Software Testing
In this chapter we present the fundamentals of testing. For better understanding we shall describe the meaning of terms: Unit Test, Test Framework, Unit Test Framework (UTF) and Test Driven Development (TDD).
Testing is the most important phase in the development of an application because it verifies the correctness of the code. Testing should uncover any problems which reduce the quality of the program. The testing of an application is closely related to the following terms: Verification and Validation (V&V):
Verification - verifying the internal consistency of the software (whether the product conforms to design, design analysis, analysis of requirements). It is a process of evaluating software to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase.
Validation - checking (an independent external source) the whole solution to the expectations of users or clients. It is a process of evaluating software during or at the end of the development process to determine whether it satisfies specified requirements.
V&V is process of checking that a software system meets specifications and that it fulfils its intended purpose. It is normally part of the software testing process of a project [2].
The main purpose of testing is to reduce the risk of error. In principle, the tests can be created with either the development or before the development of application to verify the correctness of the blank test. The second type of testing is described in detail in the next chapter Test Driven Development. Testing verifies functionality and if an error can not be found, we can assume that the application does not have any errors. Today, there are lots of types of testing as we shall describe in the next chapter.
The testing has a relationship with test case; it is a set of conditions or variables according to which a tester determines whether an application or software system is working correctly or not. Written test cases are usually collected into test suites. Test cases are: used for verification and validation of testing prepared for verification - to determine if the process that was followed to develop the final product is right
5 executed for validation - if the product is built according to the requirements of the user
2.1 Types of Testing
The development process involves various types of testing. Each test type addresses a specific testing requirement. The most common types of testing involved in the development process are: Unit Test, System Test, Integration Test, Functional Test, Performance Test, Acceptance Test. Tests can be divided to various criteria [3].
Testing methods of creation - two basic types of software testing
Manual testing - tests are created manually by testers. This approach is the most common and often it is the only way to implement the testing program. The main disadvantage of this approach is that it is repeatable and often very time consuming. o Examples: performance tests of system in cluster environment where no benefit from automatization is expected (e.g. the tests need to be done until a performance level is reached); another example may be testing of web applications under different browsers like Opera/Chrome/Firefox/IE 6.0, 7.0 where each one processes web page differently Automated testing - another possibility is the automatic testing. Many programming groups are relying more and more on automated testing, especially groups that use Test Driven Development. There are many frameworks to write tests in, and Continuous Integration software will run tests automatically every time code is checked into a version control system. While automation cannot reproduce everything that a tester can do, it can be very useful for regression testing. However, it does require a well-developed test suite of testing scripts in order to be truly useful.
Testing methods - the amount of information available to us when creating tests Black box testing – the tester who creates a test may not have information about how the application works at the level where the test is created. White box testing – for the development of the tests it is necessary to know about the operation of the test. This testing is based on knowledge of the internal logic of an application’s code.
Testing levels: the level at which we perform tests Unit Testing (unit tests) – refers to tests that verify the functionality of a specific section of code, usually at the function level. In an object-oriented environment, this is usually at the
6 class level, and the minimal unit tests include the constructors and destructors. This type of tests are usually written by developers as they work on code (white-box style), to ensure that the specific function is working as expected. Unit testing alone cannot verify the functionality of a piece of software, but rather is used to assure that the building blocks the software uses work independently of each other. Integration Testing - is any type of software testing that seeks to verify the interfaces between components against a software design. Software components may be integrated in an iterative way or all together ("big bang"). Normally the former is considered a better practice since it allows interface issues to be localised more quickly and fixed. Integration testing works to expose defects in the interfaces and interaction between integrated components (modules). Progressively larger groups of tested software components corresponding to elements of the architectural design are integrated and tested until the software works as a system. System Testing – usually created by testers (a group of programmers, which is separated from a group of developers making applications). It tests the system as installed by the contracting authority. All the means necessary for running applications (database, network resources and others) are expected to be present. System tests include verification for example of functionality, user interface, security and another. System testing tests a completely integrated system to verify that it meets its requirements. Acceptance Testing (Customer Testing) - is a black-box testing performed on a system prior to its delivery. It is also known as functional testing, black-box testing, release acceptance, QA testing, application testing, confidence testing, final testing, validation testing, or factory acceptance testing. Acceptance testing can mean one of two things: o A smoke test is used as an acceptance test prior to introducing a new build to the main testing process, i.e. before integration or regression. o Acceptance testing performed by the customer, often in their lab environment on their own HW, is known as user acceptance testing (UAT). Acceptance testing may be performed as part of the hand-off process between any two phases of development.
7
Figure 1: Test strategy for developing software
This thesis focuses solely on one part of testing and this is Unit Testing. The next chapters describe more details and connections regarding Unit Testing and its possibilities.
2.2 The Test Environment (TE)
Today there are many environments for testing which is used for the software project testing. The Test Environment can contain programs, simulators, and other support software necessary for testing developed software. For example, the Test Framework can use other programs to improve testing. The choice of how to use environment for testing depends on many factors. For example, on what programming language is used. The environment for testing should at least:
be able to runs a series of tests decide whether the tests were successful. Individual tests should be independent of the others prints out the reason (return value) when the test is done with an error summarizes the obtained results
Ideally, the environment for testing should be independent of its own tests. The testing process could be divided into three phases: 1. Preparation before testing – before testing we should have plan, which tests what we want to run. Tests can be divided into logical groups. For example by depending on the test part of the
8 application or by depending on the possibility to run tests in parallel. Also, we should prepare data. For example, we can generate random data. If tests need to simulate the activity of some external sources (e.g. database), we should prepare it in this part. The environment for testing should also be ready. Before testing the responsibility for certain parts of the application should be defined. 2. Running tests - running of one or a group of tests. If tests are run parallel or any activities are run in the background, so these activities should be synchronized. The testing environment can help with the management of machines or platforms, when multiple computers or multiple platforms are used. Another function of environment should be saving input and output tests into version control system for testing. 3. After the tests – it is very important to generate a report summarizing the results of the test. This report should provide an overview of tested files and how they have been tested. We should be able to distinguish between the failed tests and an error made during the testing. We should be able to artificially create a bug too, if test is finished as successful before modification.
2.3 The Unit Test (UT)
The Unit Test (UT) is an automate software verification and validation method which is used by a programmer to test weather individual units of source code are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual function or procedure [1]. The Unit Test is an automated test, also known as developer test. The term “Unit” refers to low-level test cases written in the same programming language as the production code, which directly access objects and members. The term “Test” can be stated as the process of validating and verifying that a software program/application/product:
1. meets the business and technical requirements that guided its design and development; 2. works as expected; and 3. can be implemented with the same characteristics.
The failure of a Unit Test implies a mistake in particular tested part of code. We should know exactly where to search to find the bug, because the simple Unit Test examines only part of source code. The goal of Unit Testing is to isolate each part of the program and show that the individual parts are correct. Unit Tests find problems early in the development cycle and from the above text it follows that Unit Test allows the developer to refactor the code at a later date, and make sure the module still works correctly (i.e. regression testing). Developers write tests for every class they produce. The tests are intended to test every expression of the class. All the Unit Tests are combined into a single huge suite. The Unit Tests are run before developers release a new code. All the test cases must be
9 successful. If any test fails the problem should be fixed. It might occur that some part of tests is passed, although it should not. When this happens, developers should enhance the Unit Tests. This problem or any similar one that might occur should not happen again.
The Unit Testing is the cornerstone of Extreme Programming, which counts on an automated Unit Testing Framework. This automated UTF provides an automated solution with no need to write the same tests many times, and no need to remember what should be the result of each test. Extreme Programming uses the creation of Unit Tests for TDD. The details regarding software development technique TDD are described below. The developer writes a Unit Test that exposes either a software requirement or a defect. This test will fail because either the requirement isn't implemented yet, or because it intentionally exposes a defect in the existing code. Then developer writes the simplest code to create new test. This one and other tests should finish as successful for verification correct implementation.
2.4 The Test Framework (TF)
The Test Framework is an object-oriented software package for software testing. For better understanding we shall know more about test, let the object (TestCase) manage the execution of a single test. Each test should be isolated from the others, so it can create data before executing and destroys it when it is done. Composites of tests (TestSuite) allow many tests to be run together. Passing a CollectingParameter (TestResult) to the tests as they run allows the collection of statistics. That is the framework, three objects: TestCase, TestSuite, TestResult. There are simpler ways of running automated tests, and many more complicated ways, but this architecture is basic model.
2.5 The Unit Test Framework (UTF)
Most people who write software have at least some experience with Unit Testing. If somebody wrote a few lines of throwaway code just to try something out, they would build a Unit Test for verification of the correctness of code. On the other end of the software spectrum, many large-scale software applications have a huge number of test cases that are repeatedly executed and added to throughout the development process. Unit Tests are useful at all levels of programming. The UTF is software program to support writing and running Unit Tests, including a foundation on which to build tests and the functionality to execute the tests and report their results. They are not solely programs for testing. They can also be used as development programs on a border line with pre- processors and debuggers. UTFs can contribute to almost every stage of software development, including software architecture and design, code implementation and debugging, performance optimization, and quality assurance.
10 Unit Tests are usually developed in parallel with production code, but they are not built into the final software product. The relationship of Unit Tests to code production is shown in Figure 2.
Figure 2: Production application and Unit Test Framework
Unit Tests use the application's objects, but exist inside the UTF. The production code is verified by using built-in Unit Tests. The size of the compiled application tends to be kept smaller for the same reason. The tests can be run separately from the application, so the objects can be tested in isolation. A single Unit Test should test a particular behaviour within the production code. Its success or failure validates a single unit of code. Well written tests set up an environment or scenario that is independent of any other conditions, then perform a distinct schedule and check a final result. These tests should avoid dependencies on the results of other tests (called test coupling), and they should be short and simple. An UTF can be used to verify very complex architectures if tests are started of the basic functionalities and then gradually building to tests of compound objects and behaviours. The Test Framework is much easier than developing standalone tests and also produces more thorough, effective tests. A comprehensive suite of Unit Tests enables rapid application development, since the effects of every change can be immediately and thoroughly verified [5].
UTF helps to simplify the process of Unit Testing. It has been developed for a wide variety of languages. For more details we refer to the list of Unit Testing Frameworks [6][6]. It is generally possible to perform Unit Testing without the support of specific framework by writing a client code that exercises the units under test and uses assertions, exception handling or other control flow mechanisms to signal failure. When framework is created for software development then adding Unit Tests becomes relatively easy. However, in some frameworks many advanced Unit Test features are missing and must be manually coded.
For example the CppUnit is one of UTF for testing. This framework is primarily intended for creating tests in C ++ language, but other UTFs could be used. We shall describe what is needed for testing related to this framework for better understanding. The CppUnit could be divided into two parts:
11 1. The first part defines how well-written tests should look like and provides a set of classes which should be used for creating tests. 2. The second part provides an environment for testing. To be able to run tests in the CppUnit we can use one of the user interfaces.
The CppUnit is continuously expanded with new functions and features. Unit Test in this framework is an inherited class that extends the class TestFixture. Two methods are the most important for testing:
void setUp() - this method is used to initialize. It is runs before each test. void tearDown() - this method is finalized. It is runs after each test.
A similar function can be replaced by the class constructor and class destructor respectively. The use of the constructor in the class is not suitable, because test classes need only one initialization between start and end of testing. The testing is executed as environment for running tests. This environment should allow re-testing of tests and re-selecting group of tests. Each test class contains several test functions. These testing methods start with prefix test - void testX(), where X is the name of the tested property (properties). Various conditions are verified inside these methods. In principle, each test can be divided into three parts:
1. Data preparation - preparing the input data and values. 2. Preparation of conditions for the test - preparing part of the application (unit), which we want to test 3. Verification of test parts - there are several assert methods in the CppUnit (beginning with prefix assert). There are assertTrue, assertFalse, assertEquals, assertNotEquals, assertNull methods and other. For example, the method assertTrue(a) tests whether that condition “a” is true. The method assertEquals(b, c) determines whether two arguments “b” and “c” are equal. By these methods we should verify whether the application behaves as expected.
Tests can be grouped with class TestSuite. The test result can be obtained using the TestResult class, which allows the detailed description of result when the test is finished. This result we can see in the chapter regarding the analysis of the Unit Test Frameworks. The test terminates unsuccessfully when any of the assert conditions is not met. This test is run using the environment for testing. We can get three types of results:
1. Test ended successfully - the whole test is done and all the conditions described by assert methods are met.
12 2. Test ended with error during running - usually known as the result of an error. Some of the tests done with mistake, for example, when the code uses division by zero or some exception caused. 3. Test ended with error - one of the conditions described in assert methods is not met. This test is usually marked as fail by environment for testing. In this case, we should get information which error condition is become on the standard output, what value we obtained and what value we expected. We can attach some description to this error message and it is written on the standard output.
We generally want to verify test functionalities of developed product. In ideal case the written tests should uncover errors in the application.
2.6 The Test Driven Development (TDD)
UTFs are key elements of TDD, also known as "test-first programming." TDD is one of the most important and widely used practices in Extreme Programming (XP) and other Agile Development methodologies. Test Frameworks are used by TDD and they achieve their maximum utility, but can be useful alone. We shall concentrate on UTF as a family of programs, rather than specifically on TDD, but the two topics are closely related. The key rule of TDD can be summarized as “test twice, code once”, where this rule refers to the three-step procedure involved when a code is changed:
1. Write a test for a new non-existent definition of the source code. Run all tests and find out weather the new one has failed, other tests should be successful. 2. Write the new code, doing “the simplest thing that could possibly work”. 3. Run all tests and see them all succeed, and refactor the code. Remove duplication of tests.
Step 1 is to write a test, run it, and verify the resulting failure. The failure is important because it validates that the test fails as expected. It is often tempting to skip running the test and see the failure. This step should not be ignored.
In Step 2, code is written to make the test succeed. We should keep the basic rule: the simplest thing that could possibly work. This may be a completely trivial implementation, such as having the new code return a constant value or copying and pasting code from one place to another. It should pass the test because new code written using copy and paste method could contain lots of mistakes. The temptation in this step is to do a little extra work and make some additional code change not directly related to passing the test. This step should not be ignored too.
13 In Step 3, the test succeeds, verifying both the new code and its test. At this point, the new code may be refactored. Refactoring is a software engineering concept defined as “behaviour-preserving transformation”. More formally, refactoring is the process of transforming the code to improve its internal design without changing its external functionality. Within the TDD cycle, refactoring starts with the inelegant code that is written to pass the Unit Test and improves it by removing duplication or other not nice parts. Since the Unit Test is in place, the details of how the code is implemented can be altered with confidence.
These three basic steps are the TDD cycle. These activities of steps create loops of Extreme Programming (Continuous programming), as illustrated Figure 3. Tests help us keep promises regarding the quality, cost, and existence of previously installed features.
Figure 3: Test Driven Development (TDD) cycle – inner (coding loop) and outer (release loop) loops of Extreme Programming
New code should only be written when a test fails. Code changes are only expected to occur when we are refactoring, adding new functionality, or debugging. Continuously repeating the TDD cycle is the most atomic level of the software development process. Software changes generally fall under two categories: adding new functionality or fixing bugs.
When the TDD cycle is respected, it is possible to write a correct code on the first attempt: “code once”. This process gives a clear indication that a piece of work is done. When a new Unit Test is written and then fails, the task is halfway completed. It can not move on to something else until the test succeeds [4].
14 The Test Driven Development (TDD) is one possible model how to develop the application. The main purpose is to ensure that all tests are passed. In addition, the various cycles can be modified and extended the functionality that is contained in developed applications. The product can be passed on to customers, when all successful tests describe all the functions that are required by customers. Important part of TDD is refactoring. It is the transformation of source code, which leads to improvement of its readability or structure, but without changing the meaning or behaviour. Typical activities in the refactoring are renaming variables, optimization of code (minimization), replacing the command “if” by polymorphism or optimization of the hierarchy or inheritance. Refactoring is an important part not only to TDD, but it is also an essential element of a different model for software development - Extreme Programming.
15 3 The Comparison of the Unit Test Frameworks
In this chapter we shall present the use of existing Unit Test Frameworks (CppUnit, CppUnitLite, CppUTest, CxxTest) and programs for projects which are created in the programming language C++. For better understanding we shall describe the concepts closely related to one of UTs, i.e. CppUnit.
When the software is developed these questions might arise: How can we test our new/extended/changed code? How can we verify whether the code is working? How can we do it?
There are many ways how to find answers to these questions. Printing code with stream output calls or stepping through a debugger, are two of the simpler ways, but they both have drawbacks. The print- out of streaming out text is fine, but it makes code ugly and it generates far more information than is needed. Stepping through code is a good idea, but it is not automatic. There are Test Automation Frameworks which can be used for unit automation testing; these frameworks are called as Unit Test Frameworks. For C++ programming languages exists lost of UTFs, for example CppUnit. It can be run automatically, it can be easily set up and it is always there to help to keep confidence in the quality of code.
Another question might be the following: How do we choose a UTF? It depends on how we are going to use it and what we are going to do with it. For example, Kent Beck illustrated TDD by building a Test Automation Framework in Python: Test-Driven Development – By Example [7]. He used the emerging Test Automation Framework to run the tests and he wrote test for each new capability. This application is a very good example of both TDD and his extension. It illustrates that we should analyze existing frameworks for Unit Testing. There is xUnit family, generic name for any Test Automation Framework for Unit Testing, which is usually automated by using the same programming language as is used for building the developed software, because less effort is required to learn how to automate tests [8]. Most programming languages in widespread use today have at least one implementation of xUnit. The family is separated into language groups where each member of group can be used differently. How to choose a correct UTF from members of C++ group will be discussed further in next chapter. We shall explore only UTF based on the xUnit family. A detailed description is presented in the following chapters.
16 3.1 The xUnit
Before we begin to describe C++ members of xUnit we should create a list of features that are important for our work [9]. In particular, we will set up Test Driven Development (TDD), which means that we will be writing and running many small tests. It is going to be used for software development. It should also fit personal TDD style. The following list summarizes the features we would like to use in a UTF in order of importance. We shall evaluate each framework on the basis of these features.
1. Minimal amount of work needed to add new tests. The shorter and easier it is to write, the easier it will be to refactor, which is important when we are doing TDD. 2. Easy to modify and portable. The framework should have not dependencies on the non standard libraries, and it should not contain non C++ STL features. To verify this one, we will utilize a set of Unit Tests using each library under Linux with g++. 3. Supports setup/teardown steps (fixtures). We will adopt the style recommended by David Astels in his book Test Driven Development: A Practical Guide about using only one assertion per test. It makes tests a lot easier to understand and maintain, but it requires heavy use of fixtures. A framework without them is ruled out immediately. 4. Handles exceptions and crashes correctly. We do not want the tests to stop just because some code that was executed accessed some error (invalid memory location, a division by zero). The UTF should report the exception of tests. 5. Good assert functionality. Failing assert statements should print the content of the variables that were compared. It should also provide a good set of assert statements. Bonus points for providing ways to check whether exceptions were or were not thrown. 6. Supports different outputs. By default, we would like to have a format that can be understood and parsed by IDEs for example like Visual Studio, so it is easy to navigate to any test failures as if they were syntax errors. But we would like to have ways to display different outputs (more detailed ones, shorter ones, parsing-friendly ones, etc) too. 7. Supports suites. This feature permits to have many libraries in the UTF, where each of them has own set of tests. It would be nice, but it is not necessarily. 8. Timing support. Both times for total running time of tests, and for individual ones. We would like to keep an eye on running times. Not for performance reasons, but to prevent other mistakes. Ideally, we would also like to see a warning printed if any single test goes over a certain amount of time, which can be configured.
Easy to install was not considered a priority, since we only once went through that.
17 3.1.1 Our Ideal Framework
Before we start going over each of the major and a few minor C++ UTF, we decided we will apply the philosophy of Test Driven Development to this analysis and start by thinking what we would like to have. We decided to write the set of sample tests in some ideal UTF. In the ideal world, this is what we would like from Unit Tests. The simplest possible test should be trivial to create. Just one line to declare the test and then the test body:
TEST (SimplestTest){ float someNum = 2.00001f; ASSERT_CLOSE (someNum, 2.0f); }
A test with fixtures is going to be a bit more complicated, but it should still be really easy to set up:
SETUP (FixtureTestSuite){ float someNum = 2.0f; std::string str = "Hello"; MyClass someObject("somename"); someObject.doSomething(); } TEARDOWN (FixtureTestSuite){ someObject.doSomethingElse(); } TEST (FixtureTestSuite, Test1){ ASSERT_CLOSE (someNum, 2.0f); someNum = 0.0f; } TEST (FixtureTestSuite, Test2){ ASSERT_CLOSE (someNum, 2.0f); } TEST (FixtureTestSuite, Test3){ ASSERT_EQUAL(str, "Hello"); }
The first point of view is that there is a minimum amount of code spent in anything other than the tests themselves. The simplest possible test takes a couple of lines and needs not support other than a main file that runs all the tests. Setting up a fixture with setup/teardown calls should also be trivial. We do not want to inherit from any classes, override any functions, or anything. Just write the setup step and move on.
18 We should look at the setup function again. The variables that are going to be used in the tests are not dynamically created. Additionally, we should know that those objects should only be created correctly before each test, and not before all tests start. We do not know how the tests will be used, but it is described above how we would like to use it. That is reason why this is an ideal framework. Now we can contrast real UTFs. For each of the frameworks we look at the list of wanted features and we try to implement the tests with this ideal framework.
3.2 The CppUnit
Because the CppUnit is one of basic UTF we shall describe the possibility of using programs, architecture and how we can use it for testing. By this description we will get better knowledge of UTF. CppUnit is a UTF module for C++. The CppUnit is used for driving testing and is a member of xUnit family. CppUnit is a port of JUnit to C++, and was originally authored by Michael Feathers and Jerome Lacoste. Its basic architecture and usage closely follow the xUnit model. The implementation details differ from JUnit as a result of design choices by CppUnit's developers and language differences between Java and C++. CppUnit's implementation makes full use of advanced C++ language features, including templates, abstract classes, nested classes, and the Standard Template Library (STL). It also makes extensive use of C macros, which some consider inelegant and error- prone, but are definitely useful here. CppUnit is designed to be thread-safe. CppUnit is open source software released under the GNU Lesser General Public License. This license makes the code free for use, modification, and redistribution. For details, refer to http://www.gnu.org. The main purpose of CppUnit is to support developers in doing their Unit Testing of C++ programs. The one of the required program quality is reliability for each project [3]. Test input does not exist, because the test cases are created by writing in C++. Test output is in XML or text format for automatic testing and GUI based on supervised tests. For a quick tour of Unit Testing with CppUnit we shall depict how we can use it as describe in chapter 3.2.2 . Before that we should know which platform will be used for testing. The CppUnit can be used for testing in majority platforms, but this testing will concentrate on the Linux platform only. Other types of platform- specific build are described on the website BuildingCppUnit1. CppUnit can use lots of features to permit easier and faster automatic testing. The list of features for CppUnit is described below (for Version CppUnit 1.12.1 0): 1. XML output with hooks for additional data (XSL format available in release 1.10.2 needs some fixing) 2. Compiler-like text output to integrate with an IDE 3. Helper macros for easier test suite declaration 4. Hierarchical test fixture support
19 5. Test registry to reduce recompilation need 6. Test plug-in for faster compile/test cycle (self testable dynamic library) 7. Protector to encapsulate test execution (allow capture of exception not derived from std::exception) 8. MfcTestRunner 9. QtTestRunner, a Qt based graphic test runner 10. CursesTestRunner 11. WxWidgetsTestRunner (formerly: WxWindowsTestRunner)
3.2.1 The Architecture
CppUnit contains 24 ordinary classes, 4 abstract classes, 7 template classes, and several nested classes and helper macros. Classes are divided into modules. The division of CppUnit modules is shown in Figure 4.
Figure 4: The CppUnit modules
The Writing test fixture includes TestFixture and TestCaller classes. The module Making assertions and Creating TestSuite include unsupported classes. The Executing test describes how CppUnit can run tests, this module includes User Interface (UI) namespaces containing text, MFC, and QT versions of TestRunner. Tracking test execution includes the class Listener for test progress and result that prints the name of each test before running it. The Writing test result prints an output to stream in text or XML format. The classes which record a failed Test execution represent a source line location and catch exceptions thrown by failed assertions are included in the module: Browsing collected test result. A set of functions to help writing assertion macros and an additional Message for assertions are resolved in the module Creating custom assertions. The Writing Test Plug-in describes classes for default implementation of test plug-in interface and possibility for test.
20
Just as in JUnit, CppUnit's central design element is an abstract interface called Test implemented by classes named TestCase as shown in Figure 5 and TestSuite as shown in Figure 6.
Figure 5: The key TestCase
Figure 6: The key TestSuite
TestCase represents an actual test object, and TestSuite is a composite of other Test objects. The class TestFixture represents the test fixture interface implemented by TestCase, with setUp() and tearDown( ) methods.
The object architecture for collecting test results is more complex than in JUnit. The key classes belonging to test result handling are shown in Figure 7.
21
Figure 7: Classes to collect test results
The class TestResult receives test results as Test objects are run. However, it does not store the results, but instead uses the TestListener interface to inform observers of test results. The TestListener subclass TestResultCollector stores the test results for reporting. As shown in Figure 7, these classes are derived from SynchronizedObject. This allows their operations to be mutex-protected, so that tests and listeners may execute safely in separate, concurrent threads. Output of test results is handled by classes implementing the Outputter interface, shown in Figure 8.
Figure 8: The Outputter classes for printing test results
The Outputter objects print the test results in human-readable text format, as an XML document, or in a compiler compatible format. The class TestRunner provides a convenient interface for running tests. CppUnit includes a generic text TestRunner as well as GUI versions of TestRunner for use in Qt and MFC development environments.
3.2.2 How to Use CppUnit for Testing
Assuming that we would like to test a class called Parser. The following are the general steps to use the CppUnit framework to test this class: 1. Write a class (let's call it TestParser) to test the Parser class. This class must inherit the class TestCase which is defined by the CppUnit framework. 2. Create a constructor for this class, passing a name that is representative of the set of tests for this class as the parameter.
22 3. Create a fixture. A test fixture is a set of sample objects which is reused during the testing. For example, can be created a few sample source files for the Parser to parse. CppUnit provides a setUp and a tearDown method to manage the fixture. Therefore, we can e.g. create file objects in setUp to open the source files and release these resources in the tearDown method. The important thing to note is that setUp and tearDown will be called for every test which is run. 4. Each performed test is represented by the implementation of a method in the test class. For example, when testing whether the parser extracts the tokens correctly, a method test called testGetToken can be implemented. The collection of test methods implemented forms a test suite. 5. In each created test method the assertion mechanism provided by CppUnit is used to compare the results of a running of the test and the expected results. For future testing we should verify and save the expected results. 6. Finally, use the text version of the TestRunner program to run the tests and collect the results. As each test is run, CppUnit will provide feedback on whether the test ran successfully, or the test failed, or an exception has occurred.
For better idea of the use of CppUnit for testing, it is advised to consult the CppUnit’s cookbook.
3.2.3 The Analysis
CppUnit is probably the most widely used UTF in C++. It is a good framework to compare with other Unit Tests.
1. Minimal amount of work needed to add new tests. CppUnit requires quite a bit of work for the simplest possible test.
#include
CPPUNIT_TEST_SUITE_REGISTRATION( SimplestCase );
void SimplestCase::MyTest(){ float fnum = 2.00001f; CPPUNIT_ASSERT_DOUBLES_EQUAL( fnum, 2.0f, 0.0005 );
23 }
2. Easy to modify and port. It runs under Windows or Linux. The functionality is sensibly well modularized (runners, results, outputs). The CppUnit requires the Standard Template Library (STL), Run-Time Type Information (RTTI) and exception handling. Even Stroustrup did not include RTTI in the original C++ design because he thought this mechanism are frequently misused and are created dependences. These requirements could be problematic if we want to link libraries that have no RTTI enabled, or if we do not want to use the STL. The main problem is in the dependences on another code. Regarding the exceptions so should be tested for unusual situations become when software is used. 3. Supports fixtures. Yes. The fixture contains a test class, which is created in setUp( ) and deleted in tearDown() eventually. This handling is not ideal, because it can be dynamically allocated in the setUp() function and create some type of memory leak when is forgot to clean.
#include
24 4. Handles the exceptions and the crashes. It uses the concept of “protectors” which are wrappers around tests. The default one attempts to catch all exceptions (and identify some of them). We can write own protectors and push them on the stack to combine them with the ones already there. It did not catch system exceptions under Linux, but it would have been trivial to add with a new protector. We do not think it had a way to easily turn off exception handling. 5. Good assert functionality. It has the minimum set of assert statements, including one for comparing floating-point numbers. The contents of the variables compared are printed to a stream if the assert fails, giving us as much information as possible about the failed test. 6. Supports different outputs. It has well defined functionality for “outputters“ writing on the standard output (display the results of the tests), as well as “listeners” (which get notified while the tests are happening). It supports Graphical User Interface (GNU) progress bar. 7. Supports suites. Yes. Figure 9 shows how we can run a suite for CppUnit.
Figure 9: The standard output obtained by running command make for CppUnit
Overall, CppUnit is ideal framework for Unit Testing because it is exactly what we want, and what we expected from our point of view. For creating new tests, we must write all code by hand. There is not mechanism which can be easier for creating new tests. Other than that, the main complaint is the need for RTTI or exceptions, and the relative complexity of the source code, which could make it challenging to port to different platforms.
Advantage Ideal UTF Extension for automation of Test Cases Widespread use of programs to automate
25
Disadvantage Dependences onto: RTTI, sometimes on the STL
3.3 The CppUnitLite
CppUnitLite has similar architecture as CppUnit, but much easier for little software project and it is customized to necessaries with whatever developers need. Michael Feathers, the original author of CppUnit got fed up with the complexity of CppUnit and how it did not fit somebody’s needs, so he wrote the ultra light weight framework CppUnitLite. The advantages lie in features such as complexity and size. This framework is composed of few files which are very clear to understand. There we can see how the expenditure of local effort will achieve various requirements. The main disadvantage is that CppUnitLite does not have any documentation; we can only find some web sites used for some software project or web sites for open source software for this framework.
3.3.1 The Analysis
1. Minimal amount of work needed to add new tests. Of all the UTFs, this is the one that can be ideal. It really fits idea of minimum amount of work required to set up a simple test or even one with a fixture.
#include "lib/TestHarness.h" TEST (Whatever,MyTest){ float fnum = 2.00001f; CHECK_DOUBLES_EQUAL (fnum, 2.0f); }
2. Easy to modify and port. Yes. It gets best of the class award in those UTF. No other UTF comes close to being this simple, easy to modify and port, and at the same time having reasonably well separated functionality. 3. Supports fixtures. This is point where the CppUnitLite starts running into trouble. It is so lightweight that it does not have room for many features. Unfortunately, it suffers from the problem that objects need to be created dynamically if we want them to be created right before each test. To be fair though, every single UTF in this evaluation has that requirement.
#include "lib/TestHarness.h" #include "MyTestClass.h" class MyFixtureSetup : public TestSetup{ public:
26 void setup(){ someValue = 2.0; } void teardown() {} protected: float someValue; MyTestClass myObject; };
TESTWITHSETUP (MyFixture,Test1){ CHECK_DOUBLES_EQUAL (someValue, 2.0f); someValue = 0; myObject.UseBadPointer(); myObject.ThrowException(); }
4. Handles exceptions and crashes well. The original CppUnitLite did not handle them at all, but were extended by easy way. To run the tests without exception support it requires recompiling the tests with a special define turned on, it is not at slick as the command line argument. 5. Good assert functionality. The assert macros are definitely the worst of the lot. They do not use a stream to print out the contents of their variables, so we need custom macros for each object type we want to use. It comes with support for doubles, longs, and strings, but anything else we need to add by hand. Also, it doesn’t have any checks for anything other than equality. 6. It supports different outputs. Again, the original only had one type of output, but it is trivial way how we can add more. 7. Supports suites. Probably the only framework that does not support suites. We never really needed them, but they would probably be very easy to add into source file. Below we can see the result of running a suite for CppUnit, it is illustrated in Figure 10.
27
Figure 10: The standard output obtained by running command make for CppUnitLite
CppUnitLite is not so good framework, but with a few modifications it hits the mark in all the important categories. If it had better support for assert statements, it would come very close to our ideal framework. Still, it is one usable framework of very many.
Advantage Size and with it related installation, modifiability, extension
Disadvantage The assert macros do not support a stream to print out the contents of their variables Does not support templates, suites, RTTI, exceptions and crashes handling
3.4 The CppUTest
CppUTest is a UTF based on CppUnitLite. The main goal of CppUTest is to provide a simple UTF that can also work well with embedded systems. This means it will just use a minimum set of C++ features. Lately it has been ported to Symbian OS used in Nokia S60 smart phones. The framework is xUnit compatible, extremely simple and easy to use. It can print the test results both to console and to a JUnit like xml file.
28 3.4.1 The Analysis
1. Minimal amount of work needed to add new tests. There is minimum amount of work required to set up a simple test or even one with a fixture. 2. Easy to modify and port code, because this framework is similar as CppUnitLite. 3. Supports fixtures. This is point where the CppUTest does not have problem to create right test case with setup and teardown methods.
TEST_GROUP(Utest){ TestTestingFixture* fixture; void setup(){ fixture = new TestTestingFixture(); afterCheck = false; } void teardown(){ delete fixture; } void testFailureWith(void (*method)()){ fixture->setTestFunction(method); fixture->runAllTests(); LONGS_EQUAL(1, fixture->getFailureCount()); CHECK(!afterCheck); } };
4. Handles exceptions and crashes well. The original framework handles it by easily and can be extended easily. 5. Good assert functionality. The assert functions are written for different variable type of assertions and they do not use a stream to print out the contents of their variables. It comes with support for bool (testing true), const char* (testing equality and containing), long (testing equality), void* (testing equality), double (testing equality). Also, it doesn’t have any checks for anything other than is described above. 6. It supports different outputs. Again, the original only had one type of output, but there is trivial way how we can add more. 7. Supports suites. Yes. The suite running is illustrated in Figure 11.
29
Figure 11: The standard output obtained by running command make for CppUTest
Advantage Memory leak detection Command line switches Initial code generation scripts - there are some scripts that are helpful in creating initial header files, source, and test files
Disadvantage Does not support templates Catches exception only for running one test, setup, test body, teardown
3.5 The CxxTest
CxxTest uses a simple C++ parser and code generator (requiring Perl or Python) for test registration. It has a framework for generating mocks of global functions, but not for generating mocks of objects. The code generation step is also trivial to integrate into the regular build system. The good documentation gives explicit step by step instructions on how to integrate it with make files. Other languages, such as Java, are better suited to what we want to do in a UTF because they have good reflection capabilities. In C++ we are forced to use manual registration of tests, ugly macros and other requirements. By CxxTest framework we get around that by parsing our simple tests and generating a C++ test runner which calls tests. We will get flexibility without the need for any ugly macros, libraries, or language features.
30 3.5.1 The Analysis
1. Minimal amount of work needed to add new tests. It is almost as simple as the best of them. Since we are doing processing with a Perl script, there is no reason for writing C++ main program.
class SimplestTestSuite : public CxxTest::TestSuite{ public: void testMyTest(){ float fnum = 2.00001f; TS_ASSERT_DELTA (fnum, 2.0f, 0.0001f); } };
2. Easy to modify and port. CxxTest requires the simplest set of language features and does not require any external libraries. It is distributed simply as a set of header files. 3. Supports fixtures. CxxTest support setup and teardown steps on test level, but it also supports them at the suite and at the global level.
#include "MyTestClass.h" class FixtureSuite : public CxxTest::TestSuite{ public: float someValue; std::string str; void setUp(){ someValue = 2.0; str = "Hello"; } void tearDown(){} void test1(){ TS_ASSERT_DELTA (someValue, 2.0f, 0.0001f); someValue = 13.0f; myObject.ThrowException(); } };
4. Handles exceptions and crashes well. It catches all exceptions and prints information about them formatted like any other error (no system exceptions under Linux though). We can easily rerun the tests with a command line argument to the Perl script to avoid catching exceptions and catch them in the debugger instead.
31 5. Good assert functionality. It has a whole suite of very comprehensive assert functions, including ones for exception handling, checking predicates, and arbitrary relations. It even has a way to print out warnings which can be used to differentiate between two parts of the code calling the same test, or to print some messages. 6. Supports different outputs. Different outputs are supported by passing a parameter indicating which type of output we want to the Perl processing step. Adding new output formatting is very straightforward and it is even covered in the documentation. 7. Supports suites. Yes. All tests are part of a suite. Figure 12 shows how to run a suite for CxxTest.
Figure 12: The standard output obtained by running command make for CxxTest
CxxTest supports mock objects. Anybody doing TDD knows the value of mock objects when it comes to testing the interactions between a set of objects. The CxxTest allows for us to override global functions with specific mock functions. This overriding does not help for regular classes. By Perl script we generate a test runner; it adds a main function and some global variables, so linking with other similar runners gives all sorts of problems. If we follow the examples in the documentation, we will create a single runner for all the tests. This can be problematic if we are going to be having thousands of tests, and then making one small change. Then is regenerating a test runner and compiled of all our code.
Advantage: Using Perl along the way to generate a test runner Supports mock objects
Disadvantage:
32 Using Perl along the way to regenerate a test runner or a part of test runner Does not support RTTI, exception handling, template functions.
3.6 The Conclusion of Analysis
Having gone through all four C++ UTFs, we found out that there are two frameworks that do not need to be modified for our use: CppUnit and CxxTest. The CxxTest is our favourite because it fits very closely the requirements of our ideal framework by using the special handling of an external scripting language. It is very useful framework and it has advanced features and great assert functionality. It requires the use of a scripting language as part of the building process for testing. CppUnit is a solid, complete framework. The major disadvantages are the relative complication when adding new tests and fixtures, as well as the trust on STL and some advanced language issues. If we wanted to use a simple framework, we should start with CppUnitLite and improve it to fit our needs. It has good structure and ultra-light framework with no external dependencies, so it is extremely easy to modify. Its main disadvantage is the lack of features and the primitive assert functionality - the same disadvantage that the framework CppUTest has, which is based on the CppUnitLite. Should we create our own UTF? We know that Kent Beck recommends it in his book Test- Driven Development: By Example [7], and it might be a great learning experience, but this is very time consuming. It is probably good idea to write a new framework which would be created to anybody’s needs but we would not recommend actually doing that in production code. We recommend starting with one of the two UTFs mentioned above. If anybody feels the need to create their own framework, they should use CppUnitLite and extent it. Before building a new framework, we should find out weather there is not a UTF corresponding with our requirements. We found the CppUnit as ideal framework for our requirements. It is the main reason why not to make a new framework in our case. In case we are to choose one, we can really do no wrong with one of those two frameworks. The most important thing is that we are writing Unit Tests, or in better case doing TDD. As Michael Feathers wrote, “the code without Unit Tests is legacy code, and we do not want to be writing legacy code”. I decided to use CppUnit framework for our extension because this framework is the most advanced even though it has some disadvantages as mentioned above in this chapter.
33 4 Programs for UTF
In addition to UTF for functionality testing of developing applications there are programs that are used for testing applications and can be used for testing other properties. The following list shows the additional programs that we could use to test applications:
1. Memory analyzer - program monitors work with memory at runtime. Example of programs: Purify, libumem, DTrace, Electric Fence etc. 2. Coverage programs - program detects how much of the code testing application is used for testing. It measures quantities of tested source code. Example of programs: TCOV, Pure Coverage. 3. Performance testing - program calls profiler. These programs monitor the number of calls of functions and record time spent in monitored function. With this program we are able to identify parts of the program where we spend the most time. This output can be served as a cornerstone for optimizing or changing algorithm. These programs are sometimes a part of the profiler development environments. Profiling can support a compiler or we can use a separated program. Example of programs: gprof. 4. Code analyzers (Static code analyzers) – program tests various static properties of the source code. For example, we can test whether the source code meets: 4.1. the standards for the programming language (ANSI C) 4.2. the safety - we are looking for potentially dangerous commands 4.3. the correctness - some languages allow the properties to be mathematically proved or to look for patterns of frequent errors (FindBug). 4.4. the size or complexity
We may also analyze the documentation that is part of the source code. And we can automatically generate the documentation, for example, doxygen documentation. We can also test the API stability. Stable API is the one that is not changed. Not only programs can be employed for testing developed software, we can use the User Interfaces (UIs) for CppUnit as well. The UI is the program third-party software that extends the functionality of UTF.
4.1 CppUnit Base Programs
As we described in the chapter regarding above, the CppUnit has UIs which are used for running test(s). We should know how to use them, because UIs are ideal for testing functions themselves. We are going to examine how we can use UIs.
34 4.1.1 The Text Test Runner
The test runner manages the life cycle of the added tests and it can run only one of the added tests or all the tests. TestRunner prints out a trace as the tests are executed followed by a summary at the end. The trace and summary print are optional. Here is an example of its use:
CppUnit::TextTestRunner runner; runner.addTest(ExampleTestCase::suite()); runner.run("", true); // Run all tests and wait
The trace is printed using a TextTestProgressListener. The summary is printed using a TextOutputter. We can specify an alternate Outputter at construction or later with setOutputter(). After construction, we can register additional TestListener to eventManager(), for a custom progress trace, for example.
CppUnit::TextTestRunner runner; runner.addTest(ExampleTestCase::suite()); runner.setOutputter(CppUnit::CompilerOutputter::defaultOutputter( &runner.result(), std::cerr)); MyCustomProgressTestListener progress; runner.eventManager().addListener(&progress); runner.run("", true); // Run all tests and wait
4.1.2 The MFC Test Runner
We use this to launch the MFC TestRunner. It is usually called from CWinApp subclass:
#include
runner.addTest(CppUnit::TestFactoryRegistry::getRegistry().makeTest()); runner.run(); }
We shall describe a MFC based UI dialog. If we want to use it we should open our application class implementation file and add lines to our InitInstance method. We define a runner instance, but it is not enough for testing. Then we should add all registered tests, which are registered through
35 CPPUNIT_TEST_SUITE_REGISTRATION macro call inside our <*.cpp> file. Once the tests are registered and added to runner, we can show the dialogs with run method. Now, we are ready to run our test cases. Just compile our application and run it. We will see a MFC based dialog as above. Just click on browse and we can see this dialog [11]:
Figure 13: MFC UI for CppUnit
Just select one test (green node), or select parent blue node to run all registered tests. This solution has many disadvantages when we need to run tests in the order defined by event scenario (sequence diagram). This sequence diagram is a kind of interaction diagram that shows how processes or rather classes operate with one another and in what order. It is a construct of a Message Sequence Chart. We do not need to verify the correctness of one function, but more than ones. Some times functions are repeated in the different scenarios. In the next chapter we will suggest a new solution.
4.1.3 The QT Test Runner
Here is an example of its use:
#include
void QDepWindow::runTests(){ CppUnit::QtUi::TestRunner runner;
runner.addTest(CppUnit::TestFactoryRegistry::getRegistry().makeTest()); runner.run(true); }
36
37 5 The Unit Test Environment (UTE)
As described in the chapter The Comparison of the Unit Test Frameworks, today there are many frameworks, where the advanced Unit Test features are missing, and they must be hand-coded for testing of an application. This is exactly our situation.
We have an application where mentioned UIs are insufficient for running the tests because we have advanced requirements which are not supported by any UTFs for software project development in the programming language C++. We decided to create an environment, which is based on the scripts and UTF’s interface used for interconnecting programs. The extension of one chosen framework is decided on the basis of chapters The Conclusion of Analysis of UTFs and Programs for UTF, where CppUnit was selected as well as programs for ensuring the important point of source code quality. We did not find ideal framework for our aim, since our requirements are not corresponding with none of the existing UTFs and there is not any environment as well. To be able to use programs in combination with one of the selected source code to UTF quality software, we use script language. It is a programming language that allows control over one or more programs. Scripts are distinct from the core code of the application, which is usually written in a different language, and are often created or at least modified by the developer. Bellow there are our requirements for our new environment which are not accomplished neither UTFs nor environment:
1. Bash script – allows the control over interconnection of UTF and programs 2. Perl script – allows easier manipulation of the text which is useful for UTF 3. Use the power of Regular Expressions for 3.1. selecting Test Case (Cases) 3.2. substitution of output and expected files (e.g. lines in accordance with described RE can be substituted) 4. Allow combination with another programs for debugging (e.g. gdb/dbx/mdb) 5. Run test cases separately from source code – the test cases (input text files) will have special syntax (each command will have prefix “$”), which will be read by the extension. Each command will be connected with implementation of a function (with predefined interface) that may call tested function or do some auxiliary action. 6. Allow to specify command line options for running binary file for testing 7. Allow to reuse testing functions with different parameters 8. Eliminate code recompilation when adding a new test (in most of cases)
Then we should gain these advantages with preservation of all basic characteristics of UTF:
38
1. easier way of running tests (e.g. select only a subset of tests with a RE) 2. decreased effort for a developer (e.g. when creating test with different parameters of a tested function) 3. and all other advantages mentioned above (1-8)
As we decided to expand CppUnit framework we should have knowledge of design UTF and we should define the respecting rules which we shall use to satisfy our requirements.
5.1 The Framework Design
Conditions of a framework design that are influenced by C++ standard are:
The class structure of the framework The overall responsibilities of certain classes in the framework The behaviour expected from certain member functions of classes in the framework The syntax of C++ declarations of classes in the framework
When designing a framework, we will meet many situations requiring the selection from design alternatives. This standard specifies alternatives that should be used for many situations. The benefit of having such a standard is threefold. First it raises the likelihood that a high-quality framework should be produced, because it is based on tested design alternatives. Second, developers should be able to produce a framework much faster, because design knowledge is being reused. Third, all the frameworks developed in accordance with the standard should have designs that are more consistent with each other, which is essential for building a framework repository.
Some design alternatives are imposed in a way that is very abstract. These types of alternatives are specified in design patterns. Because an applicability of patterns is often subjective we should not concentrate on it in this thesis. Other design alternatives are dictated in a very particular way, and there can be question whether an alternative should be applied to a given situation. These types of alternatives should be specified in design rules for all UTF, especially for CppUnit.
5.1.1 The Design Rules for Framework
The rule is a logical principle to be followed in software development and is based on experience. The rule is more easily understood because it is based on logic. Some rules are just conventions to improve code readability and found common coherences in interfaces. For example one of the rules can be the requirement of the one instance of the class.
39
Some design rules are really just a documentation of reasonable ways the C++ language should be used. There are many design rules that originated in various textbooks and articles, but now exist as popular styles among C++ developers. We shall use all of them. Since these rules are a part of the methodology standard, their contribution is critical to the quality of designs. We will present rules in the format of the following template. Rules have simple structure [12]:
Rule Documentation Template Intent: A short statement conveying what the rule achieves. Applicability: The situations in which the design rule must be applied. Designer responsibilities: The actions that must be taken by the designer when the situation described in the Applicability section. Rationale: The rationale, if not obvious from Intent. Example: A short source code.
Below, there are definitions of rules that should be met when framework is created or extended:
Rule 1 Intent: To minimize the likelihood that we will have to modify a base class later. Applicability: Use whenever designing a base class. Designer responsibilities: Design base classes to be “inheritance friendly”, by making data members and function “protected” instead “private”, unless we have a good reason to do otherwise. Make all “protected” member functions virtual. Make the destructor virtual.
Rule 2 Intent: To finding a consistent error by handling strategy. Applicability: Use whenever an error condition is met. Designer responsibilities: Never return error conditions from function calls. Use exception for all error reporting. Execution classes should be nested in the declaration of the class that they are thrown in. This rule only applies to failures.
Rule 3 Intent: To establish read-only access to persistent data. Applicability: Use when neither persistent singleton nor persistent data manager is sufficient. Designer responsibilities: Name each persistence object, allowing anyone to get a reference to that object by name. However, such objects should only have public member functions that are const. So, unless the “accessor” is a friend, there is read-only access.
40 Rule 4 Intent: To eliminate bugs due to dereferencing bad character-string pointers. Applicability: If possible, use when designing any function call signature that requires character strings. Designer responsibilities: Make arguments to functions string or d_String instead of const char*, to avoid core dumps due to dereferencing null. This rule applies unless we want to pass a character string as an optional argument, in which case we can use a const char* set to null to indicate that the argument is not passed.
Rule 5 Intent: To make classes easier to use when character strings are passed as arguments. Applicability: Use in all classes that have member functions with character string arguments. Designer responsibilities: Use const char* instead of char* whenever possible. Rationale: Every once in a while, we will find a class that we need to use, but it accepts char* when it is only reading the string. Therefore, we can't pass string::c_str() as an argument. Instead, we have to first dump the string into a buffer just to pass it to the function.
Rule 6 Intent: To establish a convention for returning something from a function that has been allocated on the heap. Applicability: Use whenever a function call allocates something on the heap and then returns it to the caller. Designer responsibilities: Use the following contract between callers: if non-const pointer is a return value, the caller is responsible for deleting the space pointed to when finished using the return value. Rationale: This will cut down on memory leaks and, when it becomes a commonplace idiom, it will make it easier for programmers to learn how to use new classes.
Rule 7 Intent: To improve efficiency when returning values and help establish consistency in class interfaces. Applicability: Apply whenever a member function or non transitive operator returns a value. Apply whenever a member function or no transitive operator returns a value. Designer responsibilities: When returning one small value to a caller, return it by value, using the return statement (e.g., int foo() { return l;}). To return one large value, fill in a non–const reference to an object passed in by the caller (e.g., void foo(BigType& result). This saves a copy.
Rule 8 Intent: To maintain predictability and consistency in operator semantic.
41 Applicability: Apply whenever a binary operator is overloaded. Designer responsibilities: For transitive operators, always make the value returned by the return statement a reference to the object performing the operation (e. g. return *this;).
Rule 9 Intent: To establish a convention for passing arguments by reference. Applicability: Apply whenever an argument of a non-built-in type is passed to a function. Designer Responsibilities: If a function argument is not of a built-in type, it should be passed by reference. When passing information into a function by reference, the reference must be constant. This prevents the function being called from inadvertently modifying the argument. Rationale: A non-const reference is assumed to be used to return information to the caller.
Rule 10 Intent: To keep the public part of a class interface as simple as possible. Applicability: Applies to all classes. Designer responsibilities: Never add a member function whose purpose can be satisfied with a combination of other member functions and trivial logic. Further, we should create the minimum number of member functions possible that can provide the required behaviour. If programmers really need to combine member functions them they should write an inline function. Rationale: Programmers can create classes that are too complicated, which happens as more functions are added.
Rule 11 Intent: To prevent to repeat a name and simplify names. Applicability: If our support namespace, define one for our framework. Designer responsibilities: A namespace should be assigned to every framework. The name should indicate the domain or facet that the framework addresses. Rationale: We can avoid name repeating by prefixing type of names.
Rule 12 Intent: To keep a framework portable and vendor independent on the GUI API. Applicability: This rule applies to all frameworks. Designer responsibilities: We should not build frameworks that have any dependency on a specific GUI API, it should only be accessed from within application code. Rationale: The framework GUI API should be designed independently of each other and independently of the part of framework.
42 Rule 13 Intent: To keep a framework portable and vendor independent non-standard components or programs. Applicability: This rule applies to all frameworks. Designer responsibilities: A framework should not rely on non-standard components or programs, unless we have the source code. The exception would be simple components like a regular expression or date class that we could replace without a problem, as long as there is a straightforward transformation for porting code to a new component. Rationale: It might occur when a project direction is inconsistent with the direction of the vendor’s product line.
All these rules were used in the extension of CppUnit. Some examples are in the Appendix. We presented rules in the template format for software project development in the programming language C++. However, we should define the UTE rules for Bash and Perl part.
5.1.2 The Rules for UTE
The rules for UTE are based on experience. Some rules are just conventions to improve script readability and found common coherences in interfaces. We shall use all of them. We will present rules in the same template as design rules for Framework.
Rule 1 Intent: To maintain explicitness for environment variable in the configuration file. Applicability: This rule applies to all variables. Designer responsibilities: A name of environment variable should have a prefix “UTE_” and should be divided into groups regarding the similar meaning. The variable can be exported. Rationale: The UTE’s variables should be distinguished from system variables. Example: UTE_BINARY=”test” UTE_BINARY_PATH=”$HOME/cppunit/example/test”
Rule 2 Intent: To establish internal variable in scripts. Applicability: This rule applies to all variables. Designer responsibilities: A name of environment variable should have a prefix “IUTE_” and should be divided into groups regarding the similar meaning. The variable can not be exported. Rationale: The internal UTE’s variables should be distinguished from system variables. Example: IUTE_COLOUR=0 IUTE_VERBOUSE=0
43
Rule 3 Intent: To maintain explicitness for exported environment variable in the configuration file. Applicability: This rule applies to variables which will be exported automatically. Designer responsibilities: A name of environment variables which will be exported by UTE automatically should have the prefix "EXP_UTE_", it relates to variables which we need to know in the C++ source code, Perl script or another place. Other variables are not exported. Rationale: The UTE’s variables should be distinguished from all variables. Example: EXP_UTE_ORACLE_HOME = "/app/oracle/product/10.1.0/db_1" EXP_UTE_ORACLE_LOGIN="sm001_2/sm001_2" EXP_UTE_ORACLE_SID="TEST01"
Rule 4 Intent: To establish function in the configuration file or scripts. Applicability: This rule applies to all functions. Designer responsibilities: A name of function should have a prefix “ute_”. Rationale: The UTE’s function should be distinguished from system functions. Example: ute_colour{:} ute_parse_clo{:}
We mentioned rules which should be respected when Bash, Perl scripts are made. Now we shall approach the implementation of an extension for CppUnit and create environment keeping the defined rules above.
5.2 Implementation of Extension CppUnit
Because we have specific requirements mentioned at the beginning of this chapter, we have to make a file reader for reading commands with a prefix “$” of input file (<*.in> file can define test case or scenario). The corresponding commands (e.g. $ CALL) are parsed and as well as their parameters (e.g. $STR) after being read from input file. Then the tested functions are called with those parameters. First of all we briefly describe an input file and its use by FileReader.cxx file. Then we shall describe syntax of FileReader, i.e. commands which are possible to read from input file.
The input file is a text file containing commands for FileReader. Each command is on a separate line starting with prefix $ and its own name is written in upper-case letters. The input file can describe:
which method shall be called
44 Example: $CALL test_class::test_function
what arguments shall be passed Example: $STR From: [email protected]
what return values should be returned Example: $INT 1
The input file can as well include documentation fulfilling doxygen syntax. The set of separators can be extended by developers, they should use functions fr_add_seperator(“###”) to add new separator for parsing lines of input file. The comments which could be used in documentation are several. The comment block is to use a block of at least two comment lines, where each line starts with an additional slash. Here is example of the case:
/// /// ... text ... ///
Note that a blank line ends a documentation block in this case. If we want to document the commands of FileReader and we want to put the documentation for these commands inside the compound, it is sometimes desired to place the documentation block after the command instead of before. For this purpose we have to put an additional < marker in the comment block. Note that this also works for the parameters of a function. Here is an example:
$CALL test_class::test_function ///< Brief description after the command
The FileReader reads input file line by line and parse each line. Depending on the command the members of class are filled. This simple program will be used as reader for reading each command describing the test case scenario.
Example: ///FileReader’s commands FileReader’s documentation $TEST class_name::function_name ///< the class name and their test function $STRING From: [email protected] ///< the first parametr of function_name $STRING To: [email protected] ///< the second parametr of function_name
45 FileReader supplies reading of commands and also basic and extended types of CALL command from an <*.in> file. Set of commands is fixed. Developers are able to read a type that consists of FileReader’s predefined types. Developers can use their own type of CALL, but they must extend commands in the FileReader.
Example: The extension of FileReader.cxx file case HEX: fr_read_hex(token); break;
The TC – test_hex.in file $TEST class_name::function_name ///< the class name and their test function $HEX 46726f6d3a206d69726f736c617640676d61696c2e636f6d
The basic types and commands are handled by FileReader class and are declared in FileReader.h file. There is the list of basic types handled by CReader class:
$INT number Description: An integer number An appropriate function for reading: fr_read_int()
Example of an application: int local_param_int = fr()->fr_read_int();
$BOOL true/false Description: Boolean value from input file An appropriate function for reading: fr_read_bool()
Example of an application: bool local_param_bool = pTestReader->fr_read_bool();
$STRING string Description: string, from input file read as C++ string, dummied out as an array of characters An appropriate function for reading: fr_read_string()
Example of an application: string local_param_string = fr()->fr_read_string();
46 5.3 Implementation of UTE
As mentioned above the UTE is also known as text/graphics UI for UTF which can be used for automation running. The UTE is a powerful environment allowing automated execution of TCs for UT. This environment has main purpose: software testing, which detects software failures so that defects may be uncovered and corrected in the source code. Users need to verify correctness of their own source code. The UTE for CppUnit gives developers the possibility to verify real code for some application. The UTE is written in Perl and Bash script, where Perl script is used for comparison of expected and output file and Bash script sets environment and handles automation of UTs. When users use UTE then they copy configuration files into directory where the tested class/es is/are and change one or more lines only. Changes and extensions are needed just in some parts, which differ from the default UTE configurations file, which is stored on the same place as main scripts. The results of the automation UTE can be produced every day and every night by hand or by cron, which is a time-based job scheduler. It is a Unix, Solaris utility that allows tasks to be automatically run in the background at regular intervals by the cron daemon. These tasks are often termed as cron jobs in Unix, Solaris. The results of cron verify correctness of changes in the source code. This automation script is very important for feedback after some changes in the real source codes each day and when error occurs in the some test then it can send a mail with feedback report. More details are illustrated in Figure 14.
47
Figure 14: UTE for CppUnit
The annotation of the Figure: 1. It runs CppUnit with input parameters: -i <*.in> -o <*.out> -t <*.trc> 2. It calls reader’s functions 3. It reads commands from <*.in> file 4. It fills up members of reader class 5. It returns values from reader 6. It calls writer’s functions 7. It writes values into <*.out> file before the call of tested function 8. It calls the real source code (test function) of the application 9. It writes trace into <*.trc> file 10. It writes values into <*.out> file after the call of tested function
The UTE is run by calling ute.sh with Command Line Options (CLOs). From above reasons and new requirements on the extension of UI we created new UTE for UT. The UI can be replaced by this UTE.
5.3.1 How to Run UTE
Developer can set Bash alias:
48 alias ute.sh=”$HOME/cppunit-1.12.1/ute/ute.sh” or extend Bash variable $PATH: PATH=$PATH:$HOME/cppunit-1.12.1/ute in the file ~/.bashrc.
The main script ute.sh is located in the directory $HOME/cppunit/ute, it is a place where we unpacked CppUnit to run Unit Testing. The ute.sh has to be executed from the directory where the UT is located, this directory should contain a local ute.cfg. The developer can refer to the chapter where there are more details: The Configuration File.
Example: Running main UTE script in the UT directory $ ute.sh 5.3.2 How to Specify which Test Cases to Run
We have several possibilities how to specify which TCs to test. It is described in the table: TCs defined by Description The name of directory where input files will be search dirname The name of any input TC file to run file.in The CLO "-i" defines interactive mode. When developers use it, then they can select one TC for testing from the list of TCs. -i The set of TCs. Refer to chapter The Configuration File for more details of UTE_STC_NAME definition. UTE_STC_NAME
Table 1: The definition of TC/s to run
Examples: It runs regression test, all TCs are executed: $ ute.sh
It runs only one TC from the basic directory: $ ute.sh tc/base_test.in
It runs all TC from the directory and UTE will write debug messages on the standard output (CLO: “-d”): $ ute.sh –d registration
It runs TCs which are defined in UTE ute.cfg file under tags STC_TEST and STC_CALL: $ ute.sh STC_TEST STC_CALL
49 5.3.3 The Configuration File for UTE
The rules which were mentioned for UTE should be respected when configuration files are made. Therefore, we need to configure an UTE appropriately, so there are text files <*.cfg> defining variables and function to set an environment for testing. Default configure file ute.cfg is on the same place as main UTE's script ute.sh and it loads first. This file is included in Bash script ute.sh, so it is required to have the correct syntax. When the developers would like to change the default configuration file then they should copy the default ute.cfg in the directory where are files for CppUnit's testing, then they could easily modify variables and function how they need. Unchanged variables can be commented because they are same as default. The copying and reconfiguring of this file has the influence on running tests, which will depend on the place where the developer's configuration file is stored. It causes that the UTE is directory sensitive. The loading configuration files proceed as follows; first the default configuration file (ute.cfg) is loaded. The developer's configuration file (ute.cfg) is loaded as second in the directory where the testing is run (if file exists). Should we need to redefine only certain variables or functions for these special cases testing, so we should create another configuration file containing only those values and definitions. Other <*.cfg> files can be loaded using the following command line option "- cfg":
Example: $ ute.sh –cfg db.cfg
The developer's configurable file <*.cfg> is in the same directory as CppUnit files. The table below describes all variables configurable in <*.cfg> file:
Variable Description Default value General properties for CppUnit: UTE_CPPUNIT_CLO The additional command line options for empty CppUnit for all test cases. General properties of tested binary: UTE_BINARY_FILE Tested binary, should be the same as source empty file name with main function. UTE_BINARY_VERSION Tested version, release or debug. debug UTE_BINARY_PATH The complete path, where all binaries can be $HOME/cppunit/exampl found. es/exec TC's settings: UTE_TC_BASIC_DIR Starting directory for all searches of TCs. ./
50 UTE_TC_OUT_DIR The directory where result files are written. output_files/ UTE_TC_TRC_DIR The directory where traces files are written output_files/ UTE_TC_EXP_DIR The subdirectory related to path of <*.in> expected_files/ file, where expected results can be read. UTE_TC_SUCCESSFUL The file with list of successful TCs. "$(date '+%y%m%d- Command line option to store the list of %H%M%S').OK_TCs.t TCs: "-ls". xt" UTE_TC_FAILED "$(date '+%y%m%d- The file with the list of failed TCs. %H%M%S').ERROR_T Command line option to store TCs: -lf. Cs.txt" UTE_TC_SUB The own substitution file. "ute_sub.pm" Working directories & files UTE_PURIFY_DIR Working directory for Purify program. program_logs/purify UTE_ TCOV_DIR Working directory for TCOV program. program_logs/tcov Special tags: UTE_STC_* The set of test cases (<*.in> files) or N/A directories (with <*.in> files) or parameters separated by white space and determined by name STC_*. Asterisk must be substituted by an identifier to complete the name of variable. The value of such variable will be expanded when used at command line. .Example: ute.cfg line: UTE_STC_TC="tc/" $ ute.sh UTE_STC_TC Other programs settings UTE_DEBUGGER The name of debugger. It will be used if "dbx" developer use CLO: $ ute.sh -dbx UTE_DIFF The name of diff program. It will be used if "vim -d" developer use CLO: $ ute.sh -dd
Table 2: The parameters configurable by ute.cfg
51 Functions which are contained in the configuration files are shell functions for easy extension of UTE. These functions are empty by default. Developers can override any of them in their own configuration files <*.cfg>.
UTE provides functions for developers, which are included in the ute.cfg:
1. Developers can fill up this function in the their own configuration file <*.cfg> 2. The name of functions determines for what and when is the function run
Table below describes all functions defined in <*.cfg> file:
UTE testing functions ute_testing_before() { It is called before the testing is run, just to initialize unit specific : variables or files. } ute_testing_after() { After the testing is finished, it is useful to have the possibility to clean : up. } UTE test case functions It is called before the execution of TC. This function has input parameters: $1 = input file ute_tc_before() $2 = output file { $3 = expected file : $4 = trace file } It is called after the execution of TC. The output files can be examined here. This function has input parameters: $1 = input file ute_tc_after() $2 = output file { $3 = expected file : $4 = trace file }
Table 3: The configurable functions for default ute.cfg
5.3.4 The Substitutions for UTE
Substitutions are used to eliminate differences at some lines of <*.out> and <*.exp> files e.g. time and pointer values. Each UT specific substitutions have to be defined in ute_sub.pm file. When no error
52 occurs during substitutions, no info messages will be shown on the standard output. When substitution has been unsuccessful, it is because some error occurred:
Some different lines are found The file <*.out> has a different number of lines then file <*.exp> The file <*.exp> has zero length
Then an error message is printed on the screen. If different number of lines of <*.exp> and <*.out> files is detected, an error message of a different length is shown.
The substitutions are defined as a list of substitution regular expressions (e.g.: 's/\x20$//'). The filename, package name and the name of substitution lists must stay unchanged. At the end of the file there should be value 1 for correct import. Templates of a file ute_sub.pm can be found at templates directory: $HOME/cppunit/ute/templates. The example of ute_sub.pm is presented in the Appendix.
Developers can use the different type of substitutions or skip sections: 1. permanent_substitutions - these substitutions are applied only to output file <*.out 2. substitutions - these one line substitutions are applied to output and expected file: <*.out> and <*.exp> 3. multiline_substitutions – these multi line substitutions are applied to output and expected file: <*.out> and <*.exp> 4. skip_sections – this list defines sections which should be skipped in output and expected file: <*.out> and <*.exp>
5.3.5 The Report of UTE
The purpose of this report is to examine the running of various programs. UTE Bash part has a special type of print-out which differs from others. Output is coloured for easier developer’s orientation. Each line has coloured prefix "*" determining the type of result: info (green colour), debug (blue colour), warning (yellow colour), error (red colour). The lines which are print-out on the standard output of second Perl part have coloured prefix "~" too. Different colour of this character has a different meaning, the same as the Bash prefix. The developer can use UTE print functions: ute_info, ute_debug, ute_warning, ute_error. If developers use some of these functions in their configuration file ute.cfg in the some ute_* function, then lines are written to the standard output. The switching colours is set by definition after UTE print function, we can use colours: blue, bold, cyan, grey, green, purple, red, redbg, yellow.
53
Example: function ute_testing_before() { mte_info "INFO: " red "mte_testing_before()" }
ANSI colour codes are ugly to store into logs. Colours on the output can be switched off by parameter "-nc".
54 6 Conclusion
In this work, we have focused on the testing of software project development in the programming language C++. The first type of testing is used by developers to verify the correctness of their new or changed source code. To verify the correctness of source code there are automated systems that provide various possibilities for testing. Having analysed existing UTFs, we found out that none of the most widely used UTFs have the possibilities which are demanded of frameworks. The basic problem is the fact that UT is developed in the same programming language as software project. Since we did not find an ideal framework for UT, which would satisfy our own needs, we extended one of the most suitable UTE and created a new UI for UT. Our solution separates the UT's part from the UIs which are developed in same programming languages as tested software. Therefore, we decided to extend CppUnit and create UTE for UT. We created the FileReader based on reading reader's command of input file which describes TC. The presented reader is able to read commands from input file and depending on them to call the relevant functions which we want to test. The ability of our proposed reader to read commands was proven by the use of this extension and running TCs in the new UTE. Each TC we want to use for testing must meet the syntax defined by our reader. This solution allows eventually expanding the syntax of reader for unsupported parameters of the test function. The extension of CppUnit produces the output file, whose correctness should be verified by developer. If the output file corresponds with it what developers expect, then the output file is renamed to the expected file. When the same TC is run again the output file is compared with expected file. The testing is based on this principle.
The solution has several advantages over existing UTF: The power of Regular Expressions for o selecting Test Case (Cases) o substitution of output files The combination with another programs for testing and debugging The test cases are run separately from C++ source code. Some functions with more different parameters can be tested The elimination of recompilation of creating binary file when adding a new test, which uses already defined functions
We tested our example of software project and we derived the simple report and output files for several TCs. The UTE allows running chosen TCs by hand or by automatic program: cron and reports results of UT.
55 The solution, described in this thesis, can generally be used in most existing UTFs and support programs for software project development in the programming language C++ to date which are based on Unix, Solaris operation systems. However, it is necessary to state that some other requirements are missing before total generalization can be claimed. Our extension of CppUnit does not provide complete support for all types of function parameters (some less standard types of parameters, e.g. HEX, are not supported). Concerning UTF, it does not support all the existing programs for testing and debugging, but this extension can be integrated into the UTE. The created UTE is applicable for any other UTF, although the developer would have to expand the interface between UTE and existing UTF similar to the one we developed in our case. Another possible improvement is related to multi-threads software project. In our concept of problems we assumed that all tested programs and CppUnit are thread-safe. Implementation of this issue requires a deep knowledge of multi-thread programming and intervention into UTF and is beyond the scope of this thesis.
56 7 Bibliography
[1] B. Keringham, R. Pike: The Practice of Programming, Publisher: Addison-Wesley Professional, Released: February 1999, ISBN-13: 978-0201615869 [2] Verification and Validation software, URL: http://en.wikipedia.org/wiki/Verification_and_Validation_%28software%29 November 2009 [3] Software testing, URL: http://en.wikipedia.org/wiki/Software_testing September 2009 [4] Unit testing, URL: http://en.wikipedia.org/wiki/Unit_Test, September 2009 [5] Paul Hamill: Unit Test Frameworks, Publisher: O'Reilly Media, Released: November 2004, Pages 2-4, ISBN-13: 978-0596006891 [6] List of unit testing frameworks, URL: http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C.2B.2B, September 2009 [7] Kent Beck: Test-Driven Development, Publisher: Addison Wesley, Released: November 2002, ISBN-13: 978-0321146533 [8] Noel Llopis: Exploring the C++ Unit Testing Framework Jungle, URL: http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle, November 2009 [9] Gerard Meszaro: xUnit Test Patterns: Refactoring Test Code (Hardcover), Publisher: Addison-Wesley, Released: May 2007, ISBN-13: 978-0131495050 [10] Visit project CppUnit, URL: http://sourceforge.net/apps/mediawiki/cppunit/index.php?title=Main_Page November 2009 [11] JM Navarro: Unit testing with CppUnit, URL: http://www.codeproject.com/KB/library/Using_CPPUnit.aspx November 2009 [12] Gregory F. Rogers: Framework-Based Software Development in C++, Publisher: Prentice Hall PTR, Released: January 2008, ISBN-13: 978-0135333655
57 8 Appendix
8.1 Abbreviations
ANSI American National Standards Institute API Application Programming Interface CLO Command Line Option GNU Graphical User Interface IDE Integrated Development Environment MFC Microsoft Foundation Classes RTTI Run-Time Type Information STL Standard Template Library TC Test Case TCOV Tool Coverage (cost model factor) TDD Test Driven Development TF Test Framework UI User Interface UT Unit Test UTE Unit Test Environment UTF Unit Test Framework
8.2 Glossary
Code refactoring – is the process of changing a computer program's internal structure without modifying its external functional behaviour or existing functionality, in order to improve internal quality attributes of the software. Reasons include improving code readability, to simplify code structure, to change code to adhere to a given programming paradigm, to improve maintainability, to improve performance, or to improve extensibility.
Doxygen – it is a tool for writing software reference documentation. The documentation is written within code, and is thus relatively easy to keep up to date. Doxygen can cross reference documentation and code, so that the reader of a document can easily refer to the actual code. Doxygen is a documentation generator for some programming, script, description languages.
58 Framework – is a class library that captures patterns of iteration between objects. A framework consists of a suite of concrete and abstract classes, explicitly designed to be used together. Applications are developed from a framework by completion of the implementations of the abstract classes. A framework can also include additional utilities to aid in the completion of end-user applications. A utility can be a code generator or algorithm, for example.
GUI – a Graphical User Interface is a type of user interface item that allows people to interact with programs in more ways than typing such as computers. A GUI offers graphical icons, and visual indicators, as opposed to text-based interfaces, typed command labels or text navigation to fully represent the information and actions available to a user. The actions are usually performed through direct manipulation of the graphical elements.
IDE – Integrated Development Environment also known as Integrated Design Environment or Integrated Debugging Environment is a software application that provides comprehensive facilities to computer programmers for software development. An IDE normally consists of: a source code editor a compiler and/or an interpreter build automation tools a debugger
Integration testing – (sometimes called Integration and Testing, abbreviated IT) is the activity of software testing in which individual software modules are combined and tested as a group. It occurs after Unit Testing and before system testing. Integration testing takes as its input modules that have been unit tested, groups them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing.
Qt - is a multiplatform C++ GUI toolkit. It provides application developers with all the functionality needed to build applications with state-of-the-art graphical user interfaces. Qt is fully object-oriented, easily extensible, and allows true component programming.
Software engineering – is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software.
Scenario testing – it is a software testing activity that uses scenario tests, or simply scenarios, which are based on a hypothetical story to help a person think through a complex problem or system for a testing environment. The ideal scenario has five key characteristics: it is (a) a story that is (b)
59 motivating, (c) credible, (d) complex, and (e) easy to evaluate. These tests are usually different from TCs in that TCs are single steps where as scenarios cover a number of steps. Test suites and scenarios can be used in concert for complete system testing.
Test Case (TC) – in software engineering is a set of conditions or variables under which a tester will determine whether an application or software system is working correctly or not. The mechanism for determining whether a software program or system has passed or failed such a test is known as a test oracle. In some settings, an oracle could be a requirement or use case, while in others it could be a heuristic. It may take many test cases to determine that a software program or system is functioning correctly. Test cases are often referred to as test scripts, particularly when written. Written test cases are usually collected into test suites. xUnit – the generic name for any Test Automation Framework for Unit Testing that is patterned on JUnit or SUnit. The xUnit Test Framework for most languages can be found at: http://xprogramming.com or http://en.wikipedia.org/wiki/XUnit Another place to look for both Unit Test and customer test tools is http://www.opensourcetesting.org. xUnit Family Members – this (incomplete) list of members of the xUnit family of Test Automation Frameworks is included here to illustrate the diversity of the family and the extent to which automated Unit Testing is supported in various programming languages. This appendix also includes comments about specific capabilities of some members of the family.
8.3 Source code
FileReader.cxx – Rule 11 namespace ute { class CFileReader:public CStringTokenizer { private: static FileReader* m_instance; … public: static FileReader* fr_instance(); … };
inline FileReader* fr() { return CFileReader::fr_instance(); } }
60