Automated Testing Carmine Vassallo
@ccvassallo Recommended Book
Gerard Meszaros, xUnit Test Patterns: Refactoring Test Code (Martin Fowler Signature Book) Addison-Wesley, 2007
2 Software Bugs
“A software bug is an error, flaw, failure or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways.”
https://en.wikipedia.org/wiki/Software_bug 3 How to detect bugs? Testing
• Testing is the activity of finding out whether a piece of code (a method, class, or program) produces the intended behaviour.
• We test software because we are sure it has bugs in it!
• The feedback provided by testing is very valuable, but if it comes so late in the development cycle, its value is greatly diminished.
• We need to continuously check for defects in our code.
4 The LEAN process
http://thecommunitymanager.com/2012/08/01/the-lean-community/ 5 Automated SW Testing
• Automated Software Testing consists of using special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes with predicted outcomes.
• Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or perform additional testing that would be difficult to do manually.
• Test automation is critical for continuous delivery and continuous testing.
6 Goals of Automated Testing (1/2)
• Tests should help us improve quality.
• If we are doing test-driven development or test-first development, the tests give us a way to capture what the SUT should be doing before we start building it.
• The very act of thinking through various scenarios in enough detail to turn them into tests helps us identify those areas where the requirements are ambiguous or self-contradictory, improving the quality of the specification, and consequently the quality of the software so specified.
• Tests should help us understand the software under test (SUT).
• If we want to know how the system does something, we can turn on the debugger, run the test, and single- step through the code to see how it works. In this sense, the automated tests act as a form of documentation for the SUT.
• Tests should reduce (and not introduce) risk.
7 Goals of Automated Testing (2/2)
• Tests should be easy to run.
• They must be Fully Automated Tests so they can be run without any effort.
• They must be Self-Checking Tests so they can detect and report any errors without manual inspection.
• They must be Repeatable Tests so they can be run multiple times with the same result.
• Ideally, each test should be an Independent Test that can be run by itself.
• Tests should be easy to write and maintain.
• Tests should require minimal maintenance as the system evolves around them.
8 Economics of Automated Testing
9 Types of Tests
10 Automated Unit Testing
11 Automated Unit Testing
• Unit tests verify the behavior of a single class or method that is a consequence of a design decision.
• This behavior is typically not directly related to the requirements except when a key chunk of business logic is encapsulated within the class or method in question.
• These tests are written by developers for their own use; they help developers describe what “done looks like” by summarizing the behavior of the unit in the form of tests.
12 xUnit
• xUnit is the collective name for several unit testing frameworks that derive their structure and functionality from Smalltalk's SUnit.
• Following its introduction in Smalltalk the framework was ported to Java by Kent Beck and Erich Gamma and gained wide popularity, eventually gaining ground in the majority of programming languages in current use.
• The names of many of these frameworks are a variation on "SUnit", usually replacing the "S" with the first letter (or letters) in the name of their intended language ("JUnit" for Java, "RUnit" for R etc.). These frameworks and their common architecture are collectively known as "xUnit".
13 xUnit: Key Concepts (1/2)
• Test Runner
• An executable program that runs tests implemented using an xUnit framework and reports the test results.
• Test Case
• Is the most elemental class. All unit tests are inherited from here.
• Text fixtures
• A test fixture (also known as a test context) is the set of preconditions or state needed to run a test. The developer should set up a known good state before the tests, and return to the original state after the tests.
14 xUnit: Key Concepts (2/2)
• Test suites
• A test suite is a set of tests that all share the same fixture. The order of the tests shouldn't matter.
• Test result formatter
• A test runner produces results in one or more output formats. In addition to a plain, human-readable format, there is often a test result formatter that produces XML output.
• Assertion
• An assertion is a function or macro that verifies the behavior (or the state) of the unit under test. Usually an assertion expresses a logical condition that is true for results expected in a correctly running system under test (SUT). Failure of an assertion typically throws an exception, aborting the execution of the current test.
15 xUnit: The steps
1. Setup: Set up the test fixture (the “before” picture) that is required for the SUT to exhibit the expected behavior as well as anything you need to put in place to be Setup able to observe the actual outcome. Exercise SUT Fixture 2. Exercise: We interact with the SUT. Verify Teardown 3. Verify: We do whatever is necessary to determine whether the expected outcome has been obtained.
4. Teardown: We tear down the test fixture to put the world back into the state in which we found it.
16 xUnit: Test Execution setup(); /* First, we should prepare our 'world' to make an isolated environment for testing */
...
/* Body of test - Here we make all the tests */
... teardown(); /* At the end, whether we succeed or fail, we should clean up our 'world' to not disturb other tests or code */
17 xUnit: Interaction Points
• A test interacts with the SUT through one or more interfaces or interaction points. From the test’s point of view, these interfaces can act as either control points or observation points.
Front Door Back Door 18 xUnit: DOC
• For the most part we assumed that the SUT was designed such that it could be tested easily in isolation of other pieces of software.
• When a class does not depend on any other classes, testing it is relatively straightforward.
• When a class does depend on other classes, called depended-on components (DOCs), we have two choices:
• We can test it together with all the DOCs;
• we can try to isolate it from the DOCs so that we can test it by itself.
19 Why don’t use directly DOCs?
• The depended-on component (DOC) may return values or throw exceptions that affect the behavior of the SUT, but it may prove difficult or impossible to cause certain cases to occur.
• The indirect inputs received from the DOC may be unpredictable (such as the system clock or calendar).
• the DOC may not be available in the test environment or may not even exist.
• In other cases, we need to verify that certain side effects of executing the SUT have, indeed, occurred.
• How can we test dependent classes in these circumstances?
20 Indirect Inputs
21 Indirect Outputs
22 Testing with Doubles
• A Test Double is any object or component that we install in place of the real component for the express purpose of running a test.
• There are 4 types:
• A Dummy Object is a placeholder object that is passed to the SUT as an argument (or an attribute of an argument) but is never actually used.
• A Test Stub is an object that replaces a real component on which the SUT depends so that the test can control the indirect inputs of the SUT. A Test Spy, which is a more capable version of a Test Stub, can be used to verify the indirect outputs of the SUT by giving the test a way to inspect them after exercising the SUT.
• A Mock Object is an object that replaces a real component on which the SUT depends so that the test can verify its indirect outputs.
• A Fake Object (or just “Fake” for short) is an object that replaces the functionality of the real DOC with an alternative imple- mentation of the same functionality.
23 Test Stub
24 Test Stub
• A Test Stub is an object that acts as a control point to deliver indirect inputs to the SUT when the Test Stub’s methods are called.
• Its use allows us to exercise Untested Code paths in the SUT that might otherwise be impossible to traverse during testing.
• A Responder is a basic Test Stub that is used to inject valid and invalid indirect inputs into the SUT via normal returns from method calls.
• A Saboteur is a special Test Stub that raises exceptions or errors to inject abnormal indirect inputs into the SUT.
25 Test Spy
26 Test Spy
• A Test Spy is an object that can act as an observation point for the indirect outputs of the SUT.
• To the capabilities of a Test Stub, it adds the ability to quietly record all calls made to its methods by the SUT.
27 Mock Object
28 Mock Object
• A Mock Object is also an object that can act as an observation point for the indirect outputs of the SUT.
• It differs from a Test Spy, however, in that the Mock Object compares actual calls received with the previously defined expectations using assertions and fails the test on behalf of the Test Method.
• As a consequence, we can reuse the logic employed to verify the indirect outputs of the SUT across all tests that use the same Mock Object:
• A strict Mock Object fails the test if the correct calls are received in a different order than was specified;
• A lenient Mock Object tolerates out-of-order calls. Some lenient Mock Objects tolerate or even ignore unexpected calls or missed calls. That is, the Mock Object may verify only those actual calls that correspond to expected ones.
29 Design For Testability (DFT)
• When we approach writing automated unit tests, the main difficulty we face is the need to isolate the tested parts in the system from the rest of it.
• In order to test a functionality of a class, we first need to detach it from the rest of system in which it is designed to work. In most cases, it will not be simple.
• Solution:
• DFT: Architectural and design decisions in order to enable us to easily and effectively test our system.
30 DFT Techniques
• Dependency Injection
• Dependency Lookup
• Humble Object
• Test Hook
31 Dependency Injection
32 Dependency Injection
• We need to provide a means to substitute a depended-on component (DOC) to make it easy to use a Test Double.
• Static binding—that is, specifying exact types or classes at compile time—severely limits our options regarding how the software is configured as it runs.
• Dynamic binding creates much more flexible software by deferring the decision of exactly which type or class to use until runtime.
• Exploit the dynamic binding to install a Test Double.
33 Dependency Injection
• Parameter Injection: We pass the dependency directly to the SUT as we invoke it.
• Constructor Injection: We tell the SUT which DOC to use when we construct it.
• Setter Injection: We tell the SUT about the DOC sometime between when we construct it and when we exercise it.
34 Test Coverage
35 Test Coverage
• Test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs.
• A program with high test coverage, measured as a percentage, has had more of its source code executed during testing, which suggests it has a lower chance of containing undetected software bugs compared to a program with low test coverage.
• There are different types of Test Coverage.
36 Test Coverage
• To measure the test coverage we need the availability of source code.
• Let’s focus on testing of a procedure/function
• A method can be tested similarly...
• First of all, we need a model!
37 The Model: Control Flow Graph (CFG)
• Directed graph.
• Nodes are blocks of sequential statements.
• Edges are transfers of control.
• Edges may be labeled with predicate representing the condition of control transfer.
• Several conventions for flow graphs models with subtle differences.
38 Control Flow Graph (CFG)
39 Basics of CFG
40 Coverage Criteria
• We have now our model M
• The CFG
• We should establish some coverage criteria Ci, expressed in terms of elements of the model, the CFG.
41 Statement Coverage
• Statement coverage: faults cannot be discovered if the parts containing them are not executed.
• Equivalent to covering all nodes in CFG.
• Executing a statement is a weak guarantee of correctness, but easy to achieve.
• Statement coverage may lead to incompleteness.
42 Branch Coverage
• Use the program structure, the control flow graph (CFG).
• Branch coverage criterion: Select a test set T such that, by executing P for each d in T, each edge of P’s control flow graph is traversed at least once.
• Exercise all conditions that govern control flow of the program with true and false values.
43 Condition Coverage
• Further strengthen edge coverage.
• Condition Coverage Criterion: Select a test set T such that, by executing P for each element in T, each edge of P’s control flow graph is traversed, and all possible values of the constituents of compound conditions are exercised at least once.
• Compound conditions: C1 and C2 or C3 ... where Ci’s are relational expressions or boolean variables (atomic conditions).
• Thus we need test cases where
• C1 should be FALSE at least once and true at least once
• Same for C2 and C3
44 Path Coverage
• Path Coverage Criterion: Select a test T such that, by executing P for each d in T, all paths leading from the initial to the final node of P’s control flow graph are traversed.
• In practice, however, the number of paths is too large, if not infinite.
• Some paths are infeasible.
• It is key to determine “critical paths”.
45 Path Coverage Criteria
• Each path needs to be exercised.
• I problem: The number of paths is, often, infinite.
• II problem: infeasible paths.
• Solution: Select a finite number of executable paths.
• CFG-based method.
• Data-flow methods.
46 CFG-based methods
• The set (or a finite set) of CFG paths is partitioned in a finite number of equivalence classes.
• Coverage criteria: a set of test cases to ensure that a patch for each class is exercised at least once.
• Linear independent paths method (McCabe).
47 Linear independent path method (McCabe)
• A path is said to be independent if it introduces at least a new set of statements or a new condition. In a CFG a path is independent if it crosses at least an edge not yet crossed.
• The set of all the linear independent path in a program constitute the set of base paths. All other paths are generated from a linear combination of them.
• Given a program, the set of base paths is not unique.
48 Linear independent path method (McCabe)
• The number of linear independent paths is equal to program’s McCabe cyclomatic complexity.
• V(G)= E - N+ 2 E: edges in G - N: nodes in G
• V(G) = P + 1 P: predicates in G
• V(G) = closed regions G + 1
• Test cases exercising base path guarantee that each statement is executed at least once.
49 Linear independent path method (McCabe)
1) V(G) = E - N + 2 = 14 - 11 + 2 = 5 2) V(G) = P + 1 = 4 + 1 = 5 3) V(G) = N. of closed regions + 1 = 4 +1 = 5
50 Measuring Code Coverage
• One advantage of structural criteria is that their coverage can be measured automatically.
• To control testing progress.
• To assess testing completeness in terms of remaining faults and reliability.
• To help fix targets for testers.
• High coverage is not a guarantee of fault-free software, just an element of information to increase our confidence —> statistical models.
51 Marick's Recommendations
• Brian Marick recommends the following approach:
• Generate functional tests from requirements and design to try every function;
• Check the structural coverage after the functional tests are all verified to be successful;
• Where the structural coverage is imperfect, generate functional tests (not structural) that induce the additional coverage.
• This works because form (structure) should follow function!
• Uncovered code must have some purpose, and that purpose has not been invoked, so some function is untested.
52 Code Coverage
• If every path through a program is tested, does that guarantee finding all bugs? NO.
• If not, why not? How could you go through every possible combination of program flow and not find the problem if one exists? Because even if you test all possible paths, you still haven't tested them with all possible values or all possible combinations of values. def Add(x as Int32, y as Int32) as Int32: return x + y
Test.Assert(Add(2, 2) == 4) //100% test coverage Add(MAXINT, 5) //Throws an exception, despite 100% test coverage
53 It is now two decades since it was pointed out that program testing may convincingly demonstrate the presence of bugs, but can never demonstrate their absence. After quoting this well-publicized remark devoutly, the software engineer returns to the order of the day and continues to refine his testing strategies, just like the alchemist of yore, who continued to refine his chrysocosmic purifications.
E. W. Dijkstra (Emphasis added. Written in 1988. It's been considerably more than 2 decades now.)
54 Automated Testing in action!
55 JUnit
• JUnit is a unit testing framework for the Java programming language.
• It is one of a family of unit testing frameworks which is collectively known as xUnit that originated with SUnit.
• https://junit.org/junit4/
56 Mockito
• Mockito is a mocking framework for unit tests in Java. (https://github.com/ mockito/mockito/wiki).
• It has been designed to be intuitive to use when the test needs Test Doubles.
• Programatic creation of mocks via mock() or spy()
• Programmatic stubbing via
• Mockito.when(mock.action()).thenReturn(true)
• Programmatic verification via
• Mockito.verify(mock).action()
57 Jacoco
• JaCoCo is a free code coverage library for Java.
• Coverage analysis of instructions (C0), branches (C1), lines, methods, types and cyclomatic complexity.
• Based on Java byte code and therefore works also without source files.
• http://www.jacoco.org/jacoco/
58 Tutorial Session
59 References
• Gerard Meszaros, xUnit Test Patterns: Refactoring Test Code, Addison-Wesley, 2007
• https://semaphoreci.com/community/tutorials/stubbing-and- mocking-with-mockito-2-and-junit
• https://github.com/in28minutes/JUnitIn28Minutes
• http://www.baeldung.com/jacoco
60