Automated Testing Carmine Vassallo [email protected] @ccvassallo Recommended Book Gerard Meszaros, xUnit Test Patterns: Refactoring Test Code (Martin Fowler Signature Book) Addison-Wesley, 2007 2 Software Bugs “A software bug is an error, flaw, failure or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways.” https://en.wikipedia.org/wiki/Software_bug 3 How to detect bugs? Testing • Testing is the activity of finding out whether a piece of code (a method, class, or program) produces the intended behaviour. • We test software because we are sure it has bugs in it! • The feedback provided by testing is very valuable, but if it comes so late in the development cycle, its value is greatly diminished. • We need to continuously check for defects in our code. 4 The LEAN process http://thecommunitymanager.com/2012/08/01/the-lean-community/ 5 Automated SW Testing • Automated Software Testing consists of using special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes with predicted outcomes. • Test automation can automate some repetitive but necessary tasks in a formalized testing process already in place, or perform additional testing that would be difficult to do manually. • Test automation is critical for continuous delivery and continuous testing. 6 Goals of Automated Testing (1/2) • Tests should help us improve quality. • If we are doing test-driven development or test-first development, the tests give us a way to capture what the SUT should be doing before we start building it. • The very act of thinking through various scenarios in enough detail to turn them into tests helps us identify those areas where the requirements are ambiguous or self-contradictory, improving the quality of the specification, and consequently the quality of the software so specified. • Tests should help us understand the software under test (SUT). • If we want to know how the system does something, we can turn on the debugger, run the test, and single- step through the code to see how it works. In this sense, the automated tests act as a form of documentation for the SUT. • Tests should reduce (and not introduce) risk. 7 Goals of Automated Testing (2/2) • Tests should be easy to run. • They must be Fully Automated Tests so they can be run without any effort. • They must be Self-Checking Tests so they can detect and report any errors without manual inspection. • They must be Repeatable Tests so they can be run multiple times with the same result. • Ideally, each test should be an Independent Test that can be run by itself. • Tests should be easy to write and maintain. • Tests should require minimal maintenance as the system evolves around them. 8 Economics of Automated Testing 9 Types of Tests 10 Automated Unit Testing 11 Automated Unit Testing • Unit tests verify the behavior of a single class or method that is a consequence of a design decision. • This behavior is typically not directly related to the requirements except when a key chunk of business logic is encapsulated within the class or method in question. • These tests are written by developers for their own use; they help developers describe what “done looks like” by summarizing the behavior of the unit in the form of tests. 12 xUnit • xUnit is the collective name for several unit testing frameworks that derive their structure and functionality from Smalltalk's SUnit. • Following its introduction in Smalltalk the framework was ported to Java by Kent Beck and Erich Gamma and gained wide popularity, eventually gaining ground in the majority of programming languages in current use. • The names of many of these frameworks are a variation on "SUnit", usually replacing the "S" with the first letter (or letters) in the name of their intended language ("JUnit" for Java, "RUnit" for R etc.). These frameworks and their common architecture are collectively known as "xUnit". 13 xUnit: Key Concepts (1/2) • Test Runner • An executable program that runs tests implemented using an xUnit framework and reports the test results. • Test Case • Is the most elemental class. All unit tests are inherited from here. • Text fixtures • A test fixture (also known as a test context) is the set of preconditions or state needed to run a test. The developer should set up a known good state before the tests, and return to the original state after the tests. 14 xUnit: Key Concepts (2/2) • Test suites • A test suite is a set of tests that all share the same fixture. The order of the tests shouldn't matter. • Test result formatter • A test runner produces results in one or more output formats. In addition to a plain, human-readable format, there is often a test result formatter that produces XML output. • Assertion • An assertion is a function or macro that verifies the behavior (or the state) of the unit under test. Usually an assertion expresses a logical condition that is true for results expected in a correctly running system under test (SUT). Failure of an assertion typically throws an exception, aborting the execution of the current test. 15 xUnit: The steps 1. Setup: Set up the test fixture (the “before” picture) that is required for the SUT to exhibit the expected behavior as well as anything you need to put in place to be Setup able to observe the actual outcome. Exercise SUT Fixture 2. Exercise: We interact with the SUT. Verify Teardown 3. Verify: We do whatever is necessary to determine whether the expected outcome has been obtained. 4. Teardown: We tear down the test fixture to put the world back into the state in which we found it. 16 xUnit: Test Execution setup(); /* First, we should prepare our 'world' to make an isolated environment for testing */ ... /* Body of test - Here we make all the tests */ ... teardown(); /* At the end, whether we succeed or fail, we should clean up our 'world' to not disturb other tests or code */ 17 xUnit: Interaction Points • A test interacts with the SUT through one or more interfaces or interaction points. From the test’s point of view, these interfaces can act as either control points or observation points. Front Door Back Door 18 xUnit: DOC • For the most part we assumed that the SUT was designed such that it could be tested easily in isolation of other pieces of software. • When a class does not depend on any other classes, testing it is relatively straightforward. • When a class does depend on other classes, called depended-on components (DOCs), we have two choices: • We can test it together with all the DOCs; • we can try to isolate it from the DOCs so that we can test it by itself. 19 Why don’t use directly DOCs? • The depended-on component (DOC) may return values or throw exceptions that affect the behavior of the SUT, but it may prove difficult or impossible to cause certain cases to occur. • The indirect inputs received from the DOC may be unpredictable (such as the system clock or calendar). • the DOC may not be available in the test environment or may not even exist. • In other cases, we need to verify that certain side effects of executing the SUT have, indeed, occurred. • How can we test dependent classes in these circumstances? 20 Indirect Inputs 21 Indirect Outputs 22 Testing with Doubles • A Test Double is any object or component that we install in place of the real component for the express purpose of running a test. • There are 4 types: • A Dummy Object is a placeholder object that is passed to the SUT as an argument (or an attribute of an argument) but is never actually used. • A Test Stub is an object that replaces a real component on which the SUT depends so that the test can control the indirect inputs of the SUT. A Test Spy, which is a more capable version of a Test Stub, can be used to verify the indirect outputs of the SUT by giving the test a way to inspect them after exercising the SUT. • A Mock Object is an object that replaces a real component on which the SUT depends so that the test can verify its indirect outputs. • A Fake Object (or just “Fake” for short) is an object that replaces the functionality of the real DOC with an alternative imple- mentation of the same functionality. 23 Test Stub 24 Test Stub • A Test Stub is an object that acts as a control point to deliver indirect inputs to the SUT when the Test Stub’s methods are called. • Its use allows us to exercise Untested Code paths in the SUT that might otherwise be impossible to traverse during testing. • A Responder is a basic Test Stub that is used to inject valid and invalid indirect inputs into the SUT via normal returns from method calls. • A Saboteur is a special Test Stub that raises exceptions or errors to inject abnormal indirect inputs into the SUT. 25 Test Spy 26 Test Spy • A Test Spy is an object that can act as an observation point for the indirect outputs of the SUT. • To the capabilities of a Test Stub, it adds the ability to quietly record all calls made to its methods by the SUT. 27 Mock Object 28 Mock Object • A Mock Object is also an object that can act as an observation point for the indirect outputs of the SUT. • It differs from a Test Spy, however, in that the Mock Object compares actual calls received with the previously defined expectations using assertions and fails the test on behalf of the Test Method.