Techniques to Maintain Effectiveness of Regression Test Suite

Techniques to maintain effectiveness of Regression Test Suite

Krishna H S ([email protected]) July 21, 2003 Regression Test Suite Maintenance Techniques

TABLE OF CONTENTS 1. Introduction...... 3 2. Need to determine the effectiveness of Regression Test Suites...... 3 3. Factors that diminish the effectiveness of Regression Test Suites...... 3 4. Techniques to maintain effectiveness of Regression Test Suites...... 4 5. Summary...... 7 6. References...... 7

- 2 - Regression Test Suite Maintenance Techniques

1. Introduction

A recent survey points out that up to 70% of work in software is in maintenance and enhancement of the existing software. Software products and systems maintenance take a major chunk of the above work. In all these kind of projects, as the features get added incrementally, manual testing of the entire product, specially carrying out the regression tests becomes unfeasible. This brings in a need for the project management to automate the regression tests. Automating the regression tests also helps to achieve greater test coverage on a regular basis. Re-testing becomes easier. Automation of regression tests also allows human test resources to focus at newer and more complex product functionality.

While automation provides the ability to quickly determine the impact of new changes on the existing code base and defect identification becomes that much easier, the projects might lose out test coverage and face other problems, if they do not properly maintain the automated regression test suites.

This paper discusses some of the common problems faced in maintenance of the automated regression test suites and attempts to provide workable solutions to the ones discussed. The focus is limited to regression tests suites although most of the issues discussed here is applicable to projects that have a full-fledged automated test suite.

2. Need to determine the effectiveness of Regression Test Suites

In most projects, regression tests contribute to a very small percentage of defect finds. However, if a project were to have an automated regression test suite, the costs of developing and maintaining the suite are high. So, it is vital for the stakeholders to evaluate the value add provided by the test suite on a continuous/regular basis and take necessary and appropriate steps to ensure that the test suites achieve the required objectives.

Taking corrective measures to plug all loopholes of the effectiveness of Regression Test Suites on a regular basis will help reduce defect slippage to customers and increase the quality of the product/deliverables.

3. Factors that diminish the effectiveness of Regression Test Suites

There are several factors that diminish the effectiveness of the regression test suites.

- 3 - Regression Test Suite Maintenance Techniques

The most important among them is when enhancements to the test suite (addition of tests to the test suites) get out of sync with respect to the enhancements to the product. When features get added to the product as part of a Release, there might not be an equal amount of tests added to adequately test the new functionality implemented in the Release. This in effect reduces the functionality coverage achieved by the test suite.

Another factor that affects the effectiveness of the test suites is when tests get added to the suite with a short-term perspective. One typical scenario is the case where testers/developers add tests to quickly test a particular functionality or a small part of the feature that is being developed for the current Release or to test a least significant part of a feature or module, mostly as part of their unit testing efforts. Later, these tests, if not removed from the test suite, become a liability because of the improper and insufficient scenarios they test.

A related factor affecting the test suite is the presence of redundant tests. There are many scenarios by which a test suite can end up with redundant tests or with two sets of tests that validate almost the same or similar functionality. Testers might add new tests to validate a feature, even when the same could be achieved by modifying some of the existing tests/framework. There might be a case where a feature gets implemented across Releases. When this happens, the corresponding tests also get staggered across Releases. It may so happen that the regression test suite would then contain of many light tests, each testing trivial parts of the feature, making the test suite bulky.

In most cases, as the product goes on adding new features, the regression test suite gets heavy due to the tests that get added to it over a period of time. This will pose new problems. The entire test set might take a longer time now to complete all tests. This will mean that defect identification takes more time, and so also the defect verification after a fix. For example, if it takes a week for the testers to complete all the automated tests for a Release, then it is worth spending time to analyze new ways and means of lessening the time taken for execution and analysis of these tests.

4. Techniques to maintain effectiveness of Regression Test Suites

The factors discussed in the previous section bring out the need to continuously monitor the effectiveness of the test suite and also to take necessary and appropriate steps to maintain the test suite. This section discusses some of the techniques to counter the above factors. It also discusses some initiatives that aid in increasing the efficiency of the regression test suites.

A strategy to maintain adequate functionality coverage of the product in the regression test suites across Releases is to have a good tracking mechanism between

- 4 - Regression Test Suite Maintenance Techniques the feature that gets developed and the tests that get added to validate the feature. This is a continuous activity that needs to be pursued.

A good tracking mechanism ensures that test checklists are created for every feature/enhancement that is planned for the Release. Managers will use this document as a tool to control and direct the testing activity for the feature. After the development team freezes with the specifications of the feature, the Test Lead/testing team will need to develop a test checklist for that feature. Any changes to the scope of the features after this period will affect both the developer and the testing team. When the feature is alpha ready, or is signed off for testing, the testing team needs to be ready with the test case document (a document that contains all test cases, which completely cover the scope of testing for the feature.) After a fair bit of testing and bug fixes on the feature (Beta Phase), the Test Lead gets a good insight on the tests that need to be present in the regression test suite for this particular feature. Tests can be suitably added/modified/deleted based on this assessment. This approach will ensure that the feature gets adequate test coverage in the regression test suite. By the time of final feature signoff to the testing team, both the developers and the testers will have had considerable grip of the feature being developed, so the testers can identify on what tests will make it to the regression test suite. This way, there will be controlled addition of tests per feature to the test suite. So, the functionality coverage of the test harness remains intact. The Test Manager can also use the above test checklists to validate the current test coverage for a feature that is being developed in the Release.

There will be scenarios where a major or complex feature/functionality will be implemented to the product across several Releases. These cases will have an effect on the regression test suites as well. The tests in this case, need to be developed on a Release-to-Release basis and added in tandem with the functionality that is getting incorporated. There is also a need to periodically review the effectiveness of the tests added on a feature basis, after its release to customers. There are cases where customers report lot of defects on a particular feature. On such occasions, the testing team needs to add applicable test cases to the test suite, clearly because it aids in maintaining the feature test coverage.

The effectiveness of the regression test suite can be maintained by monitoring the changes to the test suite. A clearly defined and strictly followed Process enforces/ensures or takes care of the fact that only tests that add value to the overall testing strategy get added to the test suite, thus keeping the efficiency and usability of the test harness at a high degree. This will effectively address the case where developers/testers add small or insignificant tests to the test suite. A review/walkthrough from a senior test developer /Test Lead for every change will ensure that no redundant tests get added to the test suite. This process of Inspection and review helps inexperienced testers/developers to achieve better knowledge about the framework of the test suite. It will also enhance their software engineering skills. This Process also helps in achieving a disciplined addition of tests, which adhere to the existing framework. Additionally, the developers and testers can be encouraged to

- 5 - Regression Test Suite Maintenance Techniques use configuration management tools to perform unit testing/exploratory testing. This will ensure that the regression test suite remains fairly clean.

There are other common problems that the regression test suites face over a period of time. As the product gets mature with more features, the regression test suite also gets rich with respect to the tests. This will create one undesirable side effect on the test suites. In many cases, tests execution starts taking a lot more time than expected. This will mean that more time will now be spent on defect identification and also on validation of a defect fix. There might also be cases where time taken for a simple fix will be far lesser than the time taken to regression test the fix. This clearly defeats the purpose of automation. Under such conditions, there are a variety of measures one can take up to get the test suite to be more effective. One can consider optimization of tests, categorize tests as necessary and run only a subset of tests that are required for a particular Release. Tests may be categorized based on the frequency of use, importance of the feature/functionality they test, or by any other means as deemed fit from the scope of the Release. There are many Regression Test Selection strategies that are in practice in the industry. A detailed discussion of those techniques is out of scope of this paper. Sometimes, there might be a particular focus area for a particular Release. For example, a Release might concentrate on enhancing the GUI of the product, then the tests for that release would be more GUI testing. Some Releases might not require the testers to run the entire suite. They might as well be fine with running a selected part of the test suite.

Another approach to maintain effectiveness of a feature rich regression test suite is to consider periodic cleanup of old tests. For this activity, all the existing tests in the test suite needs to be analyzed for their effectiveness under the current scenario, and tests that have got ineffective due to age need to be retired from the test suite. There will be cases where certain features will be retired/will not be supported due to a different product direction. In such cases, the corresponding tests in the regression test suites also need to be eased out. This will ensure that the regression test suite remains robust for a long period of time.

It is also prudent to assess the impact on the regression test suite if there are radical changes planned in the product feature list. A different product direction in most cases will call for a change in the framework of the regression test suites. In these cases, effective planning will ensure that there is minimal development work associated with enhancing the regression test suite and that the ability of the regression test suite to determine regressions remains intact.

It might also be useful to measure the effectiveness of the regression tests suites on a Release-to-Release basis, so that one can get to know the root cause for decrease in effectiveness of the test harness if any and take corrective action on the same. Collection of some metrics and their subsequent analysis will provide good visibility on the effectiveness of the regression test suite. Metrics such as what percentage of defects were found by the regression tests suite, how important they were (in terms of

- 6 - Regression Test Suite Maintenance Techniques business priority), did the regression tests catch some regressions, or did they miss many and so on will provide a good insight on the effectiveness of the test suite.

5. Summary

The following factors are the common causes for diminishing effectiveness of the regression test suites: a. Decrease in functionality coverage b. Tests that get added to the suite with a short-term perspective c. Presence of redundant tests in the test suites d. Decrease in test execution time due to presence of many tests

Some techniques to counter the above factors are: a. To implement a good tracking mechanism between the feature and its corresponding tests b. To monitor addition of tests to the tests suite c. Consider optimization of tests when the tests get bulky d. Evaluate different Regression Test Selection strategies to avoid time and effort overrun for a Release e. Consider periodic cleanup of tests f. Plan and explore changes in the regression test suite framework if there are major changes in the product focus g. Use metrics to evaluate the effectiveness of the test suite

6. References

1. Improving the Maintainability of Automated Test Suites [1] - Paper presented by Dr. Cem Kaner at Quality Week ‘97