4 December, 2008 printed in Germany printed The Magazine for Professional Testers Pantone 295 Pantone 279 free exemplar 100% 80% Test60% 40% Automation20% 100% - 80% 60% 40% 20% cmyk cmyk Does it make sense? c:100 c:80 c:60 c:40 c:20 c:64 c:55 c:41 c:28 c:14 m:56 m:45 m:39 m:22 m:12 m:34 m:27 m:20 m:14 m:7 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 b:34 b:28 b:20 b:14 b:7 b:0 b:0 b:0 b:0 b:0 www.testingexperience.com ISSN 1866-5705 ISSN 1866-5705 © iStockphoto Improved time to market through automated software testing by Dr Mike Bartley I have experienced many business advantages files in an automated through the automation of software testing. build system. The • Reduced costs as automation reduces the challenge is then to manual effort and thus the cost of test- prioritize the issues ing. and get them fixed. • Improved quality through both regular In [1] VDC predict running of the automated tests and hav- that the market for ing more time for running the remaining static analysis tools manual tests. will overtake that • Improving employee motivation by re- for dynamic tools for ducing the amount of tedious manual embedded software testing. testing in 2009. A test team should always keep in mind their contribution to the business, and in this article Static testing is ex- I will concentrate on improving time to mar- cellent at improving ket through test automation. Automation can the overall quality of make the following key contributions to time your code although to market improvements. its contribution to • Getting test results sooner: The results accelerating our time such as virus checkers running), and we found from an automated test suite should be to market is not direct. Also, it is not sufficient it easier to do the setup manually before run- available sooner than the manual equiva- as it does not test that the software implements ning a series of automated performance tests. lent. the required functionality. For that most com- The level of testing: I usually find it easier to • Catching bugs earlier: The earlier a bug panies turn to dynamic testing . In [2] Kaner automate unit testing than system level testing, © iStockphoto is found, the quicker (and cheaper) it is to states that “automated testing tools” only ac- because the unit test environment is less com- fix. The ability to regularly run an auto- tually automate some of the tasks required in plex. There are also a number of good testing mated suite should ensure some bugs are dynamic testing: running tests and reporting tools (many open-source) to aid unit level test found sooner. results. The test engineer will initially need automation. User acceptance testing is often • Stabilising the software sooner: The to design, create and document the tests, add not a target for automation, because this is of- regular execution of an automated regres- them into the automation framework and get ten a formal signoff procedure with the cus- sion test suite should allow us to stabilise the tests running for the first time. There are tomer. However, the ability to automate the the software sooner. tools that can automate some of these tasks types of tests that users want to run is helpful • Assessing the stability of the software: (e.g. automatic test data generation based on to ensure that user acceptance testing is likely One of the key questions is always “are source code analysis) or help with these tasks. to be successful. we ready to ship?” and the test manager For example, JUnit provides an excellent en- How often will the test need to be run? If a needs to be able to provide data to help vironment for writing new tests and tools for test is going to be run very infrequently then with that decision. The automated test automating their execution. Finally, the auto- the ROI (return on investment) from automa- suite should provide a history of failure mated tests will also need to be maintained, tion may not be so obvious. rates and can also be a means to generate and it is here that attempts at test automation Ease of automating the pass/fail criteria: structural coverage information. often falter. For example, early attempts at It is not always easy to automate detecting However, although the business benefits are automating GUI testing produced tests which whether a test passes or fails, and the effort potentially large, automated testing needs to were too brittle and failed after minor changes taken to set this up means the ROI on automat- be approached with extreme caution due to the in the interface. Such tools are now more ro- ing the test does not stack up. high risk of failure! bust and GUI testing is now an obvious choice Test stability and repeatability: For example, Automated testing tools fall into two main for test automation. if the code under test is not stable, then the lack types: static and dynamic. Static analysis tools of stability and repeatability of the test means have made huge improvements since the early One of the key decisions in accelerating time that the maintenance costs of the tests are too days of “lint” and can now find a range of com- to market through test automation is deciding high to consider automation of these tests. plex issues: from memory leaks to race condi- which tests to automate and which to leave tions; from naming conventions violations to “manual3”. I have found the following guide- When choosing a test automation tool, your uninitialised variables; from “island” code to lines useful over the years: crucial consideration should be the target user dead-end. The tools occasionally report false Complexity of the test environment: The community. A number of tools market them- errors, but these are reduced through improved more complex the test environment, the more selves on their ease of use and simple script- analysis engines and improved “context” by maintenance the automated tests will require. ing language, assuming that the users are not reading in larger chunks of source code. Static For example, one recent project I was involved familiar with software coding. Other tools can test tools are reasonably easy to automate, in required us to carefully measure perfor- link with standard programming languages, because it usually just involves reading the mance improvements gained through use of thus allowing common libraries of test code to source code into the tool. I recently automat- the product being tested. This required the test be developed and re-used. In one automation ed static analysis across the complete source environment to be set up very carefully (e.g. project I observed that the developers were code base by making a few changes to “make” to ensure the machine had no background jobs writing unit tests for automation, but found 1 It depends on what tests are automated as to what sort of bugs are found. It is unlikely that every possible bug will be found via an automated test suite. 2 There are formal mathematics-based static techniques for testing correct function, but these are not widely used. 3 Even “manual” tests can benefit from some level of automation. For example, scripts to set up the environment before running through a few manual steps. www.testingexperience.com The Magazine for Professional Testers 49 Pantone 295 Pantone 279 100% 80% 60% 40% 20% 100% 80% 60% 40% 20% cmyk cmyk c:100 c:80 c:60 c:40 c:20 c:64 c:55 c:41 c:28 c:14 m:56 m:45 m:39 m:22 m:12 m:34 m:27 m:20 m:14 m:7 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 b:34 b:28 b:20 b:14 b:7 b:0 b:0 b:0 b:0 b:0 the simple scripting language that came with you might write additional tests that wouldn’t ing and outsourcing of customer- focused test- the tool extremely frustrating. Although the have been viable in a manual environment. Fi- ing. automation ultimately helped time to market, nally, what value do you put on improved time This article has demonstrated both the advan- it could have made a much greater contribu- to market and improved quality? tages and issues surrounding the automation tion if the tool had better matched the skills of software testing. It has attempted to provide of the user community. A few other important Finally, let me finish with a case study of a practical solutions to those issues and demon- considerations in tool choice are: project I recently worked on. The software un- strated how the automated testing can improve • The ability to separate orthogonal con- der test was a mixture of IDE (an Integrated time to market thus allowing the test team to cerns such as test setup, test environ- Development Environment which included a demonstrate a significant business advantage ment, test data and test procedure. One compiler for an extended C language), code test framework I worked on allowed us running on a host server (under various favors References to use files and environment variables to of Linux and Windows) driving an external 1. “Static Analysis Demand to Drive the define the test environment, test start-up peripheral and a number of host applications Market for Test Automation Tools”, Ven- and tear-down, test data and expected using the peripheral for support functions.