sign in sign up
<<
Home , Software testing

4 December, 2008 printed in Germany printed The Magazine for Professional Testers

Pantone 295 Pantone 279 free exemplar 100% 80% Test60% 40% Automation20% 100% - 80% 60% 40% 20% cmyk cmyk Does it make sense?

:100 c:80 c:60 c:40 c:20 c:64 c:55 c:41 c:28 c:14 m:56 m:45 m:39 m:22 m:12 m:34 m:27 m:20 m:14 m:7 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 b:34 b:28 b:20 b:14 b:7 b:0 b:0 b:0 b:0 b:0 www.testingexperience.com ISSN 1866-5705 ISSN 1866-5705

© iStockphoto Improved time to market through automated testing

by Dr Mike Bartley

I have experienced many business advantages files in an automated through the automation of software testing. build system. The • Reduced costs as automation reduces the challenge is then to manual effort and thus the cost of test- prioritize the issues ing. and get them fixed. • Improved quality through both regular In [1] VDC predict running of the automated tests and hav- that the market for ing more time for running the remaining static analysis tools manual tests. will overtake that • Improving employee motivation by re- for dynamic tools for ducing the amount of tedious manual embedded software testing. testing in 2009. A test team should always keep in mind their contribution to the business, and in this article Static testing is ex- I will concentrate on improving time to mar- cellent at improving ket through . Automation can the overall quality of make the following key contributions to time your code although to market improvements. its contribution to • Getting test results sooner: The results accelerating our time such as virus checkers running), and we found from an automated test suite should be to market is not direct. Also, it is not sufficient it easier to do the setup manually before run- available sooner than the manual equiva- as it does not test that the software implements ning a series of automated performance tests. lent. the required functionality. For that most com- The level of testing: I usually find it easier to

• Catching bugs earlier: The earlier a bug panies turn to . In [2] Kaner automate than system level testing, © iStockphoto is found, the quicker (and cheaper) it is to states that “automated testing tools” only ac- because the unit test environment is less com- fix. The ability to regularly run an auto- tually automate some of the tasks required in plex. There are also a number of good testing mated suite should ensure some bugs are dynamic testing: running tests and reporting tools (many open-source) to aid unit level test found sooner. results. The test engineer will initially need automation. User is often • Stabilising the software sooner: The to design, create and document the tests, add not a target for automation, because this is of- regular execution of an automated regres- them into the automation framework and get ten a formal signoff procedure with the cus- sion test suite should allow us to stabilise the tests running for the first time. There are tomer. However, the ability to automate the the software sooner. tools that can automate some of these tasks types of tests that users want to run is helpful • Assessing the stability of the software: (e.g. automatic test data generation based on to ensure that user acceptance testing is likely One of the key questions is always “are analysis) or help with these tasks. to be successful. we ready to ship?” and the test manager For example, JUnit provides an excellent en- How often will the test need to be run? If a needs to be able to provide data to help vironment for writing new tests and tools for test is going to be run very infrequently then with that decision. The automated test automating their execution. Finally, the auto- the ROI (return on investment) from automa- suite should provide a history of failure mated tests will also need to be maintained, tion may not be so obvious. rates and can also be a means to generate and it is here that attempts at test automation Ease of automating the pass/fail criteria: structural coverage information. often falter. For example, early attempts at It is not always easy to automate detecting However, although the business benefits are automating GUI testing produced tests which whether a test passes or fails, and the effort potentially large, automated testing needs to were too brittle and failed after minor changes taken to set this up means the ROI on automat- be approached with extreme caution due to the in the interface. Such tools are now more ro- ing the test does not stack up. high risk of failure! bust and GUI testing is now an obvious choice Test stability and repeatability: For example, Automated testing tools fall into two main for test automation. if the code under test is not stable, then the lack types: static and dynamic. Static analysis tools of stability and repeatability of the test means have made huge improvements since the early One of the key decisions in accelerating time that the maintenance costs of the tests are too days of “lint” and can now find a range of com- to market through test automation is deciding high to consider automation of these tests. plex issues: from memory leaks to race condi- which tests to automate and which to leave tions; from naming conventions violations to “manual3”. I have found the following guide- When choosing a test automation tool, your uninitialised variables; from “island” code to lines useful over the years: crucial consideration should be the target user dead-end. The tools occasionally report false Complexity of the test environment: The community. A number of tools market them- errors, but these are reduced through improved more complex the test environment, the more selves on their ease of use and simple script- analysis engines and improved “context” by maintenance the automated tests will require. ing language, assuming that the users are not reading in larger chunks of source code. Static For example, one recent project I was involved familiar with software coding. Other tools can test tools are reasonably easy to automate, in required us to carefully measure perfor- link with standard programming languages, because it usually just involves reading the mance improvements gained through use of thus allowing common libraries of test code to source code into the tool. I recently automat- the product being tested. This required the test be developed and re-used. In one automation ed static analysis across the complete source environment to be set up very carefully (e.g. project I observed that the developers were code base by making a few changes to “make” to ensure the machine had no background jobs writing unit tests for automation, but found

1 It depends on what tests are automated as to what sort of bugs are found. It is unlikely that every possible bug will be found via an automated test suite. 2 There are formal mathematics-based static techniques for testing correct function, but these are not widely used. 3 Even “manual” tests can benefit from some level of automation. For example, scripts to set up the environment before running through a few manual steps. www.testingexperience.com The Magazine for Professional Testers 49

Pantone 295 Pantone 279

100% 80% 60% 40% 20% 100% 80% 60% 40% 20%

cmyk cmyk

c:100 c:80 c:60 c:40 c:20 c:64 c:55 c:41 c:28 c:14 m:56 m:45 m:39 m:22 m:12 m:34 m:27 m:20 m:14 m:7 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 b:34 b:28 b:20 b:14 b:7 b:0 b:0 b:0 b:0 b:0 the simple scripting language that came with you might write additional tests that wouldn’t ing and of customer- focused test- the tool extremely frustrating. Although the have been viable in a manual environment. Fi- ing. automation ultimately helped time to market, nally, what value do you put on improved time This article has demonstrated both the advan- it could have made a much greater contribu- to market and improved quality? tages and issues surrounding the automation tion if the tool had better matched the skills of software testing. It has attempted to provide of the user community. A few other important Finally, let me finish with a case study of a practical solutions to those issues and demon- considerations in tool choice are: project I recently worked on. The software un- strated how the automated testing can improve • The ability to separate orthogonal con- der test was a mixture of IDE (an Integrated time to market thus allowing the test team to cerns such as test setup, test environ- Development Environment which included a demonstrate a significant business advantage ment, test data and test procedure. One for an extended C language), code test framework I worked on allowed us running on a host server (under various favors References to use files and environment variables to of and Windows) driving an external 1. “Static Analysis Demand to Drive the define the test environment, test start-up and a number of host applications Market for Test Automation Tools”, Ven- and tear-down, test data and expected using the peripheral for support functions. ture Development Corporation, 2008 output. This easily facilitated reuse in the • The automated test environment was (see http://www.vdcresearch.com/_docu- test environment. integrated with the code configuration ments/pressrelease/press-attachment- • Portability of test between environments/ management and build system. We were 1413.pdf) platforms. For example, you may need also able to structure the tests in line with 2. “Architectures of Test Automation”, Cem to test your software on both Linux and the software so that as the software was Kaner, 2000 (see http://www.kaner.com/ Windows, with a wide variety of periph- built we only tested the changed code. pdfs/testarch.pdf) erals. This significantly reduced the test execu- • The ease of adding tests to the test pool tion time and the test result analysis time and temporarily banning tests (for ex- and allowed us to perform a continuous ample whilst a particular piece of source overnight build and test cycle. Thus we code is updated). were able to track the stability of the soft- • Ability to integrate the automation with ware, which not only meant improved the software configuration management time to market but enabled better esti- and build system. mates of the time required to stabilise • The cost of the tool is obviously a consid- the software after significant changes and eration, but I would caution against rating enhancements. this factor too high. The cost of owner- • The automation framework was devel- ship of the tool will usually outweigh the oped in-house by the same team that had initial purchase cost. developed the in-house software version My last two automation projects have both control and build system. been home-grown using scripts and make • We mainly automated the unit level tests files. This gave us several advantages in terms and of the driver and of flexibility in our test generation, but we host application code with did observe a major drawback: employing across the range of platforms. external test development resource (through • We outsourced the execution of the re- either outsourcing or contractors) meant that maining tests. These tended to be higher delivery of the tests was harder as they were level customer focused tests, which we required to understand our bespoke test system found harder to automate. and we obviously couldn’t find suppliers who • We also outsourced some of the auto- had prior experience. mated tests, but did find it was harder to import the tests into our automation en- Once you have your tool, you will need to vironment as the outsource organisation maintain the tool (as well as the automated did not have knowledge of our test auto- Biography tests). In my experience it is not always advis- mation tool. able to give this task to the test team, as they • We integrated a static analysis tool into Mike graduated from Bristol University do not always have the appropriate knowledge the build system. This found a number of with a PhD in Mathematical Logic. Since and skills. So, for example, if you have bought potential issues which we fixed over time then he has studied with the Open in a tool, you should then consider allowing and were able to ensure they stayed fixed University obtaining an MSc in Software the IT department to maintain it. If you are by regular running of the tool. Engineering and an MBA. He has been building your own tool, then it might be best • Test maintenance was difficult, and we involved in both software testing and for the team to develop found that we lost knowledge regarding hardware verification for about 20 years. and maintain it. You will also need to integrate the older tests as people moved around He started in software testing specialis- the tool into your development process. I have or left the organisation. This can present ing in , before moving to found that integrating the automation into the significant problems when the test fails. verify hardware at ST Microelectronics build process is the most valuable approach. This is not the case with manual (or semi- and Infineon. Most recently, as Test and This has meant that whenever we built the automated) tests, as somebody is always Verification Manager at Elixent (now software we got an automated set of test re- responsible for maintaining and running Panasonic) and ClearSpeed in Bristol, he sults too. the tests. However, ownership and han- has been responsible for the sign off of dover of automated tests was not so vis- complex hardware/software products. You will inevitably be asked to measure the ible and well managed. Mike has had numerous papers pub- ROI for the tool. The cost of buying the tool The automation used in the above project sig- lished, presented at a number of confer- and licensing it is easy to measure. It is also nificantly improved time to market of the soft- ences, and has written on software possible to measure the cost-saving in terms ware: we were able to shorten release sched- testing for the Open University. He has of time saved through automatically running ules and hit the release dates. The quality of been involved in a number of successful tests vs. running them manually. However, you the software (as measured by defects in the outsourcing relationships. have to add in the cost of maintaining the tests. field) was also significantly improved mainly Also, given that you now have automation, through a combination of the automated test-

52 The Magazine for Professional Testers www.testingexperience.com

Pantone 295 Pantone 279

100% 80% 60% 40% 20% 100% 80% 60% 40% 20%

cmyk cmyk

c:100 c:80 c:60 c:40 c:20 c:64 c:55 c:41 c:28 c:14 m:56 m:45 m:39 m:22 m:12 m:34 m:27 m:20 m:14 m:7 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 b:34 b:28 b:20 b:14 b:7 b:0 b:0 b:0 b:0 b:0