DOCSLIB.ORG

Test Automation

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Test Automation 4 December, 2008 printed in Germany printed The Magazine for Professional Testers Pantone 295 Pantone 279 free exemplar 100% 80% Test60% 40% Automation20% 100% - 80% 60% 40% 20% cmyk cmyk Does it make sense? c:100 c:80 c:60 c:40 c:20 c:64 c:55 c:41 c:28 c:14 m:56 m:45 m:39 m:22 m:12 m:34 m:27 m:20 m:14 m:7 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 b:34 b:28 b:20 b:14 b:7 b:0 b:0 b:0 b:0 b:0 www.testingexperience.com ISSN 1866-5705 ISSN 1866-5705 © iStockphoto Improved time to market through automated software testing by Dr Mike Bartley I have experienced many business advantages files in an automated through the automation of software testing. build system. The • Reduced costs as automation reduces the challenge is then to manual effort and thus the cost of test- prioritize the issues ing. and get them fixed. • Improved quality through both regular In [1] VDC predict running of the automated tests and hav- that the market for ing more time for running the remaining static analysis tools manual tests. will overtake that • Improving employee motivation by re- for dynamic tools for ducing the amount of tedious manual embedded software testing. testing in 2009. A test team should always keep in mind their contribution to the business, and in this article Static testing is ex- I will concentrate on improving time to mar- cellent at improving ket through test automation. Automation can the overall quality of make the following key contributions to time your code although to market improvements. its contribution to • Getting test results sooner: The results accelerating our time such as virus checkers running), and we found from an automated test suite should be to market is not direct. Also, it is not sufficient it easier to do the setup manually before run- available sooner than the manual equiva- as it does not test that the software implements ning a series of automated performance tests. lent. the required functionality. For that most com- The level of testing: I usually find it easier to • Catching bugs earlier: The earlier a bug panies turn to dynamic testing . In [2] Kaner automate unit testing than system level testing, © iStockphoto is found, the quicker (and cheaper) it is to states that “automated testing tools” only ac- because the unit test environment is less com- fix. The ability to regularly run an auto- tually automate some of the tasks required in plex. There are also a number of good testing mated suite should ensure some bugs are dynamic testing: running tests and reporting tools (many open-source) to aid unit level test found sooner. results. The test engineer will initially need automation. User acceptance testing is often • Stabilising the software sooner: The to design, create and document the tests, add not a target for automation, because this is of- regular execution of an automated regres- them into the automation framework and get ten a formal signoff procedure with the cus- sion test suite should allow us to stabilise the tests running for the first time. There are tomer. However, the ability to automate the the software sooner. tools that can automate some of these tasks types of tests that users want to run is helpful • Assessing the stability of the software: (e.g. automatic test data generation based on to ensure that user acceptance testing is likely One of the key questions is always “are source code analysis) or help with these tasks. to be successful. we ready to ship?” and the test manager For example, JUnit provides an excellent en- How often will the test need to be run? If a needs to be able to provide data to help vironment for writing new tests and tools for test is going to be run very infrequently then with that decision. The automated test automating their execution. Finally, the auto- the ROI (return on investment) from automa- suite should provide a history of failure mated tests will also need to be maintained, tion may not be so obvious. rates and can also be a means to generate and it is here that attempts at test automation Ease of automating the pass/fail criteria: structural coverage information. often falter. For example, early attempts at It is not always easy to automate detecting However, although the business benefits are automating GUI testing produced tests which whether a test passes or fails, and the effort potentially large, automated testing needs to were too brittle and failed after minor changes taken to set this up means the ROI on automat- be approached with extreme caution due to the in the interface. Such tools are now more ro- ing the test does not stack up. high risk of failure! bust and GUI testing is now an obvious choice Test stability and repeatability: For example, Automated testing tools fall into two main for test automation. if the code under test is not stable, then the lack types: static and dynamic. Static analysis tools of stability and repeatability of the test means have made huge improvements since the early One of the key decisions in accelerating time that the maintenance costs of the tests are too days of “lint” and can now find a range of com- to market through test automation is deciding high to consider automation of these tests. plex issues: from memory leaks to race condi- which tests to automate and which to leave tions; from naming conventions violations to “manual3”. I have found the following guide- When choosing a test automation tool, your uninitialised variables; from “island” code to lines useful over the years: crucial consideration should be the target user dead-end. The tools occasionally report false Complexity of the test environment: The community. A number of tools market them- errors, but these are reduced through improved more complex the test environment, the more selves on their ease of use and simple script- analysis engines and improved “context” by maintenance the automated tests will require. ing language, assuming that the users are not reading in larger chunks of source code. Static For example, one recent project I was involved familiar with software coding. Other tools can test tools are reasonably easy to automate, in required us to carefully measure perfor- link with standard programming languages, because it usually just involves reading the mance improvements gained through use of thus allowing common libraries of test code to source code into the tool. I recently automat- the product being tested. This required the test be developed and re-used. In one automation ed static analysis across the complete source environment to be set up very carefully (e.g. project I observed that the developers were code base by making a few changes to “make” to ensure the machine had no background jobs writing unit tests for automation, but found 1 It depends on what tests are automated as to what sort of bugs are found. It is unlikely that every possible bug will be found via an automated test suite. 2 There are formal mathematics-based static techniques for testing correct function, but these are not widely used. 3 Even “manual” tests can benefit from some level of automation. For example, scripts to set up the environment before running through a few manual steps. www.testingexperience.com The Magazine for Professional Testers 49 Pantone 295 Pantone 279 100% 80% 60% 40% 20% 100% 80% 60% 40% 20% cmyk cmyk c:100 c:80 c:60 c:40 c:20 c:64 c:55 c:41 c:28 c:14 m:56 m:45 m:39 m:22 m:12 m:34 m:27 m:20 m:14 m:7 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 y:0 b:34 b:28 b:20 b:14 b:7 b:0 b:0 b:0 b:0 b:0 the simple scripting language that came with you might write additional tests that wouldn’t ing and outsourcing of customer- focused test- the tool extremely frustrating. Although the have been viable in a manual environment. Fi- ing. automation ultimately helped time to market, nally, what value do you put on improved time This article has demonstrated both the advan- it could have made a much greater contribu- to market and improved quality? tages and issues surrounding the automation tion if the tool had better matched the skills of software testing. It has attempted to provide of the user community. A few other important Finally, let me finish with a case study of a practical solutions to those issues and demon- considerations in tool choice are: project I recently worked on. The software un- strated how the automated testing can improve • The ability to separate orthogonal con- der test was a mixture of IDE (an Integrated time to market thus allowing the test team to cerns such as test setup, test environ- Development Environment which included a demonstrate a significant business advantage ment, test data and test procedure. One compiler for an extended C language), code test framework I worked on allowed us running on a host server (under various favors References to use files and environment variables to of Linux and Windows) driving an external 1. “Static Analysis Demand to Drive the define the test environment, test start-up peripheral and a number of host applications Market for Test Automation Tools”, Ven- and tear-down, test data and expected using the peripheral for support functions.
Load more

Recommended publications
  • Testing E-Commerce Systems: a Practical Guide - Wing Lam
    Testing E-Commerce Systems: A Practical Guide - Wing Lam As e-customers (whether business or consumer), we are unlikely to have confidence in a Web site that suffers frequent downtime, hangs in the middle of a transaction, or has a poor sense of usability. Testing, therefore, has a crucial role in the overall development process. Given unlimited time and resources, you could test a system to exhaustion. However, most projects operate within fixed budgets and time scales, so project managers need a systematic and cost- effective approach to testing that maximizes test confidence. This article provides a quick and practical introduction to testing medium- to large-scale transactional e-commerce systems based on project experiences developing tailored solutions for B2C Web retailing and B2B procurement. Typical of most e-commerce systems, the application architecture includes front-end content delivery and management systems, and back-end transaction processing and legacy integration. Aimed primarily at project and test managers, this article explains how to · establish a systematic test process, and · test e-commerce systems. The Test Process You would normally expect to spend between 25 to 40 percent of total project effort on testing and validation activities. Most seasoned project managers would agree that test planning needs to be carried out early in the project lifecycle. This will ensure the time needed for test preparation (establishing a test environment, finding test personnel, writing test scripts, and so on) before any testing can actually start. The different kinds of testing (unit, system, functional, black-box, and others) are well documented in the literature.
    [Show full text]
  • Interpreting Reliability and Availability Requirements for Network-Centric Systems What MCOTEA Does
    Marine Corps Operational Test and Evaluation Activity Interpreting Reliability and Availability Requirements for Network-Centric Systems What MCOTEA Does Planning Testing Reporting Expeditionary, C4ISR & Plan Naval, and IT/Business Amphibious Systems Systems Test Evaluation Plans Evaluation Reports Assessment Plans Assessment Reports Test Plans Test Data Reports Observation Plans Observation Reports Combat Ground Service Combat Support Initial Operational Test Systems Systems Follow-on Operational Test Multi-service Test Quick Reaction Test Test Observations 2 Purpose • To engage test community in a discussion about methods in testing and evaluating RAM for software-intensive systems 3 Software-intensive systems – U.S. military one of the largest users of information technology and software in the world [1] – Dependence on these types of systems is increasing – Software failures have had disastrous consequences Therefore, software must be highly reliable and available to support mission success 4 Interpreting Requirements Excerpts from capabilities documents for software intensive systems: Availability Reliability “The system is capable of achieving “Average duration of 716 hours a threshold operational without experiencing an availability of 95% with an operational mission fault” objective of 98%” “Mission duration of 24 hours” “Operationally Available in its intended operating environment “Completion of its mission in its with at least a 0.90 probability” intended operating environment with at least a 0.90 probability” 5 Defining Reliability & Availability What do we mean reliability and availability for software intensive systems? – One consideration: unlike traditional hardware systems, a highly reliable and maintainable system will not necessarily be highly available - Highly recoverable systems can be less available - A system that restarts quickly after failures can be highly available, but not necessarily reliable - Risk in inflating availability and underestimating reliability if traditional equations are used.
    [Show full text]
  • QA and Testing Have Become More of a Concurrent Process
    Special Feature: SW Testing “QA and testing have become more of a concurrent process rather than an end of lifecycle process” Manish Tandon, VP & Head– Independent Validation and Testing Solutions, Infosys Interview by: Anil Chopra, PCQuest n an exclusive interview with the PCQuest Editor, Manish talks become a software tester? about the global trends in software testing, skillsets required Software testing has evolved into a very special function and Ito enter the field, how technology has evolved in this area to anybody wanting to enter the domain requires two special provide higher ROI, and much more. An IIT and IIM graduate, skillsets. One is deep functional or domain skills to understand Manish is responsible for formulating and executing the business the functionality. Plus, you need specialized technical skills strategy for Independent Validation and Testing Solutions practice as well because you want to do automation, middleware, and at Infosys. He mentors the unit specifically in meeting business performance testing. It’s a unique combination, and we focus goals and targets. In addition, he manages critical relationships very strongly on both these dimensions. A good software tester with client executives, industry analysts, deal consultants, and should always have an eye for detail. So people who have a slightly anchors the training and development of key personnel. Provided critical eye and are willing to dig deeper always make much better here are his expert comments on the subject. testers than people who want to fly at 30k feet. Q> What are some of the global trends in the software Q> You mentioned that software testing is now required testing business? How do you see the market moving? for front-end applications.
    [Show full text]
  • Usage of Prototyping in Software Testing
    Multi-Knowledge Electronic Comprehensive Journal For Education And Science Publications (MECSJ) ISSUE (14), Nov (2018) www.mescj.com USAGE OF PROTOTYPING IN SOFTWARE TESTING 1st Khansaa Azeez Obayes Al-Husseini Babylon Technical Insitute , Al-Furat Al-Awsat Technical University,51015 Babylon,Iraq. 1st [email protected] 2nd Ali Hamzah Obaid Babylon Technical Insitute , Al-Furat Al-Awsat Technical University,51015 Babylon,Iraq. 2nd [email protected] Abstract: Prototyping process is an important part of software development. This article describes usage of prototyping using Question – and – Answer memory and visual prototype diesign to realize Prototyping software development model. It also includes review of different models of software lifecycle with comparison them with Prototyping model. Key word: Question – and – Answer , Prototype, Software Development, RAD model. 1. Introduction One of the most important parts of software development is project design. Software project designing as a process of project creation can be divided in two large parts (very conditional): design of the functionality and design of user interface. To design the functionality, tools such as UML and IDEF0 are used, which have already become industry standards for software development. In the design of the graphical user interface there are no established standards, there are separate recommendations, techniques, design features, traditions, operating conditions for software, etc. At the same time, an important, but not always properly performed, part of this process is prototyping, i.e. the creation of a prototype or prototype of a future system. Prototypes can be different: paper, presentation, imitation, etc., up to exact correspondence to the future program. Most of the modern integrated development environments for software (IDE) allows to create something similar to prototypes, but it is connected with specific knowledge of IDE and programming language.
    [Show full text]
  • Lecturer: Ting Wang (王挺)
    Lecturer: Ting Wang (王挺) 利物浦大学计算机博士 清华大学计算机博士后 电子信息技术高级工程师 上海外国语大学网络与新媒体副教授 浙江清华长三角研究院海纳认知与智能研究中心主任 New Media Product DevelopmentDesign and Dr. Ting WANG School of Journalism and Communication Haina Cognition and Intelligence Research Center Shanghai International Studies University Yangtze Delta Region Institute of Tsinghua University, Zhejiang An overview to Part 01 software testing Testing in the software development Project Manager (Requirement Analyzer) System Analysis Release Structure Designer System Analysis Quality Assurance Engineer Requirement Analyzer Structure Design Testing Structure Design Programmer Structure Designer V-model Coding Coding Programmer Waterfall model Testing Quality Assurance Engineer Release Project Manager What is software testing? Software Testing Definition according to ANSI/IEEE 1059 standard -- A process of analyzing a software item to detect the differences between existing and required conditions (i.e., defects) and to evaluate the features of the software item. https://www.softwaretestingmaterial.com/software-testing/ “Testing shows the presence, not the absence of bugs.” —Edsger W. Dijkstra Edsger W. Dijkstra (May 11, 1930 - Aug 06, 2002) was a Dutch systems scientist, programmer, software engineer, science essayist, and pioneer in computing science. Bug Fault (also called “defect” or “bug”) is an erroneous hardware or software element of a system that can cause the system to fail. Reasons why software has bugs human mistakes in software design and coding. The longer a software bug exists throughout the product life-cycle, the more it costs. In 2002, software bugs cost the United States economy approximately $59.5 billion. In 2016, that number jumped to $1.1 trillion. https://dzone.com/articles/api-testing-best-practices 20 reasons for software bugs Reasons in development Reasons in testing #1) Miscommunication or No Communication #11) Not having a proper test setup (test environment) for testing all requirements.
    [Show full text]
  • Automated Testing of Firmware Installation and Update Scenarios for Peripheral Devices
    DEGREE PROJECT IN COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2019 Automated testing of firmware installation and update scenarios for peripheral devices DAG REUTERSKIÖLD KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE Automated testing of firmware installation and update scenarios for peripheral devices DAG REUTERSKIÖLD Master in Computer Science Date: August 12, 2019 Supervisor: Hamid Faragardi Examiner: Elena Troubitsyna School of Electrical Engineering and Computer Science Host company: Tobii AB Swedish title: Automatisering av enhetsinstallation, uppdatering och testning med hjälp av virtuella maskiner iii Abstract This research presents an approach to transition from manual to automated testing of hardware specific firmware. The manual approach for firmware test- ing can be repetitive and time consuming. A significant proportion of the time is spent on cleaning and re-installing operating systems so that old firmware does not interfere with the newer firmware that is being tested. The approach in this research utilizes virtual machines and presents an automation framework. One component of the automation framework is an application to imitate con- nected peripheral devices to bypass hardware dependencies of firmware in- stallers. The framework also consists of automation and pipeline scripts with the objective to execute firmware installers and detect errors and abnormalities in the installation and updating processes. The framework can run on locally hosted virtual machines, but is most applicable using cloud hosted virtual ma- chines, where it is part of a continuous integration that builds, downloads, installs, updates and tests new firmware versions, in a completely automated manner. The framework is evaluated by measuring and comparing execution times with manually conducted installation and updating tests, and the result shows that the framework complete tests much faster than the manual approach.
    [Show full text]
  • Software Testing Revealed Training Book Second Edition by International Software Test Institute™
    SOFTWARE TESTING REVEALED TRAINING BOOK SECOND EDITION BY INTERNATIONAL SOFTWARE TEST INSTITUTE™ www.test-institute.org © COPYRIGHT INTERNATIONAL SOFTWARE TEST INSTITUTE™ Dedication To all of the International Software Test Institute™ students, thank you for inspiring us, keeping us focused, and making sure we do our best to help you grow in your career with your skills and knowhow. Without you, your engagement and your loyal support, International Software Test Institute™ could not come where it is today. TABLE OF CONTENTS CLICKABLE WELCOME ........................................................................................................................6 ABOUT INTERNATIONAL SOFTWARE TEST INSTITUTE™ .....................................................7 Introduction To Software Testing ............................................................................8 What is Software Quality Assurance? ...................................................................12 What Is Software Testing? .......................................................................................18 Fundamentals of Software Testing ........................................................................21 Software Testing Roles and Responsibilities .......................................................30 Software Testing Methods ......................................................................................36 Software Testing Levels ..........................................................................................38 Software Testing
    [Show full text]
  • Empirical Evaluation of the Effectiveness and Reliability of Software Testing Adequacy Criteria and Reference Test Systems
    Empirical Evaluation of the Effectiveness and Reliability of Software Testing Adequacy Criteria and Reference Test Systems Mark Jason Hadley PhD University of York Department of Computer Science September 2013 2 Abstract This PhD Thesis reports the results of experiments conducted to investigate the effectiveness and reliability of ‘adequacy criteria’ - criteria used by testers to determine when to stop testing. The research reported here is concerned with the empirical determination of the effectiveness and reliability of both tests sets that satisfy major general structural code coverage criteria and test sets crafted by experts for testing specific applications. We use automated test data generation and subset extraction techniques to generate multiple tests sets satisfying widely used coverage criteria (statement, branch and MC/DC coverage). The results show that confidence in the reliability of such criteria is misplaced. We also consider the fault-finding capabilities of three test suites created by the international community to serve to assure implementations of the Data Encryption Standard (a block cipher). We do this by means of mutation analysis. The results show that not all sets are mutation adequate but the test suites are generally highly effective. The block cipher implementations are also seen to be highly ‘testable’ (i.e. they do not mask faults). 3 Contents Abstract ............................................................................................................................ 3 Table of Tables ...............................................................................................................
    [Show full text]
  • Effective Methods for Software Testing
    Effective Methods for Software Testing Third Edition Effective Methods for Software Testing Third Edition William E. Perry Effective Methods for Software Testing, Third Edition Published by Wiley Publishing, Inc. 10475 Crosspoint Boulevard Indianapolis, IN 46256 www.wiley.com Copyright © 2006 by Wiley Publishing, Inc., Indianapolis, Indiana Published simultaneously in Canada ISBN-13: 978-0-7645-9837-1 ISBN-10: 0-7645-9837-6 Manufactured in the United States of America 10 9 8 7 6 5 4 3 2 1 3MA/QV/QU/QW/IN No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copy- right Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Legal Department, Wiley Publishing, Inc., 10475 Crosspoint Blvd., Indianapolis, IN 46256, (317) 572-3447, fax (317) 572-4355, or online at http://www.wiley.com/go/permissions. Limit of Liability/Disclaimer of Warranty: The publisher and the author make no repre- sentations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fit- ness for a particular purpose. No warranty may be created or extended by sales or promo- tional materials.
    [Show full text]
  • Chapter2: Testing Throughout Lifecycle
    ISTQB Certification Preparation Guide: Chapter 1 – Testing Through Lifecycle Chapter2: Testing throughout Lifecycle "What is clear from the Inquiry Team's investigations is that neither the Computer Aided Dispatch (CAD) system itself, nor its users, were ready for full implementation on 26th October 1992. The CAD software was not complete, not properly tuned, and not fully tested. The resilience of the hardware under a full load had not been tested. The fall back option to the second file server had certainly not been tested." Extract from the main conclusions of the official report into the failure of the London Ambulance Service's Computer Systems on October 26th and 27th 1992. 2.1 OVERVIEW This module covers all of the different testing activities that take place throughout the project lifecycle. We introduce various models for testing, discuss the economics of testing and then describe in detail component, integration, system and acceptance testing. We conclude with a brief look at maintenance testing. After completing this module you will: 2.2 OBJECTIVES Ø Understand the difference between verification and validation testing activities Ø Understand what benefits the V model offers over other models. Ø Be aware of other models in order to compare and contrast. Ø Understand the cost of fixing faults increases as you move the product towards live use. Ø Understand what constitutes a master test plan. Ø Understand the meaning of each testing stage. http://www.softwaretestinggenius.com/ Page 1 of 22 ISTQB Certification Preparation Guide: Chapter 1 – Testing Through Lifecycle 2.3 Models for Testing This section will discuss various models for testing.
    [Show full text]
  • Pdf: Software Testing
    Software Testing Gregory M. Kapfhammer Department of Computer Science Allegheny College [email protected] I shall not deny that the construction of these testing programs has been a major intellectual effort: to convince oneself that one has not overlooked “a relevant state” and to convince oneself that the testing programs generate them all is no simple matter. The encouraging thing is that (as far as we know!) it could be done. Edsger W. Dijkstra [Dijkstra, 1968] 1 Introduction When a program is implemented to provide a concrete representation of an algorithm, the developers of this program are naturally concerned with the correctness and performance of the implementation. Soft- ware engineers must ensure that their software systems achieve an appropriate level of quality. Software verification is the process of ensuring that a program meets its intended specification [Kaner et al., 1993]. One technique that can assist during the specification, design, and implementation of a software system is software verification through correctness proof. Software testing, or the process of assessing the func- tionality and correctness of a program through execution or analysis, is another alternative for verifying a software system. As noted by Bowen, Hinchley, and Geller, software testing can be appropriately used in conjunction with correctness proofs and other types of formal approaches in order to develop high quality software systems [Bowen and Hinchley, 1995, Geller, 1978]. Yet, it is also possible to use software testing techniques in isolation from program correctness proofs or other formal methods. Software testing is not a “silver bullet” that can guarantee the production of high quality software systems.
    [Show full text]
  • Beginners Guide to Software Testing
    Beginners Guide To Software Testing Beginners Guide To Software Testing - Padmini C Page 1 Beginners Guide To Software Testing Table of Contents: 1. Overview ........................................................................................................ 5 The Big Picture ............................................................................................... 5 What is software? Why should it be tested? ................................................. 6 What is Quality? How important is it? ........................................................... 6 What exactly does a software tester do? ...................................................... 7 What makes a good tester? ........................................................................... 8 Guidelines for new testers ............................................................................. 9 2. Introduction .................................................................................................. 11 Software Life Cycle ....................................................................................... 11 Various Life Cycle Models ............................................................................ 12 Software Testing Life Cycle .......................................................................... 13 What is a bug? Why do bugs occur? ............................................................ 15 Bug Life Cycle ............................................................................................... 16 Cost of fixing bugs .......................................................................................
    [Show full text]