DOCSLIB.ORG

Software Testing CS 408

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Software Testing CS 408 Software Testing CS 408 Lecture 5: Higher-Order Testing 1/31/17 Myers, Chapter 6 Basics • A software error occurs when the program does not do what its end user reasonably expects it to do • Even if you could perform an absolutely-perfect module test, you still could not guarantee that you have found all software errors • To complete testing, higher-order testing is necessary • Software development is largely a process of communicating information about the eventual product and translating this information from one form to another 2 Basics • Translate the software user's needs into a set of written requirements (Product Backlog in Scrum) • Design -- partitions the system into individual programs, components, or subsystems, and defines their interfaces • Translate the Design into source code Most software errors stem from breakdowns in information communication One solution is to orient distinct testing processes toward particular classes of errors Need for higher-order testing increases along with the size of the program 3 Basics 4 Function Testing • Function testing is the process of attempting to find discrepancies between the program and the external specification • An external specification is a precise description of the program's behavior from the end-user point of view • Function testing is normally a black-box activity Equivalence partitioning, boundary value analysis, cause-effect graphing, and error-guessing methods Covered back in Chapter 5 5 System Testing • System testing is not a process of testing the functions of the complete system or program • System testing has a particular purpose: to compare the system or program to its original objectives • Several System Test categories can be used. Many of these are related to non-functional Software Quality Attributes Some of these will be important in CS 40800 (e.g., Stress Testing, Performance Testing, Usability Testing, ....) But, you are not required to do all (or even any!) of these 6 Sanity and Smoke Testing Sanity testing is a very brief run-through of the functionality of a program to assure that the software works roughly as expected. This is often prior to a more exhaustive round of testing Smoke Testing Origin -- physical tests using smoke made to closed systems of pipes to detect cracks or breaks Subset of test cases that cover the most important functionality of a component or system are selected and run to ascertain if the most crucial functions of a program work correctly Purpose is to determine whether the application is so badly- broken that further testing is unnecessary 7 Performance Testing Many programs have specific performance or efficiency objectives, stating such properties as response times and throughput rates under certain workload and configuration conditions Test cases should be designed to show that the program does not satisfy its performance objectives Volume Testing Subject the program to heavy volumes of data Purpose of volume testing is to show that the program cannot handle the volume of data specified in its objectives Example: The system is supposed to be able to store, retrieve, and modify information concerning 1.5 million customers 8 Stress Testing Stress testing subjects the program to heavy loads or stresses A heavy stress is a peak volume of data, or activity, encountered over a short span of time If an air traffic control system is supposed to keep track of up to 200 planes in its sector, you could stress-test it by simulating the presence of 200 planes ... or more. If an operating system is supposed to support a maximum of 150 concurrent jobs, the system could be stressed by attempting to run 150 jobs simultaneously ... or more. Web-based applications are common subjects of stress testing (Chapter 10) You could stress a mobile device application -- a mobile phone operating system, for example -- by launching multiple applications that run and stay resident, then try making or receiving one or more telephone calls 9 Usability Testing Tasking the ultimate end user of an application with testing the software in a real-world environment Covered in next chapter (Chapter 7) Security Testing Many programs now have specific security objectives Security testing is the process of attempting to devise test cases that subvert the program's security checks One way to devise such test cases is to study known security problems in similar systems and generate test cases that attempt to demonstrate comparable problems in the system you are testing Web-based applications often need a higher level of security testing than do most applications (Chapter 10) 10 Storage Testing Programs occasionally have storage objectives that state, for example, the amount of system memory the program uses and the size of temporary or log files Configuration Testing Some software must support a variety of hardware configurations, including various types and numbers of I/0 devices and communications lines, or different memory sizes Often, the number of possible configurations is too large to test each one Test a representative subset 11 Hey, You Have Given Me Too Many Knobs! Understanding and Dealing with Over-Designed ConfigurationinSystemSoftware Tianyin Xu*, Long Jin*, Xuepeng Fan*‡,YuanyuanZhou*, Shankar Pasupathy†,andRukmaTalwadker† *University of California San Diego, USA ‡Huazhong Univ. of Science & Technology, China †NetApp, USA {tixu, longjin, xuf001, yyzhou}@cs.ucsd.edu {Shankar.Pasupathy, Rukma.Talwadker}@netapp.com Configuration Testing ABSTRACT 700 500 s Storage-A s r 600 r MySQL e e t 400 Configuration problems are not only prevalent, but also severely t e 500 e m 5.6.2 m impair the reliability of today’s system software. One fundamental a a r r 300 5.5.0 a 400 a 5.1.3 p reason is the ever-increasing complexity of configuration, reflected p f f 5.0.16 o o 300 by the large number of configuration parameters (“knobs”). With 200 4.1.0 r r e e 4.0.12 b hundreds of knobs, configuring system software to ensure highre- b 200 3.23.0 m m 100 u u 100 liability and performance becomes a daunting, error-prone task. N N This paper makes a first step in understanding a fundamental 0 0 1/1999 1/2003 1/2007 1/2011 1/2014 question of configuration design: “do users really need so many 7/2006 7/2008 7/2010 7/2012 7/2014 Releasetime Releasetime knobs?”Toprovidethequantitativelyanswer,westudythecon- 600 200 figuration settings of real-world users, including thousands of cus- s Apache s r Hadoop 500 r 2.0.0 e e 160 t tomers of a commercial storage system (Storage-A), and hundreds t e 2.3.4 e 1.0.0 m 400 m of users of two widely-used open-source system software projects. a a r r 120 a a 0.19.0 p Our study reveals a series of interesting findings to motivatesoft- p 300 f f 2.2.14 o o ware architects and developers to be more cautious and disciplined 2.0.35 80 r r e e 200 b in configuration design. Motivated by these findings, we provide b 1.3.24 0.1.0 m m 40 u afewconcrete,practicalguidelineswhichcansignificantlyreduce u 100 1.3.14 MapReduce N N HDFS the configuration space. Take Storage-A as an example, the guide- 0 0 lines can remove 51.9% of its parameters and simplify 19.7% of 1/1998 1/2002 1/2006 1/2010 1/2014 1/2006 1/2008 1/2010 1/2012 1/2014 Releasetime Releasetime the remaining ones with little impact on existing users. Also, we study the existing configuration navigation methods in the context Figure 1: The increasing number of configuration parameters with of “too many knobs” to understand their effectiveness in deaXuling et. al.,software FSE’15 evolution. Storage-A is a commercial storage system from a with the over-designed configuration, and to provide practices for major storage company in the U.S. 12 building navigation support in system software. Categories and Subject Descriptors: D.2.10 [Software Engineer- all the customer-support cases in a major storage company in the ing]: Methodologies U.S., and were the most significant contributor (31%) among all the high-severity cases [75]. Rabkin and Katz reported that config- General Terms: Design, Human Factors, Reliability uration issues were the dominant source of support cost in Hadoop clusters (based on data from Cloudera Inc.), in terms of both the Keywords: Configuration, Complexity, Simplification, Navigation, number of support cases and the amount of supporting time [46]. Parameter, Difficulty, Error Moreover, configuration errors, the after-effects of configuration difficulties, have become one of the major causes of system fail- 1. INTRODUCTION ures. Barroso and Hölzle reported that configuration errors were the second major cause of service-level disruptions at one of Google’s 1.1 Motivation main services [16]. Recently, a number of outages of Internetand In recent years, configuration problems have drawn tremendous cloud services, including Google, LinkedIn, Microsoft Azure, and attention for their increasing prevalence and severity. Forexample, Amazon EC2, were caused by configuration errors [35, 59, 63, 68]. Yin et al. reported that configuration issues accounted for 27% of One fundamental reason for today’s prevalent configuration is- sues is the ever-increasing complexity of configuration, especially in system software. This is reflected by the large and still increasing Permission to make digital or hard copies of all or part of this work for personal or number of configuration parameters (“knobs”), as well as various classroom use is granted without fee provided that copies are not made or distributed configuration constraints and consistency requirements [32, 39, 45, for profit or commercial advantage and that copies bear this notice and the full citation on the first page.
Load more

Recommended publications
  • Studying the Feasibility and Importance of Software Testing: an Analysis
    Dr. S.S.Riaz Ahamed / Internatinal Journal of Engineering Science and Technology Vol.1(3), 2009, 119-128 STUDYING THE FEASIBILITY AND IMPORTANCE OF SOFTWARE TESTING: AN ANALYSIS Dr.S.S.Riaz Ahamed Principal, Sathak Institute of Technology, Ramanathapuram,India. Email:[email protected], [email protected] ABSTRACT Software testing is a critical element of software quality assurance and represents the ultimate review of specification, design and coding. Software testing is the process of testing the functionality and correctness of software by running it. Software testing is usually performed for one of two reasons: defect detection, and reliability estimation. The problem of applying software testing to defect detection is that software can only suggest the presence of flaws, not their absence (unless the testing is exhaustive). The problem of applying software testing to reliability estimation is that the input distribution used for selecting test cases may be flawed. The key to software testing is trying to find the modes of failure - something that requires exhaustively testing the code on all possible inputs. Software Testing, depending on the testing method employed, can be implemented at any time in the development process. Keywords: verification and validation (V & V) 1 INTRODUCTION Testing is a set of activities that could be planned ahead and conducted systematically. The main objective of testing is to find an error by executing a program. The objective of testing is to check whether the designed software meets the customer specification. The Testing should fulfill the following criteria: ¾ Test should begin at the module level and work “outward” toward the integration of the entire computer based system.
    [Show full text]
  • Types of Software Testing
    Types of Software Testing We would be glad to have feedback from you. Drop us a line, whether it is a comment, a question, a work proposition or just a hello. You can use either the form below or the contact details on the rightt. Contact details [email protected] +91 811 386 5000 1 Software testing is the way of assessing a software product to distinguish contrasts between given information and expected result. Additionally, to evaluate the characteristic of a product. The testing process evaluates the quality of the software. You know what testing does. No need to explain further. But, are you aware of types of testing. It’s indeed a sea. But before we get to the types, let’s have a look at the standards that needs to be maintained. Standards of Testing The entire test should meet the user prerequisites. Exhaustive testing isn’t conceivable. As we require the ideal quantity of testing in view of the risk evaluation of the application. The entire test to be directed ought to be arranged before executing it. It follows 80/20 rule which expresses that 80% of defects originates from 20% of program parts. Start testing with little parts and extend it to broad components. Software testers know about the different sorts of Software Testing. In this article, we have incorporated majorly all types of software testing which testers, developers, and QA reams more often use in their everyday testing life. Let’s understand them!!! Black box Testing The black box testing is a category of strategy that disregards the interior component of the framework and spotlights on the output created against any input and performance of the system.
    [Show full text]
  • Human Performance Regression Testing
    Human Performance Regression Testing Amanda Swearngin, Myra B. Cohen Bonnie E. John, Rachel K. E. Bellamy Dept. of Computer Science & Eng. IBM T. J. Watson Research Center University of Nebraska-Lincoln, USA P.O. Box 704 Lincoln, NE 68588-0115 Yorktown Heights, NY 10598, USA faswearn,[email protected] fbejohn,[email protected] Abstract—As software systems evolve, new interface features common tasks, necessitating longer mouse movements and, in such as keyboard shortcuts and toolbars are introduced. While fact, decrease efficiency. In addition, many toolbars with small it is common to regression test the new features for functional icons may add screen clutter and may decrease a new user’s correctness, there has been less focus on systematic regression testing for usability, due to the effort and time involved in human ability to discover how to accomplish a task over a simpler studies. Cognitive modeling tools such as CogTool provide some UI design. help by computing predictions of user performance, but they still Usability testing has traditionally been empirical, bringing require manual effort to describe the user interface and tasks, end-users in to a testing facility, asking them to perform tasks limiting regression testing efforts. In recent work, we developed on the system (or prototype), and measuring such things as the CogTool-Helper to reduce the effort required to generate human performance models of existing systems. We build on this work time taken to perform the task, the percentage of end-users by providing task specific test case generation and present our who can complete the task in a fixed amount of time, and vision for human performance regression testing (HPRT) that the number and type of errors made by the end-users.
    [Show full text]
  • Web Gui Testing Checklist
    Web Gui Testing Checklist Wes recrystallizing her quinone congruously, phytophagous and sulphonic. How imponderable is Schroeder when barbate whileand soft-footed Brewer gliff Zachery some incisure yakety-yak affluently. some chatoyancy? Fulgurating and battiest Nealson blossoms her amontillados refine Wbox aims to the field to be able to the automated support data, testing web gui checklist Planned testing techniques, including scripted testing, exploratory testing, and user experience testing. This gui content will the css or dynamic values? Test all input fields for special characters. For instance, create test data assist the maximum and minimum values in those data field. Assisted by timing testing is not tested to the order to achieve true black art relying on gui testing web checklist will best. The web hosting environments you start all web testing gui checklist can provide tests has had made. The gui testing procedures are the weak factors causing delays in agile here offering, gui testing web? At anytime without giving us a testing web gui checklist can also has on. How gui testing checklist for a gui testing web checklist to induce further eliminating redundant if there is transmitted without the below to use of jobs with. Monkey testing tool that an application or even perform testing web gui changes some test android scripts behind successful only allows an. Discusses the preceding css or if a sql injections through an application penetration testing on gui testing web? How much regression testing is enough? Fully automated attack simulations and highly automated fuzzing tests are appropriate here, and testers might also use domain testing to pursue intuitions.
    [Show full text]
  • Testing Guide 4Deets
    TESTING GUIDE Automated Testing for SuccessFactors Introduction This guide will give you an overview of the different aspects of testing your SuccessFactors environment. As most employees interact in some way or form with the system, it’s crucial to test your system regularly and around crucial events to make sure you provide an excellent employee experience. The following topics will be explored: • Testing During An Implementation • Testing For Configuration Changes • Testing For Quarterly Releases • Manual vs. Automated Testing © 2019 Page | 2 Automated Testing For SuccessFactors Testing During An Implementation During an implementation, testing is critical to make sure the system is designed and configured according to expectations. One of the most known test is User Acceptance Testing (UAT) in which a client signs off that the system works as expected. Before UAT, there are many different tests you would like to perform. These tests are done during different phases of a project. Some only once, others multiple times. Let’s start with looking at what phases there are during a SuccessFactors implementation. This may differ slightly, but all SuccessFactors implementations basically follow the same path. The reason for this is that the Agile methodology, which is the preferred method defined by SAP, needs to be followed by SAP professional services and all the certified implementation partners. This methodology has been proven to be most successful. The Agile methodology is an iterative approach. Iterative approach is a way of breaking down the system development of a large application into smaller chunks. In iterative development, functionality is designed, configured and tested in repeated cycles.
    [Show full text]
  • Software Testing Training Module
    MAST MARKET ALIGNED SKILLS TRAINING SOFTWARE TESTING TRAINING MODULE In partnership with Supported by: INDIA: 1003-1005,DLF City Court, MG Road, Gurgaon 122002 Tel (91) 124 4551850 Fax (91) 124 4551888 NEW YORK: 216 E.45th Street, 7th Floor, New York, NY 10017 www.aif.org SOFTWARE TESTING TRAINING MODULE About the American India Foundation The American India Foundation is committed to catalyzing social and economic change in India, andbuilding a lasting bridge between the United States and India through high impact interventions ineducation, livelihoods, public health, and leadership development. Working closely with localcommunities, AIF partners with NGOs to develop and test innovative solutions and withgovernments to create and scale sustainable impact. Founded in 2001 at the initiative of PresidentBill Clinton following a suggestion from Indian Prime Minister Vajpayee, AIF has impacted the lives of 4.6million of India’s poor. Learn more at www.AIF.org About the Market Aligned Skills Training (MAST) program Market Aligned Skills Training (MAST) provides unemployed young people with a comprehensive skillstraining that equips them with the knowledge and skills needed to secure employment and succeed on thejob. MAST not only meets the growing demands of the diversifying local industries across the country, itharnesses India's youth population to become powerful engines of the economy. AIF Team: Hanumant Rawat, Aamir Aijaz & Rowena Kay Mascarenhas American India Foundation 10th Floor, DLF City Court, MG Road, Near Sikanderpur Metro Station, Gurgaon 122002 216 E. 45th Street, 7th Floor New York, NY 10017 530 Lytton Avenue, Palo Alto, CA 9430 This document is created for the use of underprivileged youth under American India Foundation’s Market Aligned Skills Training (MAST) Program.
    [Show full text]
  • Testing Management Plan
    AHCA Florida Health Care Connections (FX) <<Insert Project Name Here>> Testing Management Plan Version: 300 Date: Month dd, yyyy Author: [Author] Submitted To: AHCA FX Project Management Revision History DATE VERSION DESCRIPTION AUTHOR <<Insert Project Name Here>> Testing MM/DD/YYYY 001 Management Plan first draft version Modifications to the approved baseline version (100) of this artifact must be made in accordance with the Artifact Management Standards. Quality Review History DATE REVIEWER COMMENTS MM/DD/YYYY Agency for Health Care Administration Page ii <<Insert Project Name Here>> Testing Management Plan Table of Contents Section 1 Introduction ............................................................................................................ 1 1.1 Purpose ........................................................................................................................ 1 1.2 Scope Statement .......................................................................................................... 1 1.3 Goals and Objectives ................................................................................................... 2 1.4 Referenced Documents ................................................................................................ 2 Section 2 Roles and Responsibilities .................................................................................... 3 Section 3 Assumptions, Contraints, and Risks ...................................................................... 4 3.1 Assumptions ................................................................................................................
    [Show full text]
  • A Brief Essay on Software Testing
    1 A Brief Essay on Software Testing Antonia Bertolino, Eda Marchetti Abstract— Testing is an important and critical part of the software development process, on which the quality and reliability of the delivered product strictly depend. Testing is not limited to the detection of “bugs” in the software, but also increases confidence in its proper functioning and assists with the evaluation of functional and nonfunctional properties. Testing related activities encompass the entire development process and may consume a large part of the effort required for producing software. In this chapter we provide a comprehensive overview of software testing, from its definition to its organization, from test levels to test techniques, from test execution to the analysis of test cases effectiveness. Emphasis is more on breadth than depth: due to the vastness of the topic, in the attempt to be all-embracing, for each covered subject we can only provide a brief description and references useful for further reading. Index Terms — D.2.4 Software/Program Verification, D.2.5 Testing and Debugging. —————————— u —————————— 1. INTRODUCTION esting is a crucial part of the software life cycle, and related issues, we can only briefly expand on each argu- T recent trends in software engineering evidence the ment, however plenty of references are also provided importance of this activity all along the development throughout for further reading. The remainder of the chap- process. Testing activities have to start already at the re- ter is organized as follows: we present some basic concepts quirements specification stage, with ahead planning of test in Section 2, and the different types of test (static and dy- strategies and procedures, and propagate down, with deri- namic) with the objectives characterizing the testing activity vation and refinement of test cases, all along the various in Section 3.
    [Show full text]
  • Software Engineering
    SOFTWARE ENGINEERING FUNDAMENTALS OF SOFTWARE TESTING Software testing is the evaluation of a system with the intention of finding an error or fault or a bug. It also checks for the functionalities of the system so that it meets the specified requirements. LEARNING OBJECTIVES • To execute a program with the intent of finding an error. • To check if the system meets the requirements and be executed successfully in the intended environment. • To check if the system is “Fit for purpose”. • To check if the system does what it is expected to do. STAGES OF TESTING Various testing is done during various phases of the development cycle such as: Module or unit testing. Integration testing, Function testing. Performance testing. Acceptance testing. Installation testing. UNIT TESTING Unit testing is the testing and validation at the unit level. Unit testing validates and tests the following: Algorithms and logic Data structures (global and local) Interfaces Independent paths Boundary conditions Error handling Formal verification. Testing the program itself by performing black box and white box testing. INTEGRATION TESTING One module can have an adverse effect on another. Sub-functions, when combined, may not produce the desired major function. Individually acceptable imprecision in calculations may be magnified to unacceptable levels. Interfacing errors not detected in unit testing may appear in this phase. Timing problems (in real-time systems) and resource contention problems are not detectable by unit testing. Top-Down Integration The main control module is used as a driver, and stubs are substituted for all modules directly subordinate to the main module. Depending on the integration approach selected (depth or breadth first), subordinate stubs are replaced by modules one at a time.
    [Show full text]
  • Software Testing: Way to Develop Quality Software Product 1Dipti Pawade, 2Harshada Sonkamble, 3Pranchal Chaudhari, 4Shubhangi Rathod 1,2,3Dept
    ISSN : 0976-8491 (Online) | ISSN : 2229-4333 (Print) IJCST VOL . 4, Iss UE SPL - 1, JAN - MAR C H 2013 Software Testing: Way to Develop Quality Software Product 1Dipti Pawade, 2Harshada Sonkamble, 3Pranchal Chaudhari, 4Shubhangi Rathod 1,2,3Dept. of I T, K. J. Somaiya College of Engineering, Mumbai, India 4Dept. of IT, P. I. I. T. M. S. R., Navi Mumbai, India Abstract reaction to the input. The testing is a process of comparing the Software testing is one of the most important phase of software behaviour of the software against oracles principles by which one development life cycle. No one can underestimate the importance can recognize a problem. The good practice is to test software of software testing process on software quality assurance. as early as it has been written. The testing concept is evolved Organization pays 40% of its efforts on testing process. A powerful with time. Table 1 illustrate the concept evolution of testing [1]. testing technique results in reduced software development cost Software testing life cycle comprises of the different phases [2] and time and improved performance. That is why choosing an mentioned in Table 2. appropriate testing technique is very important. In this paper we have discussed various testing approaches and methods, their Table 2: Software Testing Life Cycle peculiarities and finally discussed functional and non-functional Phase Activity testing. Phase I: Software requirements/design is Requirements/ reviewed in detail and basic idea of Keywords Design Review what needs to be tested is derived. Testing Evolution, Software Testing Life Cycle, Testing Approach, Testing Technique, Functional Testing, Non-functional Testing Phase II: Detailed test plan is derived.
    [Show full text]
  • Testing Guidelines
    Guidance Testing Guidelines Purpose This document provides guidelines for testing changes to the BSC Software, Systems and Processes. It also provides a high level view of test procedures that are followed by various parties involved in testing. The management of testing and test deliverables is detailed in the Test Management procedure (Reference 1). The scope of testing for each software release is detailed in the relevant Release Test Strategy document, which is developed by ELEXON and the relevant Service Providers. The scope is agreed by all test participants and captured in the Test Strategy. This document is specifically written as a guideline for testing the BSC Systems. However, the principles described may be used when planning for testing of other software systems. Contents 1. Testing Process Overview ...................................................................................................... 2 2. Application Manager and Developer Testing Overview ........................................................ 4 3. Business Process Operator Testing Overview ....................................................................... 9 4. Terms Used ............................................................................................................................ 13 5. References ............................................................................................................................. 13 6. Appendix A – Test Results ...................................................................................................
    [Show full text]
  • 1. Can You Explain the PDCA Cycle and Where Testing Fits In?
    1. Can you explain the PDCA cycle and where testing fits in? Software testing is an important part of the software development process. In normal software development there are four important steps, also referred to, in short, as the PDCA (Plan, Do, Check, Act) cycle. Let's review the four steps in detail. 1. Plan: Define the goal and the plan for achieving that goal. 2. Do/Execute: Depending on the plan strategy decided during the plan stage we do execution accordingly in this phase. 3. Check: Check/Test to ensure that we are moving according to plan and are getting the desired results. 4. Act: During the check cycle, if any issues are there, then we take appropriate action accordingly and revise our plan again. So developers and other stakeholders of the project do the "planning and building," while testers do the check part of the cycle. Therefore, software testing is done in check part of the PDCA cyle. 2. What is the difference between white box, black box, and gray box testing? Black box testing is a testing strategy based solely on requirements and specifications. Black box testing requires no knowledge of internal paths, structures, or implementation of the software being tested. White box testing is a testing strategy based on internal paths, code structures, and implementation of the software being tested. White box testing generally requires detailed programming skills. There is one more type of testing called gray box testing. In this we look into the "box" being tested just long enough to understand how it has been implemented.
    [Show full text]