Topic # 10 Software Testing: Strategies (Ch. 17) 1 Objectives 1. Software Testing Strategy 2. Unit Testing 3. Integration Testing 4. Validation Testing 5. System Testing 2 Software Testing Strategy The goal of software testing is to uncover errors. To achieve this goal objective, the following strategic approach should be used: a series of 4 types (levels) of software tests – 1) unit tests, 2) integration tests, 3) validation tests, and 4) system tests. – should be planned and executed. Software testing accounts for the largest percentage (up to 25-30%) of technical effort in the software engineering process. Testing -- a systematic, planned activity Debugging -- an art (luck, intuition, etc). 3 Verification and Validation Verification -- process-related term corresponds to “Are we building the product right?” (Is software engineering process correct?) Validation - product-related term corresponds to “Are we building the right product?” (Is software product correct: Does it meet all customer requirements?) Ex: somebody used software engineering theory, software design procedures, graphic user interface, testing procedures, BUT the final software product is useless for a customer (or, maybe it is not popular among vast majority of customers, users). In this case: - verification – YES, - validation – NO. 4 Components of Software Testing Strategy 1. Unit tests 2. Unit integration tests concentrate on functional verification of a component concentrate on functional and incorporation verification of a component and incorporation of components into a program structure. 4. System tests 3. Validation tests demonstrates traceability validates software once it has been validates software once it has been to software requirements Incorporated into a larger system (actual environment) 5 Question: Why do we need so many types of tests? Answers: Due to various types of bugs of software system. Due to various consequences of SW bugs. Bug Categories: 1) variables-related bugs, 2) input/output data-related bugs, 3) coding (syntax)-related bugs, 4) system-related bugs, 5) functionality (functions)-related bugs, 6) design-related bugs, 7) standards violation-related bugs, 8) documentation-related bugs, etc. Consequences (Types) of SW Bugs infectious In 2002, a study damage commissioned by the US Department of Commerce' catastrophic National Institute of Standards and Technology concluded extreme that software bugs, or errors, serious are so prevalent and so harmful that they cost the US disturbing economy an estimated $59 billion annually, or about annoying 0.6 percent of the gross mild domestic product. Bug Type 6 What Testing Shows (Outcomes) syntax (coding) errors, logic errors, input date errors, etc. requirements conformance performance quality of final product 7 1. Unit Testing Testing begins at the unit (module) level and works “outward” toward the integration of the entire SW system Testing is conducted by the developer in the developer’s environment (alpha-testing) and by independent testing group outside development environment (beta-testing). module to be tested results software engineer test cases Unit testing focuses verification effort on the smallest unit. Parallel testing on multiple modules is possible. 8 Unit Testing (cont.) module to be tested We should test: • scope of all declared variables (local, global) • data types of variables (integer, real, string, char, etc.) • boundary conditions (what will happen with variables when we leave this unit or module) • all functional tests (to test all required functions inside this unit) • independent paths inside this unit or module • all interfaces inside unit/module • identify and test all error-handling paths test cases 9 2. Integration Testing Strategies The goal of integration testing is to ensure that all interacting software units/modules/subsystems in a system interface correctly with one another to produce the desired results. Furthermore, in trying to attain this goal, integration tests will ensure that the introduction of one or more subsystems into the system does not have an adverse affect on existing functionality. An integration test covers the testing of interface points between subsystems. Integration testing is performed once unit testing has been completed for all units contained in the subsystems being tested. 2 main approaches: • top-down integration (the “big bang” approach) • bottom-up integration (an incremental construction strategy) 10 Top Down Integration (cont.) A top module is tested with stubs B F G stubs are replaced one at a time, "depth first" C as new modules are integrated, some subset of tests is re-run D E 11 Bottom-Up Integration (cont.) A B F G drivers are replaced one at a time, "depth first" C working modules are grouped into building blocks and integrated D E cluster 12 3, 4, … and other High Order Testing other specialized testing Validation testing: Focus is on software requirements 4. system test System testing: Focus is on system integration Alpha/Beta testing: Focus is on customer usage Recovery testing: forces the software to fail in a variety of ways and verifies that recovery is properly performed Security testing: verifies that protection mechanisms built into 3. validation test a system will, in fact, protect it from improper penetration Stress testing: executes a system in a manner that demands resources in abnormal quantity, frequency, or volume Performance Testing: test the run-time performance of software within the context of an integrated system 13 Software Testing: Tools by IBM Company (examples) 14 Topic # 10 Software Testing: Strategies Ch. 17: Additional Information 15 Software Testing: Additional Information 1. Software Testing Standards and Procedures http://it.toolbox.com/blogs/enterprise-solutions/sample-software-testing-standards-and- procedures-12772 2. Unit Testing http://mauriziostorani.wordpress.com/2008/07/09/unit-testing-examples-concepts-and- frameworks/ 3. Integration Testing http://www.exforsys.com/tutorials/testing/integration-testing-whywhathow.html 16 Topic # 10: Software Testing: SW Testing Techniques (Ch. 18) 17 Objectives 1. Software Testing Principles 2. Black-Box Testing technique 3. White-Box Testing technique 18 SW Testing Principles 1. All tests should traceable to customer requirements. *) most severe problems are those that cause program to fail to meet customer requirements 2. Tests should be planned long before actual testing process begins. *) think about testing as soon as customer requirements are completed 3. The Pareto principle applies to SW testing. *) 80% of all errors uncovered during testing will likely be traceable to 20 percent of all program components OR 20% of components generate 80% of errors 4. Testing should begin “in the small” (modules) and progress toward testing “in the large” (subsystems and systems). *) think about different testing procedures and content (tests) for each SW level -- system, subsystem, unit, component as soon as requirements are completed 5. Exhaustive testing is NOT possible. *) number of path permutations even in a small program is exceptionally large Æ if required reliability is greater than 0.9… 6. To be most effective, testing should be conducted by both a developer (alpha-testing) and an independent tester (beta-testing). 19 Who Tests the Software? SW developer • understands the system • is driven by "delivery“ • will test "gently" Alpha-Testing Independent tester • must learn about the system, • is driven by quality • will attempt to test as much Beta-Testing as possible; even break software system 20 Software Testing Techniques white-box black-box technique technique Techniques Strategies 21 White Box Testing Technique The goal of white-box testing (inside- module) is to 1) exercise all program logic paths within a module at least once, 2) check all loop execution constraints on both TRUE and FALSE (YES/NO) sides, and 3) check all internal data structure boundaries to ensure their validity. It focuses on the program control structure within a single module. ... our goal is to ensure that all It is a very time- and effort- statements and conditions have been executed at least once ... consuming testing method. 22 White-Box Testing Mechanics Unit/Module Environment Input Data Output Data Code Example: Currency (EURO to DOLLAR) converter unit/module (1-999 Euros -- 1.4; 1000-9999 – 1.45; >10000 – 1.45) Input: Amount (in EUROs) Output: Amount in U.S. DOLARS (with a rate based on amount of EURos) 23 Example 24 Exhaustive Testing vs Selective Testing Selective Testing: applications of the the Pareto principle : 80% of all errors uncovered during testing will likely be traceable to 20 percent of all components OR 20% of components generate 80% of errors Knowledge ? Experience ! Intuition! 25 Topic # 10 Software Testing: SW Testing Techniques In-Classroom Exercise # 1 26 Black-Box Testing Technique Black-box testing focuses on the requirements functional requirements of the software without regard to the internal workings of a module or program. It is not an alternative to white- outputs box testing; it is a complementary approach. Both white-box and black-box techniques uncover different inputs events classes (types) of errors. 27 Black-Box Testing Environment Input Data (of specified data Output Data (of specified data types and structures) types and structures) Unit/module Code(with code) Events Example: Webster system Input: Alpha-numeric UserName and Password Output: a) BUID number (integer data type), b) CORRECT (Yes, Enter)/INCORRECT