CH09: Testing the System More System Tests • to ensure the system does what the customer wants it • * to do: • * Installation Testing • * Principles of • * Automated System Testing • * Function Testing • * Test Documentation • * Performance Testing • * Testing Safety-critical systems • * Reliability, Availability, and Maintainability

TECH Computer Science

Principles of System Testing Sources of Software Faults Requirement Incorrect, missing or unclear requirements • Sources of Software Faults s Incorrect or unclear translation analysis System • System Testing Process Incorrect or unclear design specification design • Process Objectives Program Incorrect or unclear design specification • Configuration Management design Misinterpretation of system design • Test Team Program Misinterpretation of program design implementation Incorrect documentation Incorrect syntax or semantics Unit/integration Incomplete test procedures testing New faults introduced when old ones correcte

System Incomplete test procedures testing Incorrect user documentation Maintenance Poor human factors New faults introduced when old one corrected Changes in requirements

System Testing Process Process Objectives System Other Customer User • A function test checks that the integrated system functional software requirementsenvironment requirements requirementsspecification performs its functions as specified in the requirements. • A performance test compares the integrated components with the nonfunctional system requirements, Integrated Function Performance Acceptance Installation SYSTEM 4such as security, accuracy, speed, and reliability modules test test test test IN USE! Functioning Verified, Accepted system validated system software Verified vs. Validated System Tests for Customers • A verified system meets the requirement • An acceptance test assures the system meet the specifications (designer’s view). customers’ satisfaction. • A validated system meets the requirement definitions • An installation test assures the system is installed (customer’s view). correctly and working at the actual customer’s hardware.

Configuration Management Version and Release • A system configuration is a collection of system • A configuration for a particular system is sometimes components delivered to a particular customer or called a version. operating system. • A new release of the software is an improved (?) • Configuration management helps us keep track on system intended to replace the old one. the difference system configurations. • A regression test is a test applied to a new version or 4Also keep track of change: called change control release to verify that it still performs the same functions in the same manner as an older one.

Tracking the Changes: Deltas, Separate files, and Conditional compilation Test Team • Separate files: Keep separate files for each different • Everybody needs to be involved. version or release. • Developers usually do unit and . • Deltas: Keep one main version. Other versions are • Testers (testing group) do functional and stored as differences from the main version. The performance tests. difference file is called a delta. • Testers will ask analysts or front-end people to clarify • Conditional compilation: One single code addresses requirements. all version. Use compiler to determine which • Users, technical writers, ... statements apply to which versions. Function Testing Performance Testing • Test what the system is supposed to do based on the • addresses nonfunctional requirements. system’s functional requirements. Test should: • Types include: 4have a high probability of detecting a fault 4Stress tests, Volume tests -- users and data 4use a test team independent of the designers and 4Configuration test -- software and hardware programmers configurations 4know the expected actions and output 4Compatibility test -- between systems 4test both valid and invalid input 4Regression tests -- between versions 4never modify the system just to make testing easier 4Security test 4have stopping criteria

Lot more performance tests Reliability, Availability, and Maintainability 4Timing test -- respond time • Definitions 4Environmental test -- perform at installation site • Failure Data 4Quality tests -- reliability, availability • Measuring Reliability, Availability, and 4Recovery tests -- restart Maintainability 4Maintenance tests -- diagnostic tools and tracing • Reliability Stability and Growth 4documentation tests -- e.g. follow the installation guide 4Human factors test, usability tests. • Reliability Prediction • Importance of the operational Environment

Definitions More formal definitions • Reliability involves behavior over a period of time. 4Software reliability is the probability that a system will operate without failure under given condition for a give • Availability describes something at a given point in time interval. time. 4Software availability is the probability that a system is • Maintainability describes what need to be done to operating successfully according to specification at a keep the system functioning. given point in time. 4Software maintainability is the probability that a maintenance activity can be carried out within a stated time interval and using stated procedures and resources. Failure Data MTTF, MTTR, MTBF • Reliability, availability, and maintainability are • Mean time to failure (MTTF) is the average value of measured based on the working history of a complete inter-failure times. system. • Mean time to repair (MTTR) is the average value of • Failure data must be kept to allow the measurements: time taking to fix each failure. 4time between each failure (“inter-failure time”) • Mean time between failures (MTBF): 4time for each maintenance • MTBF = MTTF + MTTR

Measuring Reliability, Availability, and Maintainability Reliability Stability and Growth • Reliability = MTBF/(1+MTBF) • tell us whether the software is improving • Availability = MTBF/(MTBF + MTTR) • If the inter-failure times stay the same, • Maintainability = 1/(1 + MTTR) • then we have reliability stability!!! • If the inter-failure times increase, • then we have reliability growth (getting better)!!!

Reliability Prediction Importance of the operational Environment • Use historical information about failures to build a • Capture operational profile that describes likely user predictive models of reliability. input over time. • = assuming the change in system behavior is the same • e.g. % of create, % of delete, or % of modify used by by fixing one fault as by fixing another a user • - fixing one fault might introduce a lot more fault • (statistical testing) Test cases are created to test them (predictive models can not take account of this according to the %. problem) 4Testing concentrates on the parts of the system most likely to be used 4Reliability from these test gives the reliability as seen by the user. Acceptance Testing Alpha and Beta tests • Now the customer leads testing and defines the cases • In house test is called an alpha test. to be tested. • The customer’s pilot test is called beta test. • test uses a set of test cases that represent • Parallel testing: the new system operates in parallel typical conditions of usage. with the previous version. (Something to fall back to • Pilot test installs the system on an experimental basis in case the new one does not work!) and rely on the everyday working of the system to test all functions.

Installation Testing Automated System Testing • Final round of testing! • Simulator presents to a system all characteristics of a 4involves installing the system at customer’s sites. device or system without actually having the device • After installation, run regression test (most important or system available. test cases) to insure the software is working “in the • Sometimes a device simulator is more helpful than field”. the device itself. • When the customer is satisfied with the results, • Record and Play-back testing tools to simulate users. testing is complete and the system is formally delivered. • Done for now!!!

Test Documentation Test Analysis Report • Test Plans (covered earier) • When a test has been administered, we analyze the • Test Analysis Report results to determine if the function or performance • Problem Report Forms tested meets the requirements. Anaysis Report: 4documets the results of tests 4if a failure occurs, it provides information needed to duplicate the fauilure and to locate and fix the source of the problem. 4provides info to determine if the probject is complete 4establishes confidnece in the system’s performance Problem Report Forms Discrepancy report form • capture data about faults and failures in problem DISCREPANCY REPORT FORM DRF Number:______Tester name:______report forms Date: ______Time: ______Test Number: ______• discrepancy report form (it is called MR Script step executed when failure occurred: ______Description of failure: ______(modification request) in Lucent) ______4is a problem report that describes occurrences of ______Activities before occurrence of failure: problems where actual system behaviors or attributes ______do not match with what we expect. Expected results: ______• fault report form explains how a fault was found ______Requirements affected: and fixed, often in response to filing a discrepancy ______report form. Effect of failure on test: ______Effect of failure on system: ______Severity level: (LOW)12345 (HIGH)

FAULT REPORT S.P0204.6.10.3016

FaultORIGINATOR: report Joe form Bloggs Testing Safety-critical systems BRIEF TITLE: Exception 1 in dps_c.c line 620 raised by NAS FULL DESCRIPTION Started NAS endurance and allowed it to run for a few minutes. Disabled the active NAS • Safety-critical systems: failure of the system can link (emulator switched to standby link), then re-enabled the disabled link and CDIS exceptioned as above. (I think the re-enabling is a red herring.) (during database load) harm or kill people! ASSIGNED FOR EVALUATION TO: DATE: • Ultrahigh reliability system: has at most one failure in

CATEGORISATION: 0 1 2 3 Design Spec Docn SEND COPIES FOR INFORMATION TO: 10^9 hours! 8/7/92 EVALUATOR: DATE: 4i.e. the system can fail at most once in over 100,000 CONFIGURATION ID ASSIGNED TO PART dpo_s.c years of operation. • Problem: if a program has worked failure-free for x COMMENTS: dpo_s.c appears to try to use an invalid CID, instead of rejecting the message. AWJ hours, there is about a 50:50 chance that it will ITEMS CHANGED survive the next x hours before failing! CONFIGURATION ID IMPLEMENTOR/DATE REVIEWER/DATE BUILD/ISSUE NUM INTEGRATOR/DATE dpo_s.c v.10 AWJ 8/7/92 MAR 8/7/92 6.120 RA 8-7-92

COMMENTS:

CLOSED FAULT CONTROLLER: DATE: 9/7/92

Methods to help understand and assure reliability Design Diversity • Design Diversity • built software using same requirements specifications, • Software Safety Cases • but in several independent different designs to form • Cleanroom several independent systems. • Each system runs in parallel and • a voting scheme coordinates actions when one system’s results differ from the others’ • e.g. U.S. space shuttle Software Safety Cases Cleanroom • assign failure rates or constraints to each component • address two principles: of the system 4to certify the software with respect to the specifications • then estimate the failure rates or constraints of the 4to produce zero-fault or near-zero-fault software entire system • use formal proofs to certified with respect to • e.g. use fault-tree analysis specifications 4Not • use statistical usage testing to determine the expected MTTF and other quality measure.