Software Testing Overview Software Testing Overview: Dr. Andrea Arcuri Part I Simu la R esearc h La boratory Oslo,,y Norway [email protected] Based on slides provided by Prof. Lionel Briand 1 2 Lionel Briand 2009 Lionel Briand 2009 Software has become prevalent in all aspects of our lives Qualities of Software Products • Correctness • Repairability • Reliability • Evolvability • Robustness • Reusability • Performance • Portability • User Friendliness • Understandability • Verifiability • Interoperability • Maintainability 3 4 Lionel Briand 2009 Lionel Briand 2009 Pervasive Problems Examples of Software Failures • Communications: Loss or corruption of • Software is commonly delivered late, way over communication media, non delivery of data. budgg,et, and of unsatisfactor yqy quality • Space Applications: Lost lives , launch delays , e .g ., • Software validation and verification are rarely European Ariane 5 shuttle, 1996: systematic and are usually not based on sound, – From the official disaster report: “Due to a well-defined techniques malfunction in the control software, the rocket veered off its flight path 37 seconds after • Software development processes are commonly launch.” unstable an d uncontro lled • Defense and Warfare: Misidentification of friend or • Software quality is poorly measured, monitored, foe. andtlldd controlled. • Transportation: Deaths, delays, sudden acceleration, • Software failure examples: inability to brake. http://www. cs. bc. edu/~gtan/bug/softwarebug. html • Electric Power: Death , injuries , power outages , long-term health hazards (radiation). 5 6 Lionel Briand 2009 Lionel Briand 2009 Examples of Software Failures Ariane 5 – ESA (()cont.) • Money Management: Fraud, violation of privacy, shutdown of stock exchanges and banks, negative interest rates . • Control of Elections: Wrong results (intentional or non- On June 4, 1996, the flight of the intentional). Ariane 5 launcher ended in a • Control of Jails: Technology-aided escape attempts and successes, failure. failures in software-controlled locks. • Law Enforcement: False arrests and imprisonments . Only about 40 seconds after initiation of the flight sequence, at an altitude of about 3,700 m, the launcher veered off its flight path, broke up and exploded . 7 8 Lionel Briand 2009 Lionel Briand 2009 Ariane 5 – Root Cause Ariane 5 – Lessons Learned • Source: ARIANE 5 Fligg,pyqyht 501 Failure, Report by the Inquiry Board • Adequate exception handling and redundancy strategies A program segment for converting a floating point number to a (real function of a backup system, degraded modes?) signe d 16 bit i n teger was execut ed with an i nput d at a val ue out sid e the range representable by a signed 16-bit integer. • Clear, complete, documented specifications (e.g., This run time error (out of range , overflow) , which arose in both preconditions, post-conditions) the active and the backup computers at about the same time, was • But perhaps more importantly: usage-based testing detected and both computers shut themselves down. (based on operational profiles), in this case actual This resulted in the total loss of attitude control. The Ariane 5 Ariane 5 trajectories turned uncontrollably and aerodynamic forces broke the vehicle apart. • Note this was not a complex, computing problem, but a deficiency of the software engineering practices in place This breakup was detected by an on-board monitor which ignited the explosive charges to destroy the vehicle in the air. Ironically, … the result of this format conversion was no longer needed after lift 9 10 off. Lionel Briand 2009 Lionel Briand 2009 F-18 cras h Fatal Therac-25 Radiation • An F-18 crashed because of a missing exception condition: An if ... then ... block without the else clause that was • In 1986, a man in Texas received between 16,500- thought could not possibly arise. 25, 000 radiations in less than 10 sec , over an area of about 1 cm. • In simulation, an F-16 program bug caused the virtual plane to flip over whenever it crossed the equator, as a • He lost his left arm, and died of complications 5 result of a missing minus sign to indicate south latitude. months later. 11 12 Lionel Briand 2009 Lionel Briand 2009 Power Shutdown in 2003 Consequences of Poor Quality 508 generating units and 256 power plants shut down • Standish Group surveyed 350 companies, over 8000 projects, in 1994 Affected 10 million people in Ontario, • 31% cancelled before completed, 9-16% were delivered Canada within cost and budget Affected 40 million • US study (1995): 81 billion US$ spend per year for failing people in 8 US software development projects states • NIST stu dy (2002) : bugs cost $ 59. 5 billion a year. Ear lier Financial losses of detection could save $22 billion. $6 Billion USD The alarm system in the energy management system failed due to a software error and operators were not informed of the power overload in the system 13 14 Lionel Briand 2009 Lionel Briand 2009 Quality Assurance Dealing with SW Faults • Uncover faults in the documents where they are FltHdliFault Handling introduced, in a systematic way, in order to avoid ripple effects. Systematic, structured reviews of software FltTlFault Tolerance dtfdtdocuments are referred to as itiinspections. Fault Avoidance Fault Detection • Derive, in a systematic way, effective test cases to uncover faults Design Atomic Modular Inspections • Automate testing and inspection activities, to the Methodology Transactions Redundancy maximum extent possible Configuration Verification • Monitor and control quality, e.g., reliability, Management maintainability, safety, across all project phases and Testing Debugging activities • All this implies the quality measurement of SW products and processes Component Integration System Correctness Performance Testing Testing Testing Debugging Debugging 15 16 Lionel Briand 2009 Lionel Briand 2009 Testing Definition Basic Testing Definition • Errors: People commit errors • SW Testing: Techniques to execute programs • Fault: A fault is the result of an error in the software with the intent of finding as many defects as docu mentation, code , etc . possible and/or gaining sufficient confidence • Failure: A failure occurs when a fault executes • Many people use the above three terms inter -changeably. It in the software system under test. should be avoided – “Program testing can show the presence of • Incident: Consequences of failures – Failure occurrence bugs, never their absence” (Dijkstra) may or may not be apparent to the user • The fundamental chain of SW dependability threats: propagation causation results in Error Fault Failure Incident ... 17 18 Lionel Briand 2009 Lionel Briand 2009 Why is SW testing important? • According to some estimates: ~50% of development costs Testing • A study by (the American) NIST in 2002: – The annual national cost of inadequate testing is as Definitions & Objectives much as $59 Billion US! – The report i s t it led : “The E conomi c I mpacts of Inadequate Infrastructure for Software Testing” 19 20 Lionel Briand 2009 Lionel Briand 2009 Test Stubs and Drivers Summary of Definitions •Test Stub: Partial implementation of a component on which a unit under test Test suite depends. Test S tub exercises is re vised by Depends * * 1…n Component a Component b Test case Component Correction U nder T es t * * * Test stub finds • Test Driver: Partial implementation of a component that depends on a unit under repairs test. * Test Driver Test driver Depends C om ponent j C om ponent k * * Failure * * Fault ** Error Under Test is caused by is caused by • Test stubs and drivers enable components to be isolated from the rest of the system for testing. 21 22 Lionel Briand 2009 Lionel Briand 2009 Motivations The Testing Dilemma Available • No matter how rigorous we are, software is going testing to be faulty Limited resources resources All Software System • Testing represent a functionality substantial percentage of expertis Time Peopl e software development Money Potentially costs and time to market e • Impossible to test under thousands all operating conditions – bdbased on incomp ltlete of items testing, we must gain confidence that the system to test has the des ired be hav ior o • Testing large systems is complex – it requires strategy and technology- and is often done inefficiently in practice 23 Faulty functionality 24 Lionel Briand 2009 Lionel Briand 2009 TtiPTesting Process Overvi ew Qualities of Testing SW Representation (e.g., models, requirements) • Effective at uncovering faults DiDerive TtTest cases Estimate •Help locate faults for debugging Expected SW Code • Repeatable so that a precise understanding Results Execute Test cases of the fault can be gained Get Test Results • Automated so as to lower the cost and Test Oracle Compare timescale [Test Result==Oracle] [Test Result!=Oracle] • SiSystematic so as to bdiblibe predictable in terms of its effect on dependability 25 26 Lionel Briand 2009 Lionel Briand 2009 Subtleties of Software Continuity Property • Problem: Test a bridge ability to sustain a Dependability certain weight • Continuity Property: If a bridge can sustain a • Dependability: Correctness, reliability, safety, weight equal to W1, then it will sustain any robustness weight W2 <= W1 • A program is correct if it obeys its specification. • EtilltiittEssentially, continuity property= small differences in operating conditions should not • Reliability is a way of statistically approximating result in dramatically different behavior correctness. • Safety