Introduction to Software Testing CS 3250 Software Testing [Ammann and Offutt, “Introduction to Software Testing,” Ch. 1, Ch. 2.1] Fall 2021 – University of Virginia © Praphamontripong 1 Bug? “ ‘Bug’ – as such little faults and difficulties are called – show themselves, and months of anxious watching, study, and labor are requisite before commercial success – or failure – is certainly reached.” [Thomas Edison, 1878] [Ref: Did You Know? Edison Coined the Term “Bug”, http://theinstitute.ieee.org/tech-history/technology-history/did- you-know-edison-coined-the-term-bug, IEEE 2013] “A software bug is an error, flaw, failure or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways.” [Ref: https://en.wikipedia.org/wiki/Software_bug] • “Bug” is used informally. BUG • Fault? Error? Or failure? • This course will try to use words that have precise, defined, and unambiguous meaning – and avoid using the term “bug” Fall 2021 – University of Virginia © Praphamontripong 2 Fault, Error, and Failure • Fault: a static defect in the software’s source code Cause of a problem – “fault location” • Error: An incorrect internal state that is the manifestation of some fault Erroneous program state caused by execution of the defect • Failure: External, incorrect behavior with respect to the requirements or other descriptions of the expected behavior Propagation of erroneous state to the program outputs Fall 2021 – University of Virginia © Praphamontripong 3 First Example (Testing) • Imagine we are testing a simply “add” functionality of a calculator program (e.g., the adder() function below) • The “add” functionality takes 2 numbers that the user enters • Each number should be one or two digits • The program displays the sum of the 2 numbers • How would you test the adder() function? What are test case inputs? How many test cases do we need? def adder(): n1 = int(input('Enter number1: ')) n2 = int(input('Enter number2: ')) return n1 + n2 end = 'n' while end != 'y': print('sum =', adder()) end = input('Enter \'y\' to end the program: ') Fall 2021 – University of Virginia © Praphamontripong 4 First Example (Debugging) How would you normally debug a program? Refer to “Rubber Duck Debugging” Activity on the schedule page There is a problem with patternIndex.java. It fails to produce the expected output. Rubber duck debugging (http://www.cs.virginia.edu/~up3f/cs3250/inclass/activity-debugging.html) Talk to me Fall 2021 – University of Virginia © Praphamontripong 5 What is Software Testing? • Testing = process of finding input values to check against a software (focus of this course) Test case consists of test values and expected results Goal – reveal faults • Debugging = process of finding a fault given a failure Test values Program Actual vs Expected (inputs) results results 1. Testing is fundamentally about choosing finite sets of values from the input domain of the software being tested 2. Given the test inputs, compare the actual results with the expected results Fall 2021 – University of Virginia © Praphamontripong 6 Testing Categories Static testing Dynamic testing • Testing without executing the • Testing by executing the program program with real inputs Software inspection and • Unit testing, integration some forms of analysis testing, system testing, Effective at finding certain acceptance testing, … kinds of problems such as problems that can lead to faults when the program is modified • Inspection, walkthrough, code review, informal review, … Fall 2021 – University of Virginia © Praphamontripong 7 Testing Categories (2) White-box Gray-box Black-box testing testing testing Te s t w i t h Te s t w i t h Test from the outside complete information incomplete information (functionality and of the software of the software or behavior). (architecture, how limited knowledge of Not require components interact, internal details source code, functions and (may know how architecture, detailed operations, code, components interact knowledge about rationale, …) but not have detailed internal program knowledge about functions, internal program rationale, …) functions, source code, rationale, …) Fall 2021 – University of Virginia © Praphamontripong 8 White-box vs. Black-box • Which one should we use? • Each technique has its strengths – use both Generally, first use black-box Then, white-box for missed code • Accept that we may not be able to detect all faults When to stop?? Adequacy à Coverage level Coverage criteria provides “stopping rule” Fall 2021 – University of Virginia © Praphamontripong 9 Testing Categories (3) Functional testing Non-functional testing • Unit testing • Performance testing • Integration testing • Load testing • System testing • Stress testing • Smoke testing • Security testing • Acceptance testing (Alpha / • Compatibility testing Beta • Reliability testing • Agile testing • Usability testing • Regressing testing • Compliance testing • Continuous integration • Conformance testing testing … and many more … … and many more … Fall 2021 – University of Virginia © Praphamontripong 10 Validation and Verification (IEEE) Validation Verification • Evaluate software at the end • Evaluate software at a given of software development phase of the development process • Ensure compliance with the intended usage • Fulfill the requirements established during the • Done by experts in the previous phase intended usage of the software, not developers • Requires technical background on the software • Done by developers at the various stages of development IV&V stands for “independent verification and validation” Fall 2021 – University of Virginia © Praphamontripong 11 Testing and SW Development Lifecycle Requirements Acceptance Check if software does what the user Analysis Te s t Te s t needs Design Information Check overall Architectural System behavior w.r.t. Design Te s t specs Check interface Subsystem Integration between modules Design Te s t in the same subsystem Check interactions Detailed Module of units and Design Te s t associated data structures Check each unit Unit Implementation (method) Te s t individually [based in part on AO, p.23] Fall 2021 – University of Virginia © Praphamontripong 12 Testing and SW Development Lifecycle Requirements Acceptance Validation Analysis Te s t Te s t Design Information Architectural System Design Te s t Subsystem Integration Design Te s t Verification Detailed Module Design Te s t Unit Implementation Te s t [based in part on AO, p.23] Fall 2021 – University of Virginia © Praphamontripong 13 Goals based on Test Process Maturity Beizer’s scale for test process maturity § Level 0: There is no difference between testing and debugging § Level 1: The purpose of testing is to show correctness § Level 2: The purpose of testing is to show that the software does not work § Level 3: The purpose of testing is not to prove anything specific, but to reduce the risk of using the software § Level 4: Testing is a mental discipline that helps all IT professionals develop higher quality software [AO, p.9] Fall 2021 – University of Virginia © Praphamontripong 14 Level 0 – Testing is Debugging § Level 0: Testing is the same as debugging § Not distinguish between incorrect behavior and mistakes in the program § Not help develop software that is reliable § Adopted by most CS students L § Get programs to compile § Debug with few arbitrarily chosen inputs or those provided by instructors [image: http://softwaretestingandqa.blogspot.com/2007/12/software-bug.html] Fall 2021 – University of Virginia © Praphamontripong 15 Levels 1 – Software Works § Level 1: To show correctness (developer-biased view) § Correctness is impossible to establish or demonstrate § What do we know if “no failures”? § Good software? § Bad tests? § No strict goal, no stopping rule or formal test technique § No quantitatively way to evaluate; test managers are powerless [image: http://simply-the-test.blogspot.com/2010/] Fall 2021 – University of Virginia © Praphamontripong 16 Levels 2 – Software Doesn’t Work § Level 2: To show failure (tester-biased view) § A negative view puts testers and developers into an adversarial relationship – bad team morale § What do we know if “no failures”? Good software? Bad tests? [image: http://simply-the-test.blogspot.com/2010/] Fall 2021 – University of Virginia © Praphamontripong 17 “Mature” Testing Correctness cannot generally be achieved or demonstrated through testing. Te s t in g can only show the presence of failure, not the absence. Developers and testers should be on the same boat. Fall 2021 – University of Virginia © Praphamontripong 18 Levels 3 – Risk Reduction § Level 3: To reduce the risk of using the software § There are risks when using software § Some may be small with unimportant consequences § Some may be big with important consequences, or even catastrophic § Testers and developers cooperate to reduce risk Fall 2021 – University of Virginia © Praphamontripong 19 Levels 4 – Quality Improvement § Level 4: To increase quality of the software § Testing should be an integral part of the development process § Testers become technical leaders, measuring and improving software quality § Help developers improve the ability to produce quality software § Train developers § Testers and developers cooperate to improve the quality § Example: Spell checker § Purpose: to improve our ability to spell § Change of mindset: “find misspelled words” à “improve ability to spell” Fall 2021 – University of Virginia © Praphamontripong 20 How “Mature”