DOCSLIB.ORG

Intro to Software Testing

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Introduction to Software Testing CS 3250 Software Testing [Ammann and Offutt, “Introduction to Software Testing,” Ch. 1, Ch. 2.1] Fall 2021 – University of Virginia © Praphamontripong 1 Bug? “ ‘Bug’ – as such little faults and difficulties are called – show themselves, and months of anxious watching, study, and labor are requisite before commercial success – or failure – is certainly reached.” [Thomas Edison, 1878] [Ref: Did You Know? Edison Coined the Term “Bug”, http://theinstitute.ieee.org/tech-history/technology-history/did- you-know-edison-coined-the-term-bug, IEEE 2013] “A software bug is an error, flaw, failure or fault in a computer program or system that causes it to produce an incorrect or unexpected result, or to behave in unintended ways.” [Ref: https://en.wikipedia.org/wiki/Software_bug] • “Bug” is used informally. BUG • Fault? Error? Or failure? • This course will try to use words that have precise, defined, and unambiguous meaning – and avoid using the term “bug” Fall 2021 – University of Virginia © Praphamontripong 2 Fault, Error, and Failure • Fault: a static defect in the software’s source code Cause of a problem – “fault location” • Error: An incorrect internal state that is the manifestation of some fault Erroneous program state caused by execution of the defect • Failure: External, incorrect behavior with respect to the requirements or other descriptions of the expected behavior Propagation of erroneous state to the program outputs Fall 2021 – University of Virginia © Praphamontripong 3 First Example (Testing) • Imagine we are testing a simply “add” functionality of a calculator program (e.g., the adder() function below) • The “add” functionality takes 2 numbers that the user enters • Each number should be one or two digits • The program displays the sum of the 2 numbers • How would you test the adder() function? What are test case inputs? How many test cases do we need? def adder(): n1 = int(input('Enter number1: ')) n2 = int(input('Enter number2: ')) return n1 + n2 end = 'n' while end != 'y': print('sum =', adder()) end = input('Enter \'y\' to end the program: ') Fall 2021 – University of Virginia © Praphamontripong 4 First Example (Debugging) How would you normally debug a program? Refer to “Rubber Duck Debugging” Activity on the schedule page There is a problem with patternIndex.java. It fails to produce the expected output. Rubber duck debugging (http://www.cs.virginia.edu/~up3f/cs3250/inclass/activity-debugging.html) Talk to me Fall 2021 – University of Virginia © Praphamontripong 5 What is Software Testing? • Testing = process of finding input values to check against a software (focus of this course) Test case consists of test values and expected results Goal – reveal faults • Debugging = process of finding a fault given a failure Test values Program Actual vs Expected (inputs) results results 1. Testing is fundamentally about choosing finite sets of values from the input domain of the software being tested 2. Given the test inputs, compare the actual results with the expected results Fall 2021 – University of Virginia © Praphamontripong 6 Testing Categories Static testing Dynamic testing • Testing without executing the • Testing by executing the program program with real inputs Software inspection and • Unit testing, integration some forms of analysis testing, system testing, Effective at finding certain acceptance testing, … kinds of problems such as problems that can lead to faults when the program is modified • Inspection, walkthrough, code review, informal review, … Fall 2021 – University of Virginia © Praphamontripong 7 Testing Categories (2) White-box Gray-box Black-box testing testing testing Te s t w i t h Te s t w i t h Test from the outside complete information incomplete information (functionality and of the software of the software or behavior). (architecture, how limited knowledge of Not require components interact, internal details source code, functions and (may know how architecture, detailed operations, code, components interact knowledge about rationale, …) but not have detailed internal program knowledge about functions, internal program rationale, …) functions, source code, rationale, …) Fall 2021 – University of Virginia © Praphamontripong 8 White-box vs. Black-box • Which one should we use? • Each technique has its strengths – use both Generally, first use black-box Then, white-box for missed code • Accept that we may not be able to detect all faults When to stop?? Adequacy à Coverage level Coverage criteria provides “stopping rule” Fall 2021 – University of Virginia © Praphamontripong 9 Testing Categories (3) Functional testing Non-functional testing • Unit testing • Performance testing • Integration testing • Load testing • System testing • Stress testing • Smoke testing • Security testing • Acceptance testing (Alpha / • Compatibility testing Beta • Reliability testing • Agile testing • Usability testing • Regressing testing • Compliance testing • Continuous integration • Conformance testing testing … and many more … … and many more … Fall 2021 – University of Virginia © Praphamontripong 10 Validation and Verification (IEEE) Validation Verification • Evaluate software at the end • Evaluate software at a given of software development phase of the development process • Ensure compliance with the intended usage • Fulfill the requirements established during the • Done by experts in the previous phase intended usage of the software, not developers • Requires technical background on the software • Done by developers at the various stages of development IV&V stands for “independent verification and validation” Fall 2021 – University of Virginia © Praphamontripong 11 Testing and SW Development Lifecycle Requirements Acceptance Check if software does what the user Analysis Te s t Te s t needs Design Information Check overall Architectural System behavior w.r.t. Design Te s t specs Check interface Subsystem Integration between modules Design Te s t in the same subsystem Check interactions Detailed Module of units and Design Te s t associated data structures Check each unit Unit Implementation (method) Te s t individually [based in part on AO, p.23] Fall 2021 – University of Virginia © Praphamontripong 12 Testing and SW Development Lifecycle Requirements Acceptance Validation Analysis Te s t Te s t Design Information Architectural System Design Te s t Subsystem Integration Design Te s t Verification Detailed Module Design Te s t Unit Implementation Te s t [based in part on AO, p.23] Fall 2021 – University of Virginia © Praphamontripong 13 Goals based on Test Process Maturity Beizer’s scale for test process maturity § Level 0: There is no difference between testing and debugging § Level 1: The purpose of testing is to show correctness § Level 2: The purpose of testing is to show that the software does not work § Level 3: The purpose of testing is not to prove anything specific, but to reduce the risk of using the software § Level 4: Testing is a mental discipline that helps all IT professionals develop higher quality software [AO, p.9] Fall 2021 – University of Virginia © Praphamontripong 14 Level 0 – Testing is Debugging § Level 0: Testing is the same as debugging § Not distinguish between incorrect behavior and mistakes in the program § Not help develop software that is reliable § Adopted by most CS students L § Get programs to compile § Debug with few arbitrarily chosen inputs or those provided by instructors [image: http://softwaretestingandqa.blogspot.com/2007/12/software-bug.html] Fall 2021 – University of Virginia © Praphamontripong 15 Levels 1 – Software Works § Level 1: To show correctness (developer-biased view) § Correctness is impossible to establish or demonstrate § What do we know if “no failures”? § Good software? § Bad tests? § No strict goal, no stopping rule or formal test technique § No quantitatively way to evaluate; test managers are powerless [image: http://simply-the-test.blogspot.com/2010/] Fall 2021 – University of Virginia © Praphamontripong 16 Levels 2 – Software Doesn’t Work § Level 2: To show failure (tester-biased view) § A negative view puts testers and developers into an adversarial relationship – bad team morale § What do we know if “no failures”? Good software? Bad tests? [image: http://simply-the-test.blogspot.com/2010/] Fall 2021 – University of Virginia © Praphamontripong 17 “Mature” Testing Correctness cannot generally be achieved or demonstrated through testing. Te s t in g can only show the presence of failure, not the absence. Developers and testers should be on the same boat. Fall 2021 – University of Virginia © Praphamontripong 18 Levels 3 – Risk Reduction § Level 3: To reduce the risk of using the software § There are risks when using software § Some may be small with unimportant consequences § Some may be big with important consequences, or even catastrophic § Testers and developers cooperate to reduce risk Fall 2021 – University of Virginia © Praphamontripong 19 Levels 4 – Quality Improvement § Level 4: To increase quality of the software § Testing should be an integral part of the development process § Testers become technical leaders, measuring and improving software quality § Help developers improve the ability to produce quality software § Train developers § Testers and developers cooperate to improve the quality § Example: Spell checker § Purpose: to improve our ability to spell § Change of mindset: “find misspelled words” à “improve ability to spell” Fall 2021 – University of Virginia © Praphamontripong 20 How “Mature”
Recommended publications
  • Studying the Feasibility and Importance of Software Testing: an Analysis

    Studying the Feasibility and Importance of Software Testing: an Analysis

    Dr. S.S.Riaz Ahamed / Internatinal Journal of Engineering Science and Technology Vol.1(3), 2009, 119-128 STUDYING THE FEASIBILITY AND IMPORTANCE OF SOFTWARE TESTING: AN ANALYSIS Dr.S.S.Riaz Ahamed Principal, Sathak Institute of Technology, Ramanathapuram,India. Email:[email protected], [email protected] ABSTRACT Software testing is a critical element of software quality assurance and represents the ultimate review of specification, design and coding. Software testing is the process of testing the functionality and correctness of software by running it. Software testing is usually performed for one of two reasons: defect detection, and reliability estimation. The problem of applying software testing to defect detection is that software can only suggest the presence of flaws, not their absence (unless the testing is exhaustive). The problem of applying software testing to reliability estimation is that the input distribution used for selecting test cases may be flawed. The key to software testing is trying to find the modes of failure - something that requires exhaustively testing the code on all possible inputs. Software Testing, depending on the testing method employed, can be implemented at any time in the development process. Keywords: verification and validation (V & V) 1 INTRODUCTION Testing is a set of activities that could be planned ahead and conducted systematically. The main objective of testing is to find an error by executing a program. The objective of testing is to check whether the designed software meets the customer specification. The Testing should fulfill the following criteria: ¾ Test should begin at the module level and work “outward” toward the integration of the entire computer based system.
  • Customer Success Story

    Customer Success Story

    Customer Success Story Interesting Dilemma, Critical Solution Lufthansa Cargo AG The purpose of Lufthansa Cargo AG’s SDB Lufthansa Cargo AG ordered the serves more than 500 destinations world- project was to provide consistent shipment development of SDB from Lufthansa data as an infrastructure for each phase of its Systems. However, functional and load wide with passenger and cargo aircraft shipping process. Consistent shipment data testing is performed at Lufthansa Cargo as well as trucking services. Lufthansa is is a prerequisite for Lufthansa Cargo AG to AG with a core team of six business one of the leaders in the international air efficiently and effectively plan and fulfill the analysts and technical architects, headed cargo industry, and prides itself on high transport of shipments. Without it, much is at by Project Manager, Michael Herrmann. stake. quality service. Herrmann determined that he had an In instances of irregularities caused by interesting dilemma: a need to develop inconsistent shipment data, they would central, stable, and optimal-performance experience additional costs due to extra services for different applications without handling efforts, additional work to correct affecting the various front ends that THE CHALLENGE accounting information, revenue loss, and were already in place or currently under poor feedback from customers. construction. Lufthansa owns and operates a fleet of 19 MD-11F aircrafts, and charters other freight- With such critical factors in mind, Lufthansa Functional testing needed to be performed Cargo AG determined that a well-tested API on services that were independent of any carrying planes. To continue its leadership was the best solution for its central shipment front ends, along with their related test in high quality air cargo services, Lufthansa database.
  • Security Testing

    Security Testing

    ISSN 1866-5705 www.testingexperience.com free digital version print version 8,00 € printed in Germany 6 The Magazine for Professional Testers The MagazineforProfessional Security Testing © iStockphoto/alexandru_lamba June, 2009 © iStockphoto/Manu1174 ISTQB® Certified Tester Foundation Level for only 499,- € plus VAT ONLINE TRAINING English & German www.testingexperience.learntesting.com Certified Tester Advanced Level coming soon Editorial Dear readers, One of my professors at the university said once to all of us: Computer scientists are at some point criminals. What he meant was that we or some of us – computer scientists – at some point like to try things that are not that “legal”. The most of us are “clean”, but some of us are “free time hackers”! Nowadays the hackers are almost away from the 17 years old guy, trying to pen- etrate in some website and so on. They are now adults, with families, cars, pets, holidays and a job. They are professionals earning money for acting as such. Application Security is not only important and essential for the companies and their businesses, technology and employees. Application Security is a macroeco- nomic aspect for the countries. There are a lot of secret services or governments agencies working on getting technology or information by advance hacking the server and databases of top companies or governments worldwide. When we hear that some countries could be behind the penetration of the USA electricity net- work, you can imagine what is going on outside. Are we testers prepared for that job? I’m not! Last year we had the first tutorial by Manu Cohen about Application Security Testing.
  • Software Testing

    Software Testing

    Software Testing PURPOSE OF TESTING CONTENTS I. Software Testing Background II. Software Error Case Studies 1. Disney Lion King 2. Intel Pentium Floating Point Division Bug 3. NASA Mars Polar Lander 4. Patriot Missile Defense System 5. Y2K Bug III. What is Bug? 1. Terms for Software Failure 2. Software Bug: A Formal Definition 3. Why do Bugs occur? and cost of bug. 4. What exactly does a Software Tester do? 5. What makes a good Software Tester? IV. Software Development Process 1. Product Components 2. What Effort Goes into a Software Product? 3. What parts make up a Software Product? 4. Software Project Staff V. Software Development Lifecycle Models 1. Big Bang Model 2. Code and Fix Model 3. Waterfall Model 4. Spiral Model VI. The Realities of Software Testing VII. Software Testing Terms and Definition 1. Precision and Accuracy 2. Verification and Validation 3. Quality Assurance and Quality Control Anuradha Bhatia Software Testing I. Software Testing Background 1. Software is a set of instructions to perform some task. 2. Software is used in many applications of the real world. 3. Some of the examples are Application software, such as word processors, firmware in an embedded system, middleware, which controls and co-ordinates distributed systems, system software such as operating systems, video games, and websites. 4. All of these applications need to run without any error and provide a quality service to the user of the application. 5. The software has to be tested for its accurate and correct working. Software Testing: Testing can be defined in simple words as “Performing Verification and Validation of the Software Product” for its correctness and accuracy of working.
  • Leading Practice: Test Strategy and Approach in Agile Projects

    Leading Practice: Test Strategy and Approach in Agile Projects

    CA SERVICES | LEADING PRACTICE Leading Practice: Test Strategy and Approach in Agile Projects Abstract This document provides best practices on how to strategize testing CA Project and Portfolio Management (CA PPM) in an agile project. The document does not include specific test cases; the list of test cases and steps for each test case are provided in a separate document. This document should be used by the agile project team that is planning the testing activities, and by end users who perform user acceptance testing (UAT). Concepts Concept Description Test Approach Defines testing strategy, roles and responsibilities of various team members, and test types. Testing Environments Outlines which testing is carried out in which environment. Testing Automation and Tools Addresses test management and automation tools required for test execution. Risk Analysis Defines the approach for risk identification and plans to mitigate risks as well as a contingency plan. Test Planning and Execution Defines the approach to plan the test cases, test scripts, and execution. Review and Approval Lists individuals who should review, approve and sign off on test results. Test Approach The test approach defines testing strategy, roles and responsibilities of various team members, and the test types. The first step is to define the testing strategy. It should describe how and when the testing will be conducted, who will do the testing, the type of testing being conducted, features being tested, environment(s) where the testing takes place, what testing tools are used, and how are defects tracked and managed. The testing strategy should be prepared by the agile core team.
  • Magma: a Ground-Truth Fuzzing Benchmark

    Magma: a Ground-Truth Fuzzing Benchmark

    Magma: A Ground-Truth Fuzzing Benchmark Ahmad Hazimeh Mathias Payer EPFL EPFL ABSTRACT fuzzer is evaluated against a set of target programs. These target pro- High scalability and low running costs have made fuzz testing grams can be sourced from a benchmark suite—such as the Cyber the de-facto standard for discovering software bugs. Fuzzing tech- Grand Challenge [9], LAVA-M[12], the Juliet Test Suite [30], and niques are constantly being improved in a race to build the ultimate Google’s Fuzzer Test Suite [17], oss-fuzz [2], and FuzzBench [16]— bug-finding tool. However, while fuzzing excels at finding bugs, or from a set of real-world programs [34]. Performance metrics— comparing fuzzer performance is challenging due to the lack of including coverage profiles, crash counts, and bug counts—are then metrics and benchmarks. Crash count, the most common perfor- collected during the evaluation. Unfortunately, while such metrics mance metric, is inaccurate due to imperfections in de-duplication can provide insight into a fuzzer’s performance, they are often methods and heuristics. Moreover, the lack of a unified set of targets insufficient to compare different fuzzers. Moreover, the lackofa results in ad hoc evaluations that inhibit fair comparison. unified set of target programs makes for unfounded comparisons. We tackle these problems by developing Magma, a ground-truth Most fuzzer evaluations consider crash count a reasonable metric fuzzer evaluation framework enabling uniform evaluations and for assessing fuzzer performance. However, crash count is often comparison. By introducing real bugs into real software, Magma inflated [25], even after attempts at de-duplication (e.g., via coverage allows for a realistic evaluation of fuzzers against a broad set of profiles or stack hashes), highlighting the need for more accurate targets.
  • Functional Testing Functional Testing

    Functional Testing Functional Testing

    From Pressman, “Software Engineering – a practitionerʼs approach”, Chapter 14 and Pezze + Young, “Software Testing and Analysis”, Chapters 10-11 Today, weʼll talk about testing – how to test software. The question is: How do we design tests? And weʼll start with Functional Testing functional testing. Software Engineering Andreas Zeller • Saarland University 1 Functional testing is also called “black- box” testing, because we see the program as a black box – that is, we ignore how it is being written 2 in contrast to structural or “white-box” testing, where the program is the base. 3 If the program is not the base, then what is? Simple: itʼs the specification. 4 If the program is not the base, then what is? Simple: itʼs the specification. Testing Tactics Functional Structural “black box” “white box” • Tests based on spec • Tests based on code • Test covers as much • Test covers as much specified behavior implemented behavior as possible as possible 5 Why Functional? Functional Structural “black box” “white box” • Program code not necessary • Early functional test design has benefits reveals spec problems • assesses testability • gives additional explanation of spec • may even serve as spec, as in XP 6 Structural testing can not detect that some required feature is missing in the code Why Functional? Functional testing applies at all granularity levels (in contrast to structural testing, which only applies to Functional Structural unit and integration testing) “black box” “white box” • Best for missing logic defects Common problem: Some program logic was simply forgotten Structural testing would not focus on code that is not there • Applies at all granularity levels unit tests • integration tests • system tests • regression tests 7 2,510,588,971 years, 32 days, and 20 hours to be precise.
  • HP Functional Testing Software Data Sheet

    HP Functional Testing Software Data Sheet

    HP Functional Testing software Data sheet With HP Functional Testing you can automate functional and regression testing for every modern software application and environment, extend testing to a wider range of teams, and accelerate the testing process—so you can improve application quality and still make your market window. Simplifies test creation and HP Functional Testing makes it easy to insert, modify, data-drive, and remove test steps. It features: maintenance • Keyword capabilities: Using keywords, testers HP Functional Testing is advanced, automated can build test cases by capturing flows directly testing software for building functional and regression from the application screens and applying robust test suites. It captures, verifies, and replays user record/replay capturing technology. interactions automatically and helps testers quickly • Automatic updating: With new application builds, identify and report on application effects, while you only need to update one reference in the shared providing sophisticated functionality for tester repository and the update is propagated to all collaboration. The product includes HP QuickTest referencing tests. Professional and all of its add-ins. It is sold stand-alone or as part of the broader HP Unified • Easy data-driving: You can quickly data-drive any Functional Testing solution, which couples object definition, method, checkpoint, and output HP Functional Testing with HP Service Test to value through the integrated data table. address both GUI and non-GUI testing. • Timely advice: In
  • Managing Defects in an Agile Environment

    Managing Defects in an Agile Environment

    Managing Defects in an Agile environment Auteur Ron Eringa Managing Defects in an Agile environment Introduction Injecting passion, Agility Teams often struggle with answering the following and quality into your question: “How to manage our Defects in an organisation. Agile environment?”. They start using Scrum as a framework for developing their software and while implementing, they experience trouble on how to deal with the Defects they find/cause along the way. Scrum is a framework that does not explicitly tell you how to handle Defects. The strait forward answer is to treat your Defects as Product Backlog Items that should be added to the Product Backlog. When the priority is set high enough by the Product Owner, they will be picked up by the Development Team in the next Sprint. The application of this is a little bit more difficult and hence should be explained in more detail. RON ERINGA 1. What is a defect? AGILE COACH Wikipedia: “A software bug (or defect) is an error, flaw, After being graduated from the Fontys failure, or fault in a computer program or system that University in Eindhoven, I worked as a Software produces an incorrect or unexpected result, or causes Engineer/Designer for ten years. Although I it to behave in unintended ways. Most bugs arise have always enjoyed technics, helping people from mistakes and errors made by people in either a and organizations are my passion. Especially program’s source code or its design, or in frameworks to deliver better quality together. When people and operating systems used by such programs, and focus towards a common goal, interaction is a few are caused by compilers producing incorrect increasing and energy is released.
  • Devsecops DEVELOPMENT & DEVOPS INFRASTRUCTURE

    Devsecops DEVELOPMENT & DEVOPS INFRASTRUCTURE

    DevSecOps DEVELOPMENT & DEVOPS INFRASTRUCTURE CREATE SECURE APPLICATIONS PARASOFT’S APPROACH - BUILD SECURITY IN WITHOUT DISRUPTING THE Parasoft provides tools that help teams begin their security efforts as DEVELOPMENT PROCESS soon as the code is written, starting with static application security test- ing (SAST) via static code analysis, continuing through testing as part of Parasoft makes DevSecOps possible with API and the CI/CD system via dynamic application security testing (DAST) such functional testing, service virtualization, and the as functional testing, penetration testing, API testing, and supporting in- most complete support for important security stan- frastructure like service virtualization that enables security testing be- dards like CWE, OWASP, and CERT in the industry. fore the complete application is fully available. IMPLEMENT A SECURE CODING LIFECYCLE Relying on security specialists alone prevents the entire DevSecOps team from securing software and systems. Parasoft tooling enables the BENEFIT FROM THE team with security knowledge and training to reduce dependence on PARASOFT APPROACH security specialists alone. With a centralized SAST policy based on in- dustry standards, teams can leverage Parasoft’s comprehensive docs, examples, and embedded training while the code is being developed. ✓ Leverage your existing test efforts for Then, leverage existing functional/API tests to enhance the creation of security security tests – meaning less upfront cost, as well as less maintenance along the way. ✓ Combine quality and security to fully understand your software HARDEN THE CODE (“BUILD SECURITY IN”) Getting ahead of application security means moving beyond just test- ✓ Harden the code – don’t just look for ing into building secure software in the first place.
  • Continuous Quality and Testing to Accelerate Application Development

    Continuous Quality and Testing to Accelerate Application Development

    Continuous Quality and Testing to Accelerate Application Development How to assess your current testing maturity level and practice continuous testing for DevOps Continuous Quality and Testing to Accelerate Application Development // 1 Table of Contents 03 Introduction 04 Why Is Continuous Quality and Testing Maturity Important to DevOps? 05 Continuous Testing Engineers Quality into DevOps 07 Best Practices for Well- Engineered Continuous Testing 08 Continuous Testing Maturity Levels Level 1: Chaos Level 2: Continuous Integration Level 3: Continuous Flow Level 4: Continuous Feedback Level 5: Continuous Improvement 12 Continuous Testing Maturity Assessment 13 How to Get Started with DevOps Testing? 14 Continuous Testing in the Cloud Choosing the right tools for Continuous Testing On-demand Development and Testing Environments with Infrastructure as Code The Right Tests at the Right Time 20 Get Started 20 Conclusion 21 About AWS Marketplace and DevOps Institute 21 Contributors Introduction A successful DevOps implementation reduces the bottlenecks related to testing. These bottlenecks include finding and setting up test environments, test configurations, and test results implementation. These issues are not industry specific. They can be experienced in manufacturing, service businesses, and governments alike. They can be reduced by having a thorough understanding and a disciplined, mature implementation of Continuous Testing and related recommended engineering practices. The best place to start addressing these challenges is having a good understanding of what Continuous Testing is. Marc Hornbeek, the author of Engineering DevOps, describes it as: “A quality assessment strategy in which most tests are automated and integrated as a core and essential part of DevOps. Continuous Testing is much more than simply ‘automating tests.’” In this whitepaper, we’ll address the best practices you can adopt for implementing Continuous Quality and Testing on the AWS Cloud environment in the context of the DevOps model.
  • A Study of Static Bug Detectors

    A Study of Static Bug Detectors

    How Many of All Bugs Do We Find? A Study of Static Bug Detectors Andrew Habib Michael Pradel [email protected] [email protected] Department of Computer Science Department of Computer Science TU Darmstadt TU Darmstadt Germany Germany ABSTRACT International Conference on Automated Software Engineering (ASE ’18), Sep- Static bug detectors are becoming increasingly popular and are tember 3–7, 2018, Montpellier, France. ACM, New York, NY, USA, 12 pages. widely used by professional software developers. While most work https://doi.org/10.1145/3238147.3238213 on bug detectors focuses on whether they find bugs at all, and on how many false positives they report in addition to legitimate 1 INTRODUCTION warnings, the inverse question is often neglected: How many of all Finding software bugs is an important but difficult task. For average real-world bugs do static bug detectors find? This paper addresses industry code, the number of bugs per 1,000 lines of code has been this question by studying the results of applying three widely used estimated to range between 0.5 and 25 [21]. Even after years of static bug detectors to an extended version of the Defects4J dataset deployment, software still contains unnoticed bugs. For example, that consists of 15 Java projects with 594 known bugs. To decide studies of the Linux kernel show that the average bug remains in which of these bugs the tools detect, we use a novel methodology the kernel for a surprisingly long period of 1.5 to 1.8 years [8, 24]. that combines an automatic analysis of warnings and bugs with a Unfortunately, a single bug can cause serious harm, even if it has manual validation of each candidate of a detected bug.