DOCSLIB.ORG

The Advantages of Post-Link Code Coverage

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
The Advantages of Post-Link Code Coverage The Advantages of Post-link Code Coverage Orna Raz, Moshe Klausner, Nitzan Peleg, Gad Haber, Eitan Farchi, Shachar Fienblit, Yakov Filiarsky, Shay Gammer and Sergey Novikov IBM Labs in Haifa © 2006 IBM Corporation IBM Labs in Haifa What is Code Coverage? ≥ A quality measurement used in software testing ≥ A quantitative way to describe the degree to which the source code of a program has been tested ≤ Feedback for improving the tests 2 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa How is Code Coverage Done? ≥ Via instrumentation ≤ Adding additional code to collect coverage data ≤ Introduce overhead on execution time ≥ Source level instrumentation is the common practice ≤ Instrumentation done during compilation ≥ For example gnu tools ≤ gcc/gcov – compilation + instrumentation ≤ lcov – for analysis 3 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Common Code Coverage Process Obtain Code Coverage Data Analyze Code Coverage Source Code Coverage Source Code Data Compilation + Instrumentation Coverage Analysis Instrumented Executable Code Coverage Views Execution 4 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Post Link Code Coverage Process Obtain Code Coverage Data Analyze Code Coverage Source Code Coverage Source Code Data Compilation Executable Coverage Analysis Instrumentation Instrumented Executable Code Coverage Views Execution 5 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Motivation for Post-Link Instrumentation ≥ Challenging architectures like kernel & embedded systems ≤ The current solutions (e.g. GCOV) may not address all the issues ± No OS facilities ± Non-terminating code ≥ Integrate code coverage in all tests along the entire development cycle ≤ Same tests for coverage and functional ≤ Enables accumulation of coverage data between phases As a result, coverage: ≤ is done on actual (optimized) code (e.g. SVT) ≤ instrumentation must have low overhead ± The tests cycle period should be reasonable ± The behavior should not be affected (e.g. time outs) 6 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Post-link Code Coverage Characteristics ≥ Works on the running code and does not interfere with compiler optimizations ≥ The coverage data is influenced by the compiler transformation and optimization ≥Different from source level coverage ≥More information available ≥ Enables further reduction in instrumentation overhead 7 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Our Post Link Code Coverage Process Obtain Code Coverage Data Analyze Code Coverage Optimized Coverage Source Code Executable Data FDPR-Pro FoCuS Instrumentation Analysis Instrumented Executable Code Coverage Views Execution 8 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa FoCuS can work with gcov as well (and more) Obtain Code Coverage Data Analyze Code Coverage Source Code Coverage Source Code Data gcov Instrumentation FoCuS Analysis Instrumented Executable Code Coverage Views Execution 9 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Raw Coverage Information From FDPR-Pro Basic Block – stream of instructions with one entry and one exit with no control flow in the middle 10 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa FoCuS’ Source View with Post-link Mapping switch (a) { BBS1: li r4, N case 1: cmp r3, r4 a = a * 2 bgt BBD break; BBS2: li r4, 1 case 2: cmp r3,r4 a = a * 4; blt BBD break; BBS3: sub r5,r3,r4 …. ld r6,(BT+r5*4) case N: br r6 a = a * 2^(N-4) BBC1: shl r3, 1 break; b BBE White – no matching BB default: ….. Red – Uncovered Code a = 0; BBCN: shl r3, N Green – Covered code break; b BBE Yellow – Partially } BBD: li r3,0 covered, matches to c = a * b BBE: mul r12,r3,r11 covered and non- covered BBs 11 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Working with Post-Link Code Coverage ≥ At the high level same as in regular code coverage ≥ The new methodology provides a compiler transformations dictionary to help with: ≤ White lines – not in the code ± Due to compiler optimization ± No need for additional tests to cover these lines ≤ Yellow lines – partially covered ± Due to the translation of language constructs (e.g. switch statement) ± Due to compiler optimization 12 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Understanding Coverage – Macro Example Partial coverage 13 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Understanding Coverage - If Example Partial coverage 14 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Understanding Coverage - If Example (cont) Partial coverage 15 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Low Overhead Instrumentation at Post-Link ≥ Motivation: integrating coverage along the entire development process ≤ More constraints as we approach shipment ≤ And beyond – at the customer site ≥ Post link – a better starting point ≤ Works on optimized code ≤ Lend itself for reducing instrumentation overhead ± Only covered/no-covered is needed ± Using self modifying instrumentation 16 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Low Overhead Instrumentation - Results 16 ) 14 Linux on % ( 12 d SPECint2000 a 10 e h r 8 POWER e v 6 O e 4 m i T 2 n gcc 4.1 –O3 0 o i t r f f 2 c y p p n r l e i k t c u p c p -2 a e f o o i z g c v g s g m a e m z a g e r w r r b -4 t b x a l c e r E p g -6 e v p A Using Self-modifying 6 4 ) % Code ( 2 d a 0 e i t h e k d a e p u r e r i s l m g c s k e -2 i r s a p p m i a v a e a g w r a p r w u O m t s e a m -4 m p q Average a x g e i u e v s m w i -6 A T n -8 o i t Function Level INT: 4% / 0.9% u -10 c e Basic Block Level x -12 SPECfp2000 E -14 FP: 0.4% / -1.4% 17 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Conclusion ≥ Post-link code coverage is useful ≤ Can be mapped back to source code ≤ Enables compiler optimization ≤ Supported by compiler transformation dictionary ≥ In some cases post-link code coverage is better ≤ Enables integrating coverage in the development process ± Near shipment and beyond ≤ More flexible for low overhead instrumentation ≥ Future work - enhancing the visual aids to improve the understanding of more complex compiler optimizations 18 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Questions ? FoCuS: www.alphaworks.ibm.com/tech/focus FDPR-Pro: www.haifa.il.ibm.com/projects/systems/cot/fdpr 19 Haifa Verification Conference 2007 © 2007 IBM Corporation IBM Labs in Haifa Line Number Information and optimization ≥ GCC man “Unlike most other C compilers, GCC allows you to use `-g' with `-O‘” ≥ xlc man “If you specify the -qlinedebug option, the inlining option defaults to -Q! (no functions are inlined).” But inline can be forced and the line-number info will be correct 20 Haifa Verification Conference 2007 © 2007 IBM Corporation.
Load more

Recommended publications
  • Using the GNU Compiler Collection (GCC)
    Using the GNU Compiler Collection (GCC) Using the GNU Compiler Collection by Richard M. Stallman and the GCC Developer Community Last updated 23 May 2004 for GCC 3.4.6 For GCC Version 3.4.6 Published by: GNU Press Website: www.gnupress.org a division of the General: [email protected] Free Software Foundation Orders: [email protected] 59 Temple Place Suite 330 Tel 617-542-5942 Boston, MA 02111-1307 USA Fax 617-542-2652 Last printed October 2003 for GCC 3.3.1. Printed copies are available for $45 each. Copyright c 1988, 1989, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004 Free Software Foundation, Inc. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with the Invariant Sections being \GNU General Public License" and \Funding Free Software", the Front-Cover texts being (a) (see below), and with the Back-Cover Texts being (b) (see below). A copy of the license is included in the section entitled \GNU Free Documentation License". (a) The FSF's Front-Cover Text is: A GNU Manual (b) The FSF's Back-Cover Text is: You have freedom to copy and modify this GNU Manual, like GNU software. Copies published by the Free Software Foundation raise funds for GNU development. i Short Contents Introduction ...................................... 1 1 Programming Languages Supported by GCC ............ 3 2 Language Standards Supported by GCC ............... 5 3 GCC Command Options .........................
    [Show full text]
  • Opportunities and Open Problems for Static and Dynamic Program Analysis Mark Harman∗, Peter O’Hearn∗ ∗Facebook London and University College London, UK
    1 From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis Mark Harman∗, Peter O’Hearn∗ ∗Facebook London and University College London, UK Abstract—This paper1 describes some of the challenges and research questions that target the most productive intersection opportunities when deploying static and dynamic analysis at we have yet witnessed: that between exciting, intellectually scale, drawing on the authors’ experience with the Infer and challenging science, and real-world deployment impact. Sapienz Technologies at Facebook, each of which started life as a research-led start-up that was subsequently deployed at scale, Many industrialists have perhaps tended to regard it unlikely impacting billions of people worldwide. that much academic work will prove relevant to their most The paper identifies open problems that have yet to receive pressing industrial concerns. On the other hand, it is not significant attention from the scientific community, yet which uncommon for academic and scientific researchers to believe have potential for profound real world impact, formulating these that most of the problems faced by industrialists are either as research questions that, we believe, are ripe for exploration and that would make excellent topics for research projects. boring, tedious or scientifically uninteresting. This sociological phenomenon has led to a great deal of miscommunication between the academic and industrial sectors. I. INTRODUCTION We hope that we can make a small contribution by focusing on the intersection of challenging and interesting scientific How do we transition research on static and dynamic problems with pressing industrial deployment needs. Our aim analysis techniques from the testing and verification research is to move the debate beyond relatively unhelpful observations communities to industrial practice? Many have asked this we have typically encountered in, for example, conference question, and others related to it.
    [Show full text]
  • Addresssanitizer + Code Coverage Kostya Serebryany, Google Eurollvm 2014 New and Shiny -Fprofile-Instr-Generate
    AddressSanitizer + Code Coverage Kostya Serebryany, Google EuroLLVM 2014 New and shiny -fprofile-instr-generate ● Coming this year ● Fast BB-level code coverage ● Increment a counter per every (*) BB ○ Possible contention on counters ● Creates special non-code sections ○ Counters ○ Function names, line numbers Meanwhile: ASanCoverage ● Tiny prototype-ish thing: ○ Part of AddressSanitizer ○ 30 lines in LLVM, 100 in run-time ● Function- or BB- level coverage ○ Booleans only, not counters ○ No contention ○ No extra sections in the binary At compile time: if (!*BB_Guard) { __sanitizer_cov(); *BB_Guard = 1; } At run time void __sanitizer_cov() { Record(GET_CALLER_PC()); } At exit time ● For every binary/DSO in the process: ○ Dump observed PCs in a separate file as 4-byte offsets At analysis time ● Compare/Merge using 20 lines of python ● Symbolize using regular DWARF % cat cov.c int main() { } % clang -g -fsanitize=address -mllvm -asan-coverage=1 cov. c % ASAN_OPTIONS=coverage=1 ./a.out % wc -c *sancov 4 a.out.15751.sancov % sancov.py print a.out.15751.sancov sancov.py: read 1 PCs from a.out.15751.sancov sancov.py: 1 files merged; 1 PCs total 0x4850b7 % sancov.py print *.sancov | llvm-symbolizer --obj=a.out main /tmp/cov.c:1:0 Fuzzing with coverage feedback ● Test corpus: N random tests ● Randomly mutate random test ○ If new BB is covered -- add this test to the corpus ● Many new bugs in well fuzzed projects! Feedback from our customers ● Speed is paramount ● Binary size is important ○ Permanent & temporary storage, tmps, I/O ○ Stripping non-code
    [Show full text]
  • Read the Thesis
    Making compiler tests based on compiler source code coverage Jelle Witsen Elias [email protected] January 19, 2021, 33 pages Academic supervisor: Clemens Grelck, [email protected] Daily supervisors: Jos´eLuis March Cabrelles, [email protected] Remi van Veen, [email protected] Host organisation: Solid Sands, www.solidsands.nl Universiteit van Amsterdam Faculteit der Natuurwetenschappen, Wiskunde en Informatica Master Software Engineering http://www.software-engineering-amsterdam.nl Abstract Compilers are important elements in software infrastructure. It is therefore important that these complex pieces of software are tested. We found a way of making C compiler tests that is different from the usual approach of doing this. With our new process we base new tests on their coverage of compiler source code, instead of constructing them by solely looking at the rules defined in language standards. This way, we manage to make tests with constructions that were overlooked when building our existing compiler test suites. We built a tool that takes source code of a C application as input, along with a C compiler test suite. The tool outputs pieces of code which, when compiled, cover parts of compiler source code that the test suite does not yet cover. With this output, we can construct new tests for inclusion in our existing test suite. Our process is as follows: we take an open source C application and compile its sources with an open source compiler. This compiler has been instrumented to enable source code coverage analysis. After compilation, we compare the coverage of compiler source code of the application with coverage of our existing test suites.
    [Show full text]
  • Assignment 10 — 18.11.2020 – V1.2 Dynamic Program Analysis
    SMA: Software Modeling and Analysis Prof. Dr. Oscar Nierstrasz A2020 Pascal Gadient, Pooja Rani Assignment 10 — 18.11.2020 – v1.2 Dynamic Program Analysis Please submit this exercise by email to [email protected] before 25 November 2020, 10:15am. You must submit your code as editable text, i.e., use plain text file(s). Exercise 1: Theory (1 pt) a) Suppose you have an application and you want to test all code paths with dynamic analysis. Is this approach reasonable? Justify! b) Is monkey testing a dynamic analysis technique? Justify! Exercise 2: Contracts (3 pts) In this exercise, we use the syntax and features from jContracts available here. Consider the Java code below. /** * @pre number >= 100.0 * @post return == true * @inv number < 0 || number >= 0 */ public boolean add(int number) f if (number < 10) f this.number = this.number + number; return true; g else f return false; g g a) Are the contracts for add(int number) valid, i.e., exists a configuration that passes all three checks? If yes, which configuration succeeds? If no, what is the problem? b) Can the invariant contract be removed without any side effects? If yes, why do no side-effects occur? If no, which side-effects would occur? c) Suppose you have a Java method that calculates from a given number the number multiplied by itself as shown in the code below. Design a post contract that validates the result, i.e., the contract must be violated whenever a result is wrong. public float square(int number) f return number * number; g page 1 of 5 SMA: Software Modeling and Analysis Prof.
    [Show full text]
  • A Zero Kernel Operating System: Rethinking Microkernel Design by Leveraging Tagged Architectures and Memory-Safe Languages
    A Zero Kernel Operating System: Rethinking Microkernel Design by Leveraging Tagged Architectures and Memory-Safe Languages by Justin Restivo B.S., Massachusetts Institute of Technology (2019) Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Masters of Engineering in Computer Science and Engineering at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY February 2020 © Massachusetts Institute of Technology 2020. All rights reserved. Author............................................................................ Department of Electrical Engineering and Computer Science January 29, 2020 Certified by....................................................................... Dr. Howard Shrobe Principal Research Scientist, MIT CSAIL Thesis Supervisor Certified by....................................................................... Dr. Hamed Okhravi Senior Staff Member, MIT Lincoln Laboratory Thesis Supervisor Certified by....................................................................... Dr. Samuel Jero Technical Staff, MIT Lincoln Laboratory Thesis Supervisor Accepted by...................................................................... Katrina LaCurts Chair, Master of Engineering Thesis Committee DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Assistant Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions
    [Show full text]
  • Guidelines on Minimum Standards for Developer Verification of Software
    Guidelines on Minimum Standards for Developer Verification of Software Paul E. Black Barbara Guttman Vadim Okun Software and Systems Division Information Technology Laboratory July 2021 Abstract Executive Order (EO) 14028, Improving the Nation’s Cybersecurity, 12 May 2021, di- rects the National Institute of Standards and Technology (NIST) to recommend minimum standards for software testing within 60 days. This document describes eleven recommen- dations for software verification techniques as well as providing supplemental information about the techniques and references for further information. It recommends the following techniques: • Threat modeling to look for design-level security issues • Automated testing for consistency and to minimize human effort • Static code scanning to look for top bugs • Heuristic tools to look for possible hardcoded secrets • Use of built-in checks and protections • “Black box” test cases • Code-based structural test cases • Historical test cases • Fuzzing • Web app scanners, if applicable • Address included code (libraries, packages, services) The document does not address the totality of software verification, but instead recom- mends techniques that are broadly applicable and form the minimum standards. The document was developed by NIST in consultation with the National Security Agency. Additionally, we received input from numerous outside organizations through papers sub- mitted to a NIST workshop on the Executive Order held in early June, 2021 and discussion at the workshop as well as follow up with several of the submitters. Keywords software assurance; verification; testing; static analysis; fuzzing; code review; software security. Disclaimer Any mention of commercial products or reference to commercial organizations is for infor- mation only; it does not imply recommendation or endorsement by NIST, nor is it intended to imply that the products mentioned are necessarily the best available for the purpose.
    [Show full text]
  • The Impact of Software Evolution on Code Coverage Information
    University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln Computer Science and Engineering, Department CSE Conference and Workshop Papers of 2001 The Impact of Software Evolution on Code Coverage Information Sebastian Elbaum University of Nebraska-Lincoln, [email protected] David Gable University of Nebraska-Lincoln, [email protected] Gregg Rothermel University of Nebraska-Lincoln, [email protected] Follow this and additional works at: https://digitalcommons.unl.edu/cseconfwork Part of the Computer Sciences Commons Elbaum, Sebastian; Gable, David; and Rothermel, Gregg, "The Impact of Software Evolution on Code Coverage Information" (2001). CSE Conference and Workshop Papers. 132. https://digitalcommons.unl.edu/cseconfwork/132 This Article is brought to you for free and open access by the Computer Science and Engineering, Department of at DigitalCommons@University of Nebraska - Lincoln. It has been accepted for inclusion in CSE Conference and Workshop Papers by an authorized administrator of DigitalCommons@University of Nebraska - Lincoln. Proceedings. IEEE International Conference on Software Maintenance, 2001. Digital Object Identifier: 10.1109/ICSM.2001.972727 Publication Year: 2001 , Page(s): 170 - 179 The Impact of Software Evolution on Code Coverage Information Sebastian Elbaum David Gable Gregg Rothermel Dept. of Computer Dept. of Computer Computer Science Dept. Science and Engineering Science and Engineering Oregon State University University of Nebraska-Lincoln University of Nebraska-Lincoln Corvallis, OR Lincoln, Nebraska Lincoln, Nebraska [email protected] [email protected] [email protected] Abstract Often, code coverage information is collected for a ver- sion of a program to aid in some maintenance or testing task Many tools and techniques for addressing software performed on that particular version.
    [Show full text]
  • General Commands Reference Guide C
    General Commands Reference Guide C TRACE32 Online Help TRACE32 Directory TRACE32 Index TRACE32 Documents ...................................................................................................................... General Commands ...................................................................................................................... General Commands Reference Guide C .................................................................................. 1 History ...................................................................................................................................... 11 CACHE ...................................................................................................................................... 12 CACHE View and modify CPU cache contents 12 CACHE.CLEAN Clean CACHE 12 CACHE.ComPare Compare CACHE with memory 13 CACHE.DUMP Dump CACHE 14 CACHE.FLUSH Clean and invalidate CACHE 15 CACHE.GET Get CACHE contents 16 CACHE.INFO View all information related to an address 16 CACHE.INVALIDATE Invalidate CACHE 17 CACHE.List List CACHE contents 17 CACHE.ListFunc List cached functions 18 CACHE.ListLine List cached source code lines 19 CACHE.ListModule List cached modules 19 CACHE.ListVar List cached variables 20 CACHE.LOAD Load previously stored cache contents 21 CACHE.RELOAD Reload previously loaded cache contents 21 CACHE.SAVE Save cache contents for postprocessing 21 CACHE.SNAPSHOT Take cache snapshot for comparison 22 CACHE.UNLOAD Unload previously loaded cache contents 23 CACHE.view
    [Show full text]
  • Non-Intrusive Structural Coverage for Objective Caml
    Electronic Notes in Theoretical Computer Science 264 (4) (2011) 59–73 www.elsevier.com/locate/entcs Non-Intrusive Structural Coverage for Objective Caml Philippe Wang1, Adrien Jonquet2, Emmanuel Chailloux3 Equipe´ APR Laboratoire d’Informatique de Paris 6 (CNRS UMR 7606) Universit´e Pierre et Marie Curie (Paris 6) 4 place Jussieu, 75005 Paris, France Abstract This paper presents a non-intrusive method for Objective Caml code coverage analysis. While classic methods rewrite the source code to an instrumented version that will produce traces at runtime, our approach chooses not to rewrite the source code. Instead, we use a virtual machine to monitor instructions execution and produce traces. These low-level traces are used to create a machine code coverage report. Combined with control-flow debug information, they can be analyzed to produce a source code coverage report. The purpose of this approach is to make available a method to generate code coverage analysis with the same binary for testing and for production. Our customized virtual machine respects the same semantics as the original virtual machine; one of its original aspects is that it is implemented in the Objective Caml, the very language we build the tool for. This work is part of the Coverage project, which aims to develop open source tools for safety-critical embedded applications and their code generators. Keywords: Certification Tools Design, Code Coverage, Objective Caml Virtual Machine 1 Introduction One of the most demanding development process for safety-critical software was defined a couple of decades ago by the civil avionics authorities as the DO-178B standard [17].
    [Show full text]
  • A Flexible and Non-Intrusive Approach for Computing Complex Structural Coverage Metrics
    A Flexible and Non-intrusive Approach for Computing Complex Structural Coverage Metrics Michael W. Whalen∗, Suzette Persony, Neha Rungtaz, Matt Staatsx and Daniela Grijincu{ ∗University of Minnesota, Minneapolis, Minnesota, USA Email: [email protected] yNASA Langley Research Center, Hampton, Virginia, USA Email: [email protected] zNASA Ames Research Center, Moffett Field, California, USA Email: [email protected] xGoogle Inc., Zurich, Switzerland Email: [email protected] {University of St. Andrews, Scotland, UK Email: [email protected] Abstract—Software analysis tools and techniques often lever- Instrumentation based techniques can incur considerable age structural code coverage information to reason about the overhead by adding a large number of conditional statements dynamic behavior of software. Existing techniques instrument to the original code, which in turn, can significantly increase the code with the required structural obligations and then monitor the execution of the compiled code to report cover- the program’s execution space. The problem becomes even age. Instrumentation based approaches often incur considerable worse as the complexity of the coverage criterion increases, runtime overhead for complex structural coverage metrics such both in terms of the number of coverage obligations and the as Modified Condition/Decision (MC/DC). Code instrumentation, number of operators in each obligation. For metrics such as in general, has to be approached with great care to ensure it Modified Condition/Decision Coverage (MC/DC) – a metric does not modify the behavior of the original code. Furthermore, instrumented code cannot be used in conjunction with other widely used in the avionics software domain – the overhead analyses that reason about the structure and semantics of the of the instrumentation can render the cost of computing the code under test.
    [Show full text]
  • Is Bytecode Instrumentation As Good As Source Code Instrumentation: an Empirical Study with Industrial Tools (Experience Report)
    Is Bytecode Instrumentation as Good as Source Code Instrumentation: An Empirical Study with Industrial Tools (Experience Report) Nan Li, Xin Meng, Jeff Offutt, and Lin Deng Software Engineering George Mason University nli1,xmeng,offutt,[email protected] Abstract—Branch coverage (BC) is a widely used test testing); (2) using an efficient process, that is, the test criterion that is supported by many tools. Although design process is as inexpensive as practical; and (3) textbooks and the research literature agree on a standard the tests are effective at finding as many of the faults definition for BC, tools measure BC in different ways. as possible. Of course, these goals cannot be satisfied The general strategy is to “instrument” the program by completely, and are in some sense contradictory. Thus adding statements that count how many times each branch is taken. But the details for how this is done can influence testers must make engineering compromises. the measurement for whether a set of tests have satisfied One sound way to balance these goals is to use BC. For example, the standard definition is based on coverage criteria. A coverage criterion is a rule or set program source, yet some tools instrument the bytecode to of rules that are applied to a software artifact (such as reduce computation cost. A crucial question for the validity the source code, requirements, or a design document) of these tools is whether bytecode instrumentation gives to create a collection of test requirements [2]. Each test results that are the same as, or at least comparable to, requirement defines a specific element in the software source code instrumentation.
    [Show full text]