Using Fault Injection to Increase Software Test Coverage
James M. Bieman Daniel Dreilinger Lijun Lin Computer Science Department Colorado State University Fort Collins, Colorado 80523 [email protected] To appear in Proc. International Symposium on Software Reliability (ISSRE'96)
Abstract must be recompiled and then tested. This process can be too labor and time intensive for practical use in the general test- During testing, it is nearly impossible to run all statments ing of commercial software. or branches of a program. It is especially dif®cult to test Our hypothesis is that fault injection can be effective the code used to respond to exceptional conditions. This when it is directed towards solving speci®c testing prob- untested code, often the error recovery code, will tend to be lems. In particular, we use fault injection to force the exe- an error prone part of a system. We show that test cover- cution of dif®cult to reach program paths. To avoid the need age can be increased through an ªassertion violationº tech- for recompilation, we mutate program state rather than pro- nique for injecting software faults during execution. Using gram text. our prototype tool, Visual C-Patrol (VCP), we were able to To be practical, any fault injection mechanism must be substantially increase test branch coverage in four software inexpensive and be able to model a wide variety of fault systems studied. types. Much of the fault injection should be automated. Al- though a practical approach requires a limited fault model, Keywords: software testing, software test coverage, fault we want to simulate as many faults as possible. injection, error recovery, software reliability, Visual C- Patrol. 2. Fault Injection via ªAssertion Violationº
1. Introduction Pre- and post-conditions contain logical assertions or in- variants that specify the expected behavior of a program. A Developing reliable and fault tolerant software is dif®cult pre-condition is an assertion about the nature of the system and requires discipline both in specifying system functional- state that must be true before a function is invoked to be sure ity and in implementing systems correctly. Approaches for that the function will run correctly. A post-condition is an developing highly reliable software include the use of for- assertion that describes the relationship between the input mal methods [11, 14, 9], and rigorous testing methods [2, 7, state and output state of a correct function. These conditions 13]. Testing cannot guarantee that software is correct [17], are usually not explicitly expressed in the program text, but and veri®cation requires enormous human effort and is sub- rather implicitly assumed. ject to errors [6]. Automated support is necessary to help We can create a fault by dynamically changing the state ensure software correctness and fault tolerance. of a running program so that a pre- or post-condition is not Fault injection has been proposed for use in mutation test- satis®ed. Such an arti®cial fault can be used to model ini- ing primarily as a mechanism for evaluating the adequacy of tialization faults, assignment faults, condition check faults, test data [3, 8]. Mutation testing injects faults by modifying and even function faults. program text. As a result, the testing process can generate The assertion violation scheme makes use of pre- and enormous numbers of mutant programs, and each program post-condition assertions. The ®rst step is to de®ne and state these using a new programming language construct. The Research partially supported by the Colorado Advanced Software In- second step is to automatically make them false in different stitute (CASI) and Storage Technology Inc. CASI is sponsored in part by ways and at different program locations while the program the Colorado Advanced Technology Institute (CATI), an agency of the state of Colorado. CATI promotes advanced technology teaching and research at is executing. To create a fault, one of these invariants is vi- universities in Colorado for the purpose of economic development. olated or made false in some fashion. We can automatically generate a number of possible violations of a given assertion The timing impact of this scheme is quite small because using language processing tools, such as lex and yacc. the injected values can be computed statically. An optimiz-
The pre- and post-conditions provided by the tester or ing C-compiler can compute the values to be injected, even
= 4+0:01 programmer are ®rst order logic predicates about the func- if the source output of the injector is y . tion's input and output parameters, and global variables. We can check the correctness of a system's behavior They use a syntax similar to that used in the source language. before and after injecting faults. A correct pre- or post- condition can be injected as code and checked at run time Example. Suppose the pre-condition for some function using a facility like C assert statements or C-Patrol insertion
directives.
x > 0 ^ x 6= y g f(¯oat x, y) is f . We can derive several possible faults to inject. The injection is a textual insertion at the beginning of the function. For the example, the injec- 3. Implementing Assertion Violation Fault In-
tion code might be the following (using C syntax): jection f sw itch (injection status)
To demonstrate the proposed fault injection method, we
br eak ;
case 0: extended the C-Patrol assertion insertion system [18] to sup-
br eak ;
case 1: x=0; port fault injection and built a visual X Window System in-
br eak ; case 2: x=-1;
terface.
br eak ;
case 3: x=y;
br eak ; g case 4: y=x; 3.1. C-Patrol The above block of code can be automatically generated C-Patrol is a code insertion tool that can assist developers by a code pre-processing tool. When injection is off, the in the placement of software probes that are used in testing. injection status variable is set to zero. When injection is on, Such probes may be implementations of data invariants, pre- the injection status variable is set randomly to a value be-
and post-conditions, or other run time checks. An overall 4 tween 1 and . The number of injections and the probability view of the C-Patrol system is shown in Figure 1. of choosing a particular injection can be determined by the C-Patrol allows developers to de®ne and place assertions user. Typically, an injection will cause a short chain reac- in logical locations in a C program. For example, a devel- tion of violated assertions. The invalid assertions should be oper may prefer to place a data invariant along with an asso- restored by error recovery code in fault tolerant software. ciated data declaration. A preprocessor under program con- The code mutation aspect of this scheme can be per- trol inserts these assertions into the correct locations for run formed by a pre-processor, which transforms pre- and post- time monitoring of a program. The data invariant is thus conditions into case injection statements and inserts them checked at the locations where the data structure is actually into a program. used, rather than where it is de®ned. Using C-Patrol, virtual An injector tool could accept statements from the propo- code is de®ned within comments, and is inserted into regular sitional calculus. In our prototype system, we implemented code using possibly parameterized directives. A special la- a subset which speci®cally includes statements using the al- beling system matches directives to virtual code. When the phabet: probes are not needed, they remain with the program as com-
ments that do not affect program execution.
identif ier;<;<=;==; !=;>=;>;
f Our initial intention was for C-Patrol software probes to
l iter al ; f l oat l iter al ; ^g integ er . be used to check the correctness of program behavior. How- Valid strings in the language are conjunctions of boolean ever, probes can also be used to modify the system state and terms (equalities or inequalities). Ideally, the language will thus create any desired or undesired state. Probes can gen-
erate states that would result from software and hardware
; _; 8; 9g be augmented with f: . An example of a recogniz- able precondition in the system is: faults. Thus, C-Patrol can be used to inject faults at chosen
points in a program.
a<3 ^ b!=c ^ d>1:5 : C-Patrol is a pre-processor that inserts assertions, writ- ten as comments in virtual C, into speci®ed locations in C Many possible injections can be derived from this state- programs. The user places virtual C code within special C- ment. In our prototype system, we inject violations that are Patrol comments that are delimited by ª/*?ºandª?*/º.
near the boundary to simulate some common programming These comments are skipped by the C compiler and do not
3 a =3 b!=c
errors. Thus, a< is mapped to ,gets mapped affect the performance of the underlying system. Once the
= c c = b d>1:5
to two possible faults: b and , and ®nally, user activates the comments by running the pre-processor,
=1:5 is mapped to d .the pre-processor translates virtual code into regular C and
2 Special comments turned on C source code with C - Patroller special comments inserted
C - Patrol C Code (C code w/ special comments)
typedef structure LIST{ char *name; int value; sturct LIST *next; } ... /*? template ordered(alist): assert(is_ordered(alist)); ?*/ ... C Compiler main() { build_list_in_order(list1); Special comments not turned on /*?%%insert: %%calll ordered(list1; ?*/ build_list_in_order(list2); /*? %%insert: %%call ordered(list2); ?*/ list3 = merge_lists(list1, list2);
Figure 1. The C-Patrol System inserts it based on a set of instructions called directives.The 2. A run time visual interface to help monitor the execu- pre-processed program may then be compiled and run as tion of instrumented programs. normal C. Thus, a program with C-Patrol annotations can run with or without the inserted assertions as shown in Fig- 3. Fault injection based on assertion violation. ure 1. A user interface, shown in Figure 2, supports editing pro- The C-Patrol annotation language includes insertion di- grams with C-Patrol style assertions. All function names in rectives, call directives, label directives, template direc- a ®le are displayed in a special window, along with their in- tives and bind directives. Insertion directives include strumentation status. The user can easily skip around and %%insert, %%pre and %%post. The format for C- add, view, or change assertions. There is a menu, buttons Patrol pre- and post-conditions is for commonly used features, and an edit area in the inter- /*? face. A function area on the left side of the window displays %%pre:
3 Figure 2. VCP user interface the screen as shown in Figure 3.
The run time visual interface is automatically built for an instrumented program. Each instrumented function ap- pears in the run time monitor interface with pass/fail indi- cators for each assertion. These show the programmer both the assertion pass/fail status and program ¯ow of control. A pre- or post-condition indicator is green when the condition successfully executes and turns red when it fails. Execution can be slowed via a speed control bar and a step button.
Injecting a fault for a speci®c condition can be performed by using a step button to stop before entering the desired function. Then the user presses the inject toggle button and select a fault in the bottom window, and presses step. The fault will be injected. After fault injection, the pre-condition indicator turns purple. The tester can then check whether the results are as expected.
The third major VCP feature implements the assertion vi- olation mechanism for fault injection. The state of an exe- cuting program is dynamically modi®ed to an incorrect one with the goal of forcing recovery code to execute. The VCP editor allows the user to select speci®c assertions to instru- ment for fault injection, while the run time monitor allows the user to instantiate the injections and subsequently ob- Figure 3. Run time interface of VCP serve their effect on control ¯ow and assertion status. A pre- condition indicator turns purple when a fault is injected.
4 4. Effectiveness Table 1. Fault Injection Applied to Selected C We evaluated the assertion violation fault injection tech- Functions nique as implemented in Visual C-Patrol (VCP) by using it Fun. Test Init. Final to test four systems or partial systems. We measured the Name Lines Branches Cases Cover. Cover. test coverageÐthe ratio of the number of branches that were AC.1 37 18 2 66.7% 94.4% covered by a set of test cases to the total number of branches AC.2 72 46 2 23.9% 100% in a programÐbefore and while using VCP. AC.3 12 2 2 100% 100% Minix.1 26 4 6 75% 100% 4.1. Evaluation Data Minix.2 20 4 1 25% 100% Minix.3 30 5 12 41.7% 41.7% We examined the application of VCP to all or part of four Minix.4 19 10 5 50% 90% systems: Minix.5 11 10 4 75% 100% Ave. 28.3 11.6 4.3 41.9% 94.6% 1. AUTOLAND, a fault tolerant software system devel- oped at AT&T Bell laboratory. AUTOLAND consists of 3524 lines of C code in 12 C source ®les and 12 header ®les. We obtained a regression test set consist- 4.3. Function Level Evaluation ing of 5280 test cases. In our initial evaluation, we selected 8 functions from 3 2. C-Patrol, our software testing tool that we used to de- ®les. Table 1 shows the original branch coverage achieved velop VCP. We evaluated 9 of the 13 C-Patrol source during testing and the coverage achieved with fault injec- ®les consisting of 5834 lines of code and 677 branches. tion. Our data included three functions from AC and ®ve We used a regression test set consisting of 93 test cases. functions (commands) from Minix. 3. AC, a compiler for the language A, a subset of C used As Table 1 shows, we were quite successful in increasing in a compiler course. AC compiles source ®les written test coverage of most of these functions. We were able to in- in A into the MIPS processor assembly code. We eval- crease coverage to 100% in four of the functions (one func- uated 7 of the 27 source ®les using 19 test cases. tion had 100% coverage before fault injection). Coverage ended up above 90% in all cases except for the one function 4. Minix, a UNIX-like operating system used in education that we failed to induce any coverage increase. and research. We evaluated 5 Minix commands in early tests of VCP. 4.4. File and System Level Evaluation
4.2. Evaluation Process We evaluated complete source ®les to expand the evalu- ation beyond individual functions. This evaluation included First, we selected a sample of individual functions and 27 ®lesÐ9 ®les from AC, 9 ®les from C-Patrol, and all 12 evaluated fault injection for these small code components. ®les from Autoland. Tables 2, 3, and 4 show the evaluation Then, we evaluated fault injection applied to source ®les data for these ®les. which consist of many functions. For one system, Autoland, Except for the two ®les with the highest coverage, 96.3% we evaluated fault injection applied to all ®les in the system. and 88.9%, we were able to increase the branch coverage Each evaluation was conducted in two steps: in all of the ®les in AC (see Table 2). The largest change 1. We tested the software without using VCP and col- was in measure.c where coverage increased from 72.4% lected the test coverage, measured in terms of the num- to 100%. ber of control ¯ow branches reached by the test data. Branch coverage increases were more modest in the eval- We used the Generic Coverage Tool (GCT), a public uation of C-Patrol coverage (see Table 3). We were able to domain testing tool, to collect test coverage informa- increase coverage in only 4 of the 9 ®les. However, C-Patrol tion. already had 84.5% branch coverage before fault injection, which is the highest initial test coverage of the 3 systems. 2. We used VCP to insert C-Patrol pre-conditions into The higher the initial coverage, the more dif®cult it will be to the software, generate faults, and inject these faults increase coverage. When there are fewer untested branches, into the program when it is tested. The injected faults the untested branches seem to be hard to reach. change the run time state of the program and force new Autoland is the one fault tolerant system that we eval- branches to be covered. We used GCT to collect the uated. We were able to increase the test coverage of Au- test coverage obtained with fault injection. toland from 78.9% to 87.3% (see Table 4). This is the largest
5 Table 2. Fault Injection Applied to AC C Table 4. Fault Injection Applied to Autoland C Source Files Source Files Test Orig. Final Test Orig. Final File Lines Branches Cases Cover. Cover. File Lines Branches Cases Cover. Cover. array.c 255 54 19 96.3% 96.3% alt hold.c 78 4 5280 100% 100% avasg.c 652 154 19 56.5% 63.0% autoland.c 296 24 5280 91.7% 91.7% avexp.c 711 178 19 69.7% 71.3% bae gscf.c 414 22 5280 100% 100% cfg.c 491 72 19 88.9% 88.9% cmdmnt.c 117 6 5280 50.0% 83.3% gcp.c 541 145 19 83.5% 89.0% display.c 310 40 5280 85.0% 92.5% gcse.c 289 89 19 73.0% 77.5% ¯are.c 162 12 5280 83.3% 91.7% ¯at.c 877 204 19 86.3% 88.7% glideslp.c 170 8 5280 100% 100% meas.c 190 58 19 72.4% 100% innerlp.c 184 12 5280 75.0% 91.7% sem.c 637 213 19 82.6% 93.4% interface.c 678 64 5280 60.9% 75.0% Ave. 444.8 129.7 19 77.7% 83.6% mathutil.c 92 8 5280 75.0% 75.0% mode.c 245 2 5280 100% 100% racf.c 109 2 5280 100% 100% Ave. 238 17 5280 78.9% 87.3% Table 3. Fault Injection Applied to C-Patrol Source Files Test Orig. Final statements with the injector. File Lines Branches Cases Cover. Cover. Even with the ability to mutate local variable state, there bind.c 426 64 94 89.0% 89.0% are other faults that cannot be modeled. Consider the fol- call.c 595 110 94 90.9% 90.9% lowing code structure: inprt.c 498 16 94 93.8% 100% insert.c 153 38 94 84.2% 84.2% if(!do something()) then invoke recovery code prepst.c 343 46 94 97.8% 97.8% main.c 183 14 94 78.6% 78.6% where the function do something() is a library routine. patrol.c 669 114 94 87.7% 88.6% If do something() is part of the program under test, supprt.c 816 148 94 81.7% 88.5% the recovery code can be tested by violating the postcon- tmplt.c 691 127 94 73.2% 75.6% dition of do something() or possibly the precondition. Ave. 486 75.2 94 84.8% 87.0% But, when the exception handling mechanism depends on some unmodi®able function, a different approach must be taken. For example, increase that we obtained out of the three systems. Since if (p=malloc(sizeof(int))==NULL) recover(); Autoland is fault tolerant, there are many branches that are designed only to respond to exceptional situations. These is often found in C programs. It can be dif®cult to inject branches can be resistant to testing. This system came with faults into malloc, because it is a library routine and we a set of regression test data consisting of 5280 test cases, yet may have limited access to its source code. One possible only 78.9% of the branches were reached. We were able to solution is to automatically transform the if condition into test many of these previously untested branches using the something like: fault injection mechanism in VCP. if (!do something() 4.5. Potential Improvements OR active injection==372) then invoke recovery code. We were not able to increase coverage in several of the source ®les. Thus, we would like to improve our ability to Processing assertions with complicated mathematical ex-
force sections of code to run. We need to be able force the pressions, rather than just simple inequalities, should be pos-
2
b 4ac 0 violation of more expressive assertions than the simple ones sible. For example, couldbeprocessedal-
that we can now mutate. For example, now we cannot mu- gebraically to create invalid values of all three variables. An
2
= b =4a + tate local variables of a function, and, as a result, many faults example injection of this sort is: c .Ourcur- cannot be modeled. The mutation of the state of local vari- rent system uses assertions that are executable when com- ables could be achieved by processing C-Patrol %%insert piled as ordinary C code. That is, the pre-conditions work
6 after processing only by C-Patrol, as well as after process- The TAMER fault injection tool mutates source code to ing with the injector and C-Patrol. If we relax this restric- test fault-tolerant system [5]. TAMER injects possible faults tion, then we can write more descriptive assertions if we do at module interfaces using a ªfault manager.º The system
not require that they match C syntax. For example, univer- is designed so that all mutants can be created using only 9 sal and existential quanti®ers (8 and ) might be added to one compilation, and iterates the execution of the mutants. the recognized language. An experiment demonstrated that fault injection is needed to test the fault tolerance of a program. 5. Related Work 5.2. Dynamic Fault Injection Fault injection can be used to modify either a program's source code text or the machine state of an executing pro- The dynamic or state changing forms of fault injection gram. The most common static fault injection is mutation do not require multiple compilation. Dynamic fault injec- testing. Much of the recent fault injection research is con- tion is most commonly used to simulate hardware errors by cerned with dynamic injection. modifying or injecting faults into memory bits and regis- ters [15, 1, 4, 10]. The changed memory locations can con- 5.1. Static Fault InjectionÐMutation Testing tain both program instructions and data. A wide variety of faults can be emulated. However, dy- Mutation testing involves testing modi®ed or mutated namic fault injection has been commonly used to modeling program source text [8]. It is primarily applied to unit hardware faults such as bus errors or incorrect CPU instruc- testingÐtesting which involves small individual modules of tions. Such hardware faults often have such a drastic effect the program. on the system that human operator intervention is necessary, A mutant program is created by making a small syntactic thus impacting an automated testing process. Testing only change, a mutation, to the original program. For example, a for hardware faults ignores potential software faults. Four dynamic fault injection case studies are relevant to greater-than operator, >, might be changed to greater-than-
our work: = or-equal, > . To help test error recovery code, we could mutate the operators in conditional statements to cause a 1. The DEF.Injector (`De®ned Errors Fast Injector') in- program to branch into recovery code. jects a set of hardware faults by toggling memory bits The output of the original program is compared with the or bytes, changing bus addresses, and changing ma- output produced by the mutant. If the mutant and the orig- chine instructions [10]. It is a hardware device that is inal program produce the same output, then either the set physically connected to speci®c target machines. Be- of test cases is not adequate, or the mutant is a functionally cause of the dedicated hardware involved, intermit- identical to original. No automatic procedure can determine tent faults are easily modeled. The DEF.Injector can whether the original program and the mutant are equivalent achieve thorough coverage of all de®ned tests because or the set of test cases is inadequate. the address space of the target machines is quite small Mutation testing can require the creation of a vast number (8-64K). Exhaustive hardware fault injections would of mutant programs. A program with N variable references
be infeasible on a machine with an address space of 2 can have N mutant versions. In one study of a 30-line trian- several megabytes. gle program, 951 mutants were automatically created [16]. Massive computational resources can be required to repeat- 2. FIAT(`Fault Injection-based Automated Testing') uses edly recompile and run all of the mutations. fault injection to evaluate the dependability of fault tol- Weak mutation testing is less rigorous but more ef®cient erant software [1]. FIAT, implemented entirely in soft- than strong mutation testing [12]. Using weak mutation test- ware, changes bits in both program text and data re- ing, mutant and original program states are compared soon gions of machine memory to simulate hardware faults. after the mutation is executed, rather than after entire pro- grams are run. States can be compared soon after executing 3. Chillarege and Bowen manually injected 70 overlay mutated expressions, statements, or basic blocks. Weak mu- faults into a large commercial transaction processing tation testing is much less expensive than strong mutation system [4]. Overlay faults occur when a program testing, while almost as effective [16]. writes into an incorrect location due to a faulty desti- Weak mutation testing does not solve the problem of nation operand. To decrease fault and error latency and identifying equivalent mutants. Also, invalid program states increase the probability that a fault will cause an error, in fault tolerant software should not propagate to the output. a large region of memory was corrupted with a single Thus, weak mutation testing may be dif®cult to apply to ro- injection. Injection involved setting all bits in an en- bust fault tolerant software. tire page of physical memory to one. About 16% of
7 the faults immediately crashed the system; about 14% not all assignment faults are covered. Like the assignment caused a partial loss of service; half of the faults did not faults, condition check faults are modeled when the condi- cause failures. tions make use of input parameters. Invalid functions that result from complicated, abstract 4. FINE (`The Fault Injection and Monitoring Environ- programming errors are modeled quite well by assertion vi- ment') was used to study fault propagation in UNIX olation, since there are many options for generating faults operating system kernel [15]. This system modeled from complex assertions. hardware faults including memory faults, CPU faults, Some hardware faults cannot be easily modeled, such as bus faults, and I/O faults. incorrect CPU instructions or stuck-at bus faults. Model- Unlike the other studies, software faults were studied ing these faults may not be necessary, since they will prob- including initialization faults, assignment faults, con- ably crash most fault tolerant systems anyway (unless it is dition check faults, and invalid function faults. Initial- equipped with special hardware). ization faults can be detected by a compiler. Assign- The software faults modeled by assertion violation will ment and condition check faults are clearly relevant to tend to accelerate the progression from fault to error. An in- the testing of error recovery code, since an incorrect as- jected fault is likely to be encountered soon after injection signment or condition can be a condition that should because the violated pre-condition is `about' variables that force the execution of recovery code. Here, invalid are used in the function that runs immediately after execu- function faults are created by replacing function code tion. Similarly, the faulty return value changed by a post- with manually rewritten alternatives. For our purposes, condition injection will soon be detected by other pre- or invalid function code must be automatically generated, post-conditions. since manual rewriting of code is prohibitive in a large system. 6. Conclusions These experiments show some seemingly inherent prob- lems with dynamic fault injection. Some injected faults can We have developed the assertion violation mechanism crash the computer requiring operator intervention, so spe- for inserting faults. The method mutates state by violating cialized hardware is needed for a completely automated pro- speci®ed function pre- and post-conditions. cess. Some injected faults will have a very high error la- We have implemented assertion violation in our Visual tency. A system that continues to operate correctly for a long C-Patrol system (VCP). VCP includes assertion violation time after injection may or may not have recovered from the fault injection in a visual interface to C-Patrol. The ®rst step fault. It is very dif®cult to determine whether a latent error in using VCP is to use the VCP interface to add pre- and may still eventually occur. post- conditions to functions of interest. Then the software is tested by running the instrumented program. Faults can 5.3. Assertion Violation vs. Related Work be injected and monitored at run time from the run time in- terface. Pre- and post-conditions will indicate if the fault- Our assertion violation technique generates faults by mu- injected program satis®es speci®ed constraints. A tester can tating program state at run time. Thus it does not incur also determine whether the error recovery code is correct the recompilation overhead of mutation testing. Correctness from the output of the fault injected program. checking can be performed by comparing output or by com- Our evaluation demonstrates that fault injection can be paring states. It can be used to model both hardware and effective in increasing the coverage of hard to reach parts software faults. of a program. When applying fault injection to individual To model transient memory faults, the memory locations functions, coverage reached 90% or higher in all but one (variable aliases) characterized in an assertion can be cor- function. Five functions reach 100% coverage, while the rupted. The degree of corruption will depend on the speci®c coverage of only one function is unchanged. Coverage also rules which control the violation. These faults could explic- increased when we applied fault injection to entire source itly mimic the injections that others have used, for example, ®les. We applied fault injection to a total of 30 source ®les, by setting and resetting bits and groups of bits. and increased the coverage in 16 of them. Nine of the 14 Initialization faults related to function parameters are ®les that showed no improvement already had over 90% easily modeled. The parameters referenced in the pre- coverage without fault injection. Fault injection increased condition can simply be set to a random value, or some pre- the coverage in 15 of the 20 ®les (75%) that started with less determined but incorrect value, such as zero. than 90% coverage. Many assignment faults can be generated by modifying Table 5 shows that we were able to increase overall test parameter values. An assignment that makes use of a mu- coverage. In the ®le-level tests, coverage surpassed 80%. In tated parameter or global variable will be faulty. However, industrial practice 80% branch coverage is considered quite
8 Second Workshop on Software Testing, Veri®cation, and Table 5. Evaluation Summary Data Analysis, pages 142±151, July 1988. Increase [9] S. Garland, J. Guttag, and J. Horning. Debugging larch Evaluation Original Final over shared language speci®cations. IEEE Trans. Software En- Software Coverage Coverage original gineering, 16(9):1044±1057, September 1990. 8 Functions from 3 ®les 41.9% 94.6% 125.8% [10] J. Gerardin. The `def.injector' test instrument, assistance in 6 AC Compiler ®les 77.7% 83.6% 7.6% the design of reliable and safe systems. Computers in Indus- 9 C-Patrol ®les 84.8% 87.0% 2.6% try, 11(4):311±319, Feb. 1989. All 12 Autoland ®les 78.9% 87.3% 10.7% [11] A. Hall. Seven myths of formal methods. IEEE Software, 7(5):11±19, September 1990. [12] W. Howden. Weak mutation testing and completeness of test sets. IEEE Trans. Software Engineering, SE-8(4):371±379, good. Fault injection allowed us to increase test coverage July 1982. [13] W. Howden. A functional approach to program testing without adding further test cases. Thus, we can get higher and analysis. IEEE Trans. Software Engineering,SE- coverage with fewer test cases. 12(10):997±1005, October 1986. Assertion violation modi®es the global variables or pa- [14] C. Jones. Systematic Software Development Using VDM. rameters values that appear in function pre- and post- condi- Prentice-Hall International, London, 1986. tions. Many other kinds of assertions could be used to gen- [15] W.-L. Kao, R. Iyer, and D. Tang. Fine: a fault injection erate faults, and thus increase the effectiveness of fault in- and monitoring environment for tracing the unix system be- jection. Suggested future modi®cations to the assertion vio- havior under faults. IEEE Trans. Software Engineering, lation scheme are: 19(11):1105±1119, Nov. 1993. [16] A. Offutt and S. Lee. An empirical evaluation of weak mu-
Add the capability of local variable assertions, tation. IEEE Trans. Software Engineering, 20(5):337±345, May 1994.
Enrich the assertion language, [17] L. J. White. Basic mathematical de®nitions and results in testing. In B. Chandrasekaran and S. Radicchi, editors, Com-
Remove some non-deterministic aspects. puter Program Testing, pages 13±24. North-Holland, 1981. [18] H. Yin and J. Bieman. Improving software testability with References assertion insertion. In Proc. Int. Test Conf., Oct. 1994.
[1] J. Barton, E. Czeck, Z. Segall, and D. Siewiorek. Fault in- jection experiments using FIAT. (Fault Injection-based Au- tomated Testing). IEEE Trans. Computers, 39(4):575±583, April 1990. [2] B. Beizer. Black-Box Testing. John Wiley & Sons, 1995. [3] T. A. Budd. Mutation analysis: Ideas, examples, problems and prospects. In B. Chandrasekaran and S. Radicchi, ed- itors, Computer Program Testing, pages 129±134. North- Holland, 1981. [4] R. Chillarege and N. Bowen. Understanding large system failure ± a fault injection experiment. In Proc. 19th Int. Symp. Fault-Tolerant Computing (FTCS-19), pages 356± 363, Chicago, IL, June 1989. [5] R. DeMillo, T. Li, and A. Mathur. Architecture of TAMER: a tool for dependability analysis of distributed fault-tolerant systems. Technical Report SERC-TR-158-P, Software Engi- neering Research Center, Computer Science Dept., Purdue Univ., 1994. [6] R. DeMillo, R. Lipton, and A. Perlis. Social processes and proofs of theorems and programs. Communications of the ACM, 22(5):803±820, May 1979. [7] R. DeMillo, W. McCracken, R. Martin, and J. Passa®ume. Software Testing and Evaluation. Benjamin/Cummings, Menlo Park, CA, 1987. [8] R. A. DeMillo, D. S. Guindi, W. M. McCracken, A. J. Of- fut, and K. N. King. An extended overview of the mothra sofware testing environment. Proc. ACM SIGSOFT/IEEE
9