• Abstract and Figures
• Public Full-text

Using Fault Injection to Increase Software Test Coverage James M. Bieman Daniel Dreilinger Lijun Lin Computer Science Department Colorado State University Fort Collins, Colorado 80523 [email protected] To appear in Proc. International Symposium on Software Reliability (ISSRE'96) Abstract must be recompiled and then tested. This process can be too labor and time intensive for practical use in the general test- During testing, it is nearly impossible to run all statments ing of commercial software. or branches of a program. It is especially dif®cult to test Our hypothesis is that fault injection can be effective the code used to respond to exceptional conditions. This when it is directed towards solving speci®c testing prob- untested code, often the error recovery code, will tend to be lems. In particular, we use fault injection to force the exe- an error prone part of a system. We show that test cover- cution of dif®cult to reach program paths. To avoid the need age can be increased through an ªassertion violationº tech- for recompilation, we mutate program state rather than pro- nique for injecting software faults during execution. Using gram text. our prototype tool, Visual C-Patrol (VCP), we were able to To be practical, any fault injection mechanism must be substantially increase test branch coverage in four software inexpensive and be able to model a wide variety of fault systems studied. types. Much of the fault injection should be automated. Al- though a practical approach requires a limited fault model, Keywords: software testing, software test coverage, fault we want to simulate as many faults as possible. injection, error recovery, software reliability, Visual C- Patrol. 2. Fault Injection via ªAssertion Violationº 1. Introduction Pre- and post-conditions contain logical assertions or in- variants that specify the expected behavior of a program. A Developing reliable and fault tolerant software is dif®cult pre-condition is an assertion about the nature of the system and requires discipline both in specifying system functional- state that must be true before a function is invoked to be sure ity and in implementing systems correctly. Approaches for that the function will run correctly. A post-condition is an developing highly reliable software include the use of for- assertion that describes the relationship between the input mal methods [11, 14, 9], and rigorous testing methods [2, 7, state and output state of a correct function. These conditions 13]. Testing cannot guarantee that software is correct [17], are usually not explicitly expressed in the program text, but and veri®cation requires enormous human effort and is sub- rather implicitly assumed. ject to errors [6]. Automated support is necessary to help We can create a fault by dynamically changing the state ensure software correctness and fault tolerance. of a running program so that a pre- or post-condition is not Fault injection has been proposed for use in mutation test- satis®ed. Such an arti®cial fault can be used to model ini- ing primarily as a mechanism for evaluating the adequacy of tialization faults, assignment faults, condition check faults, test data [3, 8]. Mutation testing injects faults by modifying and even function faults. program text. As a result, the testing process can generate The assertion violation scheme makes use of pre- and enormous numbers of mutant programs, and each program post-condition assertions. The ®rst step is to de®ne and state these using a new programming language construct. The Research partially supported by the Colorado Advanced Software In- second step is to automatically make them false in different stitute (CASI) and Storage Technology Inc. CASI is sponsored in part by ways and at different program locations while the program the Colorado Advanced Technology Institute (CATI), an agency of the state of Colorado. CATI promotes advanced technology teaching and research at is executing. To create a fault, one of these invariants is vi- universities in Colorado for the purpose of economic development. olated or made false in some fashion. We can automatically generate a number of possible violations of a given assertion The timing impact of this scheme is quite small because using language processing tools, such as lex and yacc. the injected values can be computed statically. An optimiz- The pre- and post-conditions provided by the tester or ing C-compiler can compute the values to be injected, even = 4+0:01 programmer are ®rst order logic predicates about the func- if the source output of the injector is y . tion's input and output parameters, and global variables. We can check the correctness of a system's behavior They use a syntax similar to that used in the source language. before and after injecting faults. A correct pre- or post- condition can be injected as code and checked at run time Example. Suppose the pre-condition for some function using a facility like C assert statements or C-Patrol insertion directives. x > 0 ^ x 6= y g f(¯oat x, y) is f . We can derive several possible faults to inject. The injection is a textual insertion at the beginning of the function. For the example, the injec- 3. Implementing Assertion Violation Fault In- tion code might be the following (using C syntax): jection f sw itch (injection status) To demonstrate the proposed fault injection method, we br eak ; case 0: extended the C-Patrol assertion insertion system [18] to sup- br eak ; case 1: x=0; port fault injection and built a visual X Window System in- br eak ; case 2: x=-1; terface. br eak ; case 3: x=y; br eak ; g case 4: y=x; 3.1. C-Patrol The above block of code can be automatically generated C-Patrol is a code insertion tool that can assist developers by a code pre-processing tool. When injection is off, the in the placement of software probes that are used in testing. injection status variable is set to zero. When injection is on, Such probes may be implementations of data invariants, pre- the injection status variable is set randomly to a value be- and post-conditions, or other run time checks. An overall 4 tween 1 and . The number of injections and the probability view of the C-Patrol system is shown in Figure 1. of choosing a particular injection can be determined by the C-Patrol allows developers to de®ne and place assertions user. Typically, an injection will cause a short chain reac- in logical locations in a C program. For example, a devel- tion of violated assertions. The invalid assertions should be oper may prefer to place a data invariant along with an asso- restored by error recovery code in fault tolerant software. ciated data declaration. A preprocessor under program con- The code mutation aspect of this scheme can be per- trol inserts these assertions into the correct locations for run formed by a pre-processor, which transforms pre- and post- time monitoring of a program. The data invariant is thus conditions into case injection statements and inserts them checked at the locations where the data structure is actually into a program. used, rather than where it is de®ned. Using C-Patrol, virtual An injector tool could accept statements from the propo- code is de®ned within comments, and is inserted into regular sitional calculus. In our prototype system, we implemented code using possibly parameterized directives. A special la- a subset which speci®cally includes statements using the al- beling system matches directives to virtual code. When the phabet: probes are not needed, they remain with the program as com- ments that do not affect program execution. identif ier;<;<=;==; !=;>=;>; f Our initial intention was for C-Patrol software probes to l iter al ; f l oat l iter al ; ^g integ er . be used to check the correctness of program behavior. How- Valid strings in the language are conjunctions of boolean ever, probes can also be used to modify the system state and terms (equalities or inequalities). Ideally, the language will thus create any desired or undesired state. Probes can gen- erate states that would result from software and hardware ; _; 8; 9g be augmented with f: . An example of a recogniz- able precondition in the system is: faults. Thus, C-Patrol can be used to inject faults at chosen points in a program. a<3 ^ b!=c^d>1:5: C-Patrol is a pre-processor that inserts assertions, writ- ten as comments in virtual C, into speci®ed locations in C Many possible injections can be derived from this state- programs. The user places virtual C code within special C- ment. In our prototype system, we inject violations that are Patrol comments that are delimited by ª/*?ºandª?*/º. near the boundary to simulate some common programming These comments are skipped by the C compiler and do not 3 a =3 b!=c errors. Thus, a< is mapped to ,gets mapped affect the performance of the underlying system. Once the = c c = b d>1:5 to two possible faults: b and , and ®nally, user activates the comments by running the pre-processor, =1:5 is mapped to d .the pre-processor translates virtual code into regular C and 2 Special comments turned on C source code with C - Patroller special comments inserted C - Patrol C Code (C code w/ special comments) typedef structure LIST{ char *name; int value; sturct LIST *next; } ... /*? template ordered(alist): assert(is_ordered(alist)); ?*/ ... C Compiler main() { build_list_in_order(list1); Special comments not turned on /*?%%insert: %%calll ordered(list1; ?*/ build_list_in_order(list2); /*? %%insert: %%call ordered(list2); ?*/ list3 = merge_lists(list1, list2); Figure 1. The C-Patrol System inserts it based on a set of instructions called directives.The 2.

Load more

  9

Copy Link