University of Texas at El Paso DigitalCommons@UTEP Departmental Technical Reports (CS) Department of Computer Science 11-1-2007 A Fitness Function to Find Feasible Sequences of Method Calls for Evolutionary Testing of Object- Oriented Programs Myoung Yee Kim University of Texas at El Paso, [email protected] Yoonsik Cheon University of Texas at El Paso, [email protected] Follow this and additional works at: http://digitalcommons.utep.edu/cs_techrep Part of the Computer Engineering Commons Comments: Technical Report: UTEP-CS-07-57 Recommended Citation Kim, Myoung Yee and Cheon, Yoonsik, "A Fitness Function to Find Feasible Sequences of Method Calls for Evolutionary Testing of Object-Oriented Programs" (2007). Departmental Technical Reports (CS). Paper 226. http://digitalcommons.utep.edu/cs_techrep/226 This Article is brought to you for free and open access by the Department of Computer Science at DigitalCommons@UTEP. It has been accepted for inclusion in Departmental Technical Reports (CS) by an authorized administrator of DigitalCommons@UTEP. For more information, please contact [email protected]. A Fitness Function to Find Feasible Sequences of Method Calls for Evolutionary Testing of Object-Oriented Programs Myoung Yee Kim and Yoonsik Cheon TR #07-57 November 2007; revised January 2008 Keywords: fitness function, evolutionary testing, genetic algorithms, test data generator, pre and postcondi- tions, object-oriented programming, JML language. 1998 CR Categories: D.2.4 [Software Engineering] Software/Program Verification — class invariants, for- mal methods, programming by contract; D.2.5 [Software Engineering] Testing and Debugging — testing tools (e.g., data generators, coverage testing); D.3.2 [Programming Languages] Language Classifications — Object-oriented languages; F.3.1 [Logics and Meanings of Programs] Specifying and Verifying and Rea- soning about Programs — Assertions, invariants, pre- and post-conditions, specification techniques; I.2.8 [Artificial Intelligence] Problem Solving, Control Methods, and Search — Heuristic methods. To appear in First International Conference on Software Testing, Verification, and Validation, Norway, April 9-11, 2008. Department of Computer Science The University of Texas at El Paso 500 West University Avenue El Paso, Texas 79968-0518, U.S.A. A Fitness Function to Find Feasible Sequences of Method Calls for Evolutionary Testing of Object-Oriented Programs (An Extended Abstract) Myoung Yee Kim and Yoonsik Cheon Department of Computer Science The University of Texas at El Paso 500 West University Avenue El Paso, TX 79968-0518 fmkim2, [email protected] Abstract for example, call sequences that are likely to become feasi- ble are selected and then made to evolve by applying ge- In evolutionary testing of an object-oriented program, netic operations such as mutation and crossover. Thus, for the search objective is to find a sequence of method calls an evolution approach to be effective, it is crucial to identify that can successfully produce a test object of an interesting call sequences that have a potential for quickly becoming state. This is challenging because not all call sequences are feasible. This is the responsibility of the so-called fitness feasible; each call of a sequence has to meet the assumption function or objective function that measures the goodness of the called method. The effectiveness of an evolutionary of candidate solutions. testing thus depends in part on the quality of the so-called In this paper, we propose a new fitness function for evo- fitness function that determines the degree of the fitness of a lutionary testing of object-oriented programs. Our fitness candidate solution. In this paper, we propose a new fitness function is based on assertions such as method precondi- function based on assertions such as method preconditions tions and thus views a call sequence as a tree of assertions to find feasible sequences of method calls. We show through to satisfy. It combines multiple fitness values given by the experiments that to obtain the best search result the fitness assertions of the tree by assigning them different weights. function should consider the structures of method call se- Because of dependencies among method calls, an assertion quences, which are essentially trees of assertions. We also or some parts of it may not be evaluated, thereby producing provide a framework for combining multiple fitness values no or a partial fitness value. Our fitness function can han- and for analyzing different fitness functions. dle this kind of undefinedness in assertions by assigning a penalty for the undefined assertion or term. We performed several experiments with our fitness func- 1 Introduction tion. The main finding from the experiments is that for the best result the fitness function should consider the structures In unit testing object-oriented programs, test objects are of call sequences. In general, it suffices to assign weights constructed indirectly as sequences of method or construc- to assertions based on levels of the assertions in the tree, tor calls. However, it is difficult to find a call sequence that though the optimal weight distribution depends on the com- can successfully produce an object of an interesting state plexities of and dependencies among the assertions. As- because not all call sequences are feasible. A call sequence signing penalties to undefined assertions also improves the is feasible if each call of the sequence, including one from effectiveness of a fitness function. those for creating argument objects, terminates normally without throwing an exception. In general, each call of the sequence has to meet the assumption of the called method, 2 Formulation of the Problem which is often formally written as a runtime checkable as- sertion such as a method precondition. The specific problem of our interest is: how to define an Meta-heuristic information can be used to guide the effective fitness function based on method preconditions to search for feasible call sequences [6]. In genetic algorithms, guide the search for feasible call sequences? The fitness 1 transfer: 1 public class Account { amt > 0 && amt <= acc.bal 2 private /*@ spec_public @*/ int bal; 3 4 /*@ requires amt >= 0; this acc withdraw: amt withdraw: 5 @ assignable bal; amt > 0 && amt <= this.bal amt > 0 && amt <= this.bal 6 @ ensures bal == amt; @*/ 7 public Account(int amt) { bal = amt; } 20 8 this amt this amt 9 / @ requires amt > 0 && amt <= acc.bal; * Account: Account: 10 @ assignable bal, acc.bal; amt >= 0 10 amt >= 0 10 11 @ ensures bal == \old(bal) + amt 12 @ && acc.bal == \old(acc.bal - amt); @*/ amt amt 13 public void transfer(int amt, Account acc) { 14 acc.withdraw(amt); deposit(amt); 10 0 15 } 16 17 /*@ requires amt > 0 && amt <= bal; 18 @ assignable bal; Figure 2. Example assertion tree 19 @ ensures bal == \old(bal) - amt; @*/ 20 public void withdraw(int amt) { 21 bal -= amt; 22 } promising call sequence, thereby expediting the search for 23 a feasible call sequence. By the quality or performance of 24 // The rest of definition ... 25 } a fitness function we mean the search speed measured in terms of the number of generations needed to find a solu- tion, rather than the computational time spent for the search Figure 1. Example JML specification or the fitness calculation. In essence, the fitness function takes an assertion tree as the input and produces a fitness function takes a call sequence as the input and returns a fit- value for the tree by combining the fitness values given by ness value as the output. We assume that the class under the assertions of the tree. consideration is annotated with a runtime-checkable speci- fication; in particular, each method is annotated with a pre- 3 Issues and Approaches condition. We are not much concerned with calculating a fitness value for a single method call; for this, we adopt an There are several issues that need be addressed on calcu- existing approach such as our earlier work [2]. Instead, we lating fitness values for assertion trees, and each of these is- focus on combining multiple fitness values given by multi- sues becomes an independent design dimension in defining ple calls of a call sequence by considering the structure of a fitness function. The three issues that we consider in this the call sequence. paper are: (1) Which nodes—thus assertions—contribute to As we use only the preconditions of method calls in the the calculation of the fitness value of the tree? (2) How fitness calculation, we can abstract call sequences to tree much does each node contribute to the final fitness value of structures called assertion trees, where a leaf node repre- the tree? (3) How to handle undefinedness of the whole or sents a primitive value or a no-argument constructor call, parts of an assertion caused by dependencies among asser- and a non-leaf node represents an object as a method or tions. These issues or design dimensions can also be used constructor calls with the receiver and arguments as its chil- as a framework for analyzing an existing fitness function. dren. A node representing a method or constructor call is Ideally we’d like to consider all the nodes of the tree associated with an assertion, the precondition that the call when calculating the fitness value. However, because of has to satisfy. time and other constraints, it may be advantageous to take As an example, let us consider an Account class shown in into account only a subset of nodes; this often makes sense Figure 1; its behavior is formally specified in JML, a formal because the fitness value is an approximate value anyway. behavioral interface specification language for Java [5]. The We can think of various techniques for choosing a subset of following is an example call sequence for the Account class nodes, e.g., random sampling and selecting nodes based on of which the assertion tree is shown in Figure 2.