Using Formal Metho ds To DeriveTest Frames In Category-Partition Testing y Paul Ammann Je O utt [email protected] [email protected] Department of Information and Software Systems Engineering George Mason University,Fairfax, VA22030 Abstract category-partition testing is the formalization of the notion of a test speci cation, whichisanintermediate Testing is a standard method of assuring that software do cument designed to bridge the large gap b etween performs as intended. In this paper, we extend the functional sp eci cations and actual test cases. Some category-partition method, which is a speci cation- parts of a test sp eci cation can b e derived mechani- based testing method. Animportant aspect of category- cally from a functional sp eci cation. Other parts re- partition testing is the construction of test speci ca- quire the test engineer to make decisions or rely on tions as an intermediate between functional speci ca- exp erience. We wish to mechanize as many tasks as tions and actual tests. We de ne a minimal cover- p ossible. age criterion for category-partition test speci cations, In this pap er, we address some imp ortant asp ects identify a mechanical process to produce a test spec- of constructing a test sp eci cation that are left op en i cation that satis es the criterion, and discuss the in the category-partition metho d. Sp eci cally,we problem of resolving infeasible combinations of choices de ne a minimal coverage criterion for category- for categories. Our method uses formal schema-based partition testing, and supply a general pro cedure functional speci cations and is shown to befeasible for enumerating tests that satisfy the criterion with an example study of a simple le system. supply a pro cedure for deriving test cases from test scripts 1 Intro duction supply a metho d of identifying and eliminating infeasible tests caused by con icting choices. Testing software is a standard, though imp erfect, metho d of assuring software quality. The general aim A side e ect of our metho d is that we are sometimes of the research re ected in this pap er is to formal- able to uncover anomalies in the functional sp eci ca- ize, and mechanize where p ossible, routine asp ects of tion. This allows us to, in some cases, detect unsatis - testing. Such formalization has two b ene ts. First, it able as de ned by Kemmerer [Kem85] sp eci cations. makes it easier to analyze a given test set to ensure Weemploy formal metho ds, in particular Z sp eci- that it satis es a sp eci ed coverage criterion. Second, cations, as a to ol in our investigation of test gener- it frees the test engineer to concentrate on less formal- ation. There are several motivations for using formal izable, and often more interesting tests. metho ds. First, some of the analysis necessary to pro- Sp eci cation-based testing, or black-b ox testing, duce a test sp eci cation has already b een done for a relies on prop erties of the software that are captured in formal functional sp eci cation, and hence less e ort is the functional sp eci cation, as opp osed to the source required to pro duce a test sp eci cation from a formal co de. The category-partition metho d [BHO89, OB88 ] functional sp eci cation. Second, using formal meth- is a sp eci cation-based test metho d that has received ods makes the determination of whether part of a test considerable attention. An imp ortant contribution of sp eci cation results from a mechanical pro cess or from the test engineer's judgement more ob jective. Finally, Partially supp orted by the National Aeronautics and Space formal metho ds are well suited to describing artifacts Administration under grantNAG 1-1123-FDP. y of the testing pro cess itself. Such artifacts include Partially supp orted by the National Science Foundation un- der grant CCR-93-11967. parts of the test sp eci cation and actual test cases. 1.1 Related Work The steps in the category-partition metho d that lead to a test sp eci cation are as follows. Avariety of researchers haveinvestigated the use of formal metho ds in test generation. Kemmerer sug- 1. Analyze the sp eci cation to identify the individ- gested ways to test formal sp eci cations for such prob- ual functional units that can b e tested separately. lems as b eing unsatis able [Kem85]. In the DAISTS 2. Identify the input domain, that, is the parameters system of Gannon, McMullin, and Hamlet [GMH81], and environment variables that a ect the b ehav- axioms from an algebraic sp eci cation and test p oints ior of a functional unit. sp eci ed by a test engineer are used to sp ecify test sets for abstract data typ es. Hayes [Hay86] exploits the re- 3. Identify the categories, which are the signi - nement of an abstract Z sp eci cation to a more con- cantcharacteristics of parameter and environ- crete sp eci cation to sp ecify tests at the design level. mentvariables. Amla and Ammann [AA92] describ ed a technique in which category-partition tests are partially sp eci ed 4. Partition each category into choices. by extracting information captured in Z sp eci cations of abstract data typ es. Layco ck[Lay92] indep endently 5. Sp ecify combinations of choices to b e tested, in- derived similar results. More recently,Stocks and Car- stantiate test cases by sp ecifying actual data val- rington [SC93a, SC93b]have used formal metho ds to ues for eachchoice, and determine the corre- describ e test artifacts, sp eci cally test frames sets of sp onding results and the changes to the environ- test inputs that satisfy a common prop erty and test ment. heuristics sp eci c metho ds for testing software. This pap er fo cuses on the last step ab ove. Each sp eci ed combination of choices results in a test frame. 1.2 Outline of pap er The category-partition metho d relies on the test en- gineer to determine constraints among choices to ex- In section 2, we review the steps in the category- clude certain test frames. There are two reasons to partition metho d. The Z constructs that we need exclude a test frame from consideration. First, the are describ ed in section 3; further information may test engineer may decide that the cost of building a b e found in the Z reference manual [Spi89]oroneof test script for a test frame exceeds the likely b ene- the many Z textb o oks [Dil90, PST91, Wor92]. Sec- ts of executing that test. Second, a test frame may tion 4 de nes a minimalcoverage criterion for cate- b e infeasible, in that the intersection of the sp eci ed gory partition testing, gives a pro cedure for deriving choices is the empty set. Recently,Grochtmann and sp eci cations that satisfy the criterion, and de nes a Grimm [GG93]havedevelop ed classi cation trees,a metho d to pro duce test scripts from the resulting test hierarchical arrangement of categories that avoids the sp eci cations. Section 5 presents partial Z sp eci ca- intro duction of infeasible combinations of choices. tions, test sp eci cations, test frames, and test case The develop ers of the category-partition metho d scripts for an example system. Finally,we summarize have de ned a test sp eci cation language called TSL our results and ndings in Section 6. [BHO89]. A test case in TSL is an op eration and values for its parameters and relevantenvironment variables. A test script in TSL consists of the op era- 2 Category-Partition Metho d tions necessary to create the environmental conditions The category-partition metho d [BHO89, OB88] is a called the SETUP p ortion, the test case op eration, sp eci cation-based testing strategy that uses an in- whatever command is necessary to observe the a ect formal functional sp eci cation to pro duce formal test of the op eration VERIFY in TSL, and anyexitcom- sp eci cations. The category-partition metho d o ers mand CLEANUP in TSL. Test sp eci cations writ- the test engineer a general pro cedure for creating test ten in TSL can b e used to automatically generate test sp eci cations. The test engineer's key job is to develop scripts. The test engineer may optionally givespe- categories, which are de ned to b e the ma jor charac- ci c representative values for any given choice to aid teristics of the input domain of the function under test, the test generation to ol in deriving sp eci c test cases. and to partition each category into equivalence classes The category-partition metho d supplies little explicit of inputs called choices. By de nition, choices in each guidance as to whichcombinations of choices are de- category must b e disjoint, and together the choices in sirable { the task is left mostly to the test engineer's each category must cover the input domain. judgement. 2 1. Testable units corresp ond to op erations on the In this work, we follow the spirit of the category- ADT. partition metho d, but there are di erences in our use of the technique. First, we base our deriva- 2. Parameters inputs are explicitly identi ed with tion on formal sp eci cations of the software, since, a trailing \?". Environmentvariables ADT as has b een demonstrated in a variety of pap ers state comp onents are the variables of the state [AA92,Lay92, SC93a, SC93b], the formalityofthe schema.