Using Formal Metho ds To DeriveTest Frames In
Category-Partition Testing
y
Paul Ammann Je O utt
[email protected] [email protected]
Department of Information and Software Systems Engineering
George Mason University,Fairfax, VA22030
Abstract category-partition testing is the formalization of the
notion of a test speci cation, whichisanintermediate
Testing is a standard method of assuring that software
do cument designed to bridge the large gap b etween
performs as intended. In this paper, we extend the
functional sp eci cations and actual test cases. Some
category-partition method, which is a speci cation-
parts of a test sp eci cation can b e derived mechani-
based testing method. Animportant aspect of category-
cally from a functional sp eci cation. Other parts re-
partition testing is the construction of test speci ca-
quire the test engineer to make decisions or rely on
tions as an intermediate between functional speci ca-
exp erience. We wish to mechanize as many tasks as
tions and actual tests. We de ne a minimal cover-
p ossible.
age criterion for category-partition test speci cations,
In this pap er, we address some imp ortant asp ects
identify a mechanical process to produce a test spec-
of constructing a test sp eci cation that are left op en
i cation that satis es the criterion, and discuss the
in the category-partition metho d. Sp eci cally,we
problem of resolving infeasible combinations of choices
de ne a minimal coverage criterion for category-
for categories. Our method uses formal schema-based
partition testing, and supply a general pro cedure
functional speci cations and is shown to befeasible
for enumerating tests that satisfy the criterion
with an example study of a simple le system.
supply a pro cedure for deriving test cases from
test scripts
1 Intro duction
supply a metho d of identifying and eliminating
infeasible tests caused by con icting choices.
Testing software is a standard, though imp erfect,
metho d of assuring software quality. The general aim
A side e ect of our metho d is that we are sometimes
of the research re ected in this pap er is to formal-
able to uncover anomalies in the functional sp eci ca-
ize, and mechanize where p ossible, routine asp ects of
tion. This allows us to, in some cases, detect unsatis -
testing. Such formalization has two b ene ts. First, it
able as de ned by Kemmerer [Kem85] sp eci cations.
makes it easier to analyze a given test set to ensure
Weemploy formal metho ds, in particular Z sp eci-
that it satis es a sp eci ed coverage criterion. Second,
cations, as a to ol in our investigation of test gener-
it frees the test engineer to concentrate on less formal-
ation. There are several motivations for using formal
izable, and often more interesting tests.
metho ds. First, some of the analysis necessary to pro-
Sp eci cation-based testing, or black-b ox testing,
duce a test sp eci cation has already b een done for a
relies on prop erties of the software that are captured in
formal functional sp eci cation, and hence less e ort is
the functional sp eci cation, as opp osed to the source
required to pro duce a test sp eci cation from a formal
co de. The category-partition metho d [BHO89, OB88 ]
functional sp eci cation. Second, using formal meth-
is a sp eci cation-based test metho d that has received
ods makes the determination of whether part of a test
considerable attention. An imp ortant contribution of
sp eci cation results from a mechanical pro cess or from
the test engineer's judgement more ob jective. Finally,
Partially supp orted by the National Aeronautics and Space
formal metho ds are well suited to describing artifacts
Administration under grantNAG 1-1123-FDP.
y
of the testing pro cess itself. Such artifacts include
Partially supp orted by the National Science Foundation un-
der grant CCR-93-11967. parts of the test sp eci cation and actual test cases.
1.1 Related Work The steps in the category-partition metho d that
lead to a test sp eci cation are as follows.
Avariety of researchers haveinvestigated the use of
formal metho ds in test generation. Kemmerer sug-
1. Analyze the sp eci cation to identify the individ-
gested ways to test formal sp eci cations for such prob-
ual functional units that can b e tested separately.
lems as b eing unsatis able [Kem85]. In the DAISTS
2. Identify the input domain, that, is the parameters
system of Gannon, McMullin, and Hamlet [GMH81],
and environment variables that a ect the b ehav-
axioms from an algebraic sp eci cation and test p oints
ior of a functional unit.
sp eci ed by a test engineer are used to sp ecify test sets
for abstract data typ es. Hayes [Hay86] exploits the re-
3. Identify the categories, which are the signi -
nement of an abstract Z sp eci cation to a more con-
cantcharacteristics of parameter and environ-
crete sp eci cation to sp ecify tests at the design level.
mentvariables.
Amla and Ammann [AA92] describ ed a technique in
which category-partition tests are partially sp eci ed
4. Partition each category into choices.
by extracting information captured in Z sp eci cations
of abstract data typ es. Layco ck[Lay92] indep endently
5. Sp ecify combinations of choices to b e tested, in-
derived similar results. More recently,Stocks and Car-
stantiate test cases by sp ecifying actual data val-
rington [SC93a, SC93b]have used formal metho ds to
ues for eachchoice, and determine the corre-
describ e test artifacts, sp eci cally test frames sets of
sp onding results and the changes to the environ-
test inputs that satisfy a common prop erty and test
ment.
heuristics sp eci c metho ds for testing software .
This pap er fo cuses on the last step ab ove. Each
sp eci ed combination of choices results in a test frame.
1.2 Outline of pap er
The category-partition metho d relies on the test en-
gineer to determine constraints among choices to ex-
In section 2, we review the steps in the category-
clude certain test frames. There are two reasons to
partition metho d. The Z constructs that we need
exclude a test frame from consideration. First, the
are describ ed in section 3; further information may
test engineer may decide that the cost of building a
b e found in the Z reference manual [Spi89]oroneof
test script for a test frame exceeds the likely b ene-
the many Z textb o oks [Dil90, PST91, Wor92]. Sec-
ts of executing that test. Second, a test frame may
tion 4 de nes a minimalcoverage criterion for cate-
b e infeasible, in that the intersection of the sp eci ed
gory partition testing, gives a pro cedure for deriving
choices is the empty set. Recently,Grochtmann and
sp eci cations that satisfy the criterion, and de nes a
Grimm [GG93]havedevelop ed classi cation trees,a
metho d to pro duce test scripts from the resulting test
hierarchical arrangement of categories that avoids the
sp eci cations. Section 5 presents partial Z sp eci ca-
intro duction of infeasible combinations of choices.
tions, test sp eci cations, test frames, and test case
The develop ers of the category-partition metho d
scripts for an example system. Finally,we summarize
have de ned a test sp eci cation language called TSL
our results and ndings in Section 6.
[BHO89]. A test case in TSL is an op eration and
values for its parameters and relevantenvironment
variables. A test script in TSL consists of the op era-
2 Category-Partition Metho d
tions necessary to create the environmental conditions
The category-partition metho d [BHO89, OB88] is a called the SETUP p ortion , the test case op eration,
sp eci cation-based testing strategy that uses an in- whatever command is necessary to observe the a ect
formal functional sp eci cation to pro duce formal test of the op eration VERIFY in TSL , and anyexitcom-
sp eci cations. The category-partition metho d o ers mand CLEANUP in TSL . Test sp eci cations writ-
the test engineer a general pro cedure for creating test ten in TSL can b e used to automatically generate test
sp eci cations. The test engineer's key job is to develop scripts. The test engineer may optionally givespe-
categories, which are de ned to b e the ma jor charac- ci c representative values for any given choice to aid
teristics of the input domain of the function under test, the test generation to ol in deriving sp eci c test cases.
and to partition each category into equivalence classes The category-partition metho d supplies little explicit
of inputs called choices. By de nition, choices in each guidance as to whichcombinations of choices are de-
category must b e disjoint, and together the choices in sirable { the task is left mostly to the test engineer's
each category must cover the input domain. judgement. 2
1. Testable units corresp ond to op erations on the In this work, we follow the spirit of the category-
ADT. partition metho d, but there are di erences in our
use of the technique. First, we base our deriva-
2. Parameters inputs are explicitly identi ed with
tion on formal sp eci cations of the software, since,
a trailing \?". Environmentvariables ADT
as has b een demonstrated in a variety of pap ers
state comp onents are the variables of the state
[AA92,Lay92, SC93a, SC93b], the formalityofthe
schema.
functional sp eci cation helps to simplify and organize
the pro duction of a test sp eci cation. Second, wedo
3. Categories haveavariety of sources. Some cat-
not follow the TSL syntax, but instead format exam-
egories are derived from preconditions and typ e
ples as is convenient for our presentation. Sp eci cally,
information ab out the inputs and state com-
as has b een done by others [SC93a,SC93b], we employ
ponents. Typically, there are additional desir-
the formal sp eci cation notation to describ e asp ects of
able categories that cannot b e derived from the
the tests themselves as well as to describ e functional
sp eci cation; the test engineer must derive these
b ehavior.
from knowledge and exp erience. Recentwork
[SC93a, SC93b] p oints to other sources of cate-
gories in formal sp eci cations.
3 Z Sp eci cations
4. Some categories, particularly those that are based
Z is a mo del-based formal sp eci cation language based
on preconditions, partition naturally into normal
on typ ed set theory and logic. In this pap er, wefo-
cases and unusual or illegal cases. Partitions
cus our attention on Z sp eci cations for abstract data
for other categories dep end on the semantics of
typ es ADTs . An ADT is characterized by sp eci-
the system, and often require the test engineer's
ed states and op erations that observe and/or change
judgement.
state.
In Z, b oth states and op erations on states are de-
5. To determine whichcombinations of choices to
scrib ed with schemas.Aschema has three parts; a
test, there are few general rules to b e found in ei-
name,asignature to de ne variables, and a predicate
ther the Z sp eci cations or in the previous work in
to constrain the variables. Aschema describing an
category-partition testing [BHO89, OB88]. Sup-
ADT state has a signature whose variables de ne a
plying a minimal set of rules by de ning a cov-
set-based mo del for the ADT and a predicate that
erage criterion is one of the contributions of the
sp eci es invariants on the ADT. A schema describing
currentpaper.For the veri cation of outputs, the
an ADT op eration has a signature that de nes the
state invariants and p ostconditions are helpful.
inputs, outputs, state variables from the prior state,
and state variables from resulting state. The predicate
It has also b een observed that Z schemas are ideal
of an op eration schema constrains the variables with
constructs for describing test frames [SC93a, SC93b].
preconditions and p ostconditions.
In this case, the signature part lists the variables that
By usual convention in Z, input variables are deco-
make up p ossible test inputs and the predicate part
rated by a trailing \?", and output variables are dec-
constrains the variables as determined by the reason
orated by a trailing \!". State variables decorated by
for the test. For example, a test frame intended to
0
a trailing \ " indicate the state of the variable after
cover a statement in a program has a predicate part
an op eration is applied. By way of abbreviation, one
that gives the path expression that causes the ow
schema can b e included in another by using the name
of control to reach that statement. A Z schema used
of the included schema in the signature of the new
to describ e a test frame typically describ es a set of
schema. By convention, if the included schema name
p ossible inputs; a re nement pro cess must b e used
is pre xed with a , then the new schema maychange
to select an element from the set b efore the test can
the state variables of the included schema. If an in-
actually b e executed. If the set of p ossible inputs is
cluded schema name is pre xed with a , the op era-
empty, the test frame is infeasible.
tion may not change the state variables of the included
schema. Instances of schemas are given in the exam-
ples of Section 5.
4 Derivation of Test Scripts
A partial mapping of Z constructs to category-
partition test sp eci cations is given by Amla and Am- The previous category-partition work do es not givea
mann [AA92]. We brie y recap ma jor p oints b elow: prescription for whichcombinations of choices should 3
b e sp eci ed as test frames. Selecting the combina- by selecting unused choices in each category where it
tions of choices used is an imp ortant problem whose is p ossible to do so results in undesirable test sets.
solution a ects the strength and eciency of testing. The reason is that a system typically has some nor-
In general, there is no single solution that applies in mal mo de of op eration, and that normal mo de corre-
1
all applications, but it is nonetheless clear that some sp onds to a particular choice in each category. Call
p ossibilities can b e rejected as inadequate. the choice in each category that corresp onds to the
In TSL sp eci cations [BHO89], a sp ecial syntax in normal mo de the base choice. It is useful for a tester to
the form of conditionals in the RESULTS sections is evaluate how a system b ehaves in mo des of op eration
provided to sp ecify combinations of choices. However, obtained byvarying from the normal mo de though
the TSL syntax only supp orts a way to sp ecify combi- the non-base choices in each category.We de ne the
nations of choices. Although the authors suggest test- base-choice-coverage criterion to describ e suchatest
ing certain error conditions only once, deciding which set.
choices to sp ecify is left to the test engineer. To satisfy base-choice-coverage, for eachchoice in a
Using all combinations of categories is generally in- category,we combine that choice with the base choice
ecient, and the corresp onding tests are rep etitious for all other categories. This causes each non-base
and uninteresting. If we generate all combinations, choice to b e used at least once, and the base choices
where there are N categories and the kth category has to b e used several times.
i choices, then the numb er of resulting test frames is: The numb er of test frames generated to satisfy ei-
k
ther each-choice-used or base-choice-coverage is linear
N
Y
in the number of choices, rather than the combina-
i ; 1
k
torial numb er from formula 1. Thus there is no sig-
k =1
ni cant cost advantage of each-choice-used over base-
which is combinatorial in cost. For example, con-
choice-coverage, and so we adopt base-choice-coverage
sider a test sp eci cation with two categories X and
as our minimal coverage criterion for category parti-
Y , where X has three choices and Y has two:
tion test sets.
The exact numb er of test frames for base-choice-
Categories: X Y
coverage criterion for N categories where the category
* P * Q
1 1
k has i choices is:
k
* P * Q
2 2
* P
N
3
X
i N +1: 2
k
k =1
Let us denote a test frame that satis es choices P
i
Note that satisfying base-choice-coverage do es not
and Q by[P ; Q ]. The example sp eci cation de nes
j i j
mean that a test set is adequate for a particular ap-
six p ossible test frames: [P ; Q ], [P ; Q ], [P ; Q ],
1 1 1 2 2 1
plication. In general, given a test set that satis es
[P ; Q ], [P ; Q ], and [P ; Q ]. To construct a test
2 2 3 1 3 2
base-choice coverage, the test engineer would wantto
set, one cho oses from zero to six of the p ossible test
6
add other useful test frames. Mechanically supplying
or 64 p ossible sets of test frames, yielding a total of 2
a test set that satis es base-choice-coverage frees the
frames including the empty set . Although only six
test engineer to concentrate on these additional tests.
decisions need to b e made whether to include each
A strong argument can b e made, however, that a test
test frame , the test frames are interrelated, and so
set that do es not satisfy base-choice-coverage is inad-
the six decisions cannot b e made in isolation. Thus
equate. While this typ e of argument is not a strong
we are left with the question; which of these test sets
as one would prefer, it is similar to the arguments
should b e sp eci ed?
supp orting the use of program-based coverage crite-
We pro ceed by de ning a minimal metric that any
ria such as statementcoverage or data- owcoverage
reasonable selection of choice combinations should sat-
criteria such as all-uses coverage.
isfy. The basic principle is that eachchoice should b e
used at least once, otherwise there is no reason to have
distinguished that choice from other choices in a cat-
4.1 The Derivation Pro cedure
egory. Let a set of test frames that incorp orates each
The following pro cedure creates test scripts that are
choice at least once b e de ned to satisfy the each-
based on the category-partition metho d. This pro cess
choice-used criterion.
1
As it turns out, the each-choice-used criterion is not
Many systems havemultiple normal mo des of op eration,
very useful in practice. De ning successive test frames but we consider only one normal mo de for simplicity here. 4
gives a recip e for the last step of the metho d, and can Deciding whether a combination is infeasible is
b e used to generate the TSL's RESULTS section. equivalent to deciding whether the constraints in-
volved can b e satis ed. Although satis abilityis
1. Create a combination matrix.
a hard problem, to ols exist that are capable of
A convenientway to organize the values for the
resolving many common cases. For example, the-
inputs and environmentvariables in test frames
orem proving systems can handle prop ositional
is with an N -dimensional combination matrix,
and predicate calculus, as well as simple arith-
where N is the numb er of categories and eachdi-
metic prop erties, and hence could b e used as an
mension represents the choices of a category.The
intelligent assistant to the test engineer.
entries in the matrix are constraints that sp ecify
a test frame for the intersection of choices. This
5. Re ne test frames into test cases.
matrix is intended as a conceptual to ol that helps
Each test frame represents a combination of
describ e our pro cess, rather than something that
choices, and thus is a set of candidate test inputs
will actually b e constructed in practice.
for that frame. To actually execute a test, an ele-
ment from the test set must b e chosen. Although
An example, explained further b elow, is the com-
we do not concentrate on this asp ect of test gen-
bination matrix for the two categories, X and Y :
eration in this pap er, wedonotethatitisasub-
ject of serious inquiry.For example, DeMillo and
Y
O utt [DO91]havedevelop ed algorithms to au-
Q Q
1 2
tomatically generate test cases from constraints,
P 1
1
and Wild et al. are developing techniques to em-
X P 2 3
2
ployaccumulated knowledge to re ne test frames
P 4 5
3
[WZFC92].
2. Identify a base test frame.
6. Write op eration commands.
For each partitioned category and each op era-
For each cellinthecombination matrix that is
tion, the test engineer designates one choice to
chosen, the corresp onding op eration commands,
b e the base choice. This is typically either a de-
setup commands, verify commands, and cleanup
fault choice, or a \normal" choiceastaken from
commands must b e written. Although we exp ect
an exp ected user pro le. The base test frame is
that the test engineer needs to sp ecify the actual
constructed by selecting the base choice from each
commands, not all p ossible scripts are needed.
category. In the ab ove example, we assume that
Thus there is no need to enumerate the entire
[P ; Q ] is the base test frame.
1 1
matrix, whichiswhy this step comes towards the
3. Cho ose other combinations as test frames.
end. For many systems the cleanup command s
The combination matrix can b e used to automat-
are constant for the entire system and only need
ically cho ose the remaining test frames byvarying
to b e sp eci ed once, and only one verify command
over choices in each category. In e ect, westart
will b e needed for each op eration. Muchofthis
at the base cell in the matrix, and successively
step can b e automated, but to ols to do so are
cho ose each cell in every linear direction from the
pro ject sp eci c rather than general purp ose.
base cell. In the ab ove example, if [P ; Q ] forms
1 1
7. Create test scripts.
the base frame, then we also cho ose [P ; Q ],
1 2
Creating the actual test scripts is a straightfor-
[P ; Q ], and [P ; Q ].
2 1 3 1
ward pro cess. For eachchosen cell, the corre-
4. Identify infeasible combinations.
sp onding setup commands are chosen, the com-
In the combination matrix, each cell can b e anno-
mand is app ended, then the verify and cleanup
tated with a numb er that corresp onds to a setup
commands are attached. The commands can b e
script, or left blank if that combination is im-
given to an automated to ol, which can combine
p ossible. If an imp ossible choice a blank cell is
them to automatically generate the actual test
reached, we shift the test frame byvarying other
scripts, execute them, and provide the results to
choices until we reachacombination of choices
the tester.
that is p ossible. Since [P ; Q ] in the ab ove ex-
1 2
The complete generation of test scripts can b e done ample is blank, we shift the test frame bymoving
prior to the design; in fact, any time after the func- P to P to get test frame [P ; Q ]. Note that
1 2 2 2
tional sp eci cations are written. By reusing the pro- when we shift, we shift away from the base frame. 5
cedure, it is also easy to mo dify the test scripts once 5.1 State Description
they are created.
The state of FileSystem is represented by the direc-
tories in the system dirs , the les les , and a
5 The MiStix Example
currentworking directory cwd . The Z schema for
FileSystem is as follows:
We demonstrate our pro cedure on an example sys-
tem. MiStix is based on the Unix le system, and has
FileSystem
b een used in exercises in graduate software engineer-
les : P Ful lName
ing classes at George Mason University.TheMistix
dirs : P Ful lName
sp eci cation is similar to, although simpler than, the
cwd : Ful lName
Unix le system sp eci cation develop ed as a Z case
8 f : les [ dirs f 6= hi front f 2 dirs
study in Hayes [Hay93]. There are a total of ten op-
erations de ned in MiStix:
cwd 2 dirs
Two op erations to create and delete directories
Two op erations to create and delete les
FileSystem has three comp onents in its signature,
les , dirs , and cwd ,andtwoinvariants in the predi-
Two op erations to copyandmove les
cate. The rst comp onent, les , is the set of les that
currently exist in the system. The P in P Ful lName
One op eration to change the current directory
is the p owerset constructor and is read \set of ". The
One op eration to print the full pathname of the
second comp onent, dirs , is the set of directories that
current directory
currently exist in the system. The last comp onent,
cwd , do es not record any p ermanent feature of the le
One op eration to list les and directories
system, but is instead used to mark a user's current
working directory. The rst invariant states that, with
One op eration to log o
the exception of the ro ot directory hi, for a le or di-
Because the complete sp eci cation is quite lengthy,
rectory to exist, it must b e in a valid directory. As
we fo cus on one of the ten op erations, CreateDir.The
a note on Z notation, the in the rst invariantmay
complete functional and test sp eci cations for Mistix
b e read, \it is the case that". The second invariant
are in [AO93]. We sp ecify MiStix as an ADT, b e-
states that cwd must b e an existing directory.Note
ginning with a description of the base typ es needed.
that there is no constraint that prohibits les and di-
There are twotyp es of ob jects in the system: les and
rectories from sharing the same name, although such
directories. The typ e Name is used to lab el a simple
a constraint might b e desirable and could b e easily
le or directory name for example, the MiStix le
added, by including a third predicate les \ dirs = ;.
\fo o" :
[Name ]
5.2 Example MiStix Op eration
We denote constants of typ e Name with double quoted
By way of example, wegive the sp eci cations for
strings, as in the example ab ove.
one representative op eration, CreateDir . The English
Sequences of Name are full le or directory names
sp eci cation for the op eration CreateDir is as follows:
for example, the MiStix le \/usr/bin/fo o" :
Ful lName ::= seq Name
CreateDir n ?
If the name n ? is not already in the current direc-
The representation chosen here has the leaf elements
tory, create a new directory called n ?asasub-
at the tail of the sequence, and so, for example,
directory of the current directory, else printan
the representation of \/usr/bin/fo o" is the sequence
appropriate error message.
husr ; bin ; foo i.We use the sequence manipulation
functions front , which yields a subsequence up to the
last element e.g., front husr ; bin ; foo i =husr ; bin i The Z formal sp eci cation for CreateDir is as fol-
and last , which yields the element at the end of the lows the sp eci cation for the error message has b een
sequence e.g., last husr ; bin ; foo i = foo . omitted here : 6
CreateDir of categories is preconditions on op erations. Precondi-
FileSystem tions are go o d sources for categories b ecause they are
n ?:Name precisely the predicates on the domain of a testable
unit that distinguish normal op eration from unde ned
_
cwd hn ?i 2= dirs
or erroneous op eration. For CreateDir , the precondi-
0 _
tion is that the directory to b e created do es not al-
dirs = dirs [fcwd hn ?ig
ready exist. Twochoices for a category based on the
precondition are that the directory to b e created do es
CreateDir mo di es the state of the le system
not yet exist and that it already exists.
hence the FileSystem , and takes the new directory
Another source of categories is revealed by examin-
name, n ?, as an input. The rst predicate, the pre-
ing other parts of the MiStix sp eci cation and noting
condition for the op eration, is that the directory to b e
that variables of typ e Name can assume twospecial
created, the concatenation of the currentworking di-
values. One sp ecial value, whichwe denote PARENT ,
_
rectory with the new name cwd hn ? i , do es not al-
is the value of n ? used when referring to the parent
ready exist. The second predicate, the p ostcondition,
directory. The sp ecial value PARENT corresp onds to
0
adds the new directory to the set dirs . Remember
the \.." in Unix lename sp eci cations. The b ehavior
0
that dirs denotes the value of the dirs environment
of CreateDir with resp ect to a request to create a le
variable after execution of the op eration. Note that
named PARENT is technically allowed by the formal
we do not need to sp ecify that cwd is valid, since the
sp eci cation, but clearly represents a case that should
FileSystem predicates ensure that.
b e disallowed. This is an advantageous side e ect of
deriving test frames in this manner; deriving test data
5.3 Tests For CreateDir
based on the functional sp eci cations can lead the test
engineer to identifying anomalies in the sp eci cations
In this section we apply the metho d of Amla and Am-
themselves.
mann [AA92] to part of MiStix. The remainder of
Another sp ecial value of typ e Name ,whichwede-
the test sp eci cation for MiStix is similar, although
note ROOT ,isthevalue of n ? used when referring
lengthy, and can b e found in the technical rep ort
to the ro ot directory. The sp ecial value ROOT corre-
[AO93].
sp onds to the empty string.
The rst step in category-partition testing as given
The schema for CreateDir suggests two more cate-
in section 2 is to identify the testable units, whichin
gories. One is based on whether the currentworking
this case include CreateDir .
directory cwd istheempty sequence i.e. ro ot . The
The second step is identi cation of the inputs and
empty sequence is a typical sp ecial case for sequences.
environment state variables for CreateDir . From
The last category we employ is the state of the direc-
the syntax of the op eration, it is clear that n ?of
tories set dirs . The motivation for this category is
typ e Name is the explicit input and that dirs and
that if it matters if cwd is the empty sequence ro ot ,
cwd are the state variables of interest. Formally,we
cwd would always b e ro ot if the only existing directory
can describ e the input domain for CreateDir with the
is the ro ot directory.
schema:
The fourth step is partitioning the categories, some
CreateDir
asp ects of whichhave already b een discussed. The test
Input Domain
FileSystem
sp eci cations for CreateDir after the rst four steps of
n ?:Name
the category-partition metho d are:
Note that the schema FileSystem includes b oth the
declarations for dirs and cwd and also constraints on
Functional Unit:
2
the values dirs and cwd can take.
CreateDir
The third step is the identi cation of the categories,
or imp ortantcharacteristics of the inputs. One source
Inputs:
2
Since les is neither examined nor changed in CreateDir ,
n ?:Name
les is not a relevant state variable to the op eration. As a
technical p oint, we could capture this fact by using the Z
EnvironmentVariables:
schema hiding op erator to hide the variable les in the schema
dirs : P Ful lName
CreateDir ,butwe elect not to do so for the re-
Input Domain
mainder of the example. cwd : Ful lName 7
CreateDir
Base Test Frame
Categories: FileSystem
Category { Precondition n ?:Name
_
Choice 1 Base : cwd hn ?i 2= dirs
_
cwd hn ?i 2= dirs
_
Choice 2: cwd hn ?i2dirs
Category { Typ e of n ?
n ? 6= ROOT ^ n ? 6= PARENT
Choice 1 Base : n ? 6= ROOT ^ n ? 6= PARENT
cwd 6= hi
Choice 2: n ?=ROOT
Choice 3: n ?=PARENT
dirs 6= fhig
Category { Typ e of cwd
Choice 1 Base : cwd 6= hi
The source of each of the explicitly listed predicates
Choice 2: cwd = hi
in CreateDir is as follows. The predi-
Base Test Frame
Category { Typ e of dirs
_
cate cwd hn ?i 2= dirs comes from the schema that
Choice 1 Base : dirs 6= fhig
is Choice 1 Base for Category { Precondition. Sim-
Choice 2: dirs = fhig
ilarly, the predicate n ? 6= ROOT ^ n ? 6= PARENT
comes from the schema that is Choice 1 Base for
Category { Typ e of n?, and so on.
5.3.3 Cho osing Other Combinations
5.3.1 Creating The Combination Matrix
Applying the heuristic from section 4 to the CreateDir
op eration of MiStix gives a total of six test frame
The combination matrix is a conceptual to ol; only
schemas, the base test frame schema shown ab ove
those combinations of choices that are selected need
in CreateDir and ve additional varia-
b e explicitly enumerated. The combination matrix for
Base Test Frame
tions, one for each non-base choice. In the interest of
CreateDir has four dimensions, one for each category.
compactness we omit the declaration part of the test
There are 2 3 2 2= 24entries in the combination
frame schemas and only list the predicate parts from
matrix for CreateDir .
the choices. Note that since each test frame schema in-
cludes FileSystem , each predicate part of a test frame
5.3.2 Identifying The Base Test Frame
schema listed b elow also includes the state invariant
To identify a base test frame, a base choice is selected
from FileSystem ,even though that predicate is not
for each category. The normal case for CreateDir is
explicitly listed.
the case where the directory to b e created do es not yet
Base Test Frame exist and is neither ROOT nor PARENT, where the
Test Frame 1: The Base Test Frame schema, currentworking directory is not the ROOT directory,
CreateDir
and where the directory set contains more than just
Base Test Frame
is shown ab ove
the ROOT directory. The indications of base choices
Test Frame From Category { Precondition
are indicated by the word \Base" in the test sp eci -
_
Test Frame 2: cwd hn ?i2dirs ^ cation for CreateDir .
n ? 6= ROOT ^ n ? 6= PARENT ^
For brevity, the choices in the test sp eci cation are
cwd 6= hi ^
listed as predicates only, although a more complete
dirs 6= fhig
description is with a set of variable declarations and
predicates on those variables, i.e.,withaschema, as
Test Frames From Category { Typ e of n ?
done by Sto cks and Carrington [SC93a, SC93b]. Note
_
Test Frame 3: cwd hn ?i 2= dirs ^
that the invariantfromFileSystem , which describ es
n ?= ROOT ^
the set of valid states for the le system, must hold
cwd 6= hi ^
for every choice. The base test frame is the intersec-
dirs 6= fhig
tion of the base choice for each category, and this is
succinctly expressed with the schema conjunction of
_
Test Frame 4: cwd hn ?i 2= dirs ^
the base schema for each category.
3
For CreateDir the base test frame is the schema:
CreateDir declares the variables dirs and cwd
Base Test Frame
3
Note that the schema inclusion of FileSystem in and supplies the state invariants on these variables. 8
n ?=PARENT ^ We revise Test Frame 5 by shifting to a di erent cell
cwd 6= hi ^ in the combination matrix, namely the cell lab eled 3,
dirs 6= fhig where cwd is ROOT. As a result, we get a revised Test
Frame 5 listed in full schema form :
Test Frame From Category { Typ e of dirs
_
CreateDir
RevisedTest Frame 5
Test Frame 5: cwd hn ?i 2= dirs ^
FileSystem
n ? 6= ROOT ^ n ? 6= PARENT ^
n ?:Name
cwd 6= hi ^
dirs = fhig
_
cwd hn ?i 2= dirs
n ? 6= ROOT ^ n ? 6= PARENT
Test Frame From Category { Typ e of cwd
_
Test Frame 6: cwd hn ?i 2= dirs ^
cwd = hi
n ? 6= ROOT ^ n ? 6= PARENT ^
dirs = fhig
cwd = hi ^
dirs 6= fhig
5.3.5 Re ning Test Frames Into Test Cases
Note the source of each of the four explicitly
Re ning the test frames into test cases is the pro cess
listed predicates in a given test frame schema listed
of selecting a representative input from the set of in-
ab ove. Predicates are mechanically derived as for
puts that satisfy the selected choices. In previous work
those in CreateDir . Sp eci cally, each
Base Test Frame
such as DeMillo and O utt's [DO91], the re nement
predicate is a choice from one of the four categories
may b e based purely on the syntax of the constraints
for CreateDir .For anygiven test frame schema, one
and the typ es of the variables, whereas in a sophis-
predicate corresp onds to a non-base choice; remaining
ticated system such as the one prop osed by Wild et
predicates corresp ond to base choices.
al. [WZFC92], such a re nement mightbebasedon
a knowledge base incorp orating other pro ject sp eci c
5.3.4 Identifying Infeasible Combinations
data. Since re nement is not the fo cus of our present
work, we simply present sample test inputs that sat-
Test Frame 5, develop ed ab oveforCreateDir , is, in
isfy the necessary constraints. Each test input is a
fact, infeasible. The conjuncts
triple of n ?; dirs ; cwd .
dirs = fhig ^
All Base Choice
cwd 6= hi
Test Case 1: b, {<>,},
along with the invariant relation from FileSystem ,
Non-Base Precondition Choice
Test Case 2: b, {<>,,
cwd 2 dirs
Non-Base Type of n? Choice
Test Case 3: PARENT, {<>,},
simplify to false. Informally, if the ro ot directory is
Test Case 4: ROOT, {<>,},
the only directory,then cwd must b e set to the ro ot
Non-Base Type of dirs Choice
directory.
Test Case 5: b, {<>}, <>
We demonstrate the utility of the combination ma-
Non-Base Type of cwd Choice
trix by showing the entries for the Typeofdirs
Test Case 6: b, {<>,}, <>
Type of cwd part of the combination matrix:
Type of dirs
5.3.6 Writing Op eration Commands
Base Ro ot Only
The actual test case commands must use the parame-
Type of cwd Base 1
ters sp eci ed bythevalues of the test cases in a syn-
Ro ot 2 3
tactically correct command to the system b eing tested.
This is a straightforward pro cess that could b e auto-
The empty cell represents the combination of
mated by using the formal sp eci cation of the com-
choices for Test Frame 5 where there is only one direc-
mand. For cell 1 in the ab ove example, the op eration tory, the ro ot directory, and it contains no sub directo-
is: ries dirs = f<>g, and the currentworking directory
Op eration: CreateDir b cwd is some non-ro ot, e.g. /a. 9
5.3.7 Creating Test Scripts of mechanically generated test sp eci cations as iden-
ti ed here is to relieve the burden of routine tasks and
A test script is derived from a matrix entry bytak-
free the test engineer to concentrate on more creative
ing the corresp onding setup script, creating the syn-
tasks.
tax for the test op eration, and app ending the verify
Previous category-partition pap ers have o ered rel-
and cleanup scripts. A test script for cell 1 from the
atively little guidance as to whichcombinations of
ab ove matrix is:
choices should b e tested. In this pap er wehave de ned
Setup: CreateDir a
the base-choice coverage criterion, and demonstrated
ChangeDir a
the feasibility of applying the criterion to an example
Op eration: ...
sp eci cation. Some choices are designated as \base"
Verify: List
choices. A base test frame uses all the base choices.
Cleanup: Logoff
Other test frames are generated by systematic enu-
meration over all non-base choices. This technique is
5.4 Results of Testing MiStix
relatively inexp ensive linear in the number of choices
and ensures that eachchoice will b e used in at least
To demonstrate the feasibility of our technique, we
one test case if feasible . In future work weintend
generated and executed a complete set of test data for
to investigate the feasibility of the base-choice cover-
the MiStix system. The implementation is ab out 900
age criterion for categories derived from other sources
lines of C source in three separate mo dules.
than those investigated here.
We derived 72 test cases for the ten op erations, of
which5were duplicates. Some of the test cases were
also sup erseded by others in the sense that they were
Acknowledgements
pre xes of the other test cases. We did not eliminate
these tests although an automated to ol to supp ort
It is a pleasure to acknowledge Tom Ostrand for his
this pro cess could easily do so . The MiStix system
helpful comments and encouragement on the applica-
contained 10 known faults, of which7were detected.
tion of formal metho ds to category-partition testing.
We are also grateful to Steve Zeil for explaining the
utilityofZschemas as test frame descriptors.
6 Conclusions
Test sp eci cations are an imp ortantintermediate
representation b etween functional sp eci cations and
References
functional tests. In general, it is desirable to know
the extent to which a test sp eci cation can b e de- [AA92] N. Amla and P. Ammann. Using Z Sp eci-
rived from the functional sp eci cation and the extent cations in Category Partition Testing. In
to which the tester must rely on information exter- Proceedings of the Seventh Annual Confer-
nal to the sp eci cation. In this pap er wehave help ed ence on Computer Assurance COMPASS
to answer this question by presenting a pro cedure that 92 , Gaithersburg MD, June 1992. IEEE
will guide the tester when creating complete functional Computer So ciety Press.
tests from functional sp eci cations.
[AO93] P. Ammann and A. J. O utt. Functional
It is unreasonable to exp ect to derive a test sp eci-
cation completely from the functional sp eci cations. and test sp eci cations for the MiStix le
For example, the test engineer mightknow that pro- system. Technical rep ort ISSE-TR-93-100,
grammers often make the mistake of using xed-sized Department of Information and Software
structures when dynamically-sized structures are re- Systems Engineering, George Mason Uni-
versity,Fairfax VA, 1993.
quired. For example, in the sp eci cation for MiStix,
the depth of the directory tree is not constrained by
the Z sp eci cations. The lack of an explicit statement [BHO89] M. Balcer, W. Hasling, and T. Ostrand.
makes it dicult to derive an appropriate category Automatic Generation of Test Scripts from
e.g., directory tree depth . However, to a typical hu- Formal Test Sp eci cations. In Proceedings
man test engineer, limits on directory depth is a rel- of the ThirdSymposium on SoftwareTest-
atively obvious prop ertytocheck and indeed sucha ing, Analysis, and Veri cation, pages 210{
check can lead to tests that discover faults not found 218, Key West Florida, Decemb er 1989.
by the generated test sp eci cation . Thus the role ACM SIGSOFT 89. 10
[Dil90] A. Diller. AnIntroduction to Formal [SC93b] P. Sto cks and D. Carrington. Test
Methods. Wiley Publishing Company Inc., Templates: A Sp eci cation-Based Testing
1990. Framework. In Proceedings of the 15th In-
ternational Conference on Software Engi-
[DO91] R. A. DeMillo
neering, pages 405{414, Baltimore, MD,
and A. J. O utt. Constraint-based auto-
May 1993.
matic test data generation. IEEE Transac-
[Spi89] J.M. Spivey. The Z Notation: A Reference
tions on Software Engineering, 17 9 :900{
Manual. Prentice Hall Publishing Com-
910, September 1991.
pany Inc., 1989.
[GG93] M. Gro chtmann and K. Grimm. Classi -
[Wor92] J.B. Wordsworth. Software Development
cation Trees for Partition Testing. Journal
with Z. Addison-Wesley, 1992.
of SoftwareTesting, Veri cation, and Re-
liability, 3 1 , 1993. Toappear.
[WZFC92] C. Wild, S. Zeil, G. Feng, and J. Chen.
Employing Accumulated Knowledge to
[GMH81] J. Gannon, P. McMullin, and R. Hamlet.
Re ne Test Descriptions. Journal of Soft-
Data-Abstraction Implementation, Sp eci-
wareTesting, Veri cation, and Reliability,
cation, and Testing. ACM Transactions
2 2 :53{68, August 1992.
on Programming Languages and Systems,
3 3 :211{223, July 1981.
[Hay86] I. J. Hayes. Sp eci cation Directed Mo d-
ule Testing. IEEE Transactions on Soft-
ware Engineering, SE-12 1 :124{133, Jan-
uary 1986.
[Hay93] I. Hayes. Speci cation Case Studies. Pren-
tice Hall Publishing Company Inc., 1993.
[Kem85] R. A. Kemmerer. Testing formal sp eci ca-
tions to detect design errors. IEEE Trans-
actions on Software Engineering, SE-
11 1 :32{43, January 1985.
[Lay92] G. Layco ck. Formal Sp eci cation and
Testing: a Case Study. Journal of Soft-
wareTesting, Veri cation, and Reliability,
2:7{23, 1992.
[OB88] T. J. Ostrand and M. J. Balcer. The
Category-Partition Metho d for Sp ecifying
and Generating Functional Tests. Com-
munications of the ACM, 31 6 :676{686,
June 1988.
[PST91] B. Potter, J. Sinclair, and D. Till. An
Introduction to Formal Speci cation and
Z. Prentice Hall Publishing Company Inc.,
1991.
[SC93a] P. Sto cks and D. Carrington. Test Tem-
plate Framework: A Sp eci cation-Based
Testing Case Study. In Proceedings of the
1993 International Symposium on Soft-
wareTesting and Analysis, pages 11{18,
Cambridge, MA, June 1993. 11