QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs
Koen Claessen John Hughes Chalmers University of Technology Chalmers University of Technology
[email protected] [email protected]
are hard to test and so testing can b e done at a
ABSTRACT monad
ne grain
QuickCheck is a to ol which aids the Haskell programmer in
A testing to ol must b e able to determine whether a test
formulating and testing prop erties of programs Prop erties
is passed or failed the human tester must supply an auto
are describ ed as Haskell functions and can b e automati
matically checkable criterion of doing so We have chosen
cally tested on random input but it is also p ossible to de
to use formal sp eci cations for this purp ose We have de
ne custom test data generators We present a number of
signed a simple domain sp eci c language of testable speci
case studies in which the to ol was successfully used and
cations which the tester uses to de ne exp ected prop erties
also p oint out some pitfalls to avoid Random testing is es
of the functions under test QuickCheck then checks that the
p ecially suitable for functional programs b ecause prop erties
prop erties hold in a large number of cases The sp eci ca
can b e stated at a ne grain When a function is built from
tion language is embedded in Haskell using the class system
separately tested comp onents then random testing su ces
Prop erties are normally written in the same mo dule as the
to obtain go o d coverage of the de nition under test
functions they test where they serve also as checkable do c
umentation of the b ehaviour of the co de
testing to ol must also b e able to generate test cases au
1. INTRODUCTION A
tomatically We have chosen the simplest metho d random
Testing is by far the most commonly used approach to
testing which comp etes surprisingly favourably with
ensuring software quality It is also very lab our intensive
systematic metho ds in practice However it is meaningless
accounting for up to of the cost of software develop
to talk ab out random testing without discussing the distri
ment Despite anecdotal evidence that functional programs
bution of test data Random testing is most e ective when
require somewhat less testing Once it type checks it usu
the distribution of test data follows that of actual data but
ally works in practice it is still a ma jor part of functional
when testing reuseable co de units as opp osed to whole sys
program development
tems this is not p ossible since the distribution of actual
The cost of testing motivates e orts to automate it wholly
data in all subsequent reuses is not known A uniform dis
or partly Automatic testing to ols enable the programmer
tribution is often used instead but for data drawn from
to complete testing in a shorter time or to test more thor
in nite sets this is not even meaningful how would one
oughly in the available time and they make it easy to rep eat
choose a random closed term with a uniform distribution
tests after each mo di cation to a program In this pap er we
for example We have chosen to put distribution under the
describ e a to ol QuickCheck which we have developed for
human tester s control by de ning a test data generation
testing Haskell programs
language also embedded in Haskell and a way to observe
Functional programs are well suited to automatic testing
the distribution of test cases By programming a suitable
It is generally accepted that pure functions are much easier
generator the tester can not only control the distribution
to test than side e ecting ones b ecause one need not b e
of test cases but also ensure that they satisfy arbitrarily
concerned with a state b efore and after execution In an
complex invariants
imp erative language even if whole programs are often pure
An imp ortant design goal was that QuickCheck should b e
functions from input to output the pro cedures from which
lightweight Our implementation consists of a single pure
they are built are usually not Thus relatively large units
Haskell mo dule of ab out lines which is in practice
must b e tested at a time In a functional language pure
mainly used from the Hugs interpreter We have also writ
functions ab ound in Haskell only computations in the IO
ten a small script to invoke it which needs to know very
little ab out Haskell syntax and consequently supp orts the
full language and its extensions It is not dep endent on any
Haskell system A cost that comes with this deci
Permission to make digital or hard copies of all or part of this work for particular
is that we can only test prop erties that are expressible
personal or classroom use is granted without fee provided that copies are sion
observable within Haskell
not made or distributed for profit or commercial advantage and that copies and
is notoriously di cult to say how e ective a testing bear this notice and the full citation on the first page. To copy otherwise, to It
republish, to post on servers or to redistribute to lists, requires prior specific
metho d is in detecting faults However we have used Quick
permission and/or a fee.
in a variety of applications ranging from small exp er ICFP ’00, Montreal, Canada. Check
Copyright 2000 ACM 1-58113-202-6/00/0009 .. 5.00
In fact the programmer must provide a little more infor iments to real systems and we have found it to work well
mation the function quickCheck is actually overloaded in in practice We rep ort on some of this exp erience in this
order to b e able handle laws with a varying number of vari pap er and p oint out pitfalls to avoid
ables and the overloading cannot b e resolved if the law itself The rest of this pap er is structured as follows Section
has a p olymorphic type as in these examples Thus the pro introduces the concept of writing prop erties and checking
grammer must sp ecify a xed type at which the law is to b e them using QuickCheck Section shows how to de ne test
tested So we simply give a type signature for each law for data generators for user de ned types Section brie y dis
example cusses the implementation Section presents a number of
case studies that show the usefulness of the to ol Section
prop RevApp Int Int Bool
concludes
Of course the prop erty prop RevApp holds p olymorphically
we must sp ecify which monomorphic instance to test
2. DEFINING PROPERTIES but
it at so that we can generate test cases This turns out
b e quite imp ortant in the case of overloaded functions
2.1 A Simple Example to
For example is asso ciative for the type Int but not for
As a rst example we take the standard function reverse
Double In some cases we can use parametricity to
which reverses a list This satis es a number of useful laws
argue that a prop erty holds p olymorphically
such as
2.2 Functions
reverse x x
We are also able to formulate prop erties that quantify over
reverse xs ys reverse ys reverse xs
functions To check for example that function comp osition
reverse reverse xs xs
is asso ciative we rst de ne extensional equality by
f g x f x g x and then write
In fact the rst two of these characterise reverse uniquely
Note that these laws hold only for nite total values In
prop CompAssoc Int Int Int Int
this pap er unless sp eci cally stated otherwise we always
Int Int Int Bool
quantify over completely de ned nite values This is to
prop CompAssoc f g h
make it more likely that the prop erties are computable
f g h f g h
In order to check these laws using QuickCheck we repre
The only di culty that function types cause is that if a
sent them as Haskell functions Thus we de ne
counter example is found for example if we try to check
prop RevUnit x
that function comp osition is commutative then the func
reverse x x
tion values are printed just as function In this case
we discover that the law we are checking is false but not
prop RevApp xs ys
why
reverse xs ys reverse ys reverse xs
2.3 Conditional Laws
prop RevRev xs
Laws which are simple equations are conveniently repre
reverse reverse xs xs
sented by b o olean function as we have seen but in general
many laws hold only under certain conditions QuickCheck
Now if these functions return True for every p ossible argu
provides an implication combinator to represent such condi
ment then the prop erties hold We load them into the Hugs
tional laws For example the law
interactive Haskell interpreter and call for example
Main quickCheck prop RevApp
x y ) max x y y
OK passed tests
can b e represented by the de nition
The function quickCheck takes a law as a parameter and ap
prop MaxLe Int Int Property
plies it to a large number of randomly generated arguments
1
prop MaxLe x y x y max x y y
in fact rep orting OK if the result is True in
every case
Likewise the insertion function into ordered lists satis es
If the law fails then quickCheck rep orts the counter
the law
example For example if we mistakenly de ne
prop Insert Int Int Property
prop RevApp xs ys
prop Insert x xs
reverse xs ys reverse xs reverse ys
ordered xs ordered insert x xs
then checking the law might pro duce
Note that the result type of the prop erty is changed from
Bool to Property This is b ecause the testing semantics
Main quickCheck prop RevApp
is di erent for conditional laws Instead of checking the
Falsifiable after tests
prop erty for random test cases we try checking it for
test cases satisfying the condition If a candidate test
case do es not satify the condition it is discarded and a new
where the counter mo del can b e extracted by taking for
test case is tried
xs and for ys
Checking the laws prop MaxLe and prop Insert succeed
1
as usual but sometimes checking a conditional law pro duces
is a rather arbitrary number so our library provides a
the output way to sp ecify this as a parameter
the precondition There is a risk of this kind of problem ev Arguments exhausted after tests
ery time we use conditional laws so it is always imp ortant
If the precondition of a law is seldom satis ed then we might
to investigate the prop ortion of trivial cases among those
generate many test cases without nding any where it holds
actually tested
In such cases it is hop eless to search for cases in which
The b est solution though is to replace the condition with
the precondition holds Rather than allow test case gener
a custom test data generator for ordered lists We write
ation to run forever we generate only a limited number of
candidate test cases the default is If we do not nd
prop Insert Int Property
valid test cases among those candidates then we sim
prop Insert x
ply rep ort the number of successful tests we were able to
forAll orderedList xs
ordered insert x xs p erform
In the example we know that the law passed the test in
which sp eci es that values for xs should b e generated by
cases It is then up to the programmer to decide whether this
the test data generator orderedList Checking the law now
is enough or whether it should b e tested more thoroughly
gives OK passed tests as we would exp ect
provides supp ort for the programmer to de ne
2.4 Monitoring Test Data QuickCheck
his or her own test data generators with control over the
Perhaps it seems that we have tested the law for insert
distribution of test data which we will lo ok at in the next
thoroughly enough to establish its credibility However we
section
2
must b e careful Let us mo dify prop Insert as follows
Int Int Property
prop Insert 2.5 Infinite Structures
prop Insert x xs
The Haskell function cycle takes a non empty list and
ordered xs
returns a list that rep eats the contents of that list in nitely
classify null xs trivial
Now take a lo ok at the following law formulated in Quick
3
ordered insert x xs
Check as
Checking the law now pro duces the message
prop DoubleCycle Int Property
prop DoubleCycle xs
OK passed tests trivial
not null xs
The classify combinator do es not change the meaning of a
cycle xs cycle xs xs
law but it classi es some of the test cases in this case those
Although intuitively the law is true it cannot b e checked
where xs is the empty list were classi ed as trivial Thus
since we are comparing two in nite lists using computable
we see that a large prop ortion of the test cases only tested
equality which do es not terminate Instead we can refor
insertion into an empty list
mulate the prop erty as a logically equivalent one by using
We can get more information than just lab elling some test
the fact that two in nite lists are equal if all nite initial
cases The combinator collect will gather all values that
segments are equal
are passed to it and print out a histogram of these values
For example if we write
prop DoubleCycle Int Int Property
prop DoubleCycle xs n
prop Insert Int Int Property
not null xs n
prop Insert x xs
take n cycle xs take n cycle xs xs
ordered xs
collect length xs
Another issue related to in nite structures is quanti cation
ordered insert x xs
over them We will later see how to deal with prop erties
that for example hold for all in nite lists but in general it
we might get as a result
is not clear how to formulate and execute prop erties ab out
OK passed tests
structures containing b ottom
3. DEFINING GENERATORS
3.1 Arbitrary
The way we generate random test data of course dep ends
on the type Therefore we have introduced a type class
So we see that only cases tested insertion into a list with
Arbitrary of which a type is an instance if we can generate
more than one element While this is enough to provide
arbitrary elements in it
fairly strong evidence that the law holds it is worrying that
class Arbitrary a where
very short lists dominate the test cases so strongly After
arbitrary Gen a
all it is easy to de ne an erroneous version of insert which
nevertheless works for lists with at most one element
Gen a is an abstract type representing a generator for type
The reason for this b ehaviour of course is that the pre
a The programmer can either use the generators built in
condition ordered xs skews the distribution of test cases
to QuickCheck as instances of this class or supply a custom
towards short lists Every generated list of length or
generator using the forAll combinator which we saw in the
is ordered but only of the lists of length are Thus
previous section For now we de ne the type Gen as
test cases with longer lists are more likely to b e rejected by
3
Note that leaving the condition not null xs out results
2
is Haskell s in x function application in an error b ecause cycle is not de ned for empty lists
As another example we could generate arbitrary lists using newtype Gen a Gen Rand a
Here Rand is a random number seed a generator is just a
instance Arbitrary a Arbitrary a where
function which can manufacture an a in a pseudo random
arbitrary oneof
way But we will treat Gen as an abstract type so we de ne
return liftM arbitrary arbitrary
a primitive generator
where we use liftM to apply the cons op erator to an
choose Int Int Gen Int
arbitrary head and tail However this de nition is not really
satisfactory since it pro duces lists with an average length
to choose a random number in an interval and we program
of one element We can adjust the average length of list
other generators in terms of it
pro duced by using frequency instead which allows us to
We also need combinators to build complex generators
sp ecify the frequency with which each alternative is chosen
from simpler ones to do so we declare Gen to b e an instance
We de ne
of Haskell s class Monad This involves implementing the
metho ds of the Monad class
instance Arbitrary a Arbitrary a where
return a Gen a
arbitrary frequency
Gen a a Gen b Gen b
return
liftM arbitrary arbitrary
the rst one of which contructs a constant generator and
the second one b eing the monadic sequencing op erator which
to choose the cons case four times as often as the nil case
generates an a and passes it to its second argument to gen
leading to an average list length of four elements
erate a b The de nition of needs to pass indepen
However for more general data types it turns out that
dent random number seeds to its two arguments and is only
we need even ner control over the distribution of generated
passed one seed but luckily the Haskell random number li
values Supp ose we de ne a type of binary trees and a
brary provides an op eration to split one seed into two
generator
De ning generators for many types is now straightfor
data Tree a Leaf a Branch Tree a Tree a
ward As examples we give generators for integers and
pairs
instance Arbitrary a Arbitrary Tree a where
instance Arbitrary Int where
arbitrary frequency
arbitrary choose
liftM Leaf arbitrary
liftM Branch arbitrary arbitrary
instance Arbitrary a Arbitrary b
We want to avoid choosing a Leaf to o often hence the use
Arbitrary a b where
of frequency
arbitrary liftM arbitrary arbitrary
However this de nition only has a chance of termi
In the second case we use a standard monadic function
nating The reason is that for the generation of a Branch to
liftM which is de ned in terms of return and to
terminate two recursive generations must terminate If the
make a generator that applies the pairing op erator to
rst few recursions choose Branches then generation termi
the results of two other generators QuickCheck contains
nates only if very many recursive generations all terminate
such declarations for most of Haskell s prede ned types
and the chance of this is small Even when generation ter
the generated test data is sometimes very large
3.2 Generators for User-Defined Types minates
We want to avoid this since we p erform a large number of
Since we de ne test data generation via an instance of
tests we want each test to b e small and quick
class Arbitrary for each type then we must rely on the user
Our solution is to limit the size of generated test data
to provide instances for user de ned types In principle we
But the notion of a size is hard even to de ne in general for
could try to generate these automatically in a pre pro cessor
an arbitrary recursive datatype which may include func
or via p olytypic programming but we have chosen in
tion types anywhere We therefore give the resp onsibility
stead to leave this task to the user This is partly b ecause
for limiting sizes to the programmer de ning the test data
we want QuickCheck to b e a lightweight to ol easy to imple
generator We change the representation of generators to
ment and easy to use in a standard programming environ
ment we don t want to oblige users to run their programs
newtype Gen a Gen Int Rand a
through a pre pro cessor b etween editing them and testing
where the new parameter is to b e interpreted as some kind
them But another strong reason is that it seems to b e very
of size b ound We de ne a new combinator
hard to construct a generator for a type without knowing
something ab out the desired distribution of test cases
sized Int Gen a Gen a
Instead of pro ducing generators automatically we pro
which the programmer can use to access the size b ound
vide combinators to enable a programmer to de ne his own
sized generates an a by passing the current size b ound to
generators easily The simplest called oneof just makes a
its parameter It is then up to the programmer to inter
choice among a list of alternative generators with a uniform
pret the size b ound in some reasonable way during test data
distribution for example if the type Colour is de ned by
generation For example we might generate binary trees
data Colour Red Blue Green
using
then a suitable generator can b e de ned by
instance Arbitrary a Arbitrary Tree a where
arbitrary sized arbTree instance Arbitrary Colour where
arbitrary oneof
arbTree liftM Leaf arbitrary return Red return Blue return Green
variant Int Gen a Gen a arbTree n frequency
liftM Leaf arbitrary
where variant n g constructs a generator which transforms
liftM Branch arbTree n div
the random number seed it is passed in a way dep ending on
arbtree n div
n b efore passing it to g This function must b e de ned very
carefully so that all the generators we construct using it are
With this de nition the size b ound limits the number of
indep endent Given any list of integers n n nk we
no des in the generated trees which is quite reasonable
can construct a generator transformer
Now that we have introduced the notion of a size b ound
we can use it sensibly in the generators for other types such
variant n variant n variant nk
as integers and lists so that the absolute value resp ective
We de ne variant so that di erent lists of integers give
length is b ounded by the size So the de nitions we pre
rise to indep endent generator transformers with a very high
sented earlier need to b e mo di ed accordingly
probability
We stress that the size b ound is simply an extra global
Now we can de ne instances of coarbitrary that choose
parameter which every test data generator may access ev
b etween generator transformers dep ending on the argument
4
ery use of sized sees the same b ound We do not attempt
value For example the b o olean instance
to divide the size b ound among the generators so that for
instance Coarbitrary Bool where
example a longer generated list would have smaller elements
coarbitrary b
keeping the overall size of the structure the same The rea
if b then variant else variant
son is that we wish to avoid correlations b etween the sizes
transforms a generator in indep endent ways for True and for
of di erent parts of the test data which might distort the
False the generators coarbitrary True g and coarbitrary
test results
False g will b e indep endent In a similar way we can de
We do vary the size b etween di erent test cases we b egin
ne suitable instances for many other types For example
testing each prop erty on small test cases and then grad
the integer instance just converts its integer argument into
ually increase the size b ound as testing progresses This
a sequence of bits which are then used as generator trans
makes for a greater variety of test cases which b oth makes
formers in turn
testing more e ective and improves our chances of nding
Instances of Coarbitrary for recursive datatypes can b e
enough test cases satisfying the precondition of conditional
de ned according to a standard pattern For example the
prop erties It also makes it more likely that we will nd a
list instance is just
small counter example to a prop erty if there is one
instance Coarbitrary a Coarbitrary a where
variant
3.3 Generating Functions coarbitrary
coarbitrary x xs
If we are to check prop erties involving function valued
variant coarbitrary x coarbitrary xs
variables then we must b e able to generate arbitrary func
tions Rather surprisingly we are able to do so To un
The goal is that di erent lists should b e mapp ed to inde
derstand how notice that a function generator of type Gen
p endent generator transformers we achieve this by map
a b is represented by a function of type Int Rand
ping each constructor to an indep endent transformer and
a b By reordering parameters this is equivalent to the
comp osing these with transformers derived from each com
type a Int Rand b which represents a Gen b
p onent Other recursive datatypes can b e treated in the
We can thus de ne
same way Since the programmer is resp onsible for making
these de nitions for user de ned types it is imp ortant that
promote a Gen b Gen a b
they b e straightforward
and use it to pro duce a generator for a function type pro
Finally we can even interpret functions as generator trans
vided we can construct a generator for the result type which
formers with an instance of the form
somehow dep ends on the argument value We take care of
instance Arbitrary a Coarbitrary b
this dep endence by de ning a new class
Coarbitrary a b where
class Coarbitrary a where
coarbitrary f gen
coarbitrary a Gen b Gen b
arbitrary a coarbitrary f a gen
The idea is that we apply the given function to an arbitrary
whose metho d coarbitrary mo di es a generator in a way
argument and use the result to transform the given gener
dep ending on its rst parameter We will think of coarbitrary
ator In this way two functions which are di erent will give
as pro ducing a generator transformer from its rst argu
rise to di erent generator transformers
ment Given this class we can de ne
Note that if we had tried to avoid needing to split random
instance Coarbitrary a Arbitrary b
number seeds by de ning the Gen monad as a state trans
Arbitrary a b where
former on the random seed rather than a state reader then
arbitrary
we would not have b een able to de ne the promote func
promote a coarbitrary a arbitrary
tion and we would not have b een able to generate random
functions
which generates an arbitary function that uses its argument
to mo dify the generation of its result
order to de ne instances of Coarbitrary we need a way
In 4. IMPLEMENTING QUICKCHECK
to construct generator transformers We therefore de ne the
As we have seen the function quickCheck can handle
function
prop erties with a varying number of arguments and di er
4
ent result types To implement this we introduce the type
Unless the programmer explicitly changes it using the
resize combinator Property and we create the type class Testable
generated terms using the default test data generator for class Testable a where
strings then it is very unlikely that we would ever generate property a Property
uni able terms since it is unlikely we would generate the
The Property type represents predicates that can b e checked
same constructor name twice Instead we chose to generate
by testing This means that it needs to b e able to gener
constructor names using
ate random input and nally pro duct a test result So a
Property is a computation in the Gen monad ending in an
instance Arbitrary Name where
abstract type Result which keeps track of the b o olean re
arbitrary sized n oneof
sult of the testing the classi cations of test data and the
return Name v show i
arguments used in the test case
i log n
newtype Property Prop Gen Result
which gives us a go o d chance that generated terms will b e
at least partially uni able Likewise we limited uni cation
Let us take a lo ok at some instances of Testable An easy
variables in test data to a small set
type to check is of course Bool Further functions for which
Of course we could have used the second Term type ab ove
we can generate arbitrary arguments can b e tested And
and sp eci ed a custom test data generator with an explicit
lastly even the prop erty type itself is an instance so that
forAll in each prop erty But it is more convenient to let test
we can nest prop erty combinators
data b e automatically generated using arbitrary so one is
instance Testable Bool where
encouraged to make distinctions explicit in types There
property b Prop return resultBool b
are other advantages to doing so also it p ermits the type
checker to detect more errors So using QuickCheck changes
instance Arbitrary a Show a Testable b
the balance of convenience in the question of introducing
Testable a b where
new types in programs
property f forAll arbitrary f
5.1.2 Checking Functional Properties
instance Testable Property where
A uni er needs to manage the current substitution and
property p p
also the p ossibility of failures in recursive calls A convenient
way to do so is to use a monad We de ned a uni cation
Using the function property it b ecomes easy to de ne the
monad M represented by a function with op erations
function quickCheck Its type is
setV Var Term M
quickCheck Testable a a IO
getV Var M Term
More details of the implementation can b e found in the ap
to read and write variables among others We were able
p endix
to de ne an extensional equality op erator eqM on monadic
and check b oth the monad laws and prop erties such
5. SOME CASE STUDIES values
as
v t do setV v t getV v
5.1 Unification prop SetGet
eqM do setV v t return t
As a rst and rather pathological case study we dis
a un cation algorithm which we have developed along
cuss 5.1.3 Errors Found
with a QuickCheck sp eci cation This was quite revealing
It would b e nice to b e able to rep ort that QuickCheck
b oth as regards the impact that QuickCheck has on program
found a large number of errors in this example In fact no
ming and the pitfalls that must b e avoided It is to o large
errors at all were found in the uni er itself This is probably
to present in detail so we will just discuss the lessons we
more a re ection on the number of times the authors have
learned
programmed uni ers previously than on the e ectiveness of
we know how to do it quite simply
5.1.1 Impact on Type Definitions QuickCheck
On the other hand we did nd errors in the speci cation
First of all the use of QuickCheck had an impact on the
For example we de ned a substitution function which re
design of the types in the program We de ned the type of
p eatedly substitutes until no variables in the domain of the
terms to b e uni ed as
substitution remain and stated the obvious prop erty
data Term Var Var Constr Name Term
prop SubstIdempotent s t
newtype Var Variable Nat
subst s subst s t subst s t
newtype Name Name String
QuickCheck revealed this prop erty to b e false it holds only
rather than the equivalent
for acyclic substitutions otherwise an in nite term is gen
data Term Var Int Constr String Term
erated and the equality test lo ops This error was found
which we would probably have chosen otherwise The type
using the function verboseCheck which prints out the ar
we used distinguishes b etween a string used as a constructor
guments to every test case b efore it checks it
name and a string used in other contexts and b etween a
We were obliged to correct the sp eci cation to
natural number used as a variable name and an integer used
prop SubstIdempotent s t
in other contexts
acyclic s subst s subst s t subst s t
The reason we chose to make these distinctions in the
Thus QuickCheck made us think harder ab out the prop erties type is that it enabled us to de ne a di erent test data
of our co de and do cument them correctly generator for Names for example than for strings Had we
Furthermore the Lava system de nes combinators for cir On the downside formulating the sp eci cation correctly
cuits such as sequential comp osition parallel com turned out to b e quite a lot of work p erhaps more than
p osition and column which takes one circuit and writing the implementation This was partly b ecause pred
replicates it in a column of circuits connecting the vertical icates such as acyclic are non trivial to de ne a go o d set
wires theory library would have help ed here
5.1.4 A False Sense of Security 5.2.2 Properties in Lava
Prop erties of circuits can b e de ned in a similar way For
The most serious pitfall we uncovered with this exp eri
example to de ne the prop erty that a certain circuit is com
ment was the false sense of security that can b e engendered
mutative we say
when one s program passes a large number of tests in trivial
cases We have already referred to this problem when we
prop Commutative circ a b
discussed conditional prop erties in this example it bit us
circ a b circ b a
hard
where is logical equivalence lifted to arbitrary types
Many prop erties of uni cation apply to the case when
containing signals in this case a pair
uni cation succeeds They can b e stated conveniently in
Prop erties can b e formally veri ed We do this by provid
the form
ing symbolic inputs to the circuit or prop erty and calculat
ing a concrete expression in a Haskell datatype representing
prop Unify t t s Nothing
the circuit
where s unifier t t
We can then write this expression to a le and call an
since our uni er returns Nothing when it fails With a lit
external theorem prover All this is done by the overloaded
tle re ection we see that two randomly chosen terms are
Lava function verify Here is how we can use it to verify
fairly likely to b e uni able since variables o ccur quite often
that a so called half adder comp onent is commutative
and if either term is a variable then uni cation will almost
Main verify prop Commutative halfAdd
certainly succeed On the other hand if neither term is a
Proving Valid
variable then the probability that they will unify is small
The Lava system provides a number of functions and combi
Thus the case where one term is a variable is heavily over
nators to conveniently express prop erties and formally verify
represented among the test cases that satisfy the precondi
them
tion we found that over of test cases had this prop
erty Although QuickCheck succeeded in verifying every
5.2.3 QuickCheck in Lava
prop erty we can hardly consider that they were thoroughly
Though we are able to verify prop erties ab out circuits in
tested
Lava we greatly b ene t from extending it with a testing
The solution was to use a custom test data generator
to ol like QuickCheck There are two main reasons for that
prop Unify
The rst one is that calling an external theorem prover
forAll probablyUnifiablePair t t
is a very heavyweight pro cess When verifying say a
s Nothing
bit multiplier the formulas that we generate for external
where s unifier t t
theorem proving are quite big and we often have to wait for
a long time to get an answer
We generated probably uni able pairs by generating one
So a typical development cycle is to write down the sp ec
random term and replacing random subterms by variables
i cation of the circuit rst then make an implementation
in two di erent ways This usually generates uni able terms
QuickCheck it for obvious bugs and lastly call the external
although may fail to when variables are used inconsistently
theorem prover for verifying the correctness
in the two terms With this mo di cation the prop ortion of
Here is an example of how to use QuickCheck in Lava
trivial cases fell to a reasonable
This exp erience underlines the imp ortance of investigating
Main quickCheck prop Commutative halfAdd
the distribution of actual test cases whenever conditional
OK passed tests
prop erties are used
Adding this testing metho dology to Lava turned out to b e
straightforward b ecause Lava already had a notion of
5.2 Circuit Properties quite
prop erties Testing can b e done for all circuit types even
circuits containing latches We simply check the
5.2.1 Lava in a Nutshell sequential
circuit prop erty for a sequence of inputs
Lava is a to ol to describ e simulate and formally ver
hardware Lava is a so called embedded language which
ify 5.2.4 Higher Order Testing
means that the circuit descriptions and prop erties are all
The second reason for using testing in Lava is simply that
expressed in an existing programming language in this case
we can test more prop erties than we can formally verify
Haskell
The external theorem provers that are connected to Lava
The idea is to view a hardware circuit as a function from
can only deal with at most rst order logics and the Lava
signals of inputs to signals of outputs The Lava system
system is only able to generate formulas of that type
provides primitive circuits such as and xor and delay
Sometimes we would like to verify prop erties ab out com
More complicated circuits are de ned by combining these
binators For example proving that column distributes over
Circuits de ned in Lava can b e simulated as follows one
provides inputs and the outputs are calculated
prop ComposeSeqColumn circ circ inp
Main simulate and high low column circ circ inp
high column circ column circ inp
Note that we collect some statistics information contra is very hard to verify in Lava for al l circ and circ In
diction when the result was Nothing and the size of m in fact such prop erties are hard to verify automatically in gen
the case of Just m We also expressed that we disqualify a eral we can do it for small xed sizes however But since
test case when st almarck returns Just we can randomly generate functions we can at least test
With the help of this prop erty QuickCheck found bugs these kind of prop erties for arbitrary circuits
These bugs were due to implicit unjusti ed assumptions we A drawback is that we have to x the types of these cir
had ab out the input The implementations of b oth algo cuits whereas the combinators themselves and thus the
rithms assumed that no clauses in the input could contain prop erties ab out them are p olymorphic in the circuits in
the same literal twice and the st almarck function assumed put and output types
none of the input clauses was empty
5.2.5 Errors Found that
The data generator clauses is de ned using the same
The authors used the QuickCheck library while developing
techniques as in section Testing the prop erty to ok
5.1.1
a collection of arithmetic circuits Previously testing was
ab out seconds and from the output we could see that
already used in the development pro cess but only in a very
the distribution of Nothing vs Just m was ab out
limited and ad ho c way Now much more thorough testing p ossible
was 5.4 Pretty Printing
The errors we found in these particular circuits were of
Andy Gill rep orted an interesting story ab out using Quick
two kinds Firstly we found errors that our formal veri ca
Check to us He used it in developing a variant of Wadler s
tion metho d would have found as well logical errors in the
pretty printing combinator library in Java First he
circuits But secondly we also found errors due to the fact
implemented his variant functionally using Haskell Then
that random input also means random input size For ex
still using Haskell he used a state monad with exceptions to
ample for an n bit m bit adder we only use and formally
develop an imp erative implementation of the same library
verify the circuit for sp eci c input sizes Random testing
The idea was that the second implementation mo dels what
checks many more combinations and it often turned out
go es on in a Java implementation
that we had forgotten to de ne one of these cases
Then he expressed the relationship b etween the two dif
implementations using QuickCheck prop erties He
5.3 Propositional Theorem Proving ferent
writes This quickly points out where my reasoning is faulty
For teaching purp oses we implemented two di erent well
and provides great tests to catch the corners of the implemen
known metho ds of checking if a set of prop ositional logic
tation Three problems were found the third of which showed
clauses is contradictory One of these metho ds was the
that I had merged two concepts in my implementation that
Davis Putnam metho d which uses backtracking to gen
I should not have
erate a list of all mo dels The other one was St almarck s
Furthermore he made an improvement in the way Quick
metho d which is an incomplete metho d and uses a
Check rep orts counter examples Sometimes the counter
variant on the dilemma pro of system to gather information
examples found are very large and it is di cult to go back
ab out the literals in the clause set
to the prop erty and understand why it is a counter example
However when the counter example is an element of a tree
type Clause Lit disjunction
shap ed datatype the problem can often b e lo cated in one of
type Model Lit conjunction
the sub trees of the counter example found Gill extended
the Arbitrary class with a new metho d
davisPutnam Clause Model
st almarck Int Clause Maybe Model
smaller a a
The st almarck function takes an extra argument an Int
which is intended to return a list of smaller but similar
which is the so called saturation level a parameter which
values to its argument for example direct subtrees He
limits the depth of the pro ofs and usually lies b etween and
adapted the quickCheck function so that when a counter
If the result of st almarck is Nothing it means that there
example is found it tries to nd a smaller one using this
was a contradiction If the result is Just m it means that
function In some cases much smaller counterexamples were
every mo del of the clause set should have m as a sub mo del
found greatly reducing the time to understand the bug
Since davisPutnam is much more straight forward to im
found
plement than st almarck we wanted to check the latter
The last step Gill made in developing his Java pretty
against the rst Here is how we formulate the informal
printing library was p orting the state and exception monad
prop erty stated ab ove
mo del in Haskell to Java He then used QuickCheck to gen
prop St almarck vs DP Property
erate a large number of test inputs for the Java co de in
prop St almarck vs DP
order to check that the Java implementation was equivalent
forAll clauses cs
to the two Haskell mo dels
forAll choose sat
st almarck sat cs of
case 5.5 Edison
Nothing Chris Okasaki s Edison is a library of e cient data struc
collect contradiction tures suitable for implementation and use in functional pro
davisPutnam cs gramming languages He has used QuickCheck to state and
test prop erties of the library Every data structure in the
Just m library has b een made an instance of Arbitrary and he
not null m has included several extra mo dules esp ecially for formulating
collect length m prop erties ab out these data structures He rep orts My ex
all m subModel davisPutnam cs perience has mostly been that of a very satis ed user Quick
they call are tested indep endently So even when QuickCheck Check lets me test Edison with probably maybe less
is used to test a large program we always test a small part of the e ort of my previous test suite and does a much better
at a time Therefore we may exp ect random testing to work job to boot
particularly well Okasaki also mentions a drawback having to do with the
Given this together with the much greater di culty of Haskell mo dule system He often uses one sp eci cation of
automating systematic testing for Haskell our choice of ran a data structure together with di erent implementations
dom testing is clear A natural way to do this is to place the sp eci cation in one
mo dule and each implementation in a sep erate mo dule But
the sp eci cation refers to the implementation then the
since 6.2 Correctness Criteria
sp eci cation mo dule must imp ort the implementation one
The problem of determining whether a test is passed or
currently under test Okasaki was obliged to edit the sp ec
not is known as the oracle problem One solution is to com
i cation mo dule by hand b efore each test so as to imp ort
pare program output with that of another version of the pro
the right implementation Much preferable would b e to pa
gram p erhaps an older one or p erhaps a simpler slower
rameterise the sp eci cation on an implementation mo dule
but obviously correct version Alternatively an executable
ML style functors would b e really helpful here
sp eci cation might play the same r ole This kind of oracle
can easily b e expressed as a QuickCheck prop erty although
prop erties are much more general
6. DISCUSSION our
However often one can check that a program s output is
much more e ciently than one can compute the out
6.1 On Random Testing correct
At rst sight random selection of test cases may seem put Blum and Kannan exploit this in their work on result
a very naive approach Systematic metho ds are often pre checking a program checker is de ned to b e another
ferred in general a test adequacy criterion is de ned and program which classi es the program s output as correct or
testing pro ceeds by generating test cases which meet the buggy with a high probability of classifying correctly and
adequacy criterion For example a simple criterion is that do es so with strictly lower complexity They distinguish pro
every reachable statement should b e executed in at least gram checking from program testing their prop osal is that
one test a more complex one that every feasible control programs should always check their output and indeed in
ow path with exceptions for lo ops b e followed in at least further work Blum et al showed how programs which usu
one test A wide variety of adequacy critera have b een pro ally pro duce correct answers can even correct wrong output
p osed a recent survey is in particular domains Of course result checkers can
We have chosen not to base QuickCheck on such an ade also b e expressed as QuickCheck prop erties although we use
quacy criterion In part this is b ecause many criteria would them for testing rather than as a part of the nal program
need reinterpretation b efore they could b e applied to Haskell QuickCheck s prop erty language is however more general
programs it is much less clear for example what a control than result checking Via conditional prop erties or sp eci c
ow path is in a language with higher order functions and test data generators we can express prop erties which hold
lazy evaluation In part such a criterion would force us to only for a subset of all p ossible inputs Thus we avoid testing
use much more heavyweight metho ds even measuring path functions in cases which lead to run time errors or cases in
coverage for example would require compiler mo di cations which we do not care ab out the result For example we do
and thus tie QuickCheck to a particular implementation of not test insertion into an unordered list there is no p oint
Haskell namely the one we mo di ed to collect path infor in doing so Yet a result checker must verify that a program
mation Generating test data to exercise a particular path pro duces the correct output in al l cases even those which
requires constraint solving one must nd input values which are uninteresting Moreover QuickCheck prop erties are not
make the series of tests along the given path pro duce sp ec limited to checking the result of an individual function call
i ed results While such constraint solving may b e feasible the prop erty that an op erator is asso ciative for example
for arithmetic data for the rich symbolic datatypes found cannot really b e said to check the result of any individual use
in Haskell programs it is a di cult research problem in its of the op erator but still expresses a useful global prop erty
own right that can b e checked by testing
However apart from the di culty of automating system The idea of testing the prop erties in a sp eci cation di
atic testing metho ds for Haskell there is no clear reason to rectly was used in the DAISTS system for testing ab
b elieve that doing so would yield b etter results In stract data types which compiled equational prop erties into
Duran and Ntafos compared the fault detection probability testing co de although test cases had to b e supplied by the
of random testing with partition testing and discovered that user Lacking automatic test case generation DAISTS did
the di erences in e ectiveness were small Hamlet and not need equivalents of our conditional and quanti ed prop
Taylor rep eated their study more extensively and corrob o erties Although the language used was imp erative abstract
rated the original results Although partition testing is data type op erations had to b e forbidden to side e ect their
slightly more e ective at exp osing faults to quote Hamlet s arguments thus the programs to b e tested were essentially
excellent survey By taking more points in a ran restricted to b e functional Later work aims to relax this
dom test any advantage a partition test might have had is restriction Antoy and Hamlet describ e a technique for test
wiped out ing C classes against an algebraic sp eci cation which is
For small programs in particular it is likely that random animated in order to predict the correct result How
test cases wil l indeed exercise all paths for example so that ever the sp eci cation language must b e restricted in order
test coverage is likely to b e go o d by any measure Using to guarantee that sp eci cations can b e animated
QuickCheck we apply random testing at a ne grain we There seems to b e no published work on automatic test
check prop erties of individual functions but the functions ing of functional programs against sp eci cations We simply
But what precisely do we mean by morally We cannot observe that functional programs and prop erty based sp eci
x the problem just by reinterpreting equality for the Gen cations are a very go o d match we can use the given prop
type claiming the two sides are just di erent representations erties directly for testing Moreover embedding the sp eci
of the same abstract generator This isn t go o d enough cation language in Haskell p ermits us to write very p owerful
b ecause we can actually observe the di erence at other types and exible prop erties with a minimum of learning e ort
by supplying a random number seed something we have to required
able to do if the Gen type is to b e useful Instead we have
6.3 Test Data Generation b e
to reinterpret what we mean by program equivalence in the
Commercial random testing to ols generate test data in
presence of random number generation
limited domains with the goal of matching the distribution
We note that this di culty is by no means con ned to
of actual data for the system under test the so called oper
Haskell the imp erative program
ational pro le In this case statistical inferences ab out the
a random b random c a b
mean time b etween system failures can b e drawn from the
test results
is morally equivalent to the same program with the assign
In order to generate more complex data it is necessary
ments reversed in the same sense but of course pro duces a
to provide a description of the data s structure A p opu
di erent result There is some interesting semantic theory
lar approach to doing so uses grammars However it was
to b e done here
realised very early that context free grammars cannot ex
all the desired prop erties of test data for example
press 6.5 On Lazy Evaluation
that a generated random program contains no undeclared
We have argued in the past that lazy evaluation is an in
identi ers Therefore the grammars were enhanced with ac
valuable programming to ol that radically changes the way
tions or extended to attribute grammars This approach
programs can b e structured Yet QuickCheck is of
has b een most used for testing compilers although Maurer
course only able to test computable prop erties Is there
argues for its use in many contexts
a con ict here
Grammars have b een used for systematic testing where
In fact the con ict is much less than one might imag
for example the generated test data is required to exercise
ine As we have shown ab ove we can p erfectly well use
each pro duction at least once Such an adequacy crite
in nite structures in sp eci cations provided the prop erties
rion maybe b e particularly appropriate for compiler test
we actually test are computable for example we can test
ing Maurer also used grammars for random testing
that arbitrarily long pre xes of in nite lists are equal rather
and noted the termination problem for recursive grammars
than comparing the lists themselves Our Gen monad has a
His solution though was just to increase the probabilities of
lazy bind op eration b ecause we split the random number
generating leaves so that eventual termination is guaranteed
seed rather than threading it through rst one op erand
Our exp erience is that this results in far to o high a prop or
then the other and so we can freely de ne generators that
tion of trivial test cases and therefore ine cient testing
pro duce in nite results What we cannot do is observe non
more tests must b e run to exercise the program prop erly
termination in a test result So we cannot test for example
We b elieve our metho d of controlling sizes is much sup erior
the prop erty
It seems that the need to learn a complex language of ex
reverse xs undefined undefined
tended grammars has hindered the adoption of these meth
o ds in practice By embedding a test generator language
On the other hand in a sense a human tester cannot ob
in Haskell we provide at least the same capabilities but
serve non termination either and if we have b een able to
spare the programmer the need to learn more than a few new
test lazy programs satisfactorily by hand so far then we are
op erators At the same time we provide all the p ower and
not in a worse p osition if we use QuickCheck Yet a human
exibility needed to generate test data satisfying complex in
tester can observe that reverse xs undefined pro duces
variants in a language the programmer already knows By
an error message from the evaluation of undefined without
linking generators to types via Haskell s class system we re
pro ducing any other output rst and can thus infer that the
lieve the programmer of the need to sp ecify generators at all
prop erty ab ove holds The problem is that the Haskell stan
in many cases and where they must b e sp eci ed the pro
dard provides no way for a program to make the same ob
grammer s work is usually limited to sp ecifying generators
servation Yet there are various extensions of Haskell which
for his or her own new types
do indeed make this p ossible Some work done by Andy Gill
shown that given such extensions we could formulate
6.4 On Randomness has
and check prop erties such as the one ab ove using QuickCheck
We have encountered some interesting problems in reason
also
ing ab out programs which use random number generation
particular the Gen monad which QuickCheck is based on
In 6.6 Some Reflections
is not a monad at all Consider the rst monad law
We are convinced that one of the ma jor advantages of
return x f f x
using QuickCheck is that it encourages us to formulate for
Since our implementation of bind splits its random number mal sp eci cations thus improving our understanding of our
seed to yield the seeds passed to each op erand then f is programs While it is op en to the programmer to do this
passed di erent seeds on the two sides of the equation and anyway few really do p erhaps b ecause there is little short
may therefore pro duce di erent results So the law simply term payo and p erhaps b ecause a sp eci cation is of less
do es not hold Morally however we consider the law to b e value if there is no check at all that it corresp onds to the
true b ecause the two sides pro duce the same distribution of implemented program QuickCheck addresses b oth these is
results even if the results di er for any particular seed sues it gives us a short term payo via automated testing
some reason to b elieve that prop erties stated in a mo d
and 8. REFERENCES
ule actually hold
S Antoy and R Hamlet Automatically checking an
We have observed that the errors we nd are divided
implementation against its formal sp eci cation In
roughly evenly b etween errors in test data generators er
Irvine Software Symposium pages March
rors in the sp eci cation and errors in the program The
Roland Backhouse Patrik Jansson Johan Jeuring
rst category is useless to discover except insofar as it helps
and Lambert Meertens Generic Programming An
with further testing it tells us nothing ab out the actual
Introduction In Lecture notes in Computer Science
program The third category is obviously useful in a sense
volume
these are the errors we test in order to nd But the second
category is also imp ortant even if they do not reveal a mis
P Bjesse K Claessen M Sheeran and S Singh
take in the co de they do reveal a misunderstanding ab out
Lava Hardware Design in Haskell In International
what it do es Correcting such misunderstandings improves
Conference on Functional Programming Baltimore
our ability to make use of the tested co de correctly later
ACM
When formulating sp eci cations one rapidly discovers the
M Blum and S Kannan Designing programs that
need for a library of functions that implement common math
check their work In Proc st Symposium on the
ematical abstractions We are developing an implementa
Theory of Computing pages ACM May
tion of nite set theory for use with QuickCheck many of
M Blum M Luby and R Rubinfeld
the abstractions in it are to o ine cient to b e of much use in
Self testing correcting with applications to numerical
programs but in sp eci cations where the ob ject is to state
problems In Proc nd Symposium on the Theory of
prop erties as clearly and simply as p ossible they come into
Computing pages ACM May
their own Because of this di erence in purp ose there is a
A Celentano S C Reghizzi P Della Vigna and
need for libraries sp eci cally targeted at sp eci cations
C Ghezzi Compiler testing using a sentence
The ma jor limitation of QuickCheck is that there is no
generator Software Practice Experience
measurement of test coverage it is up to the user to in
vestigate the distribution of test data and decide whether
K Claessen and D Sands Observable Sharing for
su ciently many tests have b een run Although we provide
Functional Circuit Description In Asian Computer
ways to collect this information we cannot comp el the pro
Science Conference Phuket Thailand ACM
grammer to use them A programmer who do es not risks
Sigplan
gaining a false sense of security from a large number of in
M Davis and H Putnam A computing pro cedure for
adequate tests Perhaps we could de ne adequacy measures
quanti cation theory Journal of the Association for
just on the generated test data and thus warn the user at
Computing Machinery
least in this kind of situation
J Duran and S Ntafos An evaluation of random
testing Transactions on Software Engineering
July
7. CONCLUSIONS
J Gannon R Hamlet and P McMullin Data
We have taken two relatively old ideas namely sp eci ca
abstraction implementation sp eci cation and testing
tions as oracles and random testing and found ways to make
Trans Prog Lang and Systems
them easily available to Haskell programmers Firstly we
D Hamlet Random testing In J Marciniak editor
provide an embedded language for writing prop erties giv
Encyclopedia of Software Engineering pages
ing expressiveness without the learning cost The language
Wiley
contains convenient features such as quanti ers condition
R Hamlet and R Taylor Partition testing do es not
als and test data monitors Secondly we provide type based
inspire con dence Transactions on Software
default random test data generators including random func
Engineering December
tions greatly reducing the e ort of sp ecifying them Thirdly
J Hughes Why Functional Programming Matters In
we provide an embedded language for sp ecifying custom test
D Turner editor Research Topics in Functional
data generators which can b e based on the default genera
Programming Addison Wesley
tors giving a ner control over test data distribution We
M P Jones The Hugs distribution Currently
also introduce a novel way of controlling size when generat
available from http haskell org hugs
ing random elements of recursive data types
P M Maurer Generating test data with enhanced
Further we demonstrate that the combination of these
context free grammars IEEE Software
old techniques works extremely well for Haskell The func
tional nature allows for lo cal and ne grained prop erties
Gunnar St almarck A System for Determining
since all dep endencies of a function are explicit And pre
Prop ositional Logic Theorems by Applying Values and
cisely random testing is known to work very well for small
Rules to Triplets that are Generated from a Formula
ne grained programs and is e ective in nding faults
Swedish Patent No approved
Lastly the to ol is lightweight and easy to use and pro
U S Patent No Europ ean Patent
vides a short term payo for explicitly stating prop erties of
No
functions in a program which greatly increases the under
Philip Wadler Theorems for free In International
standing of the program for the programmer as well as for
Conference on Functional Programming and Computer
do cumentation purp oses
Architecture London September
Acknowledgements We would like to thank Andy Gill
Philip Wadler A prettier printer March Draft Chris Okasaki and the anonymous referees for their useful
pap er comments on this pap er
H Zhu P Hall and J May Software unit test instance Coarbitrary Bool where
coarbitrary b variant if b then else
coverage and adequacy Computing Surveys
December
instance Coarbitrary Int where
coarbitrary n
n variant
Appendix: Implementation
n variant coarbitrary n
otherwise variant coarbitrary n div
Here we show the implementation of the QuickCheck library
except for the function quickCheck The source co de of
instance Coarbitrary a Coarbitrary b
QuickCheck is available from www cs chalmers se rjmh
Coarbitrary a b where
coarbitrary a b coarbitrary a coarbitrary b
QuickCheck
module QuickCheck where instance Coarbitrary a Coarbitrary a where
coarbitrary variant
import Monad import Random coarbitrary a as
variant coarbitrary a coarbitrary as
Gen
instance Arbitrary a Coarbitrary b
newtype Gen a Gen Int Rand a Coarbitrary a b where
coarbitrary f gen
choose Random a a a Gen a arbitrary coarbitrary gen f
choose bounds Gen n r fst randomR bounds r
Property
variant Int Gen a Gen a
variant v Gen m Gen n r newtype Property Prop Gen Result
m n rands r v
where data Result Result
rands r r rands r where r r split r fok Maybe Bool stamp String arguments String g
promote a Gen b Gen a b nothing Result
promote f Gen n r a nothing Result
let Gen m f a in m n r fok Nothing stamp arguments g
sized Int Gen a Gen a result Result Property
sized fgen Gen n r result res Prop return res
let Gen m fgen n in m n r
class Testable a where
instance Monad Gen where property a Property
return a Gen n r a
Gen m k instance Testable Bool where
Gen n r let r r split r property b result nothingf ok Just b g
Gen m k m n r
in m n r instance Testable Property where
property prop prop
elements a Gen a
elements xs xs liftM choose length xs instance Arbitrary a Show a Testable b
Testable a b where
vector Arbitrary a Int Gen a property f forAll arbitrary f
vector n sequence arbitrary i n
evaluate Testable a a Gen Result
oneof Gen a Gen a evaluate a gen where Prop gen property a
oneof gens elements gens id
forAll Show a Testable b Gen a a b Property
frequency Int Gen a Gen a forAll gen body Prop
frequency xs choose sum map fst xs pick xs do a gen
where res evaluate body a
pick n k x xs n k x return arg a res
otherwise pick n k xs where
arg a res resf arguments show a arguments res g
Arbitrary Coarbitrary
Testable a Bool a Property
class Arbitrary a where True a property a
arbitrary Gen a False a result nothing
instance Arbitrary Bool where label Testable a String a Property
arbitrary elements True False label s a Prop add fmap evaluate a
where add res resf stamp s stamp res g
instance Arbitrary Int where
arbitrary sized n choose n n classify Testable a Bool String a Property
classify True name label name
instance Arbitrary a Arbitrary b Arbitrary a b where classify False property
arbitrary liftM arbitrary arbitrary
collect Show a Testable b a b Property
instance Arbitrary a Arbitrary a where collect v label show v
arbitrary sized n choose n vector
instance Arbitrary a Arbitrary b Arbitrary a b where
arbitrary promote coarbitrary arbitrary
class Coarbitrary a where
coarbitrary a Gen b Gen b