Quickcheck: a Lightweight Tool for Random Testing of Haskell Programs
Total Page：16
File Type：pdf, Size：1020Kb
QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs Koen Claessen John Hughes Chalmers University of Technology Chalmers University of Technology [email protected] [email protected] are hard to test and so testing can b e done at a ABSTRACT monad ne grain QuickCheck is a to ol which aids the Haskell programmer in A testing to ol must b e able to determine whether a test formulating and testing prop erties of programs Prop erties is passed or failed the human tester must supply an auto are describ ed as Haskell functions and can b e automati matically checkable criterion of doing so We have chosen cally tested on random input but it is also p ossible to de to use formal sp ecications for this purp ose We have de ne custom test data generators We present a number of signed a simple domainsp ecic language of testable speci case studies in which the to ol was successfully used and cations which the tester uses to dene exp ected prop erties also p oint out some pitfalls to avoid Random testing is es of the functions under test QuickCheck then checks that the p ecially suitable for functional programs b ecause prop erties prop erties hold in a large number of cases The sp ecica can b e stated at a ne grain When a function is built from tion language is embedded in Haskell using the class system separately tested comp onents then random testing suces Prop erties are normally written in the same mo dule as the to obtain go o d coverage of the denition under test functions they test where they serve also as checkable do c umentation of the b ehaviour of the co de testing to ol must also b e able to generate test cases au 1. INTRODUCTION A tomatically We have chosen the simplest metho d random Testing is by far the most commonly used approach to testing which comp etes surprisingly favourably with ensuring software quality It is also very lab our intensive systematic metho ds in practice However it is meaningless accounting for up to of the cost of software develop to talk ab out random testing without discussing the distri ment Despite anecdotal evidence that functional programs bution of test data Random testing is most eective when require somewhat less testing Once it typechecks it usu the distribution of test data follows that of actual data but ally works in practice it is still a ma jor part of functional when testing reuseable co de units as opp osed to whole sys program development tems this is not p ossible since the distribution of actual The cost of testing motivates eorts to automate it wholly data in all subsequent reuses is not known A uniform dis or partly Automatic testing to ols enable the programmer tribution is often used instead but for data drawn from to complete testing in a shorter time or to test more thor innite sets this is not even meaningful how would one oughly in the available time and they make it easy to rep eat choose a random closed term with a uniform distribution tests after each mo dication to a program In this pap er we for example We have chosen to put distribution under the describ e a to ol QuickCheck which we have developed for human testers control by dening a test data generation testing Haskell programs language also embedded in Haskell and a way to observe Functional programs are well suited to automatic testing the distribution of test cases By programming a suitable It is generally accepted that pure functions are much easier generator the tester can not only control the distribution to test than sideeecting ones b ecause one need not b e of test cases but also ensure that they satisfy arbitrarily concerned with a state b efore and after execution In an complex invariants imp erative language even if whole programs are often pure An imp ortant design goal was that QuickCheck should b e functions from input to output the pro cedures from which lightweight Our implementation consists of a single pure they are built are usually not Thus relatively large units Haskell mo dule of ab out lines which is in practice must b e tested at a time In a functional language pure mainly used from the Hugs interpreter We have also writ functions ab ound in Haskell only computations in the IO ten a small script to invoke it which needs to know very little ab out Haskell syntax and consequently supp orts the full language and its extensions It is not dep endent on any Haskell system A cost that comes with this deci Permission to make digital or hard copies of all or part of this work for particular is that we can only test prop erties that are expressible personal or classroom use is granted without fee provided that copies are sion observable within Haskell not made or distributed for profit or commercial advantage and that copies and is notoriously dicult to say how eective a testing bear this notice and the full citation on the first page. To copy otherwise, to It republish, to post on servers or to redistribute to lists, requires prior specific metho d is in detecting faults However we have used Quick permission and/or a fee. in a variety of applications ranging from small exp er ICFP ’00, Montreal, Canada. Check Copyright 2000 ACM 1-58113-202-6/00/0009 ..5.00 In fact the programmer must provide a little more infor iments to real systems and we have found it to work well mation the function quickCheck is actually overloaded in in practice We rep ort on some of this exp erience in this order to b e able handle laws with a varying number of vari pap er and p oint out pitfalls to avoid ables and the overloading cannot b e resolved if the law itself The rest of this pap er is structured as follows Section has a p olymorphic type as in these examples Thus the pro introduces the concept of writing prop erties and checking grammer must sp ecify a xed type at which the law is to b e them using QuickCheck Section shows how to dene test tested So we simply give a type signature for each law for data generators for userdened types Section briey dis example cusses the implementation Section presents a number of case studies that show the usefulness of the to ol Section propRevApp Int Int Bool concludes Of course the prop erty propRevApp holds p olymorphically we must sp ecify which monomorphic instance to test 2. DEFINING PROPERTIES but it at so that we can generate test cases This turns out b e quite imp ortant in the case of overloaded functions 2.1 A Simple Example to For example is asso ciative for the type Int but not for As a rst example we take the standard function reverse Double In some cases we can use parametricity to which reverses a list This satises a number of useful laws argue that a prop erty holds p olymorphically such as 2.2 Functions reverse x x We are also able to formulate prop erties that quantify over reverse xsys reverse ysreverse xs functions To check for example that function comp osition reverse reverse xs xs is asso ciative we rst dene extensional equality by f g x f x g x and then write In fact the rst two of these characterise reverse uniquely Note that these laws hold only for nite total values In propCompAssoc Int Int Int Int this pap er unless sp ecically stated otherwise we always Int Int Int Bool quantify over completely dened nite values This is to propCompAssoc f g h make it more likely that the prop erties are computable f g h f g h In order to check these laws using QuickCheck we repre The only diculty that function types cause is that if a sent them as Haskell functions Thus we dene counterexample is found for example if we try to check propRevUnit x that function comp osition is commutative then the func reverse x x tion values are printed just as function In this case we discover that the law we are checking is false but not propRevApp xs ys why reverse xsys reverse ysreverse xs 2.3 Conditional Laws propRevRev xs Laws which are simple equations are conveniently repre reverse reverse xs xs sented by b o olean function as we have seen but in general many laws hold only under certain conditions QuickCheck Now if these functions return True for every p ossible argu provides an implication combinator to represent such condi ment then the prop erties hold We load them into the Hugs tional laws For example the law interactive Haskell interpreter and call for example Main quickCheck propRevApp x y ) max x y y OK passed tests can b e represented by the denition The function quickCheck takes a law as a parameter and ap propMaxLe Int Int Property plies it to a large number of randomly generated arguments 1 propMaxLe x y x y max x y y in fact rep orting OK if the result is True in every case Likewise the insertion function into ordered lists satises If the law fails then quickCheck rep orts the counter the law example For example if we mistakenly dene propInsert Int Int Property propRevApp xs ys propInsert x xs reverse xsys reverse xsreverse ys ordered xs ordered insert x xs then checking the law might pro duce Note that the result type of the prop erty is changed from Bool to Property This is b ecause the testing semantics Main quickCheck propRevApp is dierent for conditional laws Instead of checking the Falsifiable after tests prop erty for random test cases we try checking it for test cases satisfying the condition If a candidate test case do es not satify the condition it is discarded and a new where the counter mo del can b e extracted by taking for test case is tried xs and for ys Checking the laws propMaxLe and propInsert succeed 1 as usual but sometimes checking a conditional law pro