QuickCheck, SmallCheck & Reach: Automated Testing in Haskell By Tom Shackell A Brief Introduction to Haskell ● Haskell is a purely functional language. ● Based on the idea of evaluation of mathematical functions rather than on manipulating state. ● Heavy emphasis on 'functions' and data-types. ● Higher-order functions. Passing functions to other functions. ● Very powerful static type system including type- inference. Some Simple Examples ... A program for calculating factorials: fac :: Int -> Int fac 0 = 1 fac n = n * fac (n - 1) And for calculating the length of a list: length :: [a] -> Int length [] = 0 length (x:xs) = 1 + length xs Data Types A data type for colours data Colour = Red | Green | Blue For binary trees data Tree a = Empty | Node a (Tree a) (Tree a) Higher Order Functions A function to apply some function to every element of a list: map :: (a -> b) -> [a] -> [b] map f [] = [] map f (x:xs) = f x : map f xs for example map (*2) [3,5,9] = [6,10,18] Class Polymorphism A function to tested whether a list contains a particular element: elem :: Eq a => a -> [a] -> Bool elem x [] = False elem x (y:ys) = x == y || elem x ys Classes The previous example makes use of the Eq class. class Eq a where (==) :: a -> a -> Bool (/=) :: a -> a -> Bool x /= y = not (x == y) We can define instances of this class to describe how to compare any data type for equality. instance Eq Colour where Red == Red = True Green == Green = True Blue == Blue = True _ == _ = False QuickCheck QuickCheck ● Developed by Koen Claessen and John Hughes. ● Based on the idea of specifying properties that the programmer expects to be true. ● These properties are then tested with a large number of random inputs. ● Properties are expressed in Haskell using combinators defined in the QuickCheck library. Simple QuickCheck Properties We can express a property such as: prop_RevRev :: [Int] -> Bool prop_RevRev xs = reverse (reverse xs) == xs We can then ask QuickCheck to verify this property. Main> quickCheck prop_RevRev OK: Passed 100 tests Or perhaps it fails. Falsifiable, after 12 tests: [3,5,1] Conditionals We can specify conditional properties. prop_MaxLe :: Int -> Int -> Property prop_MaxLe x y = x <= y ==> max x y == y prop_OrdInsert :: Int -> [Int] -> Property prop_OrdInsert x xs = ordered xs ==> ordered (insert x xs) This second property has a problem ... Trivial Cases ● The condition in prop_OrdInsert is quite restrictive. ● Most lists generated randomly are not ordered. ● Of those that are, shorter lists are a lot more likely that long ones. ● Our test function will likely only test very small cases. Generators In fact it's better to define this property using a 'generator'. prop_OrdInsert :: Int -> Property prop_OrdInsert x = forAll orderedList $ \ xs -> ordered (insert x xs) The generator orderedList generates a random ordered list of numbers. Generator Instances We can tell QuickCheck how to generate random data of any type by providing an instance of the arbitrary class. class Arbitrary a where arbitrary :: Gen a for example instance Arbitrary Colour where arbitrary = elements [ Red, Green, Blue ] Generator Functions We can also define generator functions such as orderedList orderedList :: (Arbitrary a, Ord a) => Gen [a] orderedList = do xs <- arbitrary return (sort xs) How QuickCheck is implemented ● QuickCheck is implemented as a Haskell library imported into the program rather than an external tool. ● quickCheck is a Haskell function that takes a function of any number of arguments. ● It generates random values for the arguments, passes them to the function and observes the result. ● Internally the implementation relies on some q uite clever use of Haske ll's class system. QuickCheck ● Used for random testing. ● Easy to use as it's a standard Haskell library. ● Generally works best for simple data structures. ● Works best for properties with simple preconditions. ● Writing generators can be time consuming and tedious. SmallCheck SmallCheck ● Developed by Colin Runciman at York. ● Design is similar to QuickCheck. ● However the focus is on finding small counter examples through exhaustive search of the space. ● Uses a depth-bound on the size of data generated to control search space explosion. Usage SmallCheck is used in a very similar way to QuickCheck prop_RevRev :: [Bool] -> Bool prop_RevRev xs = reverse (reverse xs) == xs Main> smallCheck 5 prop_RevRev Completed 63 tests without failure Here we've checked all lists of Bools with length less than or equal to 5. Existentials SmallCheck introduces the idea of existential quantification. prop_isPrefix :: [Bool] -> [Bool] -> Property prop_isPrefix xs ys = isPrefix xs ys ==> exists $ \ zs -> ys == xs ++ zs SmallCheck will search exhaustively for a zs that matches the criteria specified. SmallCheck ● Implemented in much the same way as QuickCheck but with exhaustive searching. ● Has issues with large search spaces. ● Still requires user to write generators. Reach Reach ● Developed by Matt Naylor and Colin Runciman ● Is based on the idea of trying to find an input that causes a target expression to be evaluated. ● Very much based around the idea of avoiding having to write 'generators'. ● Is implemented using constraint solving and Functional Logic Programming constructs. ● Like SmallCheck, makes use of a depth-bound. Target Expressions In Reach the user specifies a target expression that they would like to be executed. This is done using the target function. f xs ys = if ordered xs && ordered ys then target (xs ++ ys) else ... Reach will then search for an input to the function f that causes target to be evaluated. Property Testing This can be used to test properties just like in QuickCheck or SmallCheck. prop_RevRev :: [Int] -> Bool prop_RevRev xs = reverse (reverse xs) == xs main xs = refute (prop_RevRev xs) refute True = True refute False = target False Reach will search for an input that makes the property false. Constraint Solving ● Reach does not use exhaustive search as in SmallCheck. Instead it uses constraint solving. ● At the top level the main function is evaluated with unbound logical variables as the arguments. ● Evaluation proceeds as normal Haskell but instead of inspecting data structures computation introduces constraints. ● Evaluation finishes when the target is reached or the depth-bound exceeded. data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) main a {a=?} Stack data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte a (S Z) {a=?} Stack data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte Z (S Z) lte a (S Z) {a=?} Stack data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte Z (S Z) {a=Z} lte a (S Z) {a=?} Stack data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) Target not reached! True {a=Z} lte a (S Z) {a=?} Stack data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) (S Z) doesn't match Z lte a (S Z) {a=?} Stack data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte (S b) (S Z) {a=S b, b = ?} lte a (S Z) {a=?} Stack data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte (Slte x)b Z(S { aZ)=S {x=S b, b y=, y? }= ?} lte a (S Z) {a=?} Stack data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte Z Z {a=S b, b = Z} lte (Slte x)b Z(S { aZ)=S {x=S b, b y=, y? }= ?} lte a (S Z) {a=?} Stack data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) Target not reached! True {a = S b, b = Z} lte (Slte x)b Z(S { aZ)=S {x=S b, b y=, y? }= ?} lte a (S Z) {a=?} Stack data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) lte (S c) Z {a=S b, b = S c, c = ?} lte (Slte x)b Z(S { aZ)=S {x=S b, b y=, y? }= ?} lte a (S Z) {a=?} Stack data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z) Target reached! Answer found: a = S b, b = S c, c = ? a = S (S ?) target False {a=S b, b = S c, c = ?} lte (Slte x)b Z(S { aZ)=S {x=S b, b y=, y? }= ?} lte a (S Z) {a=?} Stack Reach ● Can be used for many of the same problems as SmallCheck. ● Is often much more efficient than SmallCheck, especially in the presence of complex antecedents. ● No need to write 'generators' which can be a big win in certain applications.