Quickcheck, Smallcheck & Reach: Automated Testing in Haskell

QuickCheck, SmallCheck & Reach: Automated Testing in Haskell

Tom Shackell

A Brief Introduction to Haskell

● Haskell is a purely functional language.

● Based on the idea of evaluation of mathematical functions rather than on manipulating state.

● Heavy emphasis on 'functions' and data-types.

● Higher-order functions. Passing functions to other functions.

● Very powerful static type system including type-

inference. Some Simple Examples ...

A program for calculating factorials:

fac :: Int -> Int fac 0 = 1 fac n = n * fac (n - 1)

And for calculating the length of a list:

length :: [a] -> Int length [] = 0 length (x:xs) = 1 + length xs

Data Types

A data type for colours

data Colour = Red | Green | Blue

For binary trees

data Tree a = Empty | Node a (Tree a) (Tree a)

Higher Order Functions

A function to apply some function to every element of a list:

map :: (a -> b) -> [a] -> [b] map f [] = [] map f (x:xs) = f x : map f xs for example

map (*2) [3,5,9] = [6,10,18]

Class Polymorphism

A function to tested whether a list contains a particular element:

elem :: Eq a => a -> [a] -> Bool elem x [] = False elem x (y:ys) = x == y || elem x ys

Classes The previous example makes use of the Eq class.

class Eq a where (==) :: a -> a -> Bool (/=) :: a -> a -> Bool x /= y = not (x == y) We can define instances of this class to describe how to compare any data type for equality. instance Eq Colour where Red == Red = True Green == Green = True Blue == Blue = True _ == _ = False

QuickCheck

● Developed by Koen Claessen and John Hughes.

● Based on the idea of specifying properties that the programmer expects to be true.

● These properties are then tested with a large number of random inputs.

● Properties are expressed in Haskell using combinators defined in the QuickCheck library.

Simple QuickCheck Properties

We can express a property such as: prop_RevRev :: [Int] -> Bool prop_RevRev xs = reverse (reverse xs) == xs We can then ask QuickCheck to verify this property.

Main> quickCheck prop_RevRev OK: Passed 100 tests Or perhaps it fails. Falsifiable, after 12 tests: [3,5,1]

Conditionals

We can specify conditional properties.

prop_MaxLe :: Int -> Int -> Property prop_MaxLe x y = x <= y ==> max x y == y

prop_OrdInsert :: Int -> [Int] -> Property prop_OrdInsert x xs = ordered xs ==> ordered (insert x xs)

This second property has a problem ...

Trivial Cases

● The condition in prop_OrdInsert is quite restrictive.

● Most lists generated randomly are not ordered.

● Of those that are, shorter lists are a lot more likely that long ones.

● Our test function will likely only test very small cases.

Generators

In fact it's better to define this property using a 'generator'.

prop_OrdInsert :: Int -> Property prop_OrdInsert x = forAll orderedList $ \ xs -> ordered (insert x xs)

The generator orderedList generates a random ordered list of numbers.

Generator Instances We can tell QuickCheck how to generate random data of any type by providing an instance of the arbitrary class.

class Arbitrary a where arbitrary :: Gen a for example

instance Arbitrary Colour where arbitrary = elements [ Red, Green, Blue ]

Generator Functions

We can also define generator functions such as orderedList

orderedList :: (Arbitrary a, Ord a) => Gen [a] orderedList = do xs <- arbitrary return (sort xs)

How QuickCheck is implemented

● QuickCheck is implemented as a Haskell library imported into the program rather than an external tool.

● quickCheck is a Haskell function that takes a function of any number of arguments.

● It generates random values for the arguments, passes them to the function and observes the result.

● Internally the implementation relies on some q uite clever use of Haske ll's class system. QuickCheck

● Used for random testing.

● Easy to use as it's a standard Haskell library.

● Generally works best for simple data structures.

● Works best for properties with simple preconditions.

● Writing generators can be time consuming and tedious.

SmallCheck

● Developed by Colin Runciman at York.

● Design is similar to QuickCheck.

● However the focus is on finding small counter examples through exhaustive search of the space.

● Uses a depth-bound on the size of data generated to control search space explosion.

Usage

SmallCheck is used in a very similar way to QuickCheck

prop_RevRev :: [Bool] -> Bool prop_RevRev xs = reverse (reverse xs) == xs

Main> smallCheck 5 prop_RevRev Completed 63 tests without failure

Here we've checked all lists of Bools with length less than or equal to 5.

Existentials

SmallCheck introduces the idea of existential quantification.

prop_isPrefix :: [Bool] -> [Bool] -> Property prop_isPrefix xs ys = isPrefix xs ys ==> exists $ \ zs -> ys == xs ++ zs

SmallCheck will search exhaustively for a zs that matches the criteria specified.

SmallCheck

● Implemented in much the same way as QuickCheck but with exhaustive searching.

● Has issues with large search spaces.

● Still requires user to write generators.

Reach

● Developed by Matt Naylor and Colin Runciman

● Is based on the idea of trying to find an input that causes a target expression to be evaluated.

● Very much based around the idea of avoiding having to write 'generators'.

● Is implemented using constraint solving and Functional Logic Programming constructs.

● Like SmallCheck, makes use of a depth-bound.

Target Expressions

In Reach the user specifies a target expression that they would like to be executed. This is done using the target function.

f xs ys = if ordered xs && ordered ys then target (xs ++ ys) else ... Reach will then search for an input to the function f that causes target to be evaluated.

Property Testing

This can be used to test properties just like in QuickCheck or SmallCheck.

prop_RevRev :: [Int] -> Bool prop_RevRev xs = reverse (reverse xs) == xs

main xs = refute (prop_RevRev xs)

refute True = True refute False = target False Reach will search for an input that makes the property false.

Constraint Solving

● Reach does not use exhaustive search as in SmallCheck. Instead it uses constraint solving.

● At the top level the main function is evaluated with unbound logical variables as the arguments.

● Evaluation proceeds as normal Haskell but instead of inspecting data structures computation introduces constraints.

● Evaluation finishes when the target is reached or the depth-bound exceeded. data Nat = Z | S Nat lte Z _ = True lte (S _) Z = target False lte (S x) (S y) = lte x y main a = lte a (S Z)

main a {a=?}