MLCheck– Property-Driven Testing of Machine Learning Models
Arnab Sharma Caglar Demir Department of Computer Science Data Science Group University of Oldenburg Paderborn University Oldenburg, Germany Paderborn, Germany [email protected] [email protected] Axel-Cyrille Ngonga Ngomo Heike Wehrheim Data Science Group Department of Computer Science Paderborn University University of Oldenburg Paderborn, Germany Oldenburg, Germany [email protected] [email protected] ABSTRACT 1 INTRODUCTION In recent years, we observe an increasing amount of software with The importance of quality assurance for applications developed machine learning components being deployed. This poses the ques- using machine learning (ML) increases steadily as they are being tion of quality assurance for such components: how can we validate deployed in a growing number of domains and sites. Supervised ML whether specified requirements are fulfilled by a machine learned algorithms “learn” their behaviour as generalizations of training software? Current testing and verification approaches either focus data using sophisticated statistical or mathematical methods. Still, on a single requirement (e.g., fairness) or specialize on a single type developers need to make sure that their software—whether learned of machine learning model (e.g., neural networks). or programmed—satisfies certain specified requirements. Currently, In this paper, we propose property-driven testing of machine two orthogonal approaches can be followed to achieve this goal: (A) learning models. Our approach MLCheck encompasses (1) a lan- employing an ML algorithm guaranteeing some requirement per guage for property specification, and (2) a technique for system- design, or (B) validating the requirement on the model generated atic test case generation. The specification language is compara- by the ML algorithm. ble to property-based testing languages. Test case generation em- Both approaches have their individual shortcomings: Approach ploys advanced verification technology for a systematic, property- A is only available for a handful of requirements (e.g., fairness, dependent construction of test suites, without additional user- monotonicity, robustness) [23, 33, 41]. Moreover, such algorithms supplied generator functions. We evaluate MLCheck using require- cannot ensure the complete fulfillment of the requirement. For ments and data sets from three different application areas (software example, Galhotra et al. [15] have found fairness-aware ML al- discrimination, learning on knowledge graphs and security). Our gorithms to generate unfair predictions, and Sharma et al. [35] evaluation shows that despite its generality MLCheck can even out- detected non-monotonic predictions in supposedly monotone clas- perform specialised testing approaches while having a comparable sifiers. For robustness to adversarial attacks, the algorithms can runtime. only reduce the attack surface. Approach B, on the other hand, is only possible if a validation technique exists which is applicable to KEYWORDS (1) the specific type of machine learning classifier under considera- tion (i.e., neural network, SVM, etc.) and (2) the specific property to Machine Learning Testing, Decision Tree, Neural Network, Property- be checked. Current validation techniques are restricted to either a Based Testing. single model type or a single property (or even both). ACM Reference Format: In this paper, we propose property-driven testing as a validation arXiv:2105.00741v1 [cs.SE] 3 May 2021 Arnab Sharma, Caglar Demir, Axel-Cyrille Ngonga Ngomo, and Heike technique for machine learning models overcoming the shortcom- Wehrheim. 2021. MLCheck– Property-Driven Testing of Machine Learning ings of approach B. Our technique allows developers to specify the Models . In Proceedings of ACM Conference (Conference’DD). ACM, New property under interest and—based on the property—performs a York, NY, USA, 12 pages. https:XXXX targeted generation of test cases. The target is to find test cases violating the property. The approach is applicable to arbitrary types of non-stochastic properties and arbitrary types of supervised ma- Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed chine learning models. We consider the model under test (MUT) as for profit or commercial advantage and that copies bear this notice and the full citation a black-box of which we just observe the input-output behaviour. on the first page. Copyrights for components of this work owned by others than ACM To achieve a systematic generation of test cases, specific to both must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a MUT and property, we train a second white-box model approximat- fee. Request permissions from [email protected]. ing the MUT by using its predictions as training data. Knowing Conference’DD, MM YYYY, XX the white-box’s internal structure, we can apply state-of-the-art © 2021 Association for Computing Machinery. ACM ISBN 978-x-xxxx-xxxx-x/YY/MM...$15.00 verification technology to verify the property on it. A verification https:XXXX result of “failure” (property not satisfied) is then accompanied by Conference’DD, MM YYYY, XX Arnab Sharma, Caglar Demir, Axel-Cyrille Ngonga Ngomo, and Heike Wehrheim
(one or more) counterexamples, which we subsequently store as 2 FOUNDATIONS test inputs whenever they are failures for the MUT as well. We start by introducing some basic terminology in machine learning We currently employ two types of ML models as approximating and formally defining the properties to be checked for our three white-boxes: decision trees and neural networks. While no prior application areas. knowledge is required pertaining to the internal structure of the A supervised machine learning (ML) algorithm works in two model under test, the internals of the white-box model are accessible steps. In the first (learning) phase, it is presented with a set of data to verification. Test generation proceeds by (1) encoding both prop- instances (training data) and generates a function (the predictive erty and white-box model as logical formulae and (2) using an SMT model), generalising from the training data. The generated predic- (Satisfiability Modulo Theories) solver to check their satisfiability. tive model (short, model) is then used in the second (prediction) Counterexamples in this case directly come in the form of satisfying phase to predict classes for unknown data instances. assignments to logical variables which encode feature and class Formally, the generated model is a function values. Due to the usage of an approximating white-box model, test generation is an iterative procedure: whenever a counterexample 푀 : 푋1 × ... × 푋푛 → 푍1 × ... × 푍푚 , on the white-box model is found which is not valid for the MUT, where 푋푖 is the value set of feature 푖, 1 ≤ 푖 ≤ 푛, and every 푍 푗 , 1 ≤ the white-box model gets retrained. This way the approximation 푗 ≤ 푚, contains the classes for the 푗th label. Instead of numbering quality of the white-box model is successively improved. features and labels, we also use feature names 퐹1, . . . , 퐹푛 and label We have implemented our approach in a tool called MLCheck names 퐿1, . . . , 퐿푚, and let 퐹 = {퐹1, . . . , 퐹푛 }, 퐿 = {퐿1, . . . , 퐿푚 }. We and evaluated it on requirements of three different application freely mix numbers and names in our formalizations. When 푚 > 1, areas: the learning problem is a multilabel classification problem; when |푍푖 | > 2 for some 푖, the learning problem is a multiclass classification • Software discrimination studies whether ML models give problem. In case |푍푖 | = 2 for all 푖, it is a binary classification problem. ì ì predictions which are (un)biased with respect to some at- We write 푋 for 푋1 × ... × 푋푛, 푍 for 푍1 × . . . 푍푚 and use an index tributes. Different definitions of such fairness requirements (like in 푥푖 ) to access the 푖-th component. The training data consists exist (see [40]); we exemplarily use individual discrimina- of elements from 푋ì × 푍ì, i.e., data instances with known associated tion [15]. class labels. During the prediction, the generated predictive model • Knowledge graphs are a family of knowledge representa- assigns classes 푧 ∈ 푍ì to a data instance 푥 ∈ 푋ì (which is poten- tion techniques. We consider learning classifiers for entities tially not in the training data). Based on this formalization, we based on knowledge graphs embeddings [12] and exemplar- define properties relevant to our three application areas software ily consider the properties of class disjointness and subsump- discrimination, knowledge representation and security. tion. Software discrimination. Fairness of predictive models refers • Security of machine learning applications investigates if to the absence of discrimination of individuals due to certain fea- ML models are vulnerable to attacks, i.e., can be manipu- ture values. More precisely, a model has no individual discrimina- lated as to give specific predictions. We exemplarily study tion [15] if flipping the value of a single, so called sensitive feature, vulnerability to trojan attacks [17]. while keeping the values of other features does not change the prediction. In all three areas, we compare our approach to either other tools specifically testing such properties (if they exist) or to a baseline Definition 1. A predictive model 푀 is individually fair with re- employing a property-based tester [9] for test case generation. The spect to a sensitive feature 푠 ∈ {1, . . . , 푛} if for any two data instances ì evaluation shows that MLCheck can outperform other tools with 푥,푦 ∈ 푋 the following holds: respect to effectiveness in generating property falsifying test cases. (푥푠 ≠ 푦푠 ) ∧ (∀푖, 푖 ≠ 푠, 푥푖 = 푦푖 ) ⇒ 푀(푥) = 푀 (푦) . MLCheck in particular excels at hard tasks where other tools cannot find any test case. This increased performance does moreover not Fairness is (most often) a requirement for applications which come at the prize of a much higher runtime. perform binary classification only, i.e. with 푚 = 1 and |푍1| = 2. Summarizing, this paper makes the following contributions: Knowledge graphs. Our second application area are knowledge graphs, more precisely learning to categorize entities according • we present a language for specifying properties on machine to given concepts as e.g. fixed in an ontology. Ontologies do not learning models, just describe concepts (like Animal, Dog, Cat), but also state their • we propose a method for systematic test case generation, relationships (e.g. ”is-a” relationships, “every dog is an animal”). driven by the property and ML model under consideration, In such a setting, we get a multilabel classification problem–every and concept is a label name–and for every label, we perform binary • we systematically evaluate our approach in three different classification (instance 푥 is or is not a dog). In the following, we application areas employing 56 models under test generated treat the two classes 0 and 1 of every label as boolean values. from 24 data sets. Definition 2. A concept relationship is a boolean expression over the label names 퐿. A predictive model 푀 is respecting concept The tool and all data to replicate the results mentioned in this paper relationship 휑 if for any data instance 푥 the formula can be found at https://github.com/anonymseal/MLCheck. 휑 [퐿푖 := 푀 (푥)푖, 1 ≤ 푖 ≤ 푚] MLCheck– Property-Driven Testing of Machine Learning Models Conference’DD, MM YYYY, XX
Table 1: Characteristics of properties # Model under test mut = ... # Sensitive feature Hyperproperty Binary Multiclass Multilabel s = ... Fairness ✔ ✔ ✘ ✘ # Assumption for i in range (0, f_size - 1): Subsumption ✘ ✔ ✘ ✔ if ( i == s ): Disjointness ✘ ✔ ✘ ✔ Assume('x[i] != y[i ]', i ) ✘ ✘ ✔ ✘ else : Trojan attack Assume('x[i] == y[i ]', i ) # Assertion Assert( ' mut.predict (x) == mut.predict (y )') is true. Figure 1: Property specification for fairness
Here, 휑 [퐿푖 := 푀 (푥)푖, 1 ≤ 푖 ≤ 푚] stands for the formula 휑 in which label names are replaced by the corresponding (boolean) 3.1 Property Specification values obtained from a prediction. Of frequent interest are two spe- In property-based testing [9], software developers specify prop- cific concept relationships: subsumption, the “is-a” relationship, and erties about functions (of their programs), and the testing tool disjointness. For an animal ontology, desired concept relationships generates inputs for checking such properties. Often, properties are might for instance be described by formulae 휑 : dog ⇒ animal 1 specified in an assume/assert style. The assert statement defines the (every dog is an animal) or 휑 : dog ⇒ ¬cat (a dog is not a cat). 2 conditions to be satisfied by a function’s output; the assume state- Security. Our third application area is security. Here, we exem- ment specifies conditions on inputs to the function. The property plarily consider trojan attacks. Trojan attacks are input patterns for is violated if a test input can be found which satisfies the assume which—when present in a data instance—the attacker expects to statement but where the output of the function applied to this input yield a certain prediction. violates the assert statement. Our domain-specific language follows this assume/assert style Definition 3. Let 푇 ⊆ {푖1, . . . , 푖ℓ } be a set of trigger features, t ∈ 푋ì a trigger vector and z ∈ 푍ì a target prediction. A predictive and uses Python as base language. We chose Python because ma- model 푀 is vulnerable to attack (푇, t, z) if for any data instance chine learning applications are often written in Python, using ML libraries like scikit learn1 or PyTorch2. Assume and assert state- 푥 ∈ 푋ì the following holds: ments are calls to functions Assume and Assert. These functions can be used within arbitrary Python code. To allow this Python ∀푡 ∈ 푇 : 푥푡 = t푡 ⇒ 푀(푥) = z . code to refer to characteristics of the current model under test, the Trojan attacks are often run on image classifiers. There are spe- developer can use predefined variables and functions: (1) f_size cific training techniques as well as manipulation strategies forML (the number of features 푛), F (set of all feature names) and its ele- models which make models vulnerable to trojan attacks [25]. Note ments, (2) similarly l_size (the number of labels 푚), L (set of all that trojan attacks are different from adversarial attacks [27]. label names) and its elements plus (3) the function predict (the These three areas and their properties have complementary char- model function 푀). acteristics (see also Table 1), and in the evaluation can as such Calls to the assume and assert functions take the following form: demonstrate the versatility of our specification and testing ap- Assume('
# Model under test Formula Generation. Property and white-box model are trans- mut = ... lated to logical formulae. The construction guarantees that these # Assumption Assume('true ') formulae in conjunction are satisfiable if and only if the property # Assertion does not hold for the white-box model. Our aim is the generation Assert( ' mut.predict (x)[dog] => mut.predict (x)[animal ]') of test inputs violating the property. The translation employs the SMT-LIB format3 to leverage state-of-the-art SMT (Satisfiability Figure 2: Property specification for dog ⇒ animal Modulo Theories) solvers for satisfiability checking. The translation itself is detailed in Section 4. SMT Solving and Augmentation. Next, the SMT solver Z3 [11] # Model under test mut = ... is used to check satisfiability. When the formula is satisfiable, we # Trigger features, trigger vector, target prediction extract the logical model of the formula. This logical model is a coun- T = ... t = ... terexample to the property, i.e., gives us values of data instances z = ... and predicted classes violating the property. Such a counterexample # Assumption serves as a test input and thus becomes part of the test suite (unless for f in T: Assume('x[f] == t[f ]', t , f ) it is no counterexample for the MUT). To generate several test in- # Assertion puts, we furthermore use an augmentation phase and let the SMT Assert( ' mut.predict (x) == z ', z) solver construct further logical models. This is done by adding more constraints to the logical formula ruling out previously returned Figure 3: Property specification for trojan attacks counterexamples. Retraining. As verification takes place on the white-box model, not every thus computed counterexample is also a valid counterex- first set mut and sensitive feature s. Then we specify a condition on ample for the MUT (which is only approximated by the white-box some data instances (inputs) x and y using several calls to Assume in model). Therefore, we compare the prediction of the white-box a for loop. Finally, we make an assertion about the MUT’s prediction model on generated test inputs with that of the black-box model. In on these data instances. case of differences, the test input plus MUT prediction is added to the training set for the white-box model. After having collected sev- 3.2 Test Data Generation eral such invalid counterexamples, the white-box model is retrained For test data generation, we employ a technique called verification- as to improve its approximation quality. based testing, first proposed in35 [ ]. Verification-based testing per- These steps are repeated until a user-definable maximum number forms formal verification of the property to be checked via SMT of samples has been reached. (satisfiability modulo theories) solving. Since we treat the modelun- der test (MUT) as black-box (and since we aim at a testing technique 4 ENCODINGS applicable to any kind of machine learning model), verification first The generation of the logical formula requires an encoding of the of all requires the existence of a verifiable white-box model. To this white-box model and of the specified property. end, we train a white-box model approximating the MUT using predictions of the MUT as training data. On the white-box model, 4.1 White-Box Model Encoding we verify the property, and use counterexamples to the property as Our approach currently involves two sorts of white-box models for test inputs. As the white-box model is only an approximation of the verification-based testing, decision trees and neural networks. We MUT, not all such counterexamples must be valid counterexamples briefly formalize their encodings as a number of logical constraints in the MUT. In case of counterexamples being invalid for the MUT, next. we do not include them in the test suite and instead retrain the Decision trees. The first option is to train a decision tree as white-box model to enhance its approximation quality. white-box model. A decision tree is a (not necessarily complete, nor The overall workflow of test data generation is depicted inFig- full or balanced) tree in which every edge between a node and its ure 4. Inputs are the model under test (MUT) and the property children is labelled with a boolean condition on features values, and specification, the output is a test suite. We briefly discuss all steps every leaf is labelled with a prediction giving class values for all in the sequel. (푖) White-Box Model Training. The white-box model on which labels. Formally, for every level 푖 in the tree, we let 푠푗 be the 푗-th we verify the property is generated from predictions of the black- (푖−1) − (푖) node and 푠푝푟푒 ( 푗) be its predecessor on level 푖 1. We let 푐표푛푑푝푟푒 (푗) box model (the MUT). To this end, we generate training data for the ( − ) ( ) ( ) be the condition on the edge from 푠 푖 1 to 푠 푖 , and 푝푟푒푑 푖 be the white-box model from randomly chosen data instances together 푝푟푒 ( 푗) 푗 푗 with the MUT’s predictions on these instances. (푖) prediction associated to a leaf node 푠푗 . We assume predictions to Currently, our approach employs two types of white-box models Ó take the form ℓ ∈퐿 (ℓ = 푐) where 푐 ∈ 푍ℓ . which the user can choose from: decision trees and neural networks. For the encoding, we introduce one boolean variable per node in During the evaluation, we compare them with respect to efficiency the tree and one variable 푐푙푎푠푠ℓ for every label ℓ ∈ 퐿. The constraints and effectiveness in generating test inputs (see Section 6). For train- are as follows. We get one constraint for the root of the node on ing, we take ML algorithms from the scikit-learn and PyTorch library to create white-box models. 3http://smtlib.cs.uiowa.edu/ MLCheck– Property-Driven Testing of Machine Learning Models Conference’DD, MM YYYY, XX
White-Box Model Formula SMT Solving Test MUT (Re)Training Generation & Augmentation Suite
Prop
Figure 4: Workflow of Test Data Generation level 0: of the previous layer plus the bias term. For the output layer 푘 + 1, ( ) 퐶 ≡ 푠 0 (푘+1) 푟표표푡 1 we just require constraint 퐶푖푛 . In case of a single label classifier, Thus, the boolean variable for the root node is always true. For the predicted class is determined by the output neuron with the ( ) every further inner node 푠 푖 we get one constraint maximal input, and we therefore add a constraint about the class 푗 for the single label ℓ to model this arg-max function5. (푖) (푖−1) (푖) (푖) 퐶 ≡ (푠 ∧ 푐표푛푑 ∧ 푠 ) ( + ) Û ( ) ( ) 푗 푝푟푒 (푗) 푝푟푒 (푗) 푗 푘 1 ( ) ≡ ( 푘 ≥ 푘 ) ∧ 퐶표푢푡 푐 푖푛푐 푖푛푐′ 푐푙푎푠푠ℓ = 푐 ( − ) ( ) ( ) ′ ∨ ((¬푠 푖 1 ∨ ¬푐표푛푑 푖 ) ∧ ¬푠 푖 ) 푐 ≠푐 푝푟푒 ( 푗) 푝푟푒 (푗) 푗 ′ Here 푐, 푐 are the classes of 푍1. In case of a multilabel classifier, Thus, node variables become true when their successor node is (푖) an additional threshold value th is learned and the constraint for true and the condition on the edge holds. For every leaf 푠푗 with label ℓ is Ó ( ) 4 ( + ) ( ) prediction ℓ ∈퐿 ℓ = 푐 we get the constraint : 푘 1 ( ) ≡ Ó푛푘+1 ( 푘 ≥ ∧ ) ∨ 퐶표푢푡 ℓ ℓ=1 푖푛ℓ th 푐푙푎푠푠ℓ = 1 (푖) Û 퐶 ≡ (푐푙푎푠푠ℓ = 푐) (푘) 푗 (푖푛 < th ∧ 푐푙푎푠푠ℓ = 0) ℓ ∈퐿 ℓ Neural networks. The second option is to train a neural net- The thus generated formulae employ real numbers and multipli- work as white-box model. We assume training to supply us with a cation operations. This often impairs the performance of the SMT feed forward neural network with ReLU (Rectified Linear Unit) ac- solver. For better scalability, we employ some form of quantiza- tion: we parametrize training as to obtain weights and biases in tivation functions modelling the function 푀 : 푋ì → 푍ì with 푛 = |푋ì| the interval [-10,10] only and with 3 decimal places6. Thus, we do input nodes, 푚 = |푍ì| output nodes (in case of a multilabel classifier), not altogether abandon real values, but limit them. Roundings or 푚 = |푍 | (in case of a single label), and 푘 hidden layers with 푛 1 푖 binarizations are frequent in formal neural network analysis (e.g., neurons each, 1 ≤ 푖 ≤ 푘. We set 푛 = 푛, and 푛 = 푚. Attached to 0 푘+1 [3, 28]). Note that this does not impact the soundness of our ap- each connection from neuron 푗 in layer 푖 to neuron 푙 in layer 푖 + 1 (푖) (푖) proach since the white-box model is an approximation of the MUT is a weight 푤 푗푙 . Every neuron is equipped with a bias 푏 푗 . only, and hence all generated counterexamples will be checked on The encoding of such neural networks is in spirit similar to the MUT at the end. other encodings, e.g., by Bastani et al. [4]. We use two real-valued (푖) (푖) 4.2 Property Encoding variables 푖푛푙 and 표푢푡푙 for neuron 푙 on layer 푖 and a boolean variable 푐푙푎푠푠ℓ for every label ℓ ∈ 퐿. On the encoding of the white-box model, we verify the specified For every hidden layer 푖, 1 ≤ 푖 ≤ 푘, we generate two constraints, property. Our properties take the form one describing conditions about the inputs to a neuron, the other 푎푠푠푢푚푒 ⇒ 푎푠푠푒푟푡 about the outputs. 푛 i.e., if the assume condition holds on the inputs, the outputs should Û푖 (푖) ≡ ( (푖) = 푛푖−1 (푖−1) (푖−1) + (푖) ) satisfy the assert condition. For verification we basically generate a 퐶푖푛 푖푛푙 Σ푗=1 푤 푗푙 표푢푡푗 푏푙 푙=1 logical formula 푎푠푠푢푚푒 ∧ ¬푎푠푠푒푟푡 and check the satisfiability of its 푛푖 conjunction with the white-box model encoding. If the conjunction ( ) Û ( ) ( ) 푖 ≡ ( 푖 < ∧ 푖 = ) 퐶표푢푡 푖푛푙 0 표푢푡푙 0 is satisfiable, its logical model is a counterexample to the property 푙=1 (for the white-box model). This is the basic scheme; the details are ( ) ( ) ( ) ∨ (푖푛 푖 ≥ 0 ∧ 표푢푡 푖 = 푖푛 푖 ) explained next. 푙 푙 푙 Connecting white-box model and property. First, we need Basically, 퐶표푢푡 encodes the ReLU activation function, and 퐶푖푛 to generate one copy of the white-box model formula for every data fixes the input as the weighted sum over all outputs from nodes 5For simplicity, the translation given here ignores ties. 4We assume that the decision tree makes deterministic predictions, i.e. only one leaf 6These values have been chosen after some experiments with a number of different node is chosen. values. Conference’DD, MM YYYY, XX Arnab Sharma, Caglar Demir, Axel-Cyrille Ngonga Ngomo, and Heike Wehrheim instance 푥 occurring as parameter to predict in the property. We Table 2: Parameters of MLCheck use a simple numbering scheme on variables to distinguish these copies. Second, every copy needs to be connected to the parameter Parameter Type Explanation 푥 of predict. In the decision tree encoding, this means that we model 푀 : 푋ì → 푍ì model under test replace every feature name occurring in a condition on an edge model_type {푡표푟푐ℎ,푠푐푖푘푖푡} ML library used for the model by its appropriately numbered version. In the neural network, we instance_list 푋ì∗ sequence of instance variables add a constraint equating the feature values of parameter 푥 with XML_file XML data format (0) the output of layer 0 (fixing 표푢푡 ), again using the appropriate wbm {푑푡, 푛푛} white-box model to be used version. multi boolean single or multiple CEX Translating property. For the property itself, we execute the max_samples integer size of test suite Python code containing assume and assert statements. Every exe- bound_cex boolean constrain the values of CEX cution of Assume and Assert generates one logical formula, basi- cally the condition passed as parameter (in SMT-LIB format) with program variables replaced by the corresponding arguments to Assume/Assert. The conjunction of all these formulae presents checked (parameter multi). Finally, the parameter max_samples the encoding of the property. Here we again employ appropriately controls the number of samples to be generated during test suite numbered versions of variables of the white-box model, both for construction, and the parameter bound_cex fixes whether further feature names and labels/classes. data-specific constraints on the counterexamples should be applied. As an example consider the translation7 of the property specified in Figure 1. Assuming that the model is a binary classifier on data instances with 4 features called 푎,푏, 푐 and 푑, a label called lab and 6 EVALUATION sensitive feature 푠 is 푏, the formula would be With our tool MLCheck at hand, we evaluated our approach within the already mentioned three application areas. First of all, we em- (푎1 = 푎2) ∧ (¬(푏1 = 푏2)) ∧ (푐1 = 푐2) ∧ (푑1 = 푑2) ∧ ployed our specification language for describing properties onML ¬(class1푙푎푏 = class2푙푎푏 ) models arising in these areas. All properties defined in Section 2 Here, 1 and 2 are the numbers of the two copies generated for (and more) could easily be specified. data instances x and y occurring in the property. The first line is The core part of our evaluation concerns test case generation. For generated by executing the for loop containing assume statements, this, we were interested in the following three research questions: the second line is the translation of the (negated) assert statement. RQ1 How effective is MLCheck in constructing test cases violat- ing properties compared to existing approaches? 5 TOOL IMPLEMENTATION RQ2 How efficient is MLCheck in constructing test cases violat- The entire approach is implemented as a testing tool called MLCheck. ing properties compared to existing approaches? The implementation is written in Python (v3.6.9) and contains ap- RQ3 How do our two white-box models compare to each other? proximately 3,000 lines of code. We use the scikit-learn library We performed the evaluation for answering these questions in (v0.22.1) to build our white-box model decision tree and the Py- all three application areas. Torch (v1.5.1) [30] deep learning platform for neural networks. We employ Z3 [11] for SMT solving. 6.1 Setup Table 2 presents the list of main parameters of our tool. First of all, the MUT needs to be provided to MLCheck. The format for this Evaluation requires to have (1) models under test (obtained by depends upon the type of ML library used to generate the model. training on some data sets), (2) properties to be checked (already If the MUT is generated by using scikit-learn, then the model given) and (3) tools to compare MLCheck to. can be directly provided as an input parameter to model. In case of Datasets. We use different data sets to construct MUTs inthe PyTorch, in addition the architecture (i.e., the class defining the type different application areas. Some statistics about the data setscan of activation functions, number of layers and number of neurons be found in Table 3. • per layer) of such a model has to be provided. The model_type For the fairness experiments, we have taken the Adult and German credit datasets from the UCI machine learning repos- parameter gives the library used for the generation of the MUT. 8 Further parameters are (1) a list of instance variables (the ones itory. We used “gender” as sensitive feature (for checking employed in assume and assert statements), (2) an XML file describ- individual discrimination) as this feature has also been used ing the format of the training data (features, labels, classes), and in previous works of fairness testing [1, 15]. • (3) the type of the white-box model (default: decision tree 푑푡). In For testing concept relationships, we first employed the Pyke embedding approach [12] to map entities from the DBpedia case of using a neural network as white-box model, the default is 9 to train a network with 2 hidden layers with 10 neurons each. The knowledge graph (version 3.6) to real vectors in 50 dimen- user can also specify different values for this in a configuration file. sions. Such embedding approaches compute the features of Unlike most property-based testing tools, our tool can easily be entities. Pyke achieves the best results in the class prediction configured to output multiple test cases violating the property to be 8https://archive.ics.uci.edu/ml 7Not in SMT-LIB format. 9http://dbpedia.org MLCheck– Property-Driven Testing of Machine Learning Models Conference’DD, MM YYYY, XX
0 0 Table 3: Data sets and their characteristics
2 2 Name #Features #Instances #NoClasses Adult 13 32,561 2 4 4 German credit 22 1000 2 CR1 50 450 3 6 6 CR2 50 450 3 CR3 50 450 3 8 8 CR4 50 450 3 CR5 50 450 3 0 2 4 6 8 0 2 4 6 8 CR6 50 450 3 MNIST 100 60,000 10 (a) Trigger 1 (b) Trigger 3
Figure 5: Examples of images with triggers For concept relationship testing, we also train a neural network and a random forest. Finally, for trojan attacks we just train a neural task and yields features which are well suited to the clas- network since this is the main classifier used on images. We use two sification of entities. Our experimental results suggest that different architectures for the neural network: one with 1 hidden finding counterexamples for classifiers trained on these em- layer of 100 neurons (called NN1 in Tables 7 and 8) and one with 2 beddings is a difficult task, thus such classifiers provide good hidden layers with 64 neurons (NN2). Note that there is no need to benchmarks for testing tools. We then generated 6 datasets, employ networks with several hidden layers as long as the network which each contained embeddings from 3 classes (our labels). is able to approximate the MUT well enough. In three of the datasets, 2 of the classes were known to be Baselines. For fairness testing, there are specialized tools for disjoint (e.g., persons and places). The other three datasets testing for individual discrimination. We compared our tools with 12 contained two classes of which one subsumed the other (e.g., the Symbolic Generation (SG) algorithm of Aggarwal et al. [1] persons and actors). and with AEQUITAS [38]. We do not consider THEMIS [15] for our • For testing on trojan attacks, we employ the same datasets comparison as this has already been shown to be less effective in as used by Baluta et al. [3] for quantitatively verifying neural comparison to SG and AEQUITAS as stated by Zhang et al. [44]. We networks wrt. trojan attacks. They use the MNIST10 dataset configured our tool to generate multiple counterexamples since SG containing images of hand-written digits and resize the im- and AEQUITAS also construct several failing test inputs (in order ages to 10×10. To obtain models which are vulnerable to to compute some unfairness score). trojan attacks, we further extended this training set with ad- For concept relationships and trojan attacks, there are no spe- ditional “poisened” data instances11, i.e., instances in which cialized testing tools available. Here, we have used the Python some trigger t is present and the specific target prediction z implementation (Hypothesis) of the property-based testing tool is given. This way, we train a model which is vulnerable to- qickCheck as our baseline approach to compare against. In this wards such an attack. Figure 5 shows two instances (images) case, we configured MLCheck to generate a single counterexample. with triggers set (2 out of 4 triggers). The trigger features are Parameter max_samples was set to 1000 in all cases and bound_cex the first 2 and 7 pixels, respectively, in the upper left corner to false. of the image, set to some randomly chosen color occuring Note that the ground truth about the models under test is un- in these images. The target prediction of these two attacks known in all the experiments, i.e., we do not a priori know whether in this case is class 4, i.e., we steer the MUT into falsely pre- the trained classifiers do or do not satisfy the property. dicting an image containing the triggers to show the digit All experiments were run on a machine with 2 cores Intel(R) 4. The more such “poisened” data instances we add to the Core(TM) i5-7300U CPU with 2.60GHz and 16GB memory using training set, the more likely it is that attacks succeed and Python version 3.6 with GPU as Intel(R) HD Graphics 620. the more difficult it gets to generate test cases violating the attack property. 6.2 Results ML algorithms. Out of the training sets, we generate ML models For RQ1, we compared the effectiveness of the tools in generating using scikit-learn and PyTorch, the latter for all neural net- test inputs violating the property under interest. We report on the works as it provides more sophisticated configuration options for results separately for every application area. Due to the randomness training NNs. For fairness testing, we train a random forest, a logis- in ML algorithms, we ran every experiment 20 times. Whenever tic regression classifier, a naive Bayes classifier and a decision tree. we generated multiple counterexamples (i.e., for fairness), we give Moreover, we employ two fair-aware classifiers [5, 41], i.e., classi- the mean over the 20 runs as well as the standard error of the fiers which are supposed to generate non-discriminating models. mean13. In the cases of a single counterexample (i.e., for concept
10http://yann.lecun.com/exdb/mnist/ 12We got the implementation of SG from the authors of [44]. 11Another option to obtain a “trojaned” model is to employ specific trojaning algo- 13The Standard Error of the Mean (SEM) is obtained by dividing the standard deviation rithms which however requires manipulating the model itself, i.e., requires a white-box with the total number of samples which in our case is number of times we run our model. tool. Conference’DD, MM YYYY, XX Arnab Sharma, Caglar Demir, Axel-Cyrille Ngonga Ngomo, and Heike Wehrheim
Table 4: Mean (± SEM) for Adult dataset Table 6: Probability of detected violations of subsump- tion/disjointness Classifiers MLC_DT MLC_NN SG AEQUITAS Logistic Regress. 102.30 (±16.36) 65.21 (±7.78) 30.20 (±3.27) 90.80 (±31.46) Dataset MLC_DT MLC_NN PBT Decision Tree 214.00 (±20.16) 64.30 (±1.36) 225.48 (±4.23) 112.00 (±25.14) S1/D1/D2 S1/D1/D2 S1/D1/D2 Naive Bayes 38.40 (±5.53) 69.6 (±3.93) 23.83 (±1.68) 0.00 (±0.00) Random Forest 166.14 (±22.12) 50.60 (±2.47) 19.82 (±5.59) 158.00 (±4.35) CR1 (NN) 1.00/0.00/0.80 0.25/0.00/1.00 0.00/0.00/0.00 Fair-Aware1 0.00 5.70 (±1.38) 0.00 0.00 CR1 (RF) 0.00/0.00/0.00 0.00/0.00/0.00 0.00/0.00/0.00 Fair-Aware2 80.91 (±2.67) 1.25 (±0.76) 3.87 (±0.56) 0.89 (±0.50) CR2 (NN) 1.00/1.00/1.00 1.00/1.00/1.00 1.00/1.00/1.00 CR2 (RF) 0.00/0.00/0.00 0.00/0.00/0.00 0.00/0.00/0.00 Table 5: Mean (± SEM) for Credit dataset CR3 (NN) 1.00/1.00/1.00 1.00/1.00/1.00 1.00/1.00/1.00 CR3 (RF) 0.00/0.00/0.00 0.00/0.00/0.00 0.00/0.00/0.00 CR4 (NN) 1.00/0.00/ 1.00 1.00/0.00/1.00 0.00/0.00/0.00 Classifiers MLC_DT MLC_NN SG AEQUITAS CR4 (RF) 0.00/0.00/0.00 0.00/0.00/0.00 0.00/0.00/0.00 Logistic Regress. 144.71 (±13.62) 78.60 (±7.97) 63.43 (±2.27) 63.00 (±18.65) CR5 (NN) 1.00/1.00/1.00 1.00/1.00/1.00 1.00/1.00/1.00 Decision Tree 396.17 (±28.16) 17.75 (±1.36) 239.25 (±4.71) 18.72 (±8.98) Naive Bayes 3.00 (±1.03) 39.40 (±8.76) 3.00 (±0.00) 0.00 CR5 (RF) 0.00/0.00/0.00 0.00/0.00/0.00 0.00/0.00/0.00 Random Forest 154.57 (±22.12) 69.43 (±5.91) 251.42 (±9.74) 10.20 (±9.12) CR6 (NN) 0.95/1.00/1.00 1.00/1.00/1.00 1.00/1.00/1.00 Fair-Aware1 0.00 19.89 (±1.38) 0.00 0.00 CR6 (RF) 0.00/0.00/0.00 0.00/0.00/0.20 0.00/0.00/0.00 Fair-Aware2 120.87 (±7.98) 0.00 2.54 (±0.56) 1.78 (±0.50)
Table 7: Probability of detected violations of trojan attacks relationships and trojan attacks) we give the probability of finding (data set with 1,000 poisened instances) a counterexample as calculated from the 20 runs. Tables 4 and 5 show the measures for the number of detected Trigger MLC_DT MLC_NN PBT ART unfair test cases (i.e., test input pairs) for Adult and Credit dataset, NN1/NN2 NN1/NN2 NN1/NN2 NN1/NN2 respectively. T1-4 0.00/0.00 1.00/1.00 err/err 1.00/1.00 The classifiers used for training the MUT are given in thefirst T1-5 0.10/0.00 1.00/1.00 err/err 1.00/1.00 column (Fair-Aware1 and Fair-Aware2 are the algorithms of [41] T2-4 0.05/0.00 1.00/1.00 err/err 0.00/0.10 and [5]). The next columns give the numbers for MLCheck (MLC_DT T2-5 0.20/0.00 1.00/1.00 err/err 0.00/0.25 T3-4 0.00/0.00 1.00/1.00 err/err 1.00/1.00 with decision tree and MLC_NN with neural network as white-box) T3-5 0.20/0.00 1.00/1.00 err/err 1.00/0.00 as well as SG and AEQUITAS. The largest number is shown in bold. T4-4 0.00/0.00 1.00/1.00 err/err 1.00/0.80 An entry 0.00 stands for no counterexamples found, the entry - T4-5 0.00/0.00 1.00/1.00 err/err 1.00/0.50 (for Fair-Aware1) describes the fact that SG and AEQUITAS could not work on the MUT generated by this algorithm because of the Table 8: Probability of detected violations of trojan attacks format of the model returned by it. We see that MLCheck always (data set with 10,000 poisened instances) generates the largest number of counterexamples except for a single one (Random Forest with Adult dataset). We discuss differences between the DT and NN version of MLCheck below. Trigger MLC_DT MLC_NN PBT ART Next, Table 6 shows the result of testing for concept relationships. NN1/NN2 NN1/NN2 NN1/NN2 NN1/NN2 We generated test cases for three properties (called S1, D1 and T1-4 0.00/0.00 1.00/1.00 err/err 0.00/0.00 D2), one subsumption and two disjointness relationships. The rows T1-5 0.00/0.00 1.00/1.00 err/err 0.00/0.00 T2-4 0.00/0.00 1.00/1.00 err/err 0.00/0.00 shows the results in probabilities per dataset and model type (neural T2-5 0.00/0.00 1.00/1.00 err/err 0.00/0.00 network NN or random forest RF). We see that MLCheck (in either T3-4 0.00/0.00 1.00/1.00 err/err 0.00/0.00 DT or NN version) is able to find more or an equal number of T3-5 0.00/0.00 1.00/1.00 err/err 0.00/0.00 falsifying test cases compared to property-based testing. T4-4 0.00/0.00 1.00/1.00 err/err 0.00/0.00 Tables 7 and 8 shows the results of our experiments for trojan T4-5 0.00/0.00 1.00/1.00 err/err 0.00/0.00 attacks. The tables again depict the probabilities with which the testing tool was or was not able to find a test input falsifying the property under interest. We considered two architectures for neu- the reason for this failure is the high number of features (100) in ral network models (NN1 and NN2), trained on the MNIST data this data set, i.e., the fact that Hypothesis has to generate inputs set enhanced by 1,000 and 10,000 additional “poisened” instances, for a function with 100 arguments which is likely not the setting respectively for the two tables, using 4 different trigger features envisaged by the developers of this tool. T1 to T4 and 2 different target predictions (classes 4 and 5). The In order to be able to compare our technique to other methods, triggers are hence named T1-4, T1-5 and so on. we hence decided to develop a prototype tool for adaptive random It turned out that the property-based testing tool which we testing [7] with respect to trojan attacks. Note that there is no adap- employed is not able to generate test cases at all. On all instances, tive random tester allowing to specify arbitrary properties to be it stopped with the error message “hypothesis.errors.Unsatisfiable: tested. Thus, a new implementation is required for every property, Unable to satisfy assumptions of hypothesis”, typically after trying in particular the definition and implementation of a distance metric to generate test inputs for around 40 minutes. We suspect that (for which we here used the euclidian distance metric on feature MLCheck– Property-Driven Testing of Machine Learning Models Conference’DD, MM YYYY, XX
1,000 MLCheck_dt MLCheck_dt MLCheck_nn MLCheck_nn ART SG 1,000 AQ
500 500 runtime (in seconds) runtime (in seconds) 100 100 1 1 1 2 4 6 8 10 12 14 16 1 2 3 4 5 6 7 8 9 10 11 12 Solved tasks (ordered by runtime of approach) Solved tasks (ordered by runtime of approach)
Figure 8: Runtime for checking trojan attacks Figure 6: Runtime for checking fairness
1,000 MLCheck_dt the entire tree but only some paths. This explains the lower runtime MLCheck_nn for some tasks, but also the smaller numbers of generated failing test PBT inputs. AEQUITAS has significantly larger runtimes than MLCheck on the more complex tasks. 500 Figure 7 shows the runtime for testing concept relationships on 36 tasks (2 classifiers trained on 6 data sets and 3 properties tobe checked). Here, the decision tree version of MLCheck performs best and property-based testing is comparable to the neural network runtime (in seconds) version for all but the 8 most complicated tasks. 100 For trojan attacks, Figure 8 shows the runtimes of our two 1 1 6 12 18 24 30 36 MLCheck instances as well as our own adaptive random tester. Solved tasks (ordered by runtime of approach) It shows that the runtime of ART is relatively low compared to that of MLCheck with a neural network as white-box model. The reason for this is that our approach includes the training of the Figure 7: Runtime for checking concept relationships white-box model which for neural networks requires some time. vectors). Tables 7 and 8 therefore also give the result for our pro- Summarizing, except for trojan attacks we see that the increased totype adaptive random tester (ART). Interestingly, ART is able to effectiveness of MLCheck does not come at the prize of a much find counterexamples for a number of models trained on thedata higher runtime. set enhanced with 1,000 poisened instances, even more often than MLCheck with a decision tree. However, in the harder cases with For RQ3, we take another look at the tables of detected violations models trained on the data set enhanced with 10,000 instances, ART and figures of runtimes, now comparing the decision tree and neural also produces no test inputs at all. network version of MLCheck. With respect to the number of detected violations, we see that In summary, MLCheck outperforms other tools in almost all the neural network as white-box model is–with a few exceptions– cases, even when they are specialised to the property to be only able to outperform the decision tree in case of the MUT being tested. a neural network itself. The better performance in these cases does often not come at the prize of a (much) increased runtime. For RQ2, we compared the efficiency (in terms of runtime) ofthe In some such cases the neural network white-box model can even tools in generating test inputs violating the property under interest. spectacularly outperform the decision tree, which can be seen in the We again report on the results separately for every application area trojan attack setting when trained with 10,000 additional poisened and again the given values are averaged over 20 runs. instances. For the trojan attacks with a high number of features, Figure 6 depicts the runtime of four tools on 12 fairness testing the neural network white-box is much better in approximating tasks (6 classifiers on the two datasets). SG and AEQUITAS curves the MUT. This confirmed our initial expectation that it does in end at 10 tasks as they do not run on models generated by one of fact make sense to employ two different white-box models with the fair-aware algorithms. The x-axis depicts the number of tasks different generalization abilities in test case generation. solved, ordered by runtime per tool from fastest to slowest, the y-axis is the runtime in seconds. We see that there are a number of Note that all three application areas contain hard benchmarks char- tasks for which SG has a smaller runtime, though not significantly. acterised by only few counterexamples, either generated by using SG uses SMT solving on decision trees as well, but does not consider specific ML algorithms (fair-aware algorithms) or by assembling Conference’DD, MM YYYY, XX Arnab Sharma, Caglar Demir, Axel-Cyrille Ngonga Ngomo, and Heike Wehrheim specific data sets, obtained by embeddings (for concept relation- 8 RELATED WORK ships) or by flooding the training set with property-satisfying in- We briefly discuss other approaches to the validation of machine stances (trojan attacks). We can summarize the results of the exper- learning models. iments on these benchmarks as follows. The most frequently studied type of models are deep neural net- MLCheck in particular outperforms other tools on hard testing works (DNNs). In the area of testing neural networks one focus has tasks. been on the development of appropriate coverage criteria for DNNs, ranging from neuron coverage [31] over some form of MC/DC cov- erage [37] (in concolic testing) to multi-granularity testing criteria 7 DISCUSSION on test beds [26]. The most frequently studied property of DNNs is adversarial We briefly discuss some further aspects of our approach. robustness. It describes the vulnerability of a DNN to adversarial attacks, i.e., attacks in which a small deviation from a correctly clas- 7.1 Soundness sified input yields a different class prediction. Attack methods aim Our approach is sound in the sense of only generating test inputs at generating such adversarial examples, and can be classified into which are counter examples to the property on the black-box model. white-box (e.g., [18, 27]) and black box (e.g., [29, 42]) approaches. We might generate candidate counter examples which are only Recent works have also proposed methods for computing proba- valid counter examples on the approximating white-box model, but bilistic guarantees on robustness [6]. such counter examples do not get into the test suite. All candidate Testing methods typically aim at generating counterexamples to counter examples are checked on the model under test. properties. Formal verification on the other hand aims at correct- Furthermore, our approach allows to add constraints on feature ness guarantees. For neural networks a number of verification tech- values to the test input generation process. Such constraints can for niques have been developed, based on abstract interpretation [14, instance reflect the “data semantics” (e.g., a feature ‘age’ not allow- 16, 36], by layer-wise computations of safety constraints [21] or by ing for values above 100), if such value restrictions are explicitly a combination of SAT solving and linear programming [13]. given. Currently, we derive constraints on minimal and maximal More recently, Pham et al. [32] have proposed a unifying frame- feature values from the training set and take these into account work to verify different types of neural network models (i.e. models when the parameter bound_cex is set to True. with different types of layers and activation functions) using two With the direct incorporation of such constraints in the SMT verification methods, namely optimization-based falsification and solving process, we can avoid costly transformations of counter statistical model checking. Alike us, they also provide a specification examples which other approaches need to perform [44]. language to specify several properties (such as fairness, robustness etc.) to be checked on the model. 7.2 Scalability Baluta et al. [3] propose quantitative verification for neural net- Though we have image recognition as one of our case studies, we do works, i.e., verification which gives a quantitative account onthe not intend to claim that our approach will in general be applicable number of inputs violating some property. Similar to the encoding to image classifiers. Classifiers on images with several thousands of our NN white-box models, they translate neural networks to of bits and hence several thousands of features will pose difficulties logical formulae on which approximate model counting can then for the SMT solver. Note however that the number of hidden layers provide estimates about the number of satisfying logical models. of the model under test is not a limiting factor as we employ a much To make their approach scale, they apply it to binarized NNs only, simpler neural network for approximation, and only this needs and perform further quantization. Other verification approaches translation to logical formulae. using logical encodings of DNNs together with SAT, SMT or MIP Also note that our approach can so far not be applied to test for solvers for property checking have been proposed by Narodytska statistical properties of models and to models which operate on et al. [28] (studying various properties, in particular also adver- data streams (like audio processing). sarial robustness), Pulina et al. [34] or Cheng et al. [8]. Katz et al. [22] in addition build a specific SMT solver for solving linear 7.3 Threats to Validity real arithmetic constraints arising from DNNs with ReLU activation functions. A study comparing different verification approaches of The results discussed here depend on the chosen data sets, classi- NNs and evaluating the impact of pruning techniques has been fiers and properties. We have taken data sets and non-stochastic performed by Guidotti et al. [19]; further surveys of validation ap- properties out of three areas as to demonstrate the tool’s generality. proaches for neural networks have been done by Liu et al. [24] and The classifers are taken from standard ML libraries which will also Huang et al. [20]. be used by software developers for learning. We can however not These approaches are specific to neural networks. A survey on exclude that we would get different findings on other areas using testing techniques for ML models in general, i.e., not restricted to different training algorithms. neural networks, has recently been assembled by Zhang et al. [43]. The concrete numbers in the tables are affected by the random- Model agnostic but property specific approaches most often target ness inside the ML algorithms, both for training MUTs and training fairness testing, more precisely testing for individual discrimina- our white-box models. To mitigate this thread, we ran every experi- tion. Themis [2, 15] is an automated test case generation technique ment 20 times and have given the mean together with the standard error of the mean. MLCheck– Property-Driven Testing of Machine Learning Models Conference’DD, MM YYYY, XX that uses random testing complemented by three optimization pro- Notes in Computer Science, Vol. 12032), Xin Wang, Francesca Alessandra Lisi, cedures. It allows for checking two types of fairness definitions, Guohui Xiao, and Elena Botoeva (Eds.). Springer, 192–209. https://doi.org/10. 1007/978-3-030-41407-8_13 namely causal (i.e. individual) discrimination and group discrimi- [13] Rüdiger Ehlers. 2017. Formal Verification of Piece-Wise Linear Feed-Forward nation for a given black-box model. Later, Aggarwal et al. propose Neural Networks. In ATVA (Lecture Notes in Computer Science, Vol. 10482), Deepak a symbolic approach (SG) to generate test cases for checking indi- D’Souza and K. Narayan Kumar (Eds.). Springer, 269–286. https://doi.org/10. 1007/978-3-319-68167-2_19 vidual discrimination. They use a tool called LIME to generate a [14] Yizhak Yisrael Elboher, Justin Gottschlich, and Guy Katz. 2020. An Abstraction- path of a decision tree from the MUT, on which they use dynamic Based Framework for Neural Network Verification. In CAV (Lecture Notes in Computer Science, Vol. 12224), Shuvendu K. Lahiri and Chao Wang (Eds.). Springer, symbolic execution to generate test cases. They show that their 43–65. https://doi.org/10.1007/978-3-030-53288-8_3 approach outperforms Themis and AEQUITAS [39]. We, on the [15] Sainyam Galhotra, Yuriy Brun, and Alexandra Meliou. 2017. Fairness testing: other hand, approximate the entire MUT by a white-box model testing software for discrimination. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 498–510. (either a decision tree or a neural network) and then compute the [16] Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat test cases on this model. Moreover, our approach can be used for Chaudhuri, and Martin T. Vechev. 2018. AI2: Safety and Robustness Certification checking not only individual discrimination but also other types of of Neural Networks with Abstract Interpretation. In IEEE Symposium on Security and Privacy, SP. 3–18. https://doi.org/10.1109/SP.2018.00058 fairness (e.g. fairness through awareness [40]) as well as completely [17] Arturo Geigel. 2013. Neural network Trojan. J. Comput. Secur. 21, 2 (2013), different properties. 191–232. https://doi.org/10.3233/JCS-2012-0460 [18] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In ICLR, Yoshua Bengio and Yann LeCun 9 CONCLUSION (Eds.). http://arxiv.org/abs/1412.6572 [19] Dario Guidotti, Francesco Leofante, Luca Pulina, and Armando Tacchella. 2020. In this paper, we have proposed an approach and tool for property- Verification of Neural Networks: Enhancing Scalability through Pruning. CoRR driven testing of machine learning models. The approach encom- abs/2003.07636 (2020). arXiv:2003.07636 https://arxiv.org/abs/2003.07636 passes a language for property specification and a method for tar- [20] Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi. 2020. A survey of safety and trust- geted generation of test cases falsifying the property. As future worthiness of deep neural networks: Verification, testing, adversarial attack and work, we intend to study the applicability of the more advanced defence, and interpretability. Computer Science Review 37 (2020). [21] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. 2017. Safety verification techniques on deep neural networks (e.g. [32]) for gen- Verification of Deep Neural Networks. In CAV. 3–29. https://doi.org/10.1007/ erating counterexamples for our white-box model neural network. 978-3-319-63387-9_1 [22] Guy Katz, Clark W. Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. 2017. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks. In REFERENCES CAV (Lecture Notes in Computer Science, Vol. 10426), Rupak Majumdar and Viktor [1] Aniya Aggarwal, Pranay Lohia, Seema Nagar, Kuntal Dey, and Diptikalyan Saha. Kuncak (Eds.). Springer, 97–117. https://doi.org/10.1007/978-3-319-63387-9_5 2019. Black box fairness testing of machine learning models. In Proceedings [23] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2017. Adversarial Machine of the ACM Joint Meeting on European Software Engineering Conference and Learning at Scale. In ICLR. OpenReview.net. https://openreview.net/forum?id= Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE. 625– BJm4T4Kgx 635. https://doi.org/10.1145/3338906.3338937 [24] Changliu Liu, Tomer Arnon, Christopher Lazarus, Clark W. Barrett, and Mykel J. [2] Rico Angell, Brittany Johnson, Yuriy Brun, and Alexandra Meliou. 2018. Themis: Kochenderfer. 2019. Algorithms for Verifying Deep Neural Networks. CoRR automatically testing software for discrimination. In ESEC/SIGSOFT FSE, Gary T. abs/1903.06758 (2019). arXiv:1903.06758 http://arxiv.org/abs/1903.06758 Leavens, Alessandro Garcia, and Corina S. Pasareanu (Eds.). ACM, 871–875. [25] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, https://doi.org/10.1145/3236024.3264590 and Xiangyu Zhang. 2018. Trojaning Attack on Neural Networks. In 25th Annual [3] Teodora Baluta, Shiqi Shen, Shweta Shinde, Kuldeep S. Meel, and Prateek Saxena. Network and Distributed System Security Symposium, NDSS 2018, San Diego, Cali- 2019. Quantitative Verification of Neural Networks and Its Security Applications. fornia, USA, February 18-21, 2018. The Internet Society. http://wp.internetsociety. In CCS, Lorenzo Cavallaro, Johannes Kinder, XiaoFeng Wang, and Jonathan Katz org/ndss/wp-content/uploads/sites/25/2018/02/ndss2018_03A-5_Liu_paper.pdf (Eds.). ACM, 1249–1264. https://doi.org/10.1145/3319535.3354245 [26] Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang [4] Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepGauge: Aditya V. Nori, and Antonio Criminisi. 2016. Measuring Neural Net Robustness multi-granularity testing criteria for deep learning systems. In ASE, Marianne with Constraints. In NIPS, Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Huchard, Christian Kästner, and Gordon Fraser (Eds.). ACM, 120–131. https: Isabelle Guyon, and Roman Garnett (Eds.). 2613–2621. http://papers.nips.cc/ //doi.org/10.1145/3238147.3238202 paper/6339-measuring-neural-net-robustness-with-constraints [27] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. [5] Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. 2009. Building Classifiers DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks. In CVPR. with Independency Constraints. In ICDM Workshops. 13–18. https://doi.org/10. IEEE Computer Society, 2574–2582. https://doi.org/10.1109/CVPR.2016.282 1109/ICDMW.2009.83 [28] Nina Narodytska, Shiva Prasad Kasiviswanathan, Leonid Ryzhyk, Mooly Sagiv, [6] Luca Cardelli, Marta Kwiatkowska, Luca Laurenti, and Andrea Patane. 2019. and Toby Walsh. 2018. Verifying Properties of Binarized Deep Neural Networks. Robustness Guarantees for Bayesian Inference with Gaussian Processes. In AAAI. In (AAAI-18), Sheila A. McIlraith and Kilian Q. Weinberger (Eds.). AAAI Press, AAAI Press, 7759–7768. https://doi.org/10.1609/aaai.v33i01.33017759 6615–6624. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/ [7] Tsong Yueh Chen, Hing Leung, and I. K. Mak. 2004. Adaptive Random Testing. 16898 In ASIAN. 320–329. https://doi.org/10.1007/978-3-540-30502-6_23 [29] Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay [8] Chih-Hong Cheng, Georg Nührenberg, and Harald Ruess. 2017. Maximum Celik, and Ananthram Swami. 2017. Practical Black-Box Attacks against Machine Resilience of Artificial Neural Networks. In ATVA (Lecture Notes in Computer Learning. In AsiaCCS, Ramesh Karri, Ozgur Sinanoglu, Ahmad-Reza Sadeghi, Science, Vol. 10482), Deepak D’Souza and K. Narayan Kumar (Eds.). Springer, and Xun Yi (Eds.). ACM, 506–519. https://doi.org/10.1145/3052973.3053009 251–268. https://doi.org/10.1007/978-3-319-68167-2_18 [30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, [9] Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, random testing of Haskell programs. In (ICFP ’00). 268–279. https://doi.org/10. Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, 1145/351240.351266 Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and [10] Michael R. Clarkson and Fred B. Schneider. 2010. Hyperproperties. J. Comput. Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Secur. 18, 6 (2010), 1157–1210. https://doi.org/10.3233/JCS-2009-0393 Learning Library. In Advances in Neural Information Processing Systems 32, [11] Leonardo Mendonça de Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett Solver. In Tools and Algorithms for the Construction and Analysis of Systems, (Eds.). Curran Associates, Inc., 8024–8035. http://papers.neurips.cc/paper/ 14th International Conference, TACAS 2008. 337–340. https://doi.org/10.1007/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. 978-3-540-78800-3_24 pdf [12] Caglar Demir and Axel-Cyrille Ngonga Ngomo. 2019. A Physical Embedding [31] Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. DeepXplore: Au- Model for Knowledge Graphs. In Semantic Technology - 9th Joint International tomated Whitebox Testing of Deep Learning Systems. In Proceedings of the Conference, JIST 2019, Hangzhou, China, November 25-27, 2019, Proceedings (Lecture Conference’DD, MM YYYY, XX Arnab Sharma, Caglar Demir, Axel-Cyrille Ngonga Ngomo, and Heike Wehrheim
26th Symposium on Operating Systems Principles,. 1–18. https://doi.org/10.1145/ 3132747.3132785 [32] Long H. Pham, Jiaying Li, and Jun Sun. 2020. SOCRATES: Towards a Uni- fied Platform for Neural Network Verification. CoRR abs/2007.11206 (2020). arXiv:2007.11206 https://arxiv.org/abs/2007.11206 [33] R. Potharst and A. J. Feelders. 2002. Classification Trees for Problems with Monotonicity Constraints. SIGKDD Explor. Newsl. 4, 1 (June 2002), 1–10. https: //doi.org/10.1145/568574.568577 [34] Luca Pulina and Armando Tacchella. 2010. An Abstraction-Refinement Approach to Verification of Artificial Neural Networks. In CAV (Lecture Notes in Computer Science, Vol. 6174), Tayssir Touili, Byron Cook, and Paul B. Jackson (Eds.). Springer, 243–257. https://doi.org/10.1007/978-3-642-14295-6_24 [35] Arnab Sharma and Heike Wehrheim. 2020. Higher income, larger loan? Mono- tonicity testing of machine learning models. In ISSTA, Sarfraz Khurshid and Co- rina S. Pasareanu (Eds.). ACM, 200–210. https://doi.org/10.1145/3395363.3397352 [36] Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin T. Vechev. 2019. An abstract domain for certifying neural networks. Proc. ACM Program. Lang. 3, POPL (2019), 41:1–41:30. https://doi.org/10.1145/3290354 [37] Youcheng Sun, Min Wu, Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska, and Daniel Kroening. 2018. Concolic testing for deep neural networks. In ASE. 109–119. https://doi.org/10.1145/3238147.3238172 [38] Sakshi Udeshi, Pryanshu Arora, and Sudipta Chattopadhyay. 2018. Automated directed fairness testing. In Proceedings of the 33rd ACM/IEEE International Confer- ence on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018, Marianne Huchard, Christian Kästner, and Gordon Fraser (Eds.). ACM, 98–108. https://doi.org/10.1145/3238147.3238165 [39] Sakshi Udeshi, Pryanshu Arora, and Sudipta Chattopadhyay. 2018. Automated directed fairness testing. In Proceedings of the 33rd ACM/IEEE International Confer- ence on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018, Marianne Huchard, Christian Kästner, and Gordon Fraser (Eds.). ACM, 98–108. https://doi.org/10.1145/3238147.3238165 [40] Sahil Verma and Julia Rubin. 2018. Fairness definitions explained. In International Workshop on Software Fairness, FairWare@ICSE. 1–7. http://doi.acm.org/10.1145/ 3194770.3194776 [41] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi. 2017. Fairness Constraints: Mechanisms for Fair Classification. In AISTATS. 962–970. http://proceedings.mlr.press/v54/zafar17a.html [42] Fuyuang Zhang, Sankalan Pal Chowdhury, and Maria Christakis. 2020. DeepSearch: A Simple and Effective Blackbox Attack for Deep Neural Networks. In ESEC/FSE. forthcoming. [43] Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2019. Machine Learn- ing Testing: Survey, Landscapes and Horizons. CoRR abs/1906.10742 (2019). arXiv:1906.10742 http://arxiv.org/abs/1906.10742 [44] Peixin Zhang, Jingyi Wang, Jun Sun, Guoliang Dong, Xinyu Wang, Xingen Wang, Jin Song Dong, and Ting Dai. 2020. White-box fairness testing through adversarial sampling. In ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020, Gregg Rothermel and Doo-Hwan Bae (Eds.). ACM, 949–960. https://doi.org/10.1145/3377811.3380331