Empirical Investigation of Decision Tree Extraction From

EMPIRICAL INVESTIGATION OF DECISION TREE EXTRACTION FROM

NEURAL NETWORKS

A thesis presented to

the faculty of

the Fritz J. and Dolores H. Russ

College of Engineering and Technology of Ohio University

In partial fulfillment

of the requirements for the degree

Master of Science

Maimuna H. Rangwala

June 2006

This thesis entitled

EMPIRICAL INVESTIGATION OF DECISION TREE EXTRACTION FROM

NEURAL NETWORKS

MAIMUNA H. RANGWALA

has been approved for

the Department of Industrial and Manufacturing Systems Engineering

and the Russ College of Engineering and Technology by

Gary R. Weckman

Associate Professor of Industrial & Manufacturing Systems Engineering

R. Dennis Irwin

Dean, Fritz J. and Dolores H. Russ College of Engineering and Technology ABSTRACT

RANGWALA, MAIMUNA H., M.S., June 2006. Industrial and Manufacturing Systems

Engineering

EMPIRICAL INVESTIGATION OF DECISION TREE EXTRACTION FROM

NEURAL NETWORKS (201 pp.)

Director of Thesis: Gary R. Weckman

The purpose of this thesis is to develop heuristics for employing Trepan, an

algorithm for extracting decision trees from neural networks. Typically, several

parameters need to be chosen to obtain a satisfactory performance of the algorithm. The

current understanding of the various interactions between these is not well understood.

By empirically evaluating the performance of the algorithm on a test set of databases

chosen from benchmark machine learning and real world problems, several heuristics are

proposed to explain and improve the performance of the algorithm. The experimentation

is further validated by performance statistic measures. The algorithm is extended to work

for multi-class regression problems and its ability to comprehend generalized

feedforward networks is investigated. This work thus serves to provide improvements, an

increased understanding of the behavior of the algorithm and heuristics to choose

parameters for a better performance.

Approved:

Gary R. Weckman

Associate Professor of Industrial and Manufacturing Systems Engineering

Dedicated to my Father

Prof. H. T. RANGWALA

(1942-2006)

and

my Sister

Fatema Rangwala

(1988-2006)

TABLE OF CONTENTS

ABSTRACT...... 3

LIST OF TABLES...... 8

LIST OF FIGURES ...... 11

CHAPTER 1. INTRODUCTION ...... 13

1.1 MACHINE LEARNING...... 13

1.2 CLASSIFICATION ALGORITHMS ...... 14

1.3 RESEARCH OBJECTIVES ...... 14

1.4 THESIS OVERVIEW ...... 16

CHAPTER 2. BACKGROUND AND LITERATURE REVIEW ...... 17

2.1 ARTIFICIAL NEURAL NETWORKS ...... 17

2.1.1 Neural Network Architecture...... 17

2.1.2 Neural Network Training...... 24

2.1.3 Neural Networks for Classification and Regression...... 25

2.1.4 Rule Extraction from Neural Networks ...... 26

2.2 DECISION TREES ...... 27

2.2.1 Decision Tree Classification...... 27

2.2.2 Decision Tree Applications...... 29

2.3 C4.5 ALGORITHM ...... 30 6

2.3.1 Information Gain, Entropy Measure and Gain Ratio...... 31

2.4 TREPAN ALGORITHM...... 37

2.4.1 M-of-N Splitting tests ...... 39

2.4.2 Single Test TREPAN and Disjunctive TREPAN ...... 40

CHAPTER 3. METHODOLOGY ...... 41

3.1 PHASE 1 ...... 43

3.2 PHASE 2 ...... 44

3.2.1 Datasets...... 45

3.2.2 Neural Network Modeling ...... 56

3.3 PHASE 3 ...... 57

3.4 PHASE 4 ...... 60

3.5 PERFORMANCE MEASURES ...... 62

3.5.1 Classification Accuracy...... 62

3.5.2 Comprehensibility...... 64

CHAPTER 4. RESULTS AND DISCUSSION...... 65

4.1 INVESTIGATE AND EXTEND TREPAN...... 65

4.2 DATASET ANALYSIS ...... 65

4.2.1 Corrosion...... 65

4.2.2 Outages ...... 79

4.2.3 Iris ...... 89

4.2.4 Body Fat...... 94 7

4.2.5 Saginaw Bay...... 101

4.2.6 Admissions...... 110

CHAPTER 5. CONCLUSIONS AND FUTURE RESEARCH ...... 118

5.1 SUMMARY AND DISCUSSION ...... 118

5.1.1 Accuracy ...... 119

5.1.2 Comprehensibility...... 120

5.2 HEURISTICS...... 120

5.3 CONCLUSIONS...... 122

5.4 FUTURE RESEARCH...... 123

REFERENCES ...... 125

APPENDIX A: WEIGHTS AND NETWORK FILE FORMATS (GFF)...... 131

APPENDIX B: CORROSION RESULTS...... 134

APPENDIX C: OUTAGES RESULTS...... 143

APPENDIX D: IRIS RESULTS...... 153

APPENDIX E: BODY FAT RESULTS...... 157

APPENDIX F: SAGINAW BAY RESULTS...... 161

APPENDIX G: ADMISSIONS RESULTS...... 178 8

LIST OF TABLES

Table 2.1: Activation Functions used in Neural Networks...... 21

Table 2.2: Play Tennis Example Dataset ...... 34

Table 3.1: Dataset Summary...... 44

Table 3.2: Iris--Sample Dataset ...... 45

Table 3.3: Body Fat -- Sample Dataset...... 47

Table 3.4: Body Fat Class Labels ...... 47

Table 3.5: Saginaw Bay -- Sample Dataset ...... 50

Table 3.6: Chlorophyll Level Class Labels...... 50

Table 3.7: Corrosion -- Sample Dataset...... 52

Table 3.8: Corrosion Class labels ...... 52

Table 3.9: Admissions -- Sample Dataset...... 54

Table 3.10: Outages -- Sample Dataset...... 55

Table 3.11: Reportable Outages Class Labels ...... 55

Table 3.12: Network Configurations ...... 56

Table 3.13: TREPAN Parameters...... 57

Table 3.14: Confusion Matrix -- Example...... 63

Table 4.1: Single Test TREPAN Experimental Setup...... 67

Table 4.2: TREPAN and Disjunctive TREPAN Experimental Setup ...... 68

Table 4.3: Corrosion: Single Test TREPAN -- Test Accuracies at each Node ...... 70

Table 4.4: Corrosion: TREPAN (beam width 5) -- Test Accuracies at each Node ...... 72 9

Table 4.5: Corrosion -- Summary of Experimental Runs...... 74

Table 4.6: Corrosion Confusion Matrix (TREPAN)...... 75

Table 4.7: Corrosion Confusion Matrix (C4.5) ...... 77

Table 4.8: Outages: Single Test TREPAN -- Test Accuracies at each Node ...... 81

Table 4.9: Outages-Summary of Experimental Runs ...... 84

Table 4.10: Outages Confusion Matrix (TREPAN) ...... 85

Table 4.11: Outages Confusion Matrix (C4.5) ...... 87

Table 4.12: Iris Confusion Matrix (Neural Network)...... 89

Table 4.13: Iris: Single Test TREPAN -- Test Accuracies at each Node...... 90

Table 4.14: Iris -- Summary of Experimental Runs...... 92

Table 4.15: Iris Confusion Matrix (TREPAN) ...... 93

Table 4.16: Iris Confusion Matrix (C4.5) ...... 93

Table 4.17: Body Fat: Single Test TREPAN -- Test Accuracies at each Node...... 96

Table 4.18: Body Fat -- Summary of Experimental Runs ...... 98

Table 4.19: Body Fat Confusion Matrix (TREPAN)...... 99

Table 4.20: Body Fat Confusion Matrix (C4.5)...... 99

Table 4.21: Saginaw Bay: Single Test TREPAN -- Test Accuracies at each Node ...... 103

Table 4.22: Saginaw Bay -- Summary of Experimental Runs...... 105

Table 4.23: Saginaw Bay Confusion Matrix (TREPAN) ...... 106

Table 4.24: Saginaw Bay Confusion Matrix (C4.5) ...... 108

Table 4.25: Admissions: Confusion Matrix (Neural Network) ...... 110 10

Table 4.26: Admissions: Single Test TREPAN-Test Accuracies at each Node...... 111

Table 4.27: Admissions-Summary of Experimental Runs ...... 114

Table 4.28: Admissions: Confusion Matrix (TREPAN)...... 115

Table 4.29: Admissions: Confusion Matrix (C4.5) ...... 117

Table 5.1: Summary of Results...... 119

LIST OF FIGURES

Figure 1.1: Extracting Decision Trees from Neural Networks...... 15

Figure 2.1: Model of a Single Neuron ...... 18

Figure 2.2: Neural Network Architecture ...... 20

Figure 2.3: Multi Layer Perceptron Network ...... 22

Figure 2.4: Generalized Feed Forward Network ...... 23

Figure 2.5: Use of Cross-Validation during Training...... 25

Figure 2.6: Decision Tree Architecture ...... 28

Figure 2.7: Decision Tree for the Play Tennis Problem...... 37

Figure 2.8: TREPAN Algorithm...... 39

Figure 3.1: Methodology (Phases 1-3) ...... 42

Figure 3.2: Methodology (Phase 4) ...... 43

Figure 3.3: Scope of Ecological Informatics ...... 49

Figure 3.4: TREPAN Experimentation...... 59

Figure 3.5: ARFF Format File -- Saginaw Bay ...... 61

Figure 4.1: Corrosion: Actual Output vs. Modeled Neural Network Output ...... 66

Figure 4.2: Corrosion: Single Test TREPAN -- Classification Accuracy vs. Tree Size .. 71

Figure 4.3: Corrosion: TREPAN (beam width 5)-Classification Accuracy vs. Tree Size 73

Figure 4.4: Corrosion: TREPAN Decision Tree...... 76

Figure 4.5: Corrosion: C4.5 Decision Tree...... 78

Figure 4.6: Outages: Actual Output vs. Modeled Neural Network Output ...... 80 12

Figure 4.7: Outages: Single Test TREPAN -- Classification Accuracy vs. Tree Size ..... 83

Figure 4.8: Outages: TREPAN Decision Tree...... 86

Figure 4.9: Outages: C4.5 Decision Tree...... 88

Figure 4.10: Iris: Single Test TREPAN -- Classification Accuracy vs. Tree Size ...... 91

Figure 4.11: Iris: Decision Tree Comparison ...... 94

Figure 4.12: Body Fat: Actual Output vs. Modeled Neural Network Output...... 95

Figure 4.13: Body Fat: Single Test TREPAN-Classification Accuracy vs. Tree Size..... 97

Figure 4.14: Body Fat: Decision Tree Comparison...... 100

Figure 4.15: Saginaw Bay: Actual Output vs. Modeled Neural Network Output ...... 101

Figure 4.16: Saginaw Bay: Single Test TREPAN-Classification Accuracy vs. Tree Size

...... 104

Figure 4.17: Saginaw Bay: TREPAN Decision Tree...... 107

Figure 4.18: Saginaw Bay: C4.5 Decision Tree ...... 109

Figure 4.19: Admissions: Single Test TREPAN-Classification Accuracy vs. Tree Size113

Figure 4.20: Admissions: TREPAN Decision Tree...... 116

CHAPTER 1. INTRODUCTION

1.1 Machine Learning

The human brain is the most complex data processing, storage and operations

control system known to date. It begins the process of learning a complex task by breaking it down into simple subcomponents that are easier to learn and store [1] . It is this ability of the brain that has fascinated scientists and engineers who strive to build machines that can duplicate human intelligence. A machine learns whenever it changes its program, structure or data in such a manner that its future performance improves [2] .

Machine learning refers to the ability of a system to capture knowledge from experience and analytical observations. In other words, it is a method of creating programs that allow computers to “learn” by analyzing data. Machine learning can be sub-divided into two categories:

• Unsupervised learning (Clustering) where, given a set of observations the aim is

to establish the existence of classes or clusters in the data.

• Supervised learning (Classification) where the existence and the number of

classes for a given dataset are known and the aim is to establish a rule to be able

to classify a new observation into one of the existing classes. 14

1.2 Classification Algorithms

Classification falls under the machine learning umbrella which encompasses a

wide range of aspirations. Classification is very useful when applied to data that can be used as a foundation for making future decisions, such as medical diagnosis, scientific analysis or credit risk analysis. Classification algorithms use the concept of inductive learning to classify examples into a given set of categories. Inductive learning is one of the major paradigms of machine learning which infers from a given set of examples in order to make accurate predictions for future examples. Decision trees, Neural Networks,

Case Based Reasoning, K-Nearest Neighbor Classifier, and Bayesian Networks are some

of the typical classification algorithms.

1.3 Research Objectives

One of the more pronounced drawbacks of Neural Networks is their lack of

explanation capability. They act like ‘black boxes’ and do not give any reasoning behind

the conclusion of a learning system. However it is essential to understand the basis of

decisions of this type of computer support systems for various reasons. For example,

consider the case of a credit card company: a Neural Network model may indicate that

credit should be given to a particular individual, but it is helpful to know what

characteristics about the individual led to the decision in order to make future predictions.

This information may help the company target other similar individuals in order to

expand its business. Thus classification decisions are extremely useful when they are in

human comprehensible form. Representing the extracted knowledge in the form of ‘if- 15

then’ rules is currently the best method of explaining the output of a Neural Network model. Decision trees can be easily represented in the form of ‘if-then’ rules and hence extracting decision trees is one of the best methods of explaining a Neural Network. This idea is described more effectively by Figure 1.1.

NEURAL NETWORK Can be explained better using NEURAL NETWORK DECISION TREE MODEL MODEL

'IF-THEN' Implies Rules

EXTRACT DECISION TREES FROM TRAINED NEURAL NETWORK DECISION TREES Can be easily expressed as MODELS

Figure 1.1: Extracting Decision Trees from Neural Networks

Decision trees are fast, simple to implement and can convert the learned hypothesis to a set of easily interpreted rules. The basic idea is to mimic human reasoning in a way that gives an insight into the decision process.

Schmitz, et al. [3] have proposed an algorithm ANN-DT that extracts decision trees from trained neural networks. Craven and Shavlik [4] have developed the TREPAN 16

algorithm that mimics the behavior of a neural network in the form of decision trees. One of the research objectives of this thesis is to enhance TREPAN to be able to handle not only multi-class classification type but also multi-class regression type problems.

Another objective is to investigate if TREPAN can understand and analyze generalized feedforward networks (GFF). TREPAN is tested on different datasets and best settings for TREPAN algorithm are explored based on database type to generate heuristics for various problem domains. The best TREPAN model is then compared to the baseline

C4.5 decision tree algorithm to test for accuracy and comprehensibility.

1.4 Thesis Overview

The thesis is structured into five chapters. Chapter 2 describes the problem and

provides background literature for the concepts involved as well as a review of the

current approaches in existing research. Chapter 3 outlines the methodology and

describes datasets and the tools that were used in the analysis within this thesis. Chapter 4

presents the results of the neural network models, TREPAN and C4.5 decision tree

induction. Chapter 5 presents a discussion and heuristics identified in the empirical

investigation. Finally it reports the conclusions and provides recommendations for future

research. 17

CHAPTER 2. BACKGROUND AND LITERATURE

REVIEW

2.1 Artificial Neural Networks

Artificial neural networks are modeled on the architecture of the human brain.

They offer a means of efficiently modeling large and complex problems in which there

may be hundreds of independent variables that have many interactions. Neural networks

generate their own implicit rules by learning from examples. Since their resurgence in the

1980’s, artificial neural networks have been applied to a variety of problem domains [5] such as speech recognition [6] , speech generation [7] , medical diagnostics [8] , game playing [9] and robotics [10] . The generalization ability of neural networks has proved to be on par or superior to other learning systems over a wide range of applications [11] .

2.1.1 Neural Network Architecture

A neural network consists of a large number of units called processing elements

(nodes or neurons) that are connected on a parallel scale. The network starts with an input

layer, where each node corresponds to an independent variable. Input nodes are

connected to a number of nodes in a hidden layer. There may be more than one hidden

layer and output layer. Each node in the hidden layer takes in a set of inputs (X1,

X2...... Xm), multiplies them by a connection weight (W1, W2……..Wm), adds them 18

together, applies a function, f(WTX) to them and then passes the output to the nodes of

the next layer. The connection weights are the unknown parameters that are estimated by an iterative training method and indicate the connection’s strength and excitation. The

calculation of the final outputs of the network proceeds layer by layer [12] . Each processing element of the hidden layer computes its output as a function of linear

combination of inputs from the previous layer plus a bias. This output is propagated as

input to the next layer and so on until the final layer is reached. Figure 2.1 shows the

model of a single neuron.

X1 W1 X2 W2

X3 W3 f(WTX) Y

Figure 2.1: Model of a Single Neuron

Adapted from Aldrich [13]

The output of the neuron can be expressed as

m Y = f (∑ wi xi ) , or (Eq. 2.1) i=1

Y = f (W T X ) (Eq. 2.2)

In the above equation, W is the weight vector of the neural node, defined as

T W = [w1,w2 ,w3,...... wm ] (Eq. 2.3)

and X is the input vector, defined as

X = x , x , x ,...... x T [ 1 2 3 m ] (Eq. 2.4)

f (W T X ) is called the activation function of the node

Figure 2.2 shows a typical neural network architecture representation. 20

Output Layer

Hidden Layer

Input Layer

X 1 X2 X3 X 4 X5

Figure 2.2: Neural Network Architecture

There are different types of activation functions that can be applied at the node of the network. Two of the most commonly used neural network functions are the hyperbolic and logistic (or sigmoid) functions. They are sometimes referred to as

“squashing” functions since they map the inputs into a bounded range. Table 2.2 shows a list of activation functions that are available for use in neural networks. 21

Table 2.1: Activation Functions used in Neural Networks

(Adapted from [14] )

Functions Definition Range Identity x (−∞,+∞) Logistic 1 (0,+1)

(1− e −x ) Hyperbolic e x − e −x (−1,+1)

e x + e −x Exponential e−x (0,+∞) Softmax e −x (0,+1)

∑e xi i Unit Sum x (0,+1)

∑ xi i Square root x (0,+∞) Sine Sin(x) (0,+1) Ramp ⎧−1, x ≤ −1 ⎫ (−1,+1) ⎪ ⎪ ⎨x,−1 < x < +1⎬ ⎪ ⎪ ⎩+1, x ≥ +1 ⎭ Step ⎧0, x < 0 ⎫ (0,+1) ⎨ ⎬ ⎩+1, x ≥ 0⎭

MULTILAYER PERCEPTRONS

Multilayer Perceptrons (MLPs) are layered feed forward networks typically trained with backpropagation. These networks have been used in numerous applications.

Their main advantage is that they are easy to use, and that they can approximate any input/output map. A major disadvantage is that they train slowly, and require lots of 22

training data (typically three times more training samples than network weights) [15] .

Figure 2.3 shows a schematic of a Multilayered Perceptron Network.

Hidden Layer 1

Input Layer Hidden Layer 2

Output Layer

Figure 2.3: Multi Layer Perceptron Network

GENERALIZED FEED FORWARD NETWORK (GFF)

A Generalized Feed Forward (GFF) network is a special case of a Multilayer

Perceptron wherein connections can jump over one or more layers. Although an MLP can solve any problem that a GFF can solve, in practice, a GFF network can solve the 23

problem more efficiently [15] . Figure 2.4 shows a general schematic of a Generalized

Feed Forward Network.

Output Layer

Input Layer Hidden Layer 2

Hidden Layer 1

Figure 2.4: Generalized Feed Forward Network 24

2.1.2 Neural Network Training

The neural network approach is a two stage process. In the first stage a generalized network that maps the input data to the desired output using a training algorithm is derived. The next stage is the “production” phase where the network is tested for its generalization ability against a new set of data. Rumelhart et al.’s [16] backpropagation training algorithm is the most widely used neural network training algorithm. It is an abbreviation for the term “backward propagation of errors”. The available data is fed into a network for a specified number of epochs, an epoch being the complete data set. For each epoch the training cases are fed to the network, actual and network outputs are compared and error is calculated. The weights are then adjusted to reduce this error and the process is repeated. The errors propagate backwards from the output nodes to the input nodes and hence the term “backpropagation”.

Often the neural network tends to over train and memorizes the data. To avoid this possibility, a cross-validation data set is used. The cross validation data set is a part of the data set which is set aside before training and is used to determine the level of generalization produced by the training set. As training progresses the training error drops progressively. At first the cross validation error decreases but then begins to rise as the network over trains. Best generalization ability of the network can be tapped by stopping the algorithm where the error on the cross validation set starts to rise. Figure 2.5 illustrates the use of cross-validation during training. 25

Underfitting Overfitting

Cross-validation error

Error Stop Training

Training Error

Model Complexity

Figure 2.5: Use of Cross-Validation during Training

2.1.3 Neural Networks for Classification and Regression

Neural networks are one of the most widely used algorithms for classification problems. The output layer is indicative of the decision of the classifier. The cross entropy error function is most commonly used in classification problems in combination with logistic or softmax activation functions. Cross entropy assumes that the probability of the predicted values in a classification problem lie between 0 and 1. In a classification problem each output node of a neural network represents a different hypothesis and the node activations represent the probability that each hypothesis may be true. Each output 26

node represents a probability distribution and the cross entropy measure calculates the difference between the network distribution and the actual distribution.[17] . Assigning credit risk (good or bad) is an example of a neural network classification problem.

Regression involves predicting the value of a continuous variable based on previously collected data. Mean square error is the function used for computing the error in regression networks. Projecting the profit of a company based on previous years data is a regression type neural network problem.

2.1.4 Rule Extraction from Neural Networks

Although neural networks are known to be robust classifiers, they have found limited use in decision-critical applications such as medical systems. Trained neural networks act like black boxes and are often difficult to interpret [18] . The availability of a system that would provide an explanation of the input/output mappings of a neural network in the form of rules would thus be very useful. Rule extraction is one such system that tries to elucidate to the user, how the neural network arrived at its decision in the form of if-then rules.

Two explicit approaches have been defined to date for transforming the knowledge and weights contained in a neural network into a set of symbolic rules- decompositional and pedagogical [19] . In the decompositional approach the focus is on extracting rules at an individual hidden and/or output level into a binary outcome. It involves the analysis of the weight vectors and biases associated with the processing elements in general. The Subset [20] algorithm is an example of this category. The 27

pedagogical approach treats neural networks like black boxes and aims to extract rules that map inputs directly to its outputs. The Validity Interval Analysis (VIA) [21] proposed by Thrun and TREPAN [22] is an example of one such technique. Andrews et al [23] propose a third category called eclectic which combines the elements of the two basic categories. The DEDEC [24] algorithm is representative of this category.

2.2 Decision Trees

A decision tree is a special type of graph drawn in the form of a tree structure. It consists of internal nodes each associated with a logical test and its possible consequences. Decision trees are probably the most widely used symbolic learning algorithms as are neural networks in the non-symbolic category.

2.2.1 Decision Tree Classification

Decision trees classify data through recursive partitioning of the data set into mutually exclusive subsets which best explain the variation in the dependent variable under observation[25] [26] . Decision trees classify instances (data points) by sorting them down the tree from the root node to some leaf node. This leaf node gives the classification of the instance. Each branch of the decision tree represents a possible scenario of decision and its outcome.

Decision tree algorithms depict concept descriptions in the form of a tree structure. They begin learning with a set of instances and create a tree structure that is used to classify new instances. An instance in a dataset is described by a set of feature 28

values called attributes, which can have either continuous or nominal values. Decision tree induction is best suitable for data where each example in the dataset is described by a fixed number of attributes for all examples of that dataset. Decision tree methods use a divide and conquer approach. They can be used to classify an example by starting at the root of the tree and moving through it until a leaf node is reached, which provides the classification of the instance.

Each node of a decision tree specifies a test of some attribute and each branch that descends from the node corresponds to a possible value for this attribute. The following figure shows a simple decision tree representation.

Root Node

Alternate Condition 1 Branches Condition 1

Leaf Node 1 Leaf Node 2

Class 1 Alternate

Condition 2 Condition 2

Leaf Node 3 Leaf Node 4

Class 2 Class 1

Figure 2.6: Decision Tree Architecture 29

2.2.2 Decision Tree Applications

Decision trees have been proved useful in their applications to various real world problems. Leech [27] applied a decision tree induction approach to a chemical nuclear power plant process. It involved continuous feedback and modification of the rules extracted from decision trees. This improved system resulted in reduction in inventory, increased output and increase in business of the company by more than ten million dollars per year [28] .

Michie [29] used an induction algorithm to produce a decision tree for making decisions whether to grant credit to a loan applicant or not. American Express UK immediately put this method to use because it not only gave them accurate predictions for

70% of the borderline applicants (whether they would default on their loans or not) but also gave a comprehensible explanation of the reasons for these predictions.

Evans and Fisher [30] applied decision tree induction to the problem of banding in Rotogravure printing. Banding is the occurrence of grooves or bands on the inside of the cylinder coated with ink which appear on paper during printing. As a result of banding the print run has to be halted and the cylinder cleaned or in some cases replaced causing process delays and decrease in production rates. The reason for occurrence of banding was not known. Evans and Fisher [30] extracted a decision tree and came up with a set of plausible rules that cause banding. Experts could then change the printing parameters to override these rules so as to avoid banding.

In manufacturing and production industry decision trees have been used in 30

• non-destructive testing of spot weld quality[31] ,

• increasing productivity[32] ,

• semiconductor manufacturing[33] ,

• material procurement method selection[34] ,

• process optimization [35] ,

• assembly line scheduling of printed circuit boards [36] ,

• to uncover flaws in a Boeing manufacturing process[37] ,

• separation of gas from oil [38] ,

• quality control [39] and

• hybrid system to extract rules for job shop scheduling[40] .

Other applications include the areas of Biomedical Engineering, Image

Processing, Language Processing, Law, Medicine, Molecular Biology, Pharmacology,

Physics and Plant diseases [41] . They have also been used to interpret lactation curves for nutritional analysis in Dairy farming [42]

Decision trees continue to be an active research area, the current focus being on improving methods for building, controlling and executing the decision tree algorithms to achieve maximum efficiency.

2.3 C4.5 Algorithm

The C4.5 algorithm [43] developed by Quinlan is one of the most widely used decision tree learning algorithms. It is an advanced and incremental software extension of 31

the basic ID3 algorithm [44] designed by Quinlan to address the issues that were not dealt with by ID3.

The C4.5 algorithm has its origins in Hunt’s Concept Learning Systems (CLS)

[45] . It is a non-incremental algorithm, which means that it derives its classes from an initial set of training instances. The classes derived from these instances are expected to work for all future test instances. The algorithm uses the greedy search approach i.e. selects the best attribute and never looks back to reconsider earlier choices.

The C4.5 algorithm searches through the attributes of the training instances and finds the attribute that best separates the data. If this attribute perfectly classifies the training set then it stops else it recursively works on the remaining m subsets (m = the remaining possible values of the attribute) to get their best attribute.

Some attributes split the data more purely than others. Their values correspond more consistently with instances that have particular values of the target class. Therefore it can be said that they contain more information than the other attributes. But there should be a method that helps quantify this information and compares different attributes in the data which will enable us to decide which attribute should be placed at the highest node in the tree.

2.3.1 Information Gain, Entropy Measure and Gain Ratio

A fundamental part of any algorithm that constructs a decision tree from a dataset is the method in which it selects attributes at each node of the tree for splitting so that the 32

depth of the tree is minimum. ID3 uses the concept of Information Gain which is based on Information Theory [46] to select the best attribute.

Gain measures how well a given attribute separates training examples into its target classes. The one with the highest information is selected. Information gain calculates the reduction in entropy (or gain in information) that would result from splitting the data into subsets based on an attribute.

The information gain of example set S on attribute A is defined as,

S Gain(S, A) = Entropy(S) − v Entropy(S ) (Eq. 2.5) ∑ S v

In the above equation, S is the number of instances and Sv is a subset of instances of S where A takes the value v.

Entropy is a measure of the amount of information in an attribute. The higher the entropy, the more the information is required to completely describe the data. Hence, when building the decision tree, the idea is to decrease the entropy of the dataset until we reach a subset that is pure (a leaf), has zero entropy and represents instances that all belong to one class. Entropy is given by,

Entropy(S) = − p(I)log p(I) ∑ 2 (Eq. 2.6)

where,

p(I) is the proportion of S belonging to class I

Consider the following example,

Suppose we want ID3 to construct a decision tree that will enable us to decide if the weather is favorable to play tennis. The data collected for inputs to ID3 is shown in

Table 2.2. 34

Table 2.2: Play Tennis Example Dataset

Adapted from Quinlan [43]

Day Outlook Temperature Humidity Wind Play Tennis 1 Sunny Hot High Weak No 2 Sunny Hot High Strong No 3 Overcast Hot High Weak Yes 4 Rain Mild High Weak Yes 5 Rain Cool Normal Weak Yes 6 Rain Cool Normal Strong No 7 Overcast Cool Normal Strong Yes 8 Sunny Mild High Weak No 9 Sunny Cool Normal Weak Yes 10 Rain Mild Normal Weak Yes 11 Sunny Mild Normal Strong Yes 12 Overcast Mild High Strong Yes 13 Overcast Hot Normal Weak Yes 14 Rain Mild High Strong No 35

In this example,

⎛ 9 ⎞ ⎛ 9 ⎞ ⎛ 5 ⎞ ⎛ 5 ⎞ Entropy(S) = −⎜ ⎟log 2 ⎜ ⎟ − ⎜ ⎟log 2 ⎜ ⎟ = 0.9450 ⎝14 ⎠ ⎝14 ⎠ ⎝14 ⎠ ⎝14 ⎠

(Note: Number of instances where play tennis = yes is 9 and play tennis = No is 5)

The best attribute of the four is selected by calculating the Information Gain for each attribute as follows,

5 4 5 Gain(S,Outlook) = Entr.(S) − × Entr.(Sunny) − Entr.(Overcast) − Entr.(Rain) 14 14 14

Gain(S,Outlook) = 0.940 − 0.3364 − 0 − 0.3364 = 0.2670

Similarly,Gain(S,Temp) = −0. 42 and Gain(S,Wind) = 0. 1515

The attribute outlook has the highest gain and hence it is used as the decision attribute in the root node. The root node has three branches since the attribute outlook has three possible values (Sunny, Overcast, and Rain). Only the remaining attributes are tested at the sunny branch node since outlook has already been used at the root node. This process is recursively repeated until:

• All the training instances have been classified or

• Every attribute has been utilized in the decision tree. 36

The ID3 has a strong bias in favor of tests with many outcomes. Consider an employee database that consists of an employee identification number. Every such number is intended to be unique and partitioning any set of training cases on the values of this attribute will lead to a large number of subsets, each containing only one case. Hence the C4.5 algorithm incorporates use of a statistic called the “Gain Ratio” that compensates for the number of attributes by normalizing with information encoded in the split itself.

Gain(S, A) GainRatio = (Eq. 2.7) I(A)

In the above equation,

I(A) = − p(I )log p(I ) ∑ A 2 A (Eq. 2.8)

C4.5 has another advantage over ID3; it can deal with numeric attributes, missing values and noisy data. The decision tree generated by the C4.5 is shown in Figure 2.7. 37

Outlook

Sunny Overcast Rain

Yes Humidity Wind

High Normal Strong Weak

No Yes No Yes

Figure 2.7: Decision Tree for the Play Tennis Problem.

Adapted from Quinlan [43]

The C4.5 and its variations have been a popular topic among researchers since

Quinlan introduced ID3.

2.4 TREPAN Algorithm

The TREPAN [4] [22] algorithm developed by Craven, is a novel rule-extraction algorithm that mimics the behavior of a neural network. Given a trained Neural Network, 38

TREPAN extracts decision trees that provide a close approximation to the function represented by the network. In this thesis we are concerned with its application to trained neural network models although it can be applied not only to neural networks but to a wide variety of learned models as well.

TREPAN uses a concept of recursive partitioning similar to other decision tree induction algorithms. In contrast to the depth-first growth used by other decision tree algorithms, TREPAN expands using the best first principle. That node which increases the fidelity of the tree when expanded is deemed the best.

In conventional decision tree induction algorithms the amount of training data decreases as one traverses down the tree by selecting splitting tests. Thus there is not enough data at the bottom of the tree to determine class labels and is hence poorly chosen. In contrast TREPAN uses an ‘Oracle’ to answer queries, in addition to the training samples during the inductive learning process. Since the target here is the function represented by the Neural Network, the network itself is used as the ‘Oracle’.

This learning from larger samples can avoid the lack of examples for the splitting tests at lower levels of the tree, which is usually a problem with conventional decision tree learning algorithms. It ensures that there is a minimum sample of instances (min. sample) available at a node before choosing a splitting test for that node where min. sample is one of the user specified parameters. If the number of instances at the node; say m is less than min. sample then TREPAN will make membership queries equal to (min. sample-m) 39

from the ‘Oracle’ and then make a decision at the node. Figure 2.8 provides a sketch of the TREPAN algorithm.

Algorithm: TREPAN r N Input: Trained neural network; training examples{xi , yi }i = 1, where yi is the class r label predicted by the trained neural network on the training example xi , global stopping criteria. Output: Extracted decision tree Begin Initialize the tree as a leaf node While global stopping criteria are not met and the current tree can be further refined Do pick the most promising leaf node to expand draw a sample of examples use the trained network to label these examples select a splitting test for the node for each possible outcome of the test make a new leaf node End End

Figure 2.8: TREPAN Algorithm

Adapted from Chen [47]

2.4.1 M-of-N Splitting tests

Trepan uses the m-of-n tests to partition the part of the instance space covered by a particular internal node. An m-of-n expression (a Boolean expression) is fulfilled when at least an integer threshold m of its n literals hold true. For example, consider four 40

features a, b, c and d; the m-of-n test: 3-of-{a, b > 2.3, ¬c, d} at a node signifies that if any of the 3 conditions of the given set of 4 are satisfied then an example will pass through that node. TREPAN employs a beam search method with beam width as a user defined parameter to find the best m-of-n test. Beam search is a heuristic best-first search algorithm that evaluates the first n nodes (where n is a fixed value called the ‘beam width’) at each tree depth and picks the best out of them for the split. TREPAN uses both local and global stopping criteria. The growth of the tree stops when any of the following criteria are met:

• The size of the tree which is a user specific parameter or

• When all the training examples at a node fall in the same class

2.4.2 Single Test TREPAN and Disjunctive TREPAN

In addition to TREPAN algorithm Craven has also developed two of its important variations which are investigated further in this thesis. The single test TREPAN algorithm is similar to TREPAN in all respects except that as its name suggests it uses single feature tests at the internal nodes. Disjunctive TREPAN uses disjunctive “OR” tests at the internal nodes of the tree instead of the m-of-n tests. A more detailed explanation of the TREPAN algorithm can be found in Craven’s dissertation [22] .

Baesens et al [48] have applied TREPAN to credit risk evaluation and reported that it yields very good classification accuracy as compared to the logistic regression classifier and the popular C4.5 algorithm. Hudson et al have adapted the TREPAN code into a MATLAB implementation and applied it to the field of bioinformatics [49] . 41

CHAPTER 3. METHODOLOGY

This chapter outlines the framework of the methodology which can be divided into 4 phases. They are depicted in the flowcharts in Figure 3.1 and Figure 3.2. 42

Enhance TREPAN

PHASE1 y Regression Problems y Generalized Feed Forward Networks

Standard Identify Database Machine Large Database with little Learning Databases data Databases

Preprocess Saginaw Iris Corrosion PHASE2 Data Bay

Create Body fat Admissions Outages Neural Network Model

TREPAN Parameters y Tree Size y Minimum Sample size y Beam Width

PHASE3

Identify Best Model

Heuristics (Best Settings for Database type)

Figure 3.1: Methodology (Phases 1-3) 43

Baseline TREPAN Model Best Model (C4.5) Comparison

Classification Kappa Statistic Comprehensibility Accuracy

Tree Size 7 Tree Size 5

Confusion Matrix

Figure 3.2: Methodology (Phase 4)

Each phase is described in detail in the sections that follow.

3.1 Phase 1

The first step in this research is to modify and enhance the TREPAN software in order to be able to handle a wider variety of problems. TREPAN currently works with multi-class classification type problems and two class regression problems. The aim in this research is to be able to work with classification as well as regression problems with 44

multiple classes. The next step is to run experiments to investigate if TREPAN can analyze the weights associated with generalized feed forward networks. In case TREPAN is found to be incompatible, additional programming will be done for generalization over

GFF networks.

3.2 Phase 2

The second step is to identify potential databases for TREPAN analysis. Two of each standard machine learning database and comparatively small and large real world databases selected for analysis are summarized in Table 3.1

Table 3.1: Dataset Summary

No. of Attributes Problem Dataset No. of Dataset type Numeric/ Symbolic/ size classes Real Nominal

Iris Classification 4 0 150 3 Body fat Regression 14 0 252 4 Saginaw Bay Regression 12 0 976 5 Corrosion Regression 11 0 95 5 Admissions Classification 4 3 10678 2 Outages Regression 12 0 300 5 45

A brief description with a sample of each database is described in the following sections. It should be noted that for regression problems it is required that the range of the response variable be segmented into classes. This is done by dividing the output into discrete values based on population density.

3.2.1 Datasets

IRIS

The iris plants dataset is perhaps the most widely used classification problem in machine learning research. It consists of four numerical attributes, namely the sepal length, sepal width, petal length and petal width that are that are used to predict the class of the iris plant. This database is obtained from the UCI Repository of machine learning databases [50] . Table 3.2 gives a sample of the Iris dataset.

Table 3.2: Iris--Sample Dataset

Petal Petal Sepal Sepal Sr. No. Class width length width length 1 6 2.7 5.1 1.6 Iris-versicolor 2 5 3.4 1.5 0.2 Iris-setosa 3 7.6 3 6.6 2.1 Iris-virginica 4 5.7 2.5 5 2 Iris-virginica 5 6 2.2 5 1.5 Iris-virginica 46

BODY FAT

The body fat is a regression problem for predicting the body fat percentage based on various body characteristics and measurements of body anatomy: body density, age, weight, height, neck, chest, abdomen, hip, thigh, knee, ankle, biceps, forearm and wrist.

This database is obtained from the UCI Repository of machine learning databases [50] .

A sample of the dataset and the class sizes for the regression output are shown in Table

3.3 and Table 3.4 respectively. 47

Table 3.3: Body Fat -- Sample Dataset

Sr. A B C D E F G H I J K L M N O No. 1 1.0404 40 191 74 38.3 95.4 92.4 104.3 64.6 41.1 24.8 33.6 29.5 18.5 25.8 2 1.0709 43 178.25 70.25 37.8 102.7 89.2 99.2 60.2 39.2 23.8 31.7 28.4 18.6 12.2 3 1.0575 49 171.75 71.5 35.5 97.8 90.1 95.8 57 38.7 23.2 27.5 26.5 17.6 18.1 4 1.0515 40 192.25 73.25 39.8 103.9 93.5 99.5 61.7 39 21.8 33.3 29.6 18.1 20.8 5 1.0477 38 187.25 69.25 38 102.7 92.7 101.9 64.7 39.5 24.7 34.8 30.3 18.1 22.5 Note: A=Density (Kg/L), B=Age (years), C=Weight (pounds), D=Height (inches), E=Neck (cm.), F=Chest (cm.), G=Abdomen (cm.), H=Hip (cm.), I=Thigh (cm.), J=Knee (cm.), K=Ankle (cm.), L=Biceps (cm.), M=Forearm (cm.), N=Wrist (cm.), O=Percentage Body fat.

Table 3.4: Body Fat Class Labels

Percentage Body fat Class 0-11.99 Toned 12.00-19.99 Healthy 20.00-24.99 Flabby 25 and above Obese

SAGINAW BAY

This dataset comprises of real world data collected from the Saginaw Bay ecosystem in Michigan. Various indicators include the water temperature, the level of water clarity (Secchi), total suspended solids (TSS), total phosphorous (TP), soluble reactive phosphorus (SRP), nitrate (NO3), ammonia (NH4), silica (SiO2), particulate silica (PSiO2), chloride (CL), particulate organic carbon (POC), and dissolved organic carbon (DOC). These indicators are used to predict the chlorophyll (CHL) level which is a measure for total phytoplankton biomass and is used to gauge the health of the Bay.

The Saginaw Bay is a complex dataset with a large percentage of the output data concentrated in a small area. A few instances are scattered over a large chlorophyll range.

These instances are comparable to a rare event situation where very little data is available and modeling is difficult.

Biologically inspired computational techniques for analyzing ecosystems have become increasingly popular and are studied under the term Ecological Informatics. The

International Society for Ecological Informatics defines Ecological Informatics as,

“An interdisciplinary framework promoting the use of advanced computational technology for the elucidation of principles of information processing at and between all levels of complexity of ecosystems -- from genes to ecological networks -- and aiding transparent decision-making in relation to important issues in ecology such as sustainability, biodiversity and global warming[51] .” 49

As depicted in Figure 3.3 neural networks is one of the basic concepts in

Ecological Informatics [52] .

ECOLOGICAL ECOSYSTEM INFORMATION THEORY

ECOLOGICAL DATA ECOSYSTEM ANALYSIS, ARCHIVAL, ECOLOGICAL ECOLOGICAL SYNTHESIS RETRIEVAL DECISION DATA & & SUPPORT FORECASTING VISUALIZATION

COMPUTATIONAL COMPUTATIONAL TECHNOLOGY TECHNOLOGY

Object Oriented Data Fuzzy Logic Representation Artificial Neural Networks Animation Genetic/Evolutionary Algorithms Remote sensing Hybrid Models etc. Adaptive Agents etc.

Figure 3.3: Scope of Ecological Informatics

Adapted from [51]

A sample of the Saginaw Bay dataset and the class sizes for level of chlorophyll are shown in Table 3.5 and Table 3.6 respectively. 50

Table 3.5: Saginaw Bay -- Sample Dataset

Sr. Temp Secchi TSS TP SRP NO3 NH4 SiO2 PSiO2 CL POC DOC CHL No. 1 22.11 0.9 16 32.019 1.055 0.009 17.213 0.245 1.51 23.966 3.79 4.08 20.32 2 7.8 6.5 0.76 16.2158 0.588 0.329 14.543 1.713 0.241 5.9 0.37 1.719 1.58 3 22.25 0.8 11.3 37.4068 1.07142 0.0252 7.64313 2.34418 1.01326 23.6194 1.97 3.86 9.10769 4 15.3 0.4 23.91 32.6027 1.46 0.647 18.143 0.085 3.69 20.087 1.97 3.769 22.628 5 8.77 2 4.32 13.648 0.92 1.17 13.6 1 1.36 26.3 0.95 4.12 7.45

Table 3.6: Chlorophyll Level Class Labels

CHL Class 0 – 8.999 Cl1 9 – 17.999 Cl2 18 – 26.999 Cl3 27- 35.99 Cl4 36 and above Cl5

CORROSION

Many researchers have worked on the problem of modeling the CO2 corrosion process due to its significance in predicting the corrosion rate. The corrosion dataset is a collection of the characteristic composition of 15 Venezuelan crude oils used to predict the ability of a crude oil to offer corrosion inhibition in a CO2 environment [53] . In addition to process complexity, the small size of the dataset makes it challenging to obtain a process model through traditional statistical means.

Attributes such as API, sulphur, total nitrogen, TAN (total acid number), saturates, aromatics, resins, asphaltenes, vanadium, nickel and percentage of crude oil are used for the analysis. The API (American Petroleum Institute) is a scale used by the petroleum industry to grade crude oil whereas sulphur and nitrogen content and heavy metals like vanadium and nickel are impurities found in crude oil. The TAN or total acid number indicates how much oxidation has taken place in a fluid. An in-depth description of the test conditions used to determine these oil characteristics can be found in [54] .

These attributes are used to predict the inhibiting capacity and hence the corrosion rate given by the formula,

Corrosion rate Inhibiting capacity = 1- crude oil (Eq. 3.1) Corrosion rateblank

A sample of the dataset and the class sizes for the regression output are shown in

Table 3.7 and Table 3.8 respectively. 52

Table 3.7: Corrosion -- Sample Dataset

Sr. S total N TAN Saturates Aromatics resins asphaltenes V Ni %Crude API %Inh. No. (%p/p) (ppm) (KOH/g) (%) (%) (%) (%) (ppm) (ppm) oil 1 33.3 0.566 923 0.02 62.2 34.8 3 0 5 5 20 81.40 2 20.7 1.1 3276 0.11 43.2 32.6 17.6 6.6 125 25 80 99.23 3 20.4 2.42 3450 0.74 30 45.1 16.4 8.6 362 44 50 89.1 4 8.5 3.7 6948 3.35 13.3 47.8 28.8 10.1 458 95 1 81.20 5 11 2.6 4340 4.78 21.1 50.1 19.7 9.2 409 52 20 98.6

Table 3.8: Corrosion Class labels

%Inhibition Class 0.75-0.849 Cl1 0.85-0.889 Cl2 0.89-0.949 Cl3 0.95-0.979 Cl4 0.98 and above Cl5

ADMISSIONS

The admissions database consists of attributes pertaining to an applicant at Ohio

University. Attributes such as high school rank, high school size, college of application, application day, decision day, sex and race of an applicant are used to predict if he or she will choose to attend Ohio University. Based on the results, the university can develop programs and services to help retain students. One of the reasons for such a study is to help keep a steady flow of income through tuition payments which is often the primary source of income for academic institutions. Generating and maintaining a diverse population on campus is also another criterion for this study. Various studies have been conducted to predict the attrition of students attending freshman year by institutions worldwide. If dropping out is predicted for a student, measures can be taken to help prevent that possibility. A number of researchers have developed theoretical [55] [56]

[57] [58] [59] and hybrid logistic regression models [60] to predict student retention.

The admissions database analyzed here is proprietary to Ohio University and has been masked wherever necessary. A sample of the admissions database is shown in Table 3.9. 54

Table 3.9: Admissions -- Sample Dataset

Sr. HS HS Attend College Apply_day Dec_day Sex Race No. rank size OU. 1 1 333 c1 x1 y1 F 1 no 2 1 867 c2 x2 y2 M 2 no 3 0 456 c3 x3 y3 M 3 yes 4 0 968 c4 x4 y4 F 4 no 5 1 847 c5 x5 y5 F 5 yes

OUTAGES

This database addresses the problem of evaluating the survivability of wireless networks. The data is obtained by running a simulation model on a single wireless infrastructure building (WIB) serving 100,000 customers over a 1 year period. Attributes used are the mean time to failure (MTTF) and Mean time to restore (MTR) of a set of components in a WIB. A more detailed description of the simulation runs and description of the indicators can be obtained from Weckman et al. [61] . The predicted output: reportable outages is divided into 5 classes: very good (VG), good (G), ok (O), bad (B) and very bad (VB). A sample of the dataset is shown in Table 3.10 and class sizes in

Table 3.11 respectively.

Table 3.10: Outages -- Sample Dataset

Sr. No MTTF MTTF MTTF MTTF MTTF MTTF MTR MTR MTR MTR MTR MTR Reportable BS BSC BSCBS MSC MSCBSC DB BS BSC BSCBS MSC MSCBSC DB Outages 1 2.983 4.433 4.25 9.9 4.433 2.233 1.25 1.066 3.066 0.516 1.216 0.266 8.18 2 1.416 4.05 4.883 8.933 4.183 2.25 1.066 1.483 1.516 0.333 2.166 0.166 8.2 3 1.766 3.75 4.05 6.233 3.983 3.416 1.966 1.266 1.766 0.166 1.55 0.983 8.88 4 1.05 4 4.716 5.916 4.033 2.8 1.4 1.366 2.433 0.716 6.216 0.983 8.9 5 1.25 4.566 4.566 6.05 3.933 2.133 1.116 1.333 1.383 0.35 1.133 0.933 9.12 Note: BS = Base Station, BSC = Base Station Controller, BSCBS = BSC to BS link, MSC = Mobile Switching Center, MSCBSC = MSC to BSC link, DB = Database.

Table 3.11: Reportable Outages Class Labels

Reportable Outages Class 0 – 9.99 VG 10 – 19.99 G 20 – 29.99 O 30 – 39.99 B 40 and above VB

3.2.2 Neural Network Modeling

NeuroSolutions [61] software was used to develop different neural network models for the datasets described earlier. Various networks types such as multi-layer perceptron and generalized feed forward networks were experimented with to obtain the best neural network model. Except for the corrosion dataset all networks use 60%, 15% and 25% of the data for training, cross-validation and testing respectively. Due to its limited size, corrosion is split as 70%, 15% and 15% for training, cross-validation and testing respectively in order to provide for more training data. The network configurations for each problem domain are summarized in Table 3.12.

Table 3.12: Network Configurations

Transfer Transfer No. of (Training, No. of function function processing Cross-validation, Database hidden in in elements Testing) layers hidden output (Layer1,Layer2) Data % layers layer Iris 1 4 Tanh Tanh (60,15,25)% Body fat 1 4 Tanh Tanh (60,15,25)% Saginaw 2 8,4 Tanh Tanh (60,15,25)% Bay Corrosion 2 5,3 Tanh Tanh (70,15,15)% Admissions 2 15,10 Tanh Tanh (60,15,25)% Outages 1 3 Tanh Bias (60,15,25)% 57

3.3 Phase 3

The third phase of this thesis concentrates on analysis of TREPAN. It involves evaluating the performance of TREPAN in relation to different classification and regression tasks. The classification accuracy attained by TREPAN is studied by varying user specific parameters with the objective of identifying possible heuristics. The

TREPAN tool developed by Craven [4] consists of three available decision tree induction algorithms: Single test TREPAN, TREPAN and Disjunctive TREPAN. Table

3.13 identifies the parameters that can be altered for each of these algorithms.

Table 3.13: TREPAN Parameters

Disjunctive Single Test TREPAN TREPAN TREPAN Min sample Min sample Min sample Tree Size Tree Size Tree Size Beam Width Beam Width -

A total of 66 experiments having different combinations of the above mentioned parameters were run on each of the six datasets. Beam width values of 2, 3, 5, 7, 10 and minimum sample (min. sample) sizes of 1, 10, 50, 100, 500 and 1000 were used. The tree size that resulted in best classification accuracy was obtained by plotting a graph of tree node against the classification accuracy obtained at that node. These steps were repeated 58

for each algorithm with the objective of first identifying the best set of parameters among each algorithm and then the best model among the three algorithms. The flowchart in

Figure 3.4 shows the steps for the experimental runs. 59

Figure 3.4: TREPAN Experimentation

3.4 Phase 4

In this phase, the C4.5 decision tree induction algorithm developed by Quinlan is used to analyze the datasets. The J48 algorithm available in WEKA [63] based along the lines of the C4.5 algorithm is used for the analysis. This analysis involves preparation of data files in the ARFF format which is specially developed for use with the WEKA machine learning software. An ARFF format file consists of two sections, the header information and the data information. The header information contains a list of the attributes and their type: numerical, nominal, string or date, along with the name of the relation to be classified. The data information section lists each instance in the dataset on a single line with attribute values delimited by commas. Figure 3.5 gives an example of the Saginaw dataset in the ARFF format. 61

@RELATION CHL

@ATTRIBUTE Temp NUMERIC @ATTRIBUTE Secchi NUMERIC @ATTRIBUTE TSS NUMERIC @ATTRIBUTE TP NUMERIC @ATTRIBUTE SRP NUMERIC @ATTRIBUTE NO3 NUMERIC @ATTRIBUTE NH4 NUMERIC @ATTRIBUTE SiO2 NUMERIC @ATTRIBUTE PSiO2 NUMERIC @ATTRIBUTE CL NUMERIC @ATTRIBUTE POC NUMERIC @ATTRIBUTE DOC NUMERIC

@ATTRIBUTE class {Cl1, Cl2, Cl3, Cl4, Cl5}

@DATA 14.5,5.3,0.13,3.782,0.454,0.466,24.74,0.492,0.112,11.064,1.3,2.98,Cl1 11.2,6.1,0.47,6.24606675,0.1,0.720720288,37.03606015,1.339285714,0.079867333,14.56838488,0.3 1,3.02,Cl1 …………………………………………………………………………………………………….. ……………………………………………………………………………………………………… 17,8.3,0.806666667,2.111815562,0.120261888,0.269533298,21.68514412,1.285747339,0.13257807 3,6.875,0.15,1.99,Cl1

Figure 3.5: ARFF Format File -- Saginaw Bay

A detailed illustration of the ARFF format file is available in Witten and Frank

[63] . The data classified as cross-validation and training in neural network models is together used as the training data for the C4.5 decision tree induction algorithm. The C4.5 model is then compared to the best TREPAN model for classification accuracy and comprehensibility. 62

3.5 Performance measures

3.5.1 Classification Accuracy

Assessing a classifier’s performance is a very important aspect of machine learning. It is also essential for comparing different classifiers. The classification accuracy or error rate is the percentage of correct predictions made by the model over a data set. It is assessed using the confusion matrix. A confusion matrix is a matrix plot of predicted versus actual classes with all of the correct classifications depicted along the diagonal of the matrix. It gives the number of correctly classified instances, incorrectly classified instances and overall classification accuracy.

Consider a two class problem as shown in Table 3.14. The four possible outcomes in this case are the true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). As the names suggest a false positive is when a negative instance is incorrectly classified as a positive and false negative is when a positive instance is incorrectly classified as a negative. An example of a typical confusion matrix is shown in

Table 3.14. 63

Table 3.14: Confusion Matrix -- Example

Predicted Class Class Yes No

Yes TP FN

No FP TN Actual Class

The accuracy of the classifier is given by the formula,

(TP + TN) Accuracy(%) = ×100 (Eq. 3.2) (TP + FN + FP + TN)

A confusion matrix is a primary tool in visualizing the performance of a classifier.

However it does not take into account the fact that some misclassifications are worse than others. To overcome this problem we use a measure called the Kappa Statistic [64] which considers the fact that correct values in a confusion matrix are due to chance agreement. The Kappa statistic is defined as,

P(A) − P(E) kˆ = (Eq. 3.3) 1− P(E)

In this equation,

P(A) is the proportion of times the model values were equal to the actual value and,

P(E) is the expected proportion by chance.

For perfect agreement Kappa = 1. For example: a kappa statistic of 0.84 would imply that the classification process was avoiding 84% of the errors that a completely random classification would generate.

3.5.2 Comprehensibility

The comprehensibility of the tree structure decreases with the increase in tree size and complexity. The principle of Occam’s Razor states that “when you have two competing theories which make exactly the same predictions, the one that is simpler is the better” [65] . Thus, among two trees induced by different classification algorithms and having the same prediction accuracy, the one with fewer leaves, or in other words the simpler tree is usually preferred. 65

CHAPTER 4. RESULTS AND DISCUSSION

4.1 Investigate and Extend TREPAN

A set of instructions in C language were written to enable the user to input the number of classes pertaining to the problem. Experiments were conducted to validate the code. Five of the datasets analyzed in this thesis have classes greater than two and they have been analyzed using the modified TREPAN algorithm.

A 4-4-3 generalized feedforward network was trained on the iris database for

2000 epochs in order to investigate the ability of TREPAN in comprehending GFF networks. The ‘classify_using_network’ command was used to validate that TREPAN was producing correct outputs for the network. The network and weight files with explanation of parameters for iris dataset are listed in APPENDIX A.

4.2 Dataset Analysis

Summarized results of TREPAN and C4.5 analysis on all datasets are discussed in this chapter. All other detailed information is listed in the appendix.

4.2.1 Corrosion

An 11-5-3-1 MLP network with a hyperbolic tangent function achieved the best accuracy for the corrosion model. The model was trained for 20000 epochs and gave an r 66

(correlation co-efficient) value of 0.942. Figure 4.1 shows the graph of the actual output against the modeled neural network output.

1.2

1.1

0.9 %Inhibition Output Actual Output

0.8

0.7

0.6 0.60.70.80.91 1.11.2 Modeled Neural Network Output

Figure 4.1: Corrosion: Actual Output vs. Modeled Neural Network Output

The output was then divided into 5 classes for the purpose of classification. In order to learn from the trained neural network, TREPAN requires that the instances be provided to the program in normalized form. A set of 66 experiments were setup for 67

study that included 6 runs using single test TREPAN and 30 each using TREPAN and disjunctive TREPAN as shown in Table 4.1 and Table 4.2 respectively.

Table 4.1: Single Test TREPAN Experimental Setup

Single Test TREPAN Exp. No. Min. sample 1 1 2 10 3 50 4 100 5 500 6 1000 68

Table 4.2: TREPAN and Disjunctive TREPAN Experimental Setup

TREPAN & Disjunctive TREPAN Exp. No. Beam width Min. sample 1 2 1 2 2 10 3 2 50 4 2 100 5 2 500 6 2 1000 7 3 1 8 3 10 9 3 50 10 3 100 11 3 500 12 3 1000 13 5 1 14 5 10 15 5 50 16 5 100 17 5 500 18 5 1000 19 7 1 20 7 10 21 7 50 22 7 100 23 7 500 24 7 1000 25 10 1 26 10 10 27 10 50 28 10 100 29 10 500 30 10 1000 69

The tree size was set at an initial value of 50 for all the runs since trees larger than

50 would be difficult to comprehend. The value of classification accuracy obtained over the test set when each node is added to the tree was recorded during the experiment.

From these recorded values, the tree size at which maximum accuracy is achieved was obtained. Table 4.3 depicts these values for each run of single test TREPAN.

(Calculations based on Eq. 3.2) 70

Table 4.3: Corrosion: Single Test TREPAN -- Test Accuracies at each Node

Classification Accuracy Node Min Min Min Min Min Min sample 1 sample 10 sample 50 sample 100 sample 500 sample 1000

0 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 1 0.5000 0.5000 0.5000 0.3571 0.4286 0.4286 2 0.6429 0.6429 0.6429 0.3571 0.5000 0.5000 3 0.3571 0.3571 0.4286 0.3571 0.5000 0.5000 4 0.4286 0.4286 0.6429 0.3571 0.4286 0.5000 5 0.5714 0.6429 0.6429 0.3571 0.4286 0.5000 6 0.5000 0.5714 0.7143 0.3571 0.4286 0.5000 7 0.5000 0.5714 0.6429 0.3571 0.5000 0.3571 8 0.5000 0.5714 0.6429 0.2857 0.5000 0.3571 9 0.5714 0.6429 0.6429 0.3571 0.5000 0.3571 10 0.6429 0.6429 0.6429 0.3571 0.5000 0.3571 11 0.5714 0.7143 0.6429 0.3571 0.5000 0.3571 12 0.6429 0.7143 0.5000 0.3571 0.5000 0.4286 13 0.5714 0.7857 0.5000 0.3571 0.4286 0.4286 14 0.5000 0.7857 0.5000 0.3571 0.4286 0.4286 15 0.5000 0.7857 0.5000 0.3571 0.4286 0.4286 16 0.5714 0.7143 0.5000 0.3571 0.4286 0.4286 17 0.5714 0.7143 0.5000 0.3571 0.4286 0.4286 18 0.5714 0.6429 0.5000 0.3571 0.4286 0.4286 19 0.5000 0.6429 0.5000 0.3571 0.4286 0.4286 20 0.5000 0.6429 0.5000 0.3571 0.4286 0.4286 21 0.5000 0.6429 0.5000 0.3571 0.4286 0.4286 22 - 0.6429 0.5000 0.3571 0.4286 0.4286 23 - 0.6429 0.5000 0.3571 0.4286 0.4286 24 - 0.6429 0.5000 0.3571 - -

The bold values show the maximum accuracy obtained in the respective single test TREPAN run. Following the Occam’s Razor Theory, the highest classification accuracy corresponding to the least node size was selected as the best. A graph plot of the classification accuracy at the respective node is shown in Figure 4.2. 71

0.8000

0.7000

0.6000 Min sample 1 Min sample 10 Min sample 50 Min sample 100 0.5000 Min sample 500 Min sample 1000 Classification accuracy Classification

0.4000

0.3000

0.2000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Nodes

Figure 4.2: Corrosion: Single Test TREPAN -- Classification Accuracy vs. Tree Size

From this graph, the min. sample size that gives highest classification accuracy and tree size corresponding to that accuracy can be obtained.

The same sets of experiments were run for each beam width value pertaining to

TREPAN and disjunctive TREPAN algorithm. The nodes at which highest classification accuracies were obtained in each of three runs was determined from the plots of classification accuracies vs. tree size. Among the three algorithms the best classification 72

accuracy of 85.7% was obtained in the TREPAN run with beam width 5, min sample 1 and tree size 12. Table 4.4 gives the test accuracies at the respective nodes for runs having beam width 5. The tables for the remaining runs are listed in APPENDIX B.

Table 4.4: Corrosion: TREPAN (beam width 5) -- Test Accuracies at each Node

BEAM WIDTH = 5 Classification Accuracy Node Min Min Min Min Min Min sample 1 sample 10 sample 50 sample 100 sample 500 sample 1000 0 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 1 0.5714 0.5714 0.5714 0.5000 0.4286 0.3571 2 0.5000 0.5000 0.7143 0.5000 0.4286 0.3571 3 0.5000 0.5000 0.7143 0.5000 0.4286 0.2857 4 0.5714 0.5714 0.5714 0.5000 0.5000 0.2857 5 0.4286 0.4286 0.5714 0.5000 0.5000 0.2857 6 0.6429 0.6429 0.5000 0.5000 0.5000 0.3571 7 0.6429 0.7143 0.5714 0.5000 0.5000 0.3571 8 0.6429 0.7143 0.5714 0.5000 0.5000 0.3571 9 0.6429 0.7143 0.5714 0.5000 0.5714 0.3571 10 0.6429 0.7143 0.5714 0.5000 0.5000 0.3571 11 0.7143 0.8571 0.5714 0.5000 0.5000 0.3571 12 0.8571 0.8571 0.5714 0.5000 0.5000 0.3571 13 0.7857 0.8571 0.5714 0.5714 0.5000 0.3571 14 0.7857 0.8571 0.5714 0.5714 0.5000 0.3571 15 0.7857 0.7857 0.5714 0.5714 0.5000 0.3571 16 0.7143 0.6429 0.5714 0.5714 0.5000 0.3571 17 0.7143 0.6429 0.5714 0.5714 0.5000 0.3571 18 0.6429 0.7143 0.5714 0.5714 0.5714 0.3571 19 0.5714 0.7143 0.5714 0.5714 0.5714 0.3571 20 0.5714 0.7143 0.5714 0.5714 0.5714 0.3571 21 0.5714 0.7143 0.5714 0.5000 0.5714 0.3571 22 0.5714 0.7143 0.5714 0.5000 0.5714 0.3571 23 0.5714 0.7143 0.5714 0.5714 0.5714 0.3571 24 0.5714 0.7143 0.5714 0.5714 - 0.3571 25 0.5714 0.7143 - 0.6429 - - 26 0.5714 0.6429 - - - - 73

A graph plot of classification accuracy vs. tree node size for TREPAN runs with beam width 5 is shown in Figure 4.3.

0.9000

0.8000

0.7000

Min sample 1 0.6000 Min sample 10 Min sample 50 Min sample 100 Min sample 500 0.5000 Min sample 1000 Classification accuracy Classification

0.4000

0.3000

0.2000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Nodes

Figure 4.3: Corrosion: TREPAN (beam width 5)-Classification Accuracy vs. Tree Size

The results for TREPAN and disjunctive TREPAN runs are summarized in Table

4.5. 74

Table 4.5: Corrosion -- Summary of Experimental Runs

TREPAN Disjunctive TREPAN Exp. Beam Min Sample Tree Tree No. Width Test accuracy Test accuracy Size Size 1 2 1 10 71.40% 1 57.14% 2 2 10 13 64.20% 13 71.40% 3 2 50 7 64.20% 6 64.20% 4 2 100 1 50.00% 6 64.20% 5 2 500 14 57.14% 1 42.80% 6 2 1000 9 50.00% 13 57.14% 7 3 1 10 71.40% 1 57.14% 8 3 10 13 64.20% 13 71.40% 9 3 50 1 57.14% 6 64.20% 10 3 100 18 64.20% 1 50.00% 11 3 500 14 57.14% 1 42.80% 12 3 1000 1 42.80% 13 57.14% 13 5 1 12 85.70% 1 57.14% 14 5 10 11 85.70% 13 71.40% 15 5 50 2 71.40% 10 64.20% 16 5 100 13 57.14% 1 50.00% 17 5 500 9 57.14% 1 42.80% 18 5 1000 1 35.71% 13 57.14% 19 7 1 12 85.70% 4 64.20% 20 7 10 11 85.70% 15 71.40% 21 7 50 2 71.40% 9 64.20% 22 7 100 1 50.00% 1 50.00% 23 7 500 18 50.00% 1 42.80% 24 7 1000 1 42.80% 13 50.00% 25 10 1 3 71.40% 12 57.14% 26 10 10 3 78.50% 22 71.40% 27 10 50 2 78.50% 6 57.14% 28 10 100 14 57.14% 1 50.00% 29 10 500 20 50.00% 1 42.80% 30 10 1000 1 42.80% 6 42.80%

It should be noted that although min. sample 10 with beam width 5 also gives the highest accuracy of 85.7%, it fails to classify class Cl2 and is not chosen as the best 75

model. Table 4.6 shows the confusion matrix. The values along the diagonal of the matrix are the correct classification made by the model. For example: it is observed that out of 2 instances that have the output Cl2, one is correctly classified as Cl2 and one is misclassified as Cl1.

Table 4.6: Corrosion Confusion Matrix (TREPAN)

Actual/Predicted Cl1 Cl2 Cl3 Cl4 Cl5 Cl1 4 0 0 0 0 Cl2 1 1 0 0 0 Cl3 0 0 4 0 0 Cl4 0 0 0 0 1 Cl5 0 0 0 0 3 Classification Accuracy (%) 80.00%100.00%100.00%0.00% 75.00% Total Accuracy (%) 85.71%

Figure 4.4 shows the decision tree generated by TREPAN for the corrosion model. 76

Saturates <= 60.54 Resins <=5.30 TAN <=2.18 Crude oil <=34.99 API > 17.85 Crude oil <=10.5

f N o OT 3 3 of

S <= 2.48 Aromatics > 37.35 Cl5 Nitrogen <= 2013.13

N f OT 2 o 2 of

Saturates Crude oil

.4 > 63 63 > = .4 9 3 < .9 4 4 .9 3 9 = <

Crude oil Cl1 .5 > Cl4 10 10 S <= .5

5 > 0 4 . . 4 0 = 5 Cl1 Crude oil <

9 4.9 >6 =6 4. < 99 Cl1 Cl2

S API

7 > 5 0 > . . 5 2 0 5 6 5 = 7 . .6 < 5 5 2 = <

Cl2 Saturates Cl3 Cl4 >6 4 0 .5 .5 0 4 6 = <

Cl3 API

.4 >3 31 1. = 4 <

API Cl3

5 >2 .6 5 5 .6 2 5 = <

Cl2 Cl3

Figure 4.4: Corrosion: TREPAN Decision Tree 77

The data is now modeled using the C4.5 decision tree induction algorithm which generates a tree of size 11 and classification accuracy 57.14%. Table 4.7 shows the confusion matrix for the C4.5 model.

Table 4.7: Corrosion Confusion Matrix (C4.5)

Actual/Predicted Cl1 Cl2 Cl3 Cl4 Cl5 Cl1 3 2 0 0 0 Cl2 0 0 1 0 0 Cl3 0 1 2 0 1 Cl4 0 0 0 0 0 Cl5 0 0 0 1 3 Classification Accuracy (%) 100.00% 0.00% 66.67% 0.00% 75.00% Total Accuracy (%) 57.14%

The C4.5 model completely fails to predict class Cl2 as against the TREPAN model and has higher number of misclassifications resulting in a low overall classification accuracy of 57.14%. Figure 4.5 shows the decision tree generated by C4.5 for the corrosion model. 78

Crude oil

1 > <= 1

Saturates API

.3 > 15 1 = 5. .4 > < 3 2 1 1 2 = .4 <

Cl5 Saturates

.2 Cl2 62 > Cl1 = 6 < 2 .2

Crude oil Cl1 20 >2 <= 0

Asphaltenes API

> .6 6. > 6 .3 3 = 6 3 3 < 3 .3 = <

API Cl3 Asphaltenes Cl5 .3 > 3 3 3 3 = .3 > < 3 3 <=

Cl2 Cl3 Cl3 API

> .4 20 20 .4 = <

Crude oil Cl4

0 > 5 5 = 0 <

Cl2 Cl4

Figure 4.5: Corrosion: C4.5 Decision Tree 79

Although the tree sizes generated by the two algorithms C4.5 and TREPAN were comparable, the classification accuracy attained by TREPAN was significantly higher than that for C4.5. The Kappa statistic also validated that TREPAN generated a better model than C4.5.

4.2.2 Outages

Outages is a database from the small dataset category. A 12-3-1 MLP network with a hyperbolic tangent and bias axon transfer functions in the first and the second hidden layer respectively gave the best accuracy. The model was trained for 12000 epochs and achieved an r (correlation co-efficient) value of 0.985 (or an r2 of (0.985)2).

Figure 4.6 shows the graph of the actual output against the modeled neural network output. 80

20 Reportable Outages Actual Output 15

0 0 5 10 15 20 25 30 35 40 Modeled Neural Network Output

Figure 4.6: Outages: Actual Output vs. Modeled Neural Network Output

As stated earlier, the output is divided into 5 classes for classification. The best model for this dataset was generated by Single test TREPAN algorithm having a tree size

7, min. sample size 50 and a test set classification accuracy of 85.33%. This tree is then pruned by TREPAN to tree size 6. Table 4.8 gives the test accuracies at the respective nodes for the single test TREPAN run. The tables for the remaining runs are listed in

APPENDIX C. 81

Table 4.8: Outages: Single Test TREPAN -- Test Accuracies at each Node

Classification Accuracy Node Min Min Min Min Min Min sample 1 sample 10 sample 50 sample 100 sample 500 sample 1000 0 0.6533 0.6533 0.6533 0.6533 0.6533 0.6533 1 0.6533 0.6533 0.6533 0.6533 0.5733 0.5733 2 0.7333 0.7333 0.7333 0.7333 0.5733 0.5733 3 0.7867 0.7867 0.7867 0.7333 0.7333 0.6933 4 0.7733 0.7733 0.7733 0.7333 0.7333 0.6933 5 0.8267 0.8267 0.7733 0.7467 0.7467 0.7200 6 0.8267 0.8267 0.8267 0.7867 0.7600 0.7200 7 0.8133 0.8133 0.8533 0.8000 0.8133 0.7200 8 0.7733 0.7733 0.8133 0.7867 0.8133 0.7600 9 0.7600 0.7733 0.8000 0.7867 0.8133 0.7867 10 0.7600 0.7733 0.7467 0.7467 0.8133 0.8000 11 0.7467 0.7733 0.7733 0.7600 0.8000 0.8000 12 0.7467 0.7733 0.7733 0.7600 0.8000 0.8000 13 0.7333 0.7867 0.7733 0.7467 0.7467 0.8000 14 0.7333 0.7867 0.7733 0.7467 0.7467 0.8000 15 0.7733 0.7867 0.7733 0.7467 0.7467 0.8000 16 0.7733 0.7733 0.7600 0.7600 0.7467 0.8000 17 0.7733 0.7733 0.7600 0.7600 0.7733 0.8000 18 0.7733 0.8000 0.7600 0.7600 0.7733 0.7733 19 0.7733 0.8000 0.7467 0.7733 0.7733 0.7733 20 0.7733 0.8000 0.7467 0.7733 0.7733 0.7733 21 0.7733 0.8000 0.7467 0.7733 0.7733 0.7733 22 0.7733 0.8000 0.7467 0.7733 0.7733 0.7733 23 0.7733 0.8000 0.7467 0.7733 0.7733 0.7733 24 0.7733 0.8000 0.7467 0.7733 0.7733 0.7733 25 0.7733 0.8000 0.7467 0.7733 0.7733 0.7733 26 0.7733 0.8000 0.7467 0.7733 0.7733 0.7733 27 0.7733 0.8000 0.7467 0.7733 0.7733 0.7733 28 0.7733 0.8000 0.7467 0.7733 0.7733 0.7733 29 - 0.8000 0.7467 0.7733 0.7733 0.7733 30 - 0.8000 0.7467 0.7733 0.7733 0.7733 31 - 0.8000 0.7467 - 0.7733 0.7600 32 - - 0.7467 - 0.7733 0.7600 33 - - 0.7467 - 0.7733 0.7600 34 - - 0.7467 - 0.7733 0.7600 35 - - - - 0.7733 0.7600 36 - - - - 0.7733 0.7600 37 - - - - - 0.7600 38 - - - - - 0.7600 39 - - - - - 0.7600 40 - - - - - 0.7600 82

Table 4.8: Continued

Classification Accuracy

Node Min Min Min Min Min Min sample 1 sample 10 sample 50 sample 100 sample 500 sample 1000

41 - - - - - 0.7600 42 - - - - - 0.7600 43 - - - - - 0.7600

A graph plot of classification accuracy vs. tree node size for single test TREPAN runs is shown in Figure 4.7. 83

0.9000

0.8500

0.8000

Min sample 1 0.7500 Min sample 10 Min sample 50 Min sample 100

0.7000 Min sample 500 Min sample 1000 Classification accuracy Classification

0.6500

0.6000

0.5500

0 2 4 6 8 0 2 4 6 8 0 2 4 6 8 0 2 4 6 8 2 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 40 4 Nodes

Figure 4.7: Outages: Single Test TREPAN -- Classification Accuracy vs. Tree Size

The results for TREPAN and disjunctive TREPAN runs for outages database are summarized in Table 4.9 84

Table 4.9: Outages-Summary of Experimental Runs

TREPAN Disjunctive TREPAN Exp. Beam Min Sample Tree Tree No. Width Test accuracy Test accuracy Size Size 1 2 1 2 78.66% 2 78.67% 2 2 10 15 80.00% 15 80.00% 3 2 50 16 82.66% 2 78.67% 4 2 100 2 78.66% 2 78.67% 5 2 500 15 78.66% 28 81.33% 6 2 1000 15 78.66% 6 80.00% 7 3 1 2 77.33% 2 78.67% 8 3 10 14 80.00% 15 80.00% 9 3 50 9 82.66% 2 78.67% 10 3 100 11 78.67% 2 78.67% 11 3 500 28 78.67% 7 76.00% 12 3 1000 14 78.67% 6 80.00% 13 5 1 5 74.67% 9 78.67% 14 5 10 2 77.33% 18 81.33% 15 5 50 1 72.00% 13 78.67% 16 5 100 13 81.33% 19 77.33% 17 5 500 14 80.00% 26 76.00% 18 5 1000 14 78.67% 8 78.67% 19 7 1 7 78.67% 9 78.67% 20 7 10 7 78.67% 18 81.33% 21 7 50 11 77.33% 13 78.67% 22 7 100 8 78.67% 12 82.67% 23 7 500 24 81.33% 7 76.00% 24 7 1000 13 84.00% 6 80.00% 25 10 1 2 77.33% 9 78.67% 26 10 10 2 77.33% 18 81.33% 27 10 50 16 77.33% 13 78.67% 28 10 100 8 78.67% 9 78.67% 29 10 500 10 78.67% 7 76.00% 30 10 1000 9 73.33% 16 81.33%

The confusion matrix for outages (TREPAN) is shown in Table 4.10. 85

Table 4.10: Outages Confusion Matrix (TREPAN)

Actual/Predicted Cl1 Cl2 Cl3 Cl4 Cl5 Cl1 3 0 0 0 0 Cl2 4 48 5 0 0 Cl3 0 1 8 0 0 Cl4 0 0 1 5 0 Cl5 0 0 0 0 0 Classification Accuracy (%) 42.86% 97.96% 57.14% 100.00% 0.00% Total Accuracy (%) 85.33%

Figure 4.8 is a representation of the decision tree generated by TREPAN. 86

MTTFBSCBS

8 .15 > = 3 3.1 < 58

MTRBSCBS MTRBSCBS

> 67 4 7 .7 4. 6 7 > = 7 1 2 < .0 .0 2 1 = 7 <

G MTTFBSCBS G VG 7 > 6 1. .5 5 1 6 = 7 <

B MTTFBSCBS

> 9 2 0 .4 .4 0 2 9 = <

MTTFBS O

.7 >1 1 . <= 7

O G

Figure 4.8: Outages: TREPAN Decision Tree

The C4.5 analysis resulted in a classification accuracy of 76% as shown in the confusion matrix in Table 4.11. 87

Table 4.11: Outages Confusion Matrix (C4.5)

Actual/Predicted Cl1 Cl2 Cl3 Cl4 Cl5 Cl1 2 5 0 0 0 Cl2 3 43 3 0 0 Cl3 0 6 7 1 0 Cl4 0 0 0 5 0 Cl5 0 0 0 0 0 Classification Accuracy (%) 40.00% 79.63% 70.00% 83.33% 0.00% Total Accuracy (%) 76.00%

Figure 4.9 shows a graphical representation of the decision tree generated by

C4.5. 88

MTTFBSCBS

15 >3.15 <= 3.

MTRBSCBS MTRBSCBS <=4 >4 > 75 4.7 4. 67 <= MTTFBSCBS G < MTTFBSCBS MTTFBSCBS 3 =1 43 .43 1. 3 <= >1 5 .3 .3 5 1 < = 3 = < 1 3 . 5 5 MTTFBSCBS . 3 1 3 VG = > < 3 4. MTRBSCBS G 58 58 4. 3 <= > 3 3 3 .0 3 3 0 . 3 MTTRMSCBSC MTTFBSCBS 3 3 MTTFBS = MTTFDB < 3 > 3 7 8 . 3 > 6 < .4 4 2 1 = 7 8 3 . 6 2 2 = 3 8 0 1 < . . < 0 8 4 2 2 O 3 . 3 . = = 1 G 2 3 3 3 < 6 = = .4 < < 1 6 B O G G VG O MTTRBS MTTRBS 3 > 3 1 .1 . 1 1 = 3 < 3 = < 3 1 3 . 4 4 . 3 1 3

< G MTTFMSCBSC

< VG G = 5 4 .6 .6 4 5 <=

MTTFBSCBS G

33 > 2.7 2.7 <= 33

O MTTRBSC

5 > 1. 1 = .5 <

G O

Figure 4.9: Outages: C4.5 Decision Tree 89

4.2.3 Iris

A single layer MLP network with a configuration of 4-4-3 was trained for 2000 epochs on the training data. Hyperbolic tangent transfer function was used in the hidden layer. A classification accuracy of 89.47% was observed as shown in the confusion matrix in Table 4.12.

Table 4.12: Iris Confusion Matrix (Neural Network)

Actual/Predicted Iris Setosa Iris versicolor Iris Virginica Iris Setosa 14 0 0 Iris versicolor 0 11 2 Iris Virginica 0 2 9 Classification Accuracy (%) 100.00% 84.62% 81.82% Total Accuracy (%) 89.47%

The best TREPAN model was obtained by the single test TREPAN run with a min sample size of 1000, tree size 2 and classification accuracy of 94.74%. Table 4.13 gives the test accuracies at the respective nodes for the single test TREPAN run. The tables for the remaining runs are listed in APPENDIX D. 90

Table 4.13: Iris: Single Test TREPAN -- Test Accuracies at each Node

Classification Accuracy nodes Min Min Min Min Min Min sample 1 sample 10 sample 50 sample 100 sample 500 sample 1000 0 0.2895 0.2895 0.2895 0.2895 0.2895 0.2895 1 0.6579 0.6579 0.6579 0.6579 0.6579 0.6579 2 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 3 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 4 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 5 - - - 0.8947 0.8947 0.9474 6 - - - 0.8947 0.8947 -

A graph plot of classification accuracy vs. tree node size for single test TREPAN runs is shown in Figure 4.10. 91

1.0000

0.9000

0.8000

0.7000 Min sample 1 Min sample 10 Min sample 50 0.6000 Min sample 100 Min sample 500 Min sample 1000 0.5000 Classification accuracy Classification

0.4000

0.3000

0.2000 0123456 Nodes

Figure 4.10: Iris: Single Test TREPAN -- Classification Accuracy vs. Tree Size

The results for TREPAN and disjunctive TREPAN runs for iris database are summarized in Table 4.14

. 92

Table 4.14: Iris -- Summary of Experimental Runs

Exp. TREPAN Disjunctive TREPAN Beam Width Min Sample No. Tree Size Test accuracy Tree Size Test accuracy 1 2 1 2 89.40% 2 89.40% 2 2 10 2 89.40% 2 89.40% 3 2 50 2 89.40% 2 89.40% 4 2 100 2 89.40% 2 89.40% 5 2 500 2 89.40% 2 89.40% 6 2 1000 2 94.70% 2 94.70% 7 3 1 2 89.40% 2 89.40% 8 3 10 2 89.40% 2 89.40% 9 3 50 2 89.40% 2 89.40% 10 3 100 2 89.40% 2 89.40% 11 3 500 2 89.40% 2 89.40% 12 3 1000 2 94.70% 2 94.70% 13 5 1 2 89.40% 2 89.40% 14 5 10 2 89.40% 2 89.40% 15 5 50 2 89.40% 2 89.40% 16 5 100 2 89.40% 2 89.40% 17 5 500 2 89.40% 2 89.40% 18 5 1000 2 89.40% 2 94.70% 19 7 1 2 89.40% 2 89.40% 20 7 10 2 89.40% 2 89.40% 21 7 50 2 89.40% 2 89.40% 22 7 100 2 89.40% 2 89.40% 23 7 500 2 89.40% 2 89.40% 24 7 1000 2 89.40% 2 94.70% 25 10 1 2 89.40% 2 89.40% 26 10 10 2 89.40% 2 89.40% 27 10 50 2 89.40% 2 89.40% 28 10 100 2 89.40% 2 89.40% 29 10 500 2 89.40% 2 89.40% 30 10 1000 2 89.40% 2 94.70%

The confusion matrix generated for iris by TREPAN algorithm is shown in Table

4.15. 93

Table 4.15: Iris Confusion Matrix (TREPAN)

Actual/Predicted Iris Setosa Iris versicolor Iris Virginica Iris Setosa 14 0 0 Iris versicolor 0 9 0 Iris Virginica 0 2 13 Classification Accuracy (%) 100.00% 81.82% 100.00% Total Accuracy (%) 94.74%

C4.5 algorithm attains a comparatively lower accuracy of 86.84% as seen in the confusion matrix in Table 4.16.

Table 4.16: Iris Confusion Matrix (C4.5)

Actual/Predicted Iris Setosa Iris versicolor Iris Virginica Iris Setosa 13 1 0 Iris versicolor 0 12 1 Iris Virginica 0 3 8 Classification Accuracy (%) 100.00% 75.00% 88.89% Total Accuracy (%) 86.84%

A comparison between the decision trees generated by TREPAN and C4.5 can be seen in Figure 4.11. 94

Sepal width Sepal Length

>0.5 <=2.35067 >2.35067 <=0.5

Sepal width Sepal width Iris Setosa Iris Setosa <=4.84889 >4.84889 <=4.8 >4.8

Iris Versicolor Iris Virginica Iris Versicolor Iris Virginica

TREPAN C4.5

Classification Accuracy = 94.74% Classification Accuracy = 86.84%

Figure 4.11: Iris: Decision Tree Comparison

Although the size of the decision tree in the two cases is comparable, TREPAN achieves a significantly higher classification accuracy of 94.74% as compared to C4.5.

4.2.4 Body Fat

Body fat is a regression problem in the simple machine learning dataset category.

The instances are used to predict body fat percentage based on body characteristics. A

14-4-1 MLP with hyperbolic tangent function was used to train the network for 1500 epochs giving an r (correlation co-efficient) value of 0.9882. Figure 4.12 shows the graph of the actual output against the modeled neural network output 95

20 Percentage Body fat Actual Output 15

0 0 5 10 15 20 25 30 35 40 Modeled Neural Network Output

Figure 4.12: Body Fat: Actual Output vs. Modeled Neural Network Output

The output is divided into four classes toned, healthy, flabby and obese and the dataset is run through TREPAN algorithm changing various parameters to attain the best model. A best classification accuracy of 96.8% was obtained with model parameters being single test TREPAN, min. sample size 100 and tree size 3. Table 4.17 gives the test accuracies at the respective nodes for the single test TREPAN run. The tables for the remaining runs are listed in APPENDIX E. 96

Table 4.17: Body Fat: Single Test TREPAN -- Test Accuracies at each Node

Classification Accuracy Node Min Min Min Min Min Min sample 1 sample 10 sample 50 sample 100 sample 500 sample 1000 0 0.2222 0.2222 0.2222 0.2222 0.2222 0.2222 1 0.3651 0.3651 0.3651 0.5238 0.5238 0.5238 2 0.6508 0.6508 0.6508 0.6508 0.8413 0.8413 3 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 4 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 5 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 6 - - 0.9683 0.9683 0.9683 0.9683 7 - - - - 0.9683 -

A graph plot of classification accuracy vs. tree node size for single test TREPAN runs is shown in Figure 4.13. 97

0.9

0.8

0.7

Min sample 1 0.6 Min sample 10 Min sample 50 Min sample 100 0.5 Min sample 500 Min sample 1000 Classification accuracy Classification 0.4

0.3

0.2

0.1 01234567 Nodes

Figure 4.13: Body Fat: Single Test TREPAN-Classification Accuracy vs. Tree Size

The results for TREPAN and disjunctive TREPAN runs for body fat database are summarized in Table 4.18. 98

Table 4.18: Body Fat -- Summary of Experimental Runs

Exp. TREPAN Disjunctive TREPAN Beam Width Min Sample No. Tree Size Test accuracy Tree Size Test accuracy 1 2 1 3 96.80% 3 96.80% 2 2 10 3 96.80% 3 96.80% 3 2 50 3 96.80% 3 96.80% 4 2 100 3 96.80% 3 96.80% 5 2 500 3 96.80% 3 96.80% 6 2 1000 3 96.80% 3 96.80% 7 3 1 3 96.80% 3 96.80% 8 3 10 3 96.80% 3 96.80% 9 3 50 3 96.80% 3 96.80% 10 3 100 3 96.80% 3 96.80% 11 3 500 3 96.80% 3 96.80% 12 3 1000 3 96.80% 3 96.80% 13 5 1 3 96.80% 3 96.80% 14 5 10 3 96.80% 3 96.80% 15 5 50 3 96.80% 3 96.80% 16 5 100 3 96.80% 3 96.80% 17 5 500 3 96.80% 3 96.80% 18 5 1000 3 96.80% 3 96.80% 19 7 1 3 96.80% 3 96.80% 20 7 10 3 96.80% 3 96.80% 21 7 50 3 96.80% 3 96.80% 22 7 100 3 96.80% 3 96.80% 23 7 500 3 96.80% 3 96.80% 24 7 1000 3 96.80% 3 96.80% 25 10 1 3 96.80% 3 96.80% 26 10 10 3 96.80% 3 96.80% 27 10 50 3 96.80% 3 96.80% 28 10 100 3 96.80% 3 96.80% 29 10 500 3 96.80% 3 96.80% 30 10 1000 3 96.80% 3 96.80%

Table 4.19 shows the confusion matrix of the correct classifications and misclassifications made by TREPAN. 99

Table 4.19: Body Fat Confusion Matrix (TREPAN)

Actual/Predicted Toned Healthy Flabby Obese Toned 13 0 0 0 Healthy 1 21 0 0 Flabby 0 0 9 1 Obese 0 0 0 18 Classification Accuracy (%) 92.86% 100.00% 100.00% 94.74% Total Accuracy (%) 96.83%

C4.5 gives a classification accuracy of 92.06% as shown in the confusion matrix in Table 4.20.

Table 4.20: Body Fat Confusion Matrix (C4.5)

Actual/Predicted Toned Healthy Flabby Obese Toned 13 1 0 0 Healthy 1 20 0 0 Flabby 0 0 9 0 Obese 0 0 3 16 Classification Accuracy (%) 92.86% 95.24% 75.00% 100.00% Total Accuracy (%) 92.06%

The decision trees generated by TREPAN and C4.5 are shown in Figure 4.14. 100

Density

<=1.05361 >1.05361

Density Density

<=1.0418 >1.0418 <=1.07144 >1.07144

Obese Flabby Healthy Toned

TREPAN: Classification Accuracy = 96.8% Density

<=1.0529 >1.0529

Density Density

<=1.0412 >1.0412 <=1.0713 >1.0713

Obese Flabby Healthy Toned

C4.5: Classification Accuracy = 93.65%

Figure 4.14: Body Fat: Decision Tree Comparison

In this case as well, both TREPAN and C4.5 generate identical trees in terms of size but accuracy attained by TREPAN is comparatively higher. 101

4.2.5 Saginaw Bay

A 12-8-4-1 MLP with hyperbolic tangent function was used to train the neural network for 6000 epochs and a model having an r (correlation coefficient) value of

0.8971 was obtained. Figure 4.15 shows the graph of actual chlorophyll level output against the neural network model output.

CHL

15 Actual Output

0 0 5 10 15 20 25 30 35 Modeled Neural Network Output

Figure 4.15: Saginaw Bay: Actual Output vs. Modeled Neural Network Output 102

The best accuracy for the Saginaw Bay dataset was obtained by TREPAN having tree size 2; beam width 3, min. sample size 10 and a test set classification accuracy of

89.3%. But this model failed to classify range Cl3. Therefore it would not be acceptable for generalization purposes in case it was used for testing an unknown test sample.

Classification accuracy is therefore not considered the sole criteria in testing the validity of a model. The single test TREPAN algorithm with a tree size 15, min sample size 1000 and classification accuracy 88.11% proved to be the best model. The original tree is then pruned to a size of 12. Table 4.21 presents the test accuracies at the respective nodes for the single test TREPAN run. The tables for the remaining runs are listed in APPENDIX

F. 103

Table 4.21: Saginaw Bay: Single Test TREPAN -- Test Accuracies at each Node

Classification Accuracy Node Min Min Min Min Min Min sample 1 sample 10 sample 50 sample 100 sample 500 sample 1000 0 0.7828 0.7828 0.7828 0.7828 0.7828 0.7828 1 0.8648 0.8648 0.8648 0.8648 0.8648 0.8648 2 0.8279 0.8279 0.8279 0.8279 0.8238 0.8238 3 0.8279 0.8279 0.8279 0.8648 0.8443 0.8484 4 0.8279 0.8279 0.8279 0.8689 0.8484 0.8484 5 0.8484 0.8484 0.8484 0.8730 0.8607 0.8525 6 0.8443 0.8484 0.8484 0.8689 0.8607 0.8607 7 0.8443 0.8484 0.8484 0.8689 0.8607 0.8607 8 0.8566 0.8484 0.8197 0.8689 0.8607 0.8607 9 0.8566 0.8484 0.8156 0.8607 0.8607 0.8648 10 0.8607 0.8525 0.8156 0.8607 0.8607 0.8607 11 0.8525 0.8525 0.8156 0.8566 0.8566 0.8607 12 0.8525 0.8648 0.8156 0.8566 0.8443 0.8689 13 0.8525 0.8648 0.8197 0.8607 0.8443 0.8689 14 0.8525 0.8648 0.8197 0.8607 0.8525 0.8689 15 0.8525 0.8648 0.8197 0.8607 0.8525 0.8811 16 0.8361 0.8607 0.8197 0.8607 0.8525 0.8811 17 0.8361 0.8607 0.8197 0.8607 0.8525 0.8770 18 0.8361 0.8566 0.8156 0.8566 0.8525 0.8770 19 0.8361 0.8566 0.8156 0.8566 0.8525 0.8770 20 0.8361 0.8566 0.8238 0.8566 0.8525 0.8730 21 0.8361 0.8566 0.8238 0.8566 0.8525 0.8730 22 0.8361 0.8443 0.8361 0.8566 0.8566 0.8730 23 0.8361 0.8443 0.8361 0.8566 0.8566 0.8730 24 0.8361 0.8443 0.8361 0.8566 0.8566 0.8689 25 0.8361 0.8443 0.8402 0.8566 0.8566 0.8689 26 0.8361 0.8443 0.8484 0.8566 0.8689 0.8770 27 0.8361 0.8443 0.8484 0.8566 0.8689 0.8770 28 0.8361 0.8443 0.8484 0.8566 0.8689 0.8770 29 0.8361 0.8443 0.8484 0.8566 0.8689 0.8770 30 0.8361 0.8443 0.8484 0.8566 0.8689 0.8770 31 0.8361 0.8443 0.8525 0.8566 0.8689 0.8770 32 0.8361 0.8443 0.8525 0.8566 0.8689 0.8770 33 - 0.8443 0.8525 0.8566 0.8689 0.8770 34 - 0.8443 0.8525 0.8566 0.8689 0.8770 35 - 0.8443 0.8525 0.8566 0.8689 0.8770 36 - 0.8443 0.8525 0.8566 0.8689 0.8770 37 - 0.8443 0.8525 0.8566 0.8689 0.8770 38 - 0.8443 0.8525 0.8566 0.8689 0.8770 39 - 0.8484 0.8525 0.8566 0.8689 0.8770

104

Table 4.21: Continued

Classification Accuracy Node Min Min Min Min Min Min sample 1 sample 10 sample 50 sample 100 sample 500 sample 1000 40 - 0.8484 0.8525 0.8566 - - 41 - 0.8484 0.8525 0.8566 - - 42 - 0.8484 0.8525 0.8566 - - 43 - 0.8484 - - - -

A graph plot of classification accuracy vs. tree node size for single test TREPAN runs is shown in Figure 4.16.

0.91

0.89

0.87

Min sample 1 0.85 Min sample 10 Min sample 50 Min sample 100 0.83 Min sample 500 Min sample 1000 Classification accuracy

0.81

0.79

0.77

0 2 4 6 8 2 4 2 4 10 1 1 16 18 20 2 2 26 28 30 32 34 36 38 40 42 Nodes

Figure 4.16: Saginaw Bay: Single Test TREPAN-Classification Accuracy vs. Tree Size 105

The results for TREPAN and disjunctive TREPAN runs for Saginaw Bay database are summarized in Table 4.22.

Table 4.22: Saginaw Bay -- Summary of Experimental Runs

Exp. TREPAN Disjunctive TREPAN Beam Width Min Sample No. Tree Size Test accuracy Tree Size Test accuracy 1 2 1 1 86.40% 1 86.40% 2 2 10 21 86.80% 15 86.80% 3 2 50 1 86.40% 1 86.40% 4 2 100 7 87.70% 6 87.20% 5 2 500 1 86.40% 34 88.10% 6 2 1000 1 87.70% 1 88.50% 7 3 1 2 89.30% 1 86.40% 8 3 10 2 89.30% 15 86.80% 9 3 50 2 89.30% 1 86.40% 10 3 100 1 88.90% 6 87.20% 11 3 500 1 88.90% 1 86.40% 12 3 1000 12 88.90% 1 88.50% 13 5 1 8 88.50% 11 86.00% 14 5 10 8 88.50% 11 86.00% 15 5 50 12 88.10% 5 85.60% 16 5 100 22 88.50% 18 88.50% 17 5 500 7 89.30% 1 89.30% 18 5 1000 1 87.70% 1 88.50% 19 7 1 1 88.90% 4 88.10% 20 7 10 1 88.90% 4 88.10% 21 7 50 1 88.90% 23 88.90% 22 7 100 1 88.90% 18 88.50% 23 7 500 1 88.90% 1 89.30% 24 7 1000 1 87.70% 1 88.50% 25 10 1 10 88.10% 4 88.10% 26 10 10 7 88.10% 4 88.10% 27 10 50 2 87.70% 4 88.10% 28 10 100 2 87.70% 18 88.50% 29 10 500 2 87.70% 1 89.30% 30 10 1000 2 87.70% 1 88.50% . 106

The confusion matrix depicting various correct classifications and misclassifications is shown in Table 4.23.

Table 4.23: Saginaw Bay Confusion Matrix (TREPAN)

Actual/Predicted Cl1 Cl2 Cl3 Cl4 Cl5 Cl1 183 13 0 0 0 Cl2 8 29 3 1 0 Cl3 0 1 3 1 0 Cl4 0 1 1 0 0 Cl5 0 0 0 0 0 Classification Accuracy (%) 95.81% 65.91% 42.86% 0.00% 0.00% Total Accuracy (%) 88.11%

The decision tree generated by TREPAN is shown in Figure 4.17. 107

POC

.495 > <=1 1.495

PSiO2 POC

1 > 2 2 .0 .0 =2 2 1 < 1 2.3 >2 <= .31

Cl1 NH4 PSiO2 > POC 9 1 2 5 .0 . > 5 0 5 2 1 2 8 .4 > = 9 .4 8 5 3 < 5 27 .2 =2 3. 7 < <= 5

Cl2 Cl1 TSS PSiO2 PSiO2 POC

.2 > 9 > > 3 5 > 3 1 4 2 4 1 . 0 3 3 4 4 . 9 . 3 . = . 4 3 .9 < 2 . 0 3 3 9 4 2 5 = = = < < <

POC Cl1 Cl2 Cl3 Cl2 Cl3 Cl3 Cl4 9 >1 1. .9 <=

Secchi Cl2

5 >1 2 . .2 22 1 5 = <

Cl2 Cl1

Figure 4.17: Saginaw Bay: TREPAN Decision Tree

C4.5 algorithm achieved an accuracy of 86.47% as shown in Table 4.24. 108

Table 4.24: Saginaw Bay Confusion Matrix (C4.5)

Actual/Predicted Cl1 Cl2 Cl3 Cl4 Cl5 Cl1 176 14 1 0 0 Cl2 10 33 1 0 0 Cl3 0 4 2 1 0 Cl4 0 0 2 0 0 Cl5 0 0 0 0 0 Classification Accuracy (%) 94.62% 64.71% 33.33% 0.00% 0.00% Total Accuracy (%) 86.48%

C4.5 generated a decision tree of size 24 and is shown in Figure 4.18.

109

TP 18.8 <= >18.8

Secchi PSiO2 <=1.25 >1.25 =3.51 > < 3.51 NO3 Cl1 POC >0. POC 3 403 <= 0 4 1. .4 1.5 54 0 <= = > < 19 2 2. .1 = 9 NH4 < POC NH4 POC 9 > 9 9 <= > > .1 .1 3 4 9 9 9 3 . Temp .4 2. = 9 .6 .0 0 2 4 < 9 .6 4 3 Cl4 = 9 9 = < 3 < = < 8 >9. 9. 8 <= SiO2 Cl2 Temp Cl1 Cl3 Cl2 SRP Cl2 > > 34 3 9 8 >0. 3. .3 .5 .5 7619 Cl5 Cl3 = 4 8 9 < = 9 < 1 6 .7 0 = < Temp Cl1 Cl2 Cl2 SiO2 > .2 12.2 PSiO2 =12 > < 6 0 0 .0 0. 6 = > < 1 .2 SiO2 3 1 21 3 TP 1. <=0 <= 14 .614 > 0.6 Cl2 Cl1 24 <= .5 . 4 5 2 SiO2 = < Cl1 Cl1 NO3 >0 Cl3 > 3 .43 Cl2 0. .4 8 01 0 01 8 = 0. < <=

POC Cl1 Cl1 Cl2 =1.13 <= < 1.13

Temp 9 >20 Cl2 <=20. .9

Cl2 Cl1

Figure 4.18: Saginaw Bay: C4.5 Decision Tree 110

4.2.6 Admissions

The Ohio University admissions database model comprised of a 22-15-10-2 MLP network. Two hidden layers with the hyperbolic tangent transfer functions were used for modeling. A classification accuracy of 74.04% was obtained as shown in Table 4.25.

Table 4.25: Admissions: Confusion Matrix (Neural Network)

Actual/Predicted Yes (Attend OU) No (Not Attend OU) Yes (Attend OU) 1883 1192 No (Not Attend OU) 748 3651 Classification Accuracy (%) 71.57% 75.39% Total Accuracy (%) 74.04%

The best TREPAN model was obtained by the single test TREPAN run with a min sample size of 1000, tree size 25 and classification accuracy of 72.10%. Table 4.26 gives the test accuracies at the respective nodes for the single test TREPAN run. The tables for the remaining runs are listed in APPENDIX G. 111

Table 4.26: Admissions: Single Test TREPAN-Test Accuracies at each Node

Admissions: Single test TREPAN-Test accuracies at each node Classification Accuracy Node Min Min Min Min Min Min sample 1 sample 10 sample 50 sample 100 sample 500 sample 1000 0 0.6448 0.6448 0.6448 0.6448 0.6448 0.6448 1 0.6891 0.6891 0.6891 0.6891 0.6891 0.6891 2 0.6954 0.6954 0.6954 0.6954 0.6954 0.7097 3 0.6966 0.6966 0.6966 0.6966 0.6954 0.7097 4 0.6929 0.6929 0.6929 0.6929 0.6916 0.7060 5 0.6979 0.6979 0.6979 0.6979 0.6966 0.7110 6 0.6954 0.6954 0.6954 0.6954 0.6941 0.7085 7 0.6960 0.6960 0.6960 0.6960 0.6948 0.7091 8 0.7079 0.7079 0.7079 0.7079 0.7066 0.7066 9 0.7116 0.7116 0.7116 0.7116 0.7104 0.7104 10 0.7097 0.7097 0.7097 0.7097 0.7085 0.7122 11 0.7097 0.7097 0.7097 0.7097 0.7085 0.7122 12 0.7085 0.7085 0.7085 0.7085 0.7085 0.7110 13 0.7122 0.7122 0.7122 0.7122 0.7085 0.7060 14 0.7104 0.7104 0.7104 0.7104 0.7035 0.7097 15 0.7104 0.7104 0.7104 0.7104 0.7047 0.7097 16 0.7104 0.7104 0.7104 0.7104 0.7041 0.7097 17 0.7104 0.7104 0.7104 0.7104 0.7079 0.7091 18 0.7122 0.7122 0.7122 0.7122 0.7079 0.7091 19 0.7116 0.7116 0.7116 0.7116 0.7079 0.7104 20 0.7147 0.7147 0.7147 0.7147 0.7097 0.7097 21 0.7091 0.7091 0.7091 0.7091 0.7110 0.7166 22 0.7085 0.7085 0.7085 0.7085 0.7104 0.7172 23 0.7072 0.7072 0.7072 0.7072 0.7104 0.7172 24 0.7072 0.7072 0.7072 0.7072 0.7097 0.7204 25 0.7091 0.7091 0.7091 0.7091 0.7041 0.7210 26 0.7104 0.7104 0.7104 0.7104 0.7054 0.7160 27 0.7110 0.7110 0.7110 0.7110 0.7066 0.7166 28 0.7110 0.7110 0.7110 0.7110 0.7066 0.7129 29 0.7110 0.7110 0.7110 0.7110 0.7054 0.7166 30 0.7147 0.7147 0.7147 0.7147 0.7079 0.7166 31 0.7135 0.7135 0.7135 0.7147 0.7079 0.7166 32 0.7135 0.7135 0.7135 0.7147 0.7079 0.7172 33 0.7135 0.7135 0.7135 0.7135 0.7079 0.7179 34 0.7135 0.7135 0.7135 0.7104 0.7079 0.7179 35 0.7110 0.7110 0.7110 0.7072 0.7085 0.7179 36 0.7079 0.7079 0.7079 0.7047 0.7079 0.7172 37 0.7085 0.7085 0.7085 0.7029 0.7097 0.7172 38 0.7066 0.7066 0.7066 0.7035 0.7097 0.7172 112

Table 4.26: Continued

Admissions: Single test TREPAN-Test accuracies at each node Classification Accuracy Node Min Min Min Min Min Min sample 1 sample 10 sample 50 sample 100 sample 500 sample 1000 39 0.7060 0.7060 0.7060 0.7054 0.7097 0.7191 40 0.7060 0.7060 0.7060 0.7054 0.7097 0.7210 41 0.7041 0.7041 0.7041 0.7047 0.7091 0.7191 42 0.7054 0.7054 0.7054 0.7079 0.7085 0.7185 43 0.7091 0.7091 0.7091 0.7072 0.7085 0.7204 44 0.7085 0.7085 0.7091 0.7079 0.7072 0.7204 45 0.7085 0.7085 0.7091 0.7079 0.7072 0.7204 46 0.7091 0.7091 0.7097 0.7091 0.7072 0.7204 47 0.7091 0.7091 0.7097 0.7091 0.7085 0.7204 48 0.7091 0.7091 0.7097 0.7091 0.7085 0.7204 49 0.7091 0.7091 0.7091 0.7085 0.7085 0.7204 50 0.7085 0.7085 0.7085 0.7097 0.7085 0.7204

A graph plot of classification accuracy vs. tree node size for single test TREPAN runs is shown in Figure 4.19. 113

0.73

0.72

0.71

0.7

Min sample 1 0.69 Min sample 10 Min sample 50 Min sample 100 0.68 Min sample 500 Min sample 1000 Classification accuracy Classification 0.67

0.66

0.65

0.64

0 2 4 6 8 0 2 4 8 0 2 4 6 8 0 2 4 6 8 0 4 6 8 0 1 1 1 16 1 2 2 2 2 2 3 3 3 3 3 4 42 4 4 4 5 Nodes

Figure 4.19: Admissions: Single Test TREPAN-Classification Accuracy vs. Tree Size

The results for TREPAN and disjunctive TREPAN runs for admissions database are summarized in Table 4.27. 114

Table 4.27: Admissions-Summary of Experimental Runs

Exp. TREPAN Disjunctive TREPAN Beam Width Min Sample No. Tree Size Test accuracy Tree Size Test accuracy 1 2 1 14 71.10% 18 71.10% 2 2 10 14 71.10% 18 71.10% 3 2 50 14 71.10% 18 71.10% 4 2 100 70 71.40% 18 71.10% 5 2 500 12 71.20% 28 71.60% 6 2 1000 12 71.20% 21 70.70% 7 3 1 14 71.10% 18 71.10% 8 3 10 14 71.10% 18 71.10% 9 3 50 14 71.10% 18 71.10% 10 3 100 70 71.40% 18 71.10% 11 3 500 12 71.20% 28 71.60% 12 3 1000 12 71.20% 21 70.70% 13 5 1 22 71.70% 18 71.10% 14 5 10 22 71.70% 18 71.10% 15 5 50 22 71.70% 18 71.10% 16 5 100 22 71.40% 18 71.10% 17 5 500 22 71.90% 28 71.60% 18 5 1000 92 72.10% 21 70.70% 19 7 1 12 71.70% 18 71.10% 20 7 10 12 71.03% 18 71.10% 21 7 50 12 71.03% 18 71.10% 22 7 100 12 71.03% 18 71.10% 23 7 500 84 71.60% 28 71.60% 24 7 1000 92 72.50% 21 70.70% 25 10 1 12 71.40% 13 71.30% 26 10 10 14 70.60% 13 71.30% 27 10 50 14 70.60% 13 71.30% 28 10 100 68 71.40% 13 71.30% 29 10 500 14 70.10% 13 71.30% 30 10 1000 14 71.10% 24 72%

The confusion matrix for Admissions (TREPAN) is shown in Table 4.28. 115

Table 4.28: Admissions: Confusion Matrix (TREPAN)

Actual/Predicted Yes No Yes 401 279 No 168 754 Classification Accuracy (%) 70.47% 72.99% Total Accuracy (%) 72.10%

A graphical representation of the tree corresponding to accuracy 72.10% is shown in Figure 4.20. 116

No No Yes No 8 Decision 1 >5 Yes Day

< 7 = 2 5 4 1 > 8 App day Yes e o e re w h n r Yes < T T u No = O o 4 F 2 0 7 3 8 4 2 Decision > 2 No Day HS size > < = <= e 4 9 2 iv 3 6 2 Race F 0 3 8 Yes 1 > 4 App day 4 > Six Yes < = 1 Z 3 er 6 o 9 Yes HS rank Yes 0 No

Decision No Day Yes 1 C 34 N > < U App day =

4 < =3 1 41 Yes U ED HS H Yes College FAR No

ENT Sex F HT C M C O M Yes Yes

A & S No Yes Yes No No ro Ze C One B

A wo T 7 No 7 Race 3 Three > App day Fo ur < F Yes 9 = 7 i 3 v 3 7 e Six = 7 Decision < Day <=3 No Yes HS rank 2 7 No 2 9 0 3 > No

1 App day < No = 8 32 7 1 Sex F HS size > 2 HS size No >467 < M <= = 4 17 6 7 Decision 8 Yes Day No Yes >402 App day 43 < <=3 = 4 > 0 3 2 4 Yes No 3 Decision App day 363 >363 Day > < <= =3 3 63 63 Yes No

Figure 4.20: Admissions: TREPAN Decision Tree 117

C4.5 achieved an accuracy of 71.9%, almost equal to that of TREPAN, but resulted in a significantly large decision tree having 78 nodes and 157 leaves. A text representation of the tree is listed in APPENDIX G. The confusion matrix is shown in

Table 4.29.

Table 4.29: Admissions: Confusion Matrix (C4.5)

Actual/Predicted Yes No Yes 379 190 No 259 774 Classification Accuracy (%) 59.40% 80.29% Total Accuracy (%) 71.97% 118

CHAPTER 5. CONCLUSIONS AND FUTURE RESEARCH

5.1 Summary and Discussion

The six applications studied here are:

• Identifying the class of the Iris plant

• Classifying body fat percentage

• Predicting the level of chlorophyll in Saginaw Bay

• Predicting percentage inhibition for corrosion

• Identifying students that attend Ohio University and

• Evaluating survivability of wireless networks

Table 5.1 shows the accuracies as percentage of correct classifications obtained over the test set, tree size and kappa statistic values in the six problem domains. 119

Table 5.1: Summary of Results

Classifier No. of No. of Kappa Problem Domain Method accuracy Nodes leaves Statistic TREPAN 94.70% 2 3 0.815 Iris C4.5 94.70% 2 3 0.801 TREPAN 96.80% 3 4 0.956 Body fat C4.5 92.06% 3 4 0.892 TREPAN 85.70% 12 13 0.808 Corrosion C4.5 57.14% 11 12 0.44 TREPAN 85.33% 6 7 0.689 Outages C4.5 76.00% 17 18 0.513 TREPAN 89.30% 12 13 0.651 Saginaw C4.5 84.67% 23 24 0.629 TREPAN 72.10% 23 41 0.416 Admissions C4.5 71.90% 78 157 0.404

5.1.1 Accuracy

It is observed that excluding the iris database, TREPAN performs considerably better at classifying instances in terms of accuracy. For iris the classification accuracies in both cases are equal. This may be due to the fact that iris is a comparatively simple database and can more easily identify the underlying relationships among the attributes. It is interesting to note that in case of iris TREPAN surpassed the accuracy of the neural network itself from which it learned. This can be attributed to the ability of TREPAN to generate more data during the induction process.

The most significant difference in classification accuracy can be seen in the corrosion dataset. TREPAN was able to approximate the neural network averaging 120

fidelity of over 90%. The kappa statistic which is a better indicator of a classifier’s performance than classification accuracy showed higher values for all models of

TREPAN than C4.5.

The accuracies attained by both algorithms in the admissions case are nearly equal but there is a considerable difference in the tree size, the TREPAN tree being smaller than C4.5. The TREPAN confusion matrix is also more balanced in predicting the two classes than C4.5.

5.1.2 Comprehensibility

Although C4.5 achieved equal or nearly equal accuracies to TREPAN in two cases (iris and admissions), it generated larger incomprehensible trees compared to

TREPAN. The decision tree generated by C4.5 for admissions database had over three times the number of nodes as the TREPAN tree.

5.2 Heuristics

The complexity of the database is defined here by comparing the multiple regression (R2) values for the datasets analyzed. A regression database with an R2 of 0.85 or higher is considered simple; R2 between 0.70 and 0.85 is considered complex and that with R2 of 0.70 and lower is considered as a highly complex database. The datasets fall into three categories based on size. A database with 200 instances or less is taken as small; number of instances between 200 and 1000 is considered medium and that with 121

number of instances larger than 1000 is considered to be a large database. The empirical investigation of TREPAN on these datasets reveals the following heuristics.

• A minimum sample (min. sample) size of 50 or less should be used for small

databases. A min. sample size higher than 50 results in low model accuracy.

• Best models are obtained by setting beam width values in a range from 2 to 7.

• If a high min. sample value is selected, a high beam width value should be

selected for better model accuracy.

• Beam width values higher than 7 do not result in good model accuracy.

• For complex and highly complex databases, best model accuracy is obtained

within a tree size range of number of inputs ± 5. The number of inputs mentioned

here refers to the total number of processing elements in the input layer of the

neural network after symbolic attributes have been translated.

• TREPAN generalizes better at lower min. sample sizes for highly complex

databases having little data. This can be attributed to the fact that TREPAN

generates instances and obtains class labels for those instances from the oracle

when there is not enough data (min. sample) to complete a split. In case of highly

complex databases with little data, TREPAN’s assumption of the data distribution

may be incorrect resulting in generation of bad data and hence lower classification

accuracy. The same does not hold true for highly complex databases having large

amounts of data. For such datasets, a higher min. sample size should be used as 122

there are enough instances available which increase the probability of fitting a

correct distribution

• It is observed that single test TREPAN and TREPAN algorithm perform better

than disjunctive TREPAN on most occasions.

5.3 Conclusions

The TREPAN algorithm code was modified to be able to work with multi-class regression type problems. Various experiments were run to investigate its compatibility with generalized feed forward networks. The weights and network files were restructured to present GFF networks in a format recognizable by TREPAN.

Six problem domains consisting of a representative mix of standard machine learning, small and large real world datasets were analyzed in this thesis. Neural network models were trained on each dataset varying parameters like network architecture and transfer functions.

The weights and biases obtained from the trained models of the six datasets were fed to TREPAN for decision tree learning from neural networks. Best models from among the three available TREPAN algorithms were selected. These results were compared to the results obtained from the benchmark decision tree induction algorithm

C4.5. In case of the problems discussed here, TREPAN algorithm significantly outperformed the C4.5 algorithm in terms of classification accuracy. To back the insufficiency of using classification accuracy as the only measure of performance, the results were also compared using the kappa statistic. The kappa statistic values further 123

validated the conclusion that TREPAN is a better tool at decision tree induction than the most commonly used algorithm, C4.5. The empirical investigation of TREPAN reveals certain insights into the TREPAN algorithm. Heuristics based on dataset size and complexities for obtaining the best TREPAN model were reported.

TREPAN facilitates combining the power of the highly incomprehensible neural network black box and simple to understand decision trees. By tapping the relationships learned by a neural network in the form of comprehensible decision trees it is possible to acquire “the best of both worlds”.

5.4 Future Research

One of the contributions of this research was to modify TREPAN to enable analysis of multi-class regression type problems. Although this modification was successfully implemented there is scope for improvement in this area. The regression problems in this thesis were represented as classification problems by converting the output range into a number of classes. The ranges for the classes were of equal size determined either by judgment or expert opinion which increases possibilities of error.

One area for future research is to extend TREPAN along the lines of CART

(Classification and regression trees) [66] to enable extraction of regression trees. CART uses binary recursive partitioning to estimate real-values functions (regression problems) at the leaf nodes.

A disadvantage of TREPAN is that the trees generated have normalized values.

Denormalizing these values to comprehensible form is a cumbersome task. An algorithm 124

can be incorporated into TREPAN that denormalizes the values and provides them to the user. 125

REFERENCES

[1] Cassily, J. F., Theory of sequentially timed learning, available at http://www.accd.edu/sac/psychol/2301.090/sequentlearn.htm.

[2] Nilsson, N. J., (1996), Draft of Incomplete notes on Machine Learning., available at http://ai.stanford.edu/people/nilsson/mlbook.html.

[3] Schmitz, G. P.J., Aldrich, C., & Gouws, F. S. (1999). ANN-DT: An algorithm for extraction of decision trees from artificial Neural Networks. IEEE Transactions on Neural Networks, 10(6), 1392-1401.

[4] Craven, M. W., and Shavlik, J. W., (1996) Extracting tree-structured representations of trained networks. In Advances in Neural Information Processing Systems, volume 8, pages 24–30, Denver, CO, MIT Press.

[5] Tickle A. B., Andrews R., Golea M., Diederich J. (1998). The truth will come to light: Directions and challenges in extracting the knowledge embedded within trained artificial neural networks, IEEE Trans. Neural Networks, Nov., 9 (6): 1057–1068.

[6] Waibel A. H., (1989) Modular construction of time-delay neural networks for speech recognition, Neural computation, 1: 39 – 46.

[7] Sejnowski T. J., Rosenberg C. R., (1986) A parallel network that learns to read aloud. Technical report JHU/EECS-86/01, John Hopkins Univerity.

[8] Jabri M., Pickard S., Leong P., (1992) ANN based classification for heart defibrillators. In Moody J. E., Hanson S. J.,Lippmann R., P., (Eds.), Advances in neural information processing systems, pp 637-644, San Mateo, CA.

[9] Tesauro G. J., (1992) Practical issues in temporal difference learning, Machine Learning, 8.

[10] Hsu G. T., Simmons R., (1991) Learning football evaluation for a walking robot. In proceedings of eighth international workshop on Machine learning, Jun, pp 303-307, Evanston, IL.

126

[11] Shavlik J. W., Mooney R. J., Towell G. G. (1991) Symbolic and Neural Net Learning Algorithms: An Empirical Comparison, Machine Learning, 6: 111– 143.

[12] Lohninger, H., Teach/Me Data Analysis, (1999). Springer-Verlag, Berlin-New York-Tokyo.

[13] Aldrich C. (2002). Exploratory analysis of metallurgical process data with neural networks and related methods, Elsevier.

[14] StatSoft, Inc. (2004). Electronic Statistics Textbook. Tulsa, OK: StatSoft.

[15] Nelson M. M., Illingworth W. W., (1992). A Practical Guide to Neural Nets. 4th ed. Reading, MA: Addison-Wesley Publishing Company.

[16] Rumelhart, D. E., Hinton, G. E., Williams, R. J. (1986). Learning representations by backpropagation errors. Nature, 323: 533-536.

[17] Plunkett, K., Elman, J., L. (1997). Exercises in Rethinking Innateness: A handbook for connectionist simulations, MIT Press, pp. 166.

[18] Towell G. G., Shavlik J. W. (1993). Extracting Refined Rules from Knowledge- Based Neural Networks, Machine Learning, 13: 71–101.

[19] Craven M. W, Shavlik J. W. (1995). Using sampling and queries to extract rules from trained neural networks, Machine Learning. Proceedings of the Eleventh Inter-national Conference, Cohen W. W & Hirsh H. (Eds.), San Francisco, CA: Morgan Kaufmann.

[20] Fu L. (1991). Rule learning by searching on adapted nets, In Proceedings of the 9th National Conference on Artificial Intelligence, Anaheim, CA, pp.590-595.

[21] Thrun S. (1995). Extracting rules from artificial neural networks with distributed representations, In Tesauro G.,Touretzky D., and Leen T. eds. Advances in Neural Information Processing Systems 7, Cambridge, MA:MIT Press, pp.505- 512.

[22] Craven, M. W., (1996) Extracting Comprehensible models from trained Neural Networks, PhD Thesis, Computer Science Department, University of Wisconsin, Madison, WI.

127

[23] Andrews R., Diederich J., Tickle A. B. (1995). A survey and critique of techniques for extracting rules from trained neural networks. Knowledge Based Systems, 8(6), 373-389.

[24] Tickle A. B., Orlowski M., and Diederich J. (1996). DEDEC: a methodology for extracting rule from trained artificial neural networks. In: Proceedings of the AISB’96 Workshop on Rule Extraction from Trained NeuralNetworks, Brighton, UK, pp.90-102.

[25] Biggs, D., de Ville, B. and Suen, E. (1991) A method of choosing multiway partitions for classification and decision tree. Journal of Applied Statistics 18(1): 49-62.

[26] Liepins, G., Goeltz, R. and Rush, R. (1990) Machine learning techniques for natural resource data analysis. AI Applications 4(3): 9-18.

[27] Leech, W. J. (1986) A rule based process control method with feedback. Advances in Instrumentation, 41: 169-175.

[28] Langley P., Simon, H. A. (1995) Applications of machine learning and rule induction, Communications of the ACM, Nov., 38(11): 54-64.

[29] Michie D. (1989) Problems of computer-aided concept formation, In Quinlan, J.R., (Ed). Applications of Expert Systems Volume 2. Wokingham, UK: Addison-Wesley, 310-333.

[30] Evans, B. and Fisher, D. (1994). Overcoming process delays with decision tree induction, IEEE Expert, 9: 60-66.

[31] Ercil A. (1993). Classification trees prove useful in non destructive testing of spot weld quality, Welding Journal, Sept., Issue title: Special emphasis: Rebuilding America’s roads, railways and bridges, 72(9): 59.

[32] Kennedy D. M. (1993). Decision tree bears fruit, Products Finishing, July, 57(10): 66.

[33] Irani K. ., Jie C., Fayyad U. M., Zhaogang, Q. (1993). Applying machine learning to semiconductor manufacturing, IEEE Expert, Feb., 8(1): 41-47.

[34] Das S. K., Bhambri S. (1994). A decision tree approach for selecting between demand based, reorder and JIT/Kanban methods for material procurement, Production Planning and Control, 5(4): 342. 128

[35] Famili A. (1994) Use of Decision Tree Induction for Preocess Optimization and Knowledge Refinement of an Industrial Process, Artificial Intelligence for Engineering Design, Analysis and Manufacturing (AI EDAM), Winter, 8(1): 63- 75.

[36] Piramuthu S., Raman N., Shaw M. J. (1994). Learning-based scheduling in a flexible manufacturing flow line, IEEE Trans. on Engineering Management, May, 41(2): 172-182.

[37] Riddle P., Segal R., Etzioni O. (1994). Representation, Design and Brute-force Induction in a Boeing manufacturing domain, Applied Artificial Intelligence, Jan.-Mar., 8(1):125-147.

[38] Guilfoyle, C. (1986). T en minutes to lay the foundations. Expert Systems User, Aug., 16-19.

[39] Guo Y., Dooley K. J. (1994). Distinguishing between mean, variance and autocorrelation changes in statistical quality control, International Journal of Production Research, Feb., 33(2): 497-510.

[40] Ganduri, C., G. (2004). Rule driven job shop scheduling derived from neural networks through extraction, M. S. Thesis, Department of Industrial Engineering, Ohio University, Athens, Ohio.

[41] Murthy S. K. (1998). Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey, Data Mining and Knowledge Discovery, Dec., 2(4): 345-389.

[42] Pietersma D., Lacroix R., Lefebvre D., Wade K., M. (2002) Decision tree induction to predict lactation curves, Canadian Biosystems Engineering, 44: 7.1.

[43] Quinlan, J. R. (1993). C4.5: Programs in machine learning. San Mateo, CA: Morgan Kaufmann.

[44] Quinlan, J. R.(1986), Induction of Decision trees, Machine Learning, 1:81-106.

[45] Hunt, E., Marin, J. and Stone, P., Experiments in Induction, New York, Academic Press, 1966.

[46] Shannon, C. E., (1948) A mathematical theory of communication, Bell System Technical Journal, vol. 27, pp. 379-423 and 623-656, July and October, 1948. 129

[47] Chen, F., (2004) Learning accurate and understandable rules from SVM classifiers, M. Sc. Thesis, School of computing science, Simon Fraser University.

[48] Baesens, B., Setiono, R., Mues, C., Vanthienen, J (2003). “Using Neural Network Rule Extraction and Decision Tables for Credit-Risk Evaluation.” Management Science, v. 49 issue 3, p. 312.

[49] Hudson, B., Whitley, D., Ford, M., Browne, A., (2003). Biological Data Mining : A comparison of Neural Network and Symbolic Techniques,Technical Report. BBSRC/EPSRC bioinformatics initiative, grant reference 322/BIO12005.

[50] P. M. Murphy. (1995). UCI repository of machine learning databases – a machinereadable data repository, maintained at the Department of Information and Computer Science, University of California, Irvine., anonymous FTP from ics.uci.edu in the directory pub/machinelearningdatabases.

[51] International Society for Ecological Informatics at http://www.waite.adelaide.edu.au/ISEI/.

[52] Recknagel, Friedrich, (2003), Ecological Informatics: Understanding Ecology by Biologically-Inspired Computation, New York, NY., Springer.

[53] Hernandez, S., Weckman, Nesic, S., G., Ghai, V., (2005). Use of artificial neural networks for predicting crude oil effect on CO2 corrosion of carbon steels, Corrosion, submitted and accepted for publication, April 2006.

[54] Hernandez, S., Duplat, S., Vera, J. R., Baron, E., (2002) A statistical approach for analyzing the inhibiting effects of different types of crude oil in CO2 corrosion of carbon steel, Corrosion, paper no. 02293, NACE, Denver, CO, pp 1-15.

[55] Bean, J. P. (1982). Student attrition, intentions, and confidence: Interactions effects in a path model. Research in Higher Education, 17, 291–319.

[56] Cabrera, A. F., Nora, A., & Castaneda, M. B. (1993). College persistence: Structural equations modeling test of an integrated model of student retention. Journal of Higher Education, 64(2), 123–138.

[57] Milem, J. F., & Berger, J. B. (1997). A modified model of college student persistence: Exploring the relationship between Astin’s theory of involvement 130

and Tinto’s theory of student departure. Journal of College Student Development, 38(4), 387–400.

[58] Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition. Chicago: The University of Chicago Press.

[59] Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research. Review of Educational Research, 45,89–125.

[60] Glynn, J. G., Sauer, P. L., Miller, T. E., (2003). Signaling student retention with prematriculation data, NASPA Journal, vol. 41, 1: 41-67.

[61] Rastogi, P., (2005) Assessing Wireless Network Dependability Using Neural Networks, M S., Thesis., M. S. Thesis, School of Communication, Ohio University, Athens, Ohio

[62] NeuroSolutions, (1995). Software developed and distributed by Neurodimension Incorporated [http://www.neurosolutions.com/products/ns/].

[63] Witten I., Frank E., Data Mining Practical Machine learning tools and techniques with Java Implementations, Morgan Kaufmann.

[64] Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.

[65] Thorburn, W. M. (1915). "Occam's razor", Mind, 24, pp. 287-288. available at http://math.ucr.edu/home/baez/physics/General/occam.html.

[66] Breiman, L., Friedman, J., Olshen, R., Stone, C. (1984). Classification and regression trees, Wadsworth and Brooks, Monterey, CA.

131

APPENDIX A: WEIGHTS AND NETWORK FILE

FORMATS (GFF)

Figure: A1: GFF Iris Network file 132

GFF Iris weights file

Weights Comments

-0.792494361 1.052210033 -1.785709505 -1.787385288 0.65478535 -0.830849198 1.483910905

1.945505103 Hidden Layer weights 0.593849347 -0.157927765 -0.130315699 -0.300207928 -0.042067209 0.286189009 0.06069945 -0.199078835 0.654000000 -0.251000000 Input layer node1 to output layer node1 1.170000000 1.540000000 -2.095875987 0.593051659 Hidden layer node1 to output layer node1 0.320628702 -0.131756369 0.21200000 0.19900000 Input layer node1 to output layer node2 -0.74100000 -0.60900000 0.276439692 -0.922378357 Hidden layer node1 to output layer node2 0.418258013 0.235561247 -0.181000000 0.050600000 Input layer node1 to output layer node3 -0.539000000 -0.819000000 133

GFF Iris weights file (Continued)

Weights Comments

2.037420291 2.762026753 Hidden layer node1 to output layer node3 1.325335132 0.563507107 0 0 0 0 1.224547503 1.454678887 Biases in order of occurrence -1.094480889 -0.399863754 -1.368850344 -0.287568021 -0.865051864 134

APPENDIX B: CORROSION RESULTS

Corrosion: TREPAN (beam width 2)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 1 0.5714 0.5714 0.5714 0.5000 0.4286 0.4286 2 0.5000 0.5000 0.4286 0.5000 0.3571 0.3571 3 0.5714 0.5714 0.5714 0.5000 0.3571 0.3571 4 0.3571 0.3571 0.5714 0.4286 0.3571 0.3571 5 0.4286 0.3571 0.5714 0.4286 0.3571 0.3571 6 0.5714 0.5000 0.5714 0.4286 0.4286 0.4286 7 0.5000 0.4286 0.6429 0.4286 0.4286 0.4286 8 0.5714 0.5000 0.6429 0.3571 0.4286 0.4286 9 0.6429 0.5000 0.6429 0.3571 0.4286 0.5000 10 0.7143 0.4286 0.6429 0.3571 0.4286 0.5000 11 0.7143 0.5000 0.6429 0.3571 0.4286 0.5000 12 0.7143 0.5714 0.6429 0.3571 0.4286 0.5000 13 0.7143 0.6429 0.6429 0.3571 0.4286 0.5000 14 0.7143 0.6429 0.6429 0.3571 0.5714 0.5000 15 0.6429 0.6429 0.6429 0.3571 0.5714 0.5000 16 0.6429 0.6429 0.6429 0.4286 0.5714 0.5000 17 0.7143 0.5714 0.6429 0.4286 0.5714 0.5000 18 0.6429 0.5714 0.6429 0.4286 0.5714 0.5000 19 0.5714 0.5714 0.6429 0.4286 0.5714 0.5000 20 0.4286 0.5714 0.6429 0.4286 0.5714 0.4286 21 0.5714 0.5714 0.6429 0.4286 0.5714 0.4286 22 0.5714 0.6429 0.6429 0.4286 0.5714 0.4286 23 0.5714 0.6429 0.6429 0.4286 0.5000 0.4286 24 0.5714 0.6429 - - 0.5000 0.4286 25 0.5714 0.6429 - - - 0.4286 26 0.5714 0.6429 - - - - 135

Corrosion: TREPAN (beam width 3)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 1 0.5714 0.5714 0.5714 0.5000 0.4286 0.4286 2 0.5000 0.5000 0.4286 0.5000 0.3571 0.4286 3 0.5714 0.5714 0.5000 0.4286 0.3571 0.4286 4 0.3571 0.3571 0.5000 0.5000 0.3571 0.4286 5 0.4286 0.3571 0.5714 0.5714 0.3571 0.4286 6 0.5714 0.5000 0.5714 0.5714 0.4286 0.4286 7 0.5000 0.4286 0.5714 0.5714 0.4286 0.4286 8 0.5714 0.5000 0.5714 0.5000 0.4286 0.4286 9 0.6429 0.5000 0.5714 0.5000 0.4286 0.4286 10 0.7143 0.4286 0.5714 0.5000 0.4286 0.4286 11 0.7143 0.5000 0.5714 0.5714 0.4286 0.3571 12 0.7143 0.5714 0.5714 0.5714 0.4286 0.3571 13 0.7143 0.6429 0.5714 0.5714 0.4286 0.3571 14 0.7143 0.6429 0.5714 0.5714 0.5714 0.3571 15 0.6429 0.6429 0.5714 0.5714 0.5714 0.3571 16 0.6429 0.6429 0.5714 0.5714 0.5714 0.3571 17 0.7143 0.5714 0.5714 0.5714 0.5714 0.3571 18 0.6429 0.5714 0.5714 0.6429 0.5714 0.4286 19 0.5714 0.5714 0.5714 0.6429 0.5714 0.4286 20 0.4286 0.5714 0.5000 0.6429 0.5714 0.4286 21 0.5714 0.5714 0.5000 0.6429 0.5714 0.3571 22 0.5714 0.6429 0.5000 0.6429 0.5714 0.3571 23 0.5714 0.6429 0.5000 0.5714 0.5714 0.4286 24 0.5714 0.6429 - - 0.5714 0.4286 25 0.5714 0.6429 - - - - 26 0.5714 0.6429 - - - - 136

Corrosion: TREPAN (beam width 7)-Test accuracies at each node

Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 1 0.5714 0.5714 0.5714 0.5000 0.4286 0.4286 2 0.5000 0.5000 0.7143 0.5000 0.4286 0.4286 3 0.5000 0.5000 0.7143 0.5000 0.4286 0.4286 4 0.5714 0.5714 0.5714 0.5000 0.4286 0.4286 5 0.4286 0.4286 0.5714 0.5000 0.3571 0.4286 6 0.6429 0.6429 0.5000 0.5000 0.3571 0.4286 7 0.6429 0.7143 0.5714 0.5000 0.3571 0.4286 8 0.6429 0.7143 0.5714 0.5000 0.2857 0.4286 9 0.6429 0.7143 0.5714 0.5000 0.2857 0.4286 10 0.6429 0.7143 0.5714 0.5000 0.2857 0.4286 11 0.7143 0.8571 0.5714 0.5000 0.3571 0.4286 12 0.8571 0.8571 0.5714 0.5000 0.3571 0.4286 13 0.7857 0.8571 0.5000 0.5000 0.3571 0.4286 14 0.7857 0.8571 0.5000 0.5000 0.3571 0.4286 15 0.7857 0.7857 0.5000 0.5000 0.3571 0.4286 16 0.7143 0.6429 0.5000 0.5000 0.4286 0.4286 17 0.7143 0.6429 0.5000 0.5000 0.4286 0.4286 18 0.6429 0.7143 0.5000 0.5000 0.5000 0.4286 19 0.5714 0.7143 0.5000 0.5000 0.5000 0.4286 20 0.5714 0.7143 0.5000 0.5000 0.5000 0.4286 21 0.5714 0.7143 0.5000 0.5000 0.5000 0.4286 22 0.5714 0.7143 0.5000 0.5000 0.4286 0.4286 23 0.5714 0.7143 0.5000 0.5000 0.4286 0.4286 24 0.5714 0.7143 0.5000 0.5000 0.4286 0.4286 25 0.5714 0.7143 0.5000 0.5000 - - 26 0.5714 - - - - - 137

Corrosion: TREPAN (beam width 10)-Test accuracies at each node

Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 1 0.5714 0.5714 0.5714 0.5000 0.4286 0.4286 2 0.5714 0.5714 0.7857 0.5000 0.4286 0.4286 3 0.7143 0.7857 0.6429 0.5000 0.4286 0.4286 4 0.6429 0.7143 0.6429 0.5000 0.4286 0.4286 5 0.4286 0.5000 0.6429 0.5000 0.4286 0.4286 6 0.4286 0.5000 0.6429 0.5000 0.4286 0.4286 7 0.6429 0.7143 0.6429 0.5000 0.4286 0.4286 8 0.6429 0.7143 0.6429 0.5000 0.4286 0.4286 9 0.6429 0.7143 0.7143 0.5000 0.4286 0.4286 10 0.6429 0.7143 0.7143 0.5000 0.4286 0.4286 11 0.6429 0.7143 0.6429 0.5000 0.4286 0.4286 12 0.6429 0.7143 0.6429 0.5000 0.4286 0.4286 13 0.6429 0.7143 0.6429 0.5000 0.4286 0.4286 14 0.5714 0.6429 0.6429 0.5714 0.4286 0.4286 15 0.5000 0.6429 0.6429 0.5714 0.4286 0.4286 16 0.5714 0.6429 0.6429 0.5714 0.4286 0.4286 17 0.5000 0.6429 0.6429 0.5714 0.4286 0.4286 18 0.5000 0.5714 0.6429 0.5714 0.4286 0.4286 19 0.5000 0.5714 0.6429 0.5714 0.4286 0.4286 20 0.5000 0.5714 0.6429 0.5714 0.5000 0.4286 21 0.5000 0.5714 0.6429 0.5714 0.5000 0.4286 22 0.5000 0.5714 0.6429 0.5000 0.5000 0.4286 23 0.5000 0.5714 0.6429 0.5000 0.5000 - 138

Corrosion: Disjunctive TREPAN (beam width 2)-Test accuracies at each node

Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 1 0.5714 0.5714 0.5714 0.5000 0.4286 0.3571 2 0.5714 0.5714 0.5714 0.5000 0.3571 0.3571 3 0.5714 0.5714 0.5000 0.4286 0.3571 0.5000 4 0.3571 0.3571 0.5000 0.5714 0.3571 0.4286 5 0.4286 0.4286 0.5714 0.5714 0.3571 0.4286 6 0.5714 0.5714 0.6429 0.6429 0.3571 0.4286 7 0.5000 0.5000 0.6429 0.6429 0.3571 0.5000 8 0.5000 0.5000 0.6429 0.6429 0.3571 0.5000 9 0.5714 0.5714 0.6429 0.6429 0.3571 0.5000 10 0.5714 0.5714 0.6429 0.6429 0.3571 0.5000 11 0.5000 0.5714 0.6429 0.6429 0.3571 0.5000 12 0.5000 0.6429 0.6429 0.5714 0.3571 0.5000 13 0.4286 0.7143 0.6429 0.5714 0.4286 0.5714 14 0.5000 0.7143 0.6429 0.5714 0.4286 0.5714 15 0.5000 0.7143 0.6429 0.5714 0.4286 0.5714 16 0.5000 0.7143 0.6429 0.5000 0.4286 0.5714 17 0.5000 0.7143 0.6429 0.5000 0.4286 0.5714 18 0.5000 0.7143 0.6429 0.5000 0.4286 0.5000 19 0.5000 0.6429 0.6429 0.5000 0.4286 0.5000 20 0.4286 0.5714 0.6429 0.5000 0.4286 0.5000 21 0.4286 0.5714 0.6429 0.5000 0.4286 0.5000 22 0.4286 0.5714 0.6429 0.5000 0.4286 0.5000 23 0.4286 0.5714 0.6429 0.5000 0.4286 0.5000 24 0.4286 0.5714 - - - 0.5000 25 - - - - - 0.5000 139

Corrosion: Disjunctive TREPAN (beam width 3)-Test accuracies at each node

Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 1 0.5714 0.5714 0.5714 0.5000 0.4286 0.3571 2 0.5714 0.5714 0.5714 0.5000 0.3571 0.3571 3 0.5714 0.5714 0.5000 0.5000 0.3571 0.5000 4 0.3571 0.3571 0.5000 0.5000 0.3571 0.4286 5 0.4286 0.4286 0.5714 0.5000 0.3571 0.4286 6 0.5714 0.5714 0.6429 0.5000 0.3571 0.4286 7 0.5000 0.5000 0.6429 0.5000 0.3571 0.5000 8 0.5000 0.5000 0.6429 0.5000 0.3571 0.5000 9 0.5714 0.5714 0.6429 0.5000 0.3571 0.5000 10 0.5714 0.5714 0.6429 0.5000 0.3571 0.5000 11 0.5000 0.5714 0.6429 0.5000 0.3571 0.5000 12 0.5000 0.6429 0.6429 0.4286 0.3571 0.5000 13 0.4286 0.7143 0.6429 0.3571 0.4286 0.5714 14 0.5000 0.7143 0.6429 0.3571 0.4286 0.5714 15 0.5000 0.7143 0.6429 0.3571 0.4286 0.5714 16 0.5000 0.7143 0.6429 0.3571 0.4286 0.5714 17 0.5000 0.7143 0.6429 0.3571 0.4286 0.5714 18 0.5000 0.7143 0.6429 0.3571 0.4286 0.5000 19 0.5000 0.6429 0.6429 0.3571 0.4286 0.5000 20 0.4286 0.5714 0.6429 0.3571 0.4286 0.5000 21 0.4286 0.5714 0.6429 0.3571 0.4286 0.5000 22 0.4286 0.5714 0.6429 0.3571 0.4286 0.5000 23 0.4286 0.5714 0.6429 0.3571 0.4286 0.5000 24 0.4286 0.5714 - 0.3571 - 0.5000 25 - - - - - 0.5000 140

Corrosion: Disjunctive TREPAN (beam width 5)-Test accuracies at each node

Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 1 0.5714 0.5714 0.5714 0.5000 0.4286 0.3571 2 0.5714 0.5714 0.5000 0.5000 0.3571 0.3571 3 0.5714 0.5714 0.5000 0.5000 0.3571 0.5000 4 0.3571 0.3571 0.5000 0.5000 0.3571 0.4286 5 0.4286 0.4286 0.5714 0.5000 0.3571 0.4286 6 0.5714 0.5714 0.5714 0.5000 0.3571 0.4286 7 0.5000 0.5000 0.5714 0.5000 0.3571 0.5000 8 0.5000 0.5000 0.5714 0.5000 0.3571 0.5000 9 0.5714 0.5714 0.5714 0.5000 0.3571 0.5000 10 0.5714 0.5714 0.6429 0.5000 0.3571 0.5000 11 0.5000 0.5714 0.6429 0.5000 0.3571 0.5000 12 0.5000 0.6429 0.6429 0.4286 0.3571 0.5000 13 0.4286 0.7143 0.6429 0.3571 0.4286 0.5714 14 0.5000 0.7143 0.6429 0.3571 0.4286 0.5714 15 0.5000 0.7143 0.6429 0.3571 0.4286 0.5714 16 0.5000 0.7143 0.6429 0.3571 0.4286 0.5000 17 0.5000 0.7143 0.6429 0.3571 0.4286 0.5000 18 0.5000 0.7143 0.6429 0.3571 0.4286 0.5000 19 0.5000 0.6429 0.6429 0.3571 0.4286 0.5000 20 0.4286 0.5714 0.6429 0.3571 0.4286 0.5000 21 0.4286 0.5714 0.6429 0.3571 0.4286 0.5000 22 0.4286 0.5714 0.6429 0.3571 0.4286 0.5000 23 0.4286 0.5714 0.6429 0.3571 0.4286 0.5000 24 0.4286 0.5714 - 0.3571 - 0.5000 25 - - - 0.3571 - 0.5000 26 - - - 0.3571 - - 141

Corrosion: Disjunctive TREPAN (beam width 7)-Test accuracies at each node

Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 1 0.5714 0.5714 0.5714 0.5000 0.4286 0.3571 2 0.5714 0.5714 0.5000 0.5000 0.3571 0.3571 3 0.5714 0.5714 0.4286 0.5000 0.3571 0.3571 4 0.6429 0.6429 0.4286 0.5000 0.3571 0.3571 5 0.4286 0.4286 0.5000 0.5000 0.3571 0.3571 6 0.3571 0.3571 0.5714 0.5000 0.3571 0.4286 7 0.5000 0.5000 0.5714 0.5000 0.3571 0.4286 8 0.5000 0.5000 0.5714 0.3571 0.3571 0.4286 9 0.5714 0.5714 0.6429 0.3571 0.3571 0.4286 10 0.5714 0.5714 0.6429 0.3571 0.3571 0.4286 11 0.5714 0.5714 0.6429 0.3571 0.3571 0.4286 12 0.5714 0.5714 0.6429 0.4286 0.3571 0.4286 13 0.5714 0.6429 0.6429 0.4286 0.3571 0.5000 14 0.5714 0.6429 0.6429 0.5000 0.3571 0.5000 15 0.5714 0.7143 0.6429 0.5000 0.3571 0.5000 16 0.4286 0.7143 0.6429 0.5000 0.2857 0.5000 17 0.4286 0.7143 0.6429 0.5000 0.2857 0.5000 18 0.5714 0.7143 0.6429 0.5000 0.2857 0.5000 19 0.5000 0.5714 0.6429 0.5000 0.2857 0.5000 20 0.5714 0.7143 0.6429 0.5000 0.3571 0.5000 21 0.5714 0.7143 0.6429 0.5000 0.3571 0.5000 22 0.5714 0.6429 0.6429 0.4286 0.3571 0.5000 23 0.5714 0.6429 0.6429 0.5000 0.2857 0.5000 24 0.5714 0.6429 0.6429 0.5000 0.2857 0.5000 25 - - - - - 0.5000 142

Corrosion: Disjunctive TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 1 0.3571 0.3571 0.5000 0.5000 0.4286 0.3571 2 0.3571 0.3571 0.5000 0.5000 0.4286 0.3571 3 0.3571 0.3571 0.5000 0.5000 0.4286 0.3571 4 0.2143 0.2143 0.5000 0.5000 0.3571 0.3571 5 0.3571 0.3571 0.5000 0.5000 0.3571 0.3571 6 0.4286 0.4286 0.5714 0.5000 0.3571 0.4286 7 0.4286 0.4286 0.5714 0.5000 0.3571 0.4286 8 0.4286 0.4286 0.5714 0.5000 0.3571 0.4286 9 0.3571 0.3571 0.5714 0.5000 0.3571 0.4286 10 0.3571 0.3571 0.5714 0.5000 0.3571 0.4286 11 0.5000 0.4286 0.5714 0.5000 0.4286 0.4286 12 0.5714 0.4286 0.5714 0.5000 0.4286 0.4286 13 0.5714 0.5714 0.5714 0.5000 0.4286 0.4286 14 0.5714 0.5714 0.5714 0.5000 0.4286 0.4286 15 0.5714 0.5714 0.5714 0.5000 0.4286 0.4286 16 0.5714 0.5714 0.5714 0.5000 0.4286 0.4286 17 0.5714 0.5714 0.5000 0.4286 0.4286 0.4286 18 0.5714 0.5714 0.5000 0.4286 0.4286 0.4286 19 0.5714 0.6429 0.5000 0.4286 0.4286 0.4286 20 0.5714 0.6429 0.5000 0.4286 0.4286 0.4286 21 0.5714 0.6429 0.5000 0.4286 0.4286 0.4286 22 0.5714 0.7143 0.5000 0.4286 0.4286 0.4286 23 0.5714 0.7143 0.5000 0.4286 0.4286 0.4286 24 - 0.7143 0.5000 0.4286 - 0.4286 25 - 0.7143 - - - - 143

APPENDIX C: OUTAGES RESULTS

Outages: TREPAN (beam width 2)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.653 0.653 0.653 0.653 0.653 0.653 1 0.733 0.733 0.733 0.733 0.733 0.733 2 0.787 0.787 0.787 0.787 0.733 0.733 3 0.747 0.747 0.747 0.747 0.733 0.733 4 0.747 0.747 0.747 0.747 0.733 0.733 5 0.733 0.733 0.747 0.747 0.747 0.760 6 0.733 0.733 0.747 0.760 0.760 0.773 7 0.720 0.720 0.760 0.733 0.760 0.760 8 0.773 0.773 0.800 0.733 0.760 0.747 9 0.773 0.773 0.800 0.773 0.773 0.760 10 0.773 0.773 0.800 0.773 0.773 0.760 11 0.773 0.773 0.800 0.773 0.773 0.773 12 0.773 0.773 0.813 0.760 0.760 0.760 13 0.773 0.773 0.787 0.733 0.760 0.773 14 0.773 0.773 0.787 0.720 0.773 0.773 15 0.773 0.800 0.787 0.720 0.787 0.787 16 0.773 0.800 0.827 0.720 0.787 0.787 17 0.773 0.800 0.827 0.733 0.787 0.787 18 0.773 0.800 0.827 0.733 0.787 0.787 19 0.773 0.800 0.827 0.733 0.787 0.787 20 0.773 0.800 0.827 0.733 0.787 0.787 21 0.773 0.800 0.827 0.733 0.787 0.773 22 0.773 0.800 0.827 0.733 0.787 0.773 23 0.773 0.800 0.827 0.733 0.787 0.760 24 0.773 0.800 0.827 0.733 0.787 0.760 25 0.773 0.800 0.827 0.733 0.787 0.760 26 0.773 0.800 0.827 0.733 0.787 0.760 27 0.773 0.800 0.827 0.733 0.773 0.760 28 0.773 0.800 0.827 0.733 0.773 0.747 29 0.773 0.800 0.827 0.733 0.773 0.747 30 0.773 0.800 0.827 0.733 0.773 0.747 31 0.773 0.800 0.827 0.733 0.773 0.747 32 0.773 0.800 0.827 0.733 0.773 0.747 33 0.773 0.800 0.827 0.733 0.773 0.747 34 0.773 0.800 0.827 0.733 0.773 0.747 35 - 0.800 0.827 0.733 0.773 0.747 36 - - - 0.733 0.773 0.747 37 - - - 0.733 0.773 0.747 38 - - - - 0.773 0.747 39 - - - - 0.773 - 40 - - - - 0.773 - 144

Outages: TREPAN (beam width 3)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6533 0.6533 0.6533 0.6533 0.6533 0.6533 1 0.7200 0.7200 0.7200 0.7200 0.7200 0.7333 2 0.7733 0.7733 0.6800 0.6800 0.7200 0.7333 3 0.7333 0.7333 0.6800 0.6933 0.7333 0.7333 4 0.7333 0.7333 0.6800 0.6933 0.7200 0.7333 5 0.7467 0.7467 0.6667 0.6933 0.7200 0.7600 6 0.7467 0.7467 0.7200 0.6800 0.7733 0.7733 7 0.7467 0.7467 0.7200 0.7200 0.7733 0.7600 8 0.7467 0.7467 0.7867 0.6933 0.7733 0.7467 9 0.7600 0.7600 0.8267 0.7200 0.7733 0.7600 10 0.7600 0.7600 0.8133 0.7600 0.7733 0.7600 11 0.7600 0.7600 0.8000 0.7867 0.7733 0.7733 12 0.7600 0.7600 0.8000 0.7733 0.7600 0.7600 13 0.7467 0.7600 0.8000 0.7733 0.7600 0.7733 14 0.7467 0.8000 0.8000 0.7733 0.7600 0.7867 15 0.7467 0.8000 0.8000 0.7733 0.7600 0.7733 16 0.7467 0.8000 0.8000 0.7733 0.7600 0.7733 17 0.7467 0.8000 0.8000 0.7733 0.7600 0.7733 18 0.7467 0.8000 0.8000 0.7733 0.7600 0.7733 19 0.7467 0.8000 0.8000 0.7733 0.7600 0.7733 20 0.7467 0.8000 0.8000 0.7733 0.7600 0.7733 21 0.7467 0.8000 0.8000 0.7733 0.7600 0.7600 22 0.7467 0.8000 0.8000 0.7733 0.7600 0.7600 23 0.7467 0.8000 0.8000 0.7733 0.7600 0.7467 24 0.7467 0.8000 0.8000 0.7600 0.7600 0.7467 25 0.7467 0.8000 0.8000 0.7600 0.7600 0.7467 26 0.7467 0.8000 0.8000 0.7600 0.7600 0.7467 27 0.7467 0.8000 0.8000 0.7600 0.7733 0.7467 28 0.7467 0.8000 0.8000 0.7600 0.7867 0.7467 29 0.7467 0.8000 0.8000 0.7600 0.7867 0.7467 30 0.7467 0.8000 0.8000 0.7600 0.7867 0.7333 31 0.7467 0.8000 0.8000 0.7733 0.7867 0.7333 32 0.7467 0.8000 0.8000 0.7733 0.7867 0.7333 33 - 0.8000 0.8000 0.7733 0.7867 0.7333 34 - - 0.8000 0.7733 0.7867 0.7333 35 - - - 0.7733 - 0.7333 36 - - - 0.7733 - 0.7333 37 - - - 0.7733 - 0.7333 38 - - - 0.7733 - 0.7333 39 - - - 0.7733 - - 40 - - - 0.7733 - - 145

Outages: TREPAN (beam width 5)-Test accuracies at each node Classification Accuracy Node Min sample Min sample Min sample Min sample Min sample Min sample 1 10 50 100 500 1000 0 0.6533 0.6533 0.6533 0.6533 0.6533 0.6533 1 0.7200 0.7200 0.7200 0.7200 0.6800 0.7333 2 0.7733 0.7733 0.6800 0.6800 0.6800 0.7333 3 0.7333 0.7333 0.6800 0.6933 0.6800 0.7333 4 0.7333 0.7333 0.6800 0.6933 0.6667 0.7333 5 0.7467 0.7467 0.6667 0.6933 0.7067 0.7600 6 0.7467 0.7467 0.6800 0.7067 0.7333 0.7733 7 0.7467 0.7467 0.6800 0.6933 0.7333 0.7600 8 0.7467 0.7467 0.6800 0.6933 0.7467 0.7600 9 0.7467 0.7467 0.6800 0.7200 0.7867 0.7733 10 0.7467 0.7467 0.6800 0.7333 0.7867 0.7733 11 0.7467 0.7467 0.6800 0.7867 0.7867 0.7733 12 0.7333 0.7467 0.6800 0.7867 0.7867 0.7600 13 0.7333 0.7467 0.6933 0.8133 0.7867 0.7733 14 0.7333 0.7467 0.7200 0.8133 0.8000 0.7867 15 0.7333 0.7467 0.7200 0.8133 0.8000 0.7867 16 0.7333 0.7467 0.7067 0.7867 0.8000 0.7867 17 0.7333 0.7467 0.7067 0.7867 0.7867 0.7867 18 0.7333 0.7467 0.7067 0.7867 0.7733 0.7733 19 0.7333 0.7467 0.7067 0.7867 0.7733 0.7733 20 0.7333 0.7467 0.7067 0.7867 0.8000 0.7733 21 0.7333 0.7467 0.7067 0.7867 0.8000 0.7733 22 0.7333 0.7467 0.7067 0.7867 0.8000 0.7600 23 0.7333 0.7467 0.7067 0.7867 0.8000 0.7600 24 0.7333 0.7467 0.7067 0.7867 0.8000 0.7733 25 0.7333 0.7467 0.7067 0.7867 0.7867 0.7733 26 0.7333 0.7467 0.7067 0.7867 0.7867 0.7600 27 0.7333 0.7467 0.7067 0.7867 0.7867 0.7600 28 0.7333 0.7467 0.7067 0.7867 0.7867 0.7600 29 0.7333 0.7467 0.7067 0.7867 0.7867 0.7600 30 0.7333 0.7467 - 0.7867 0.7867 0.7600 31 0.7333 0.7467 - 0.7867 0.7867 0.7600 32 - - - 0.7867 0.7867 0.7600 33 - - - 0.7867 0.7867 0.7600 34 - - - 0.7867 0.7867 0.7600 35 - - - 0.7867 0.7867 0.7600 36 - - - 0.7867 0.7867 0.7600 37 - - - - 0.7867 - 38 - - - - 0.7867 - 39 - - - - 0.7867 - 146

Outages: TREPAN (beam width 7)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

0 0.6533 0.6533 0.6533 0.6533 0.6533 0.6533 1 0.7200 0.7200 0.7200 0.7200 0.6800 0.7733 2 0.7733 0.7733 0.6800 0.6800 0.6800 0.7733 3 0.7333 0.7333 0.6800 0.6933 0.6800 0.7733 4 0.7333 0.7333 0.6800 0.6933 0.6800 0.7733 5 0.7600 0.7600 0.6800 0.6933 0.7200 0.7867 6 0.7600 0.7600 0.6533 0.7333 0.7467 0.8133 7 0.7867 0.7867 0.6933 0.7467 0.7467 0.8133 8 0.7867 0.7867 0.7467 0.7867 0.7867 0.7867 9 0.7867 0.7867 0.7467 0.7733 0.8000 0.7867 10 0.7867 0.7867 0.7467 0.7733 0.8000 0.8133 11 0.7867 0.7867 0.7733 0.7600 0.8000 0.8133 12 0.7867 0.7867 0.7733 0.7467 0.8000 0.8267 13 0.7867 0.7867 0.7733 0.7467 0.8000 0.8400 14 0.7867 0.7867 0.7733 0.7467 0.8000 0.8400 15 0.7867 0.7867 0.7733 0.7467 0.8000 0.8400 16 0.7867 0.7867 0.7733 0.7600 0.8000 0.8267 17 0.7867 0.7867 0.7733 0.7600 0.7867 0.8267 18 0.7867 0.7867 0.7733 0.7600 0.8000 0.8267 19 0.7867 0.7867 0.7733 0.7600 0.8000 0.8267 20 0.7867 0.7867 0.7733 0.7600 0.8000 0.8267 21 0.7867 0.7867 0.7733 0.7600 0.8000 0.8267 22 0.7867 0.7867 0.7733 0.7600 0.8000 0.8133 23 0.7867 0.7867 0.7733 0.7600 0.8000 0.8133 24 0.7867 0.7867 0.7733 0.7600 0.8133 0.8267 25 0.7867 0.7867 0.7733 0.7600 0.8133 0.8267 26 0.7867 0.7867 0.7733 0.7600 0.8133 0.8267 27 0.7867 0.7867 0.7733 0.7600 0.8133 0.8267 28 0.7867 0.7867 0.7733 0.7600 0.8133 0.8267 29 0.7867 0.7867 0.7733 0.7600 0.8133 0.8267 30 0.7867 0.7867 0.7733 0.7600 0.8133 0.8267 31 0.7867 0.7867 0.7733 0.7600 0.8133 0.8267 32 0.7867 0.7867 0.7733 0.7600 0.8133 0.8267 33 - 0.7867 0.7733 0.7600 0.8133 0.8267 34 - 0.7867 0.7733 0.7600 0.8133 - 35 - 0.7867 - 0.7600 - - 147

Outages: TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6533 0.6533 0.6533 0.6533 0.6533 0.6533 1 0.7200 0.7200 0.7200 0.7200 0.6800 0.7200 2 0.7733 0.7733 0.6800 0.6800 0.6800 0.7200 3 0.7333 0.7333 0.6800 0.6933 0.6800 0.7200 4 0.7333 0.7333 0.6800 0.6933 0.6533 0.6267 5 0.7467 0.7467 0.6800 0.7200 0.6933 0.6533 6 0.7467 0.7467 0.7333 0.7333 0.7200 0.6667 7 0.7467 0.7467 0.7467 0.7733 0.7733 0.6667 8 0.7467 0.7467 0.7467 0.7867 0.7733 0.6667 9 0.7467 0.7467 0.7467 0.7867 0.7733 0.7333 10 0.7467 0.7467 0.7467 0.7733 0.7867 0.7333 11 0.7467 0.7467 0.7333 0.7733 0.7867 0.7200 12 0.7333 0.7467 0.7333 0.7467 0.7867 0.7067 13 0.7333 0.7467 0.7333 0.7467 0.7867 0.7067 14 0.7333 0.7467 0.7333 0.7333 0.7867 0.7067 15 0.7333 0.7467 0.7333 0.7333 0.7867 0.7067 16 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 17 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 18 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 19 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 20 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 21 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 22 0.7333 0.7467 0.7733 0.7333 0.7867 0.6933 23 0.7333 0.7467 0.7733 0.7333 0.7867 0.6933 24 0.7333 0.7467 0.7733 0.7333 0.7867 0.6933 25 0.7333 0.7467 0.7733 0.7333 0.7867 0.6933 26 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 27 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 28 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 29 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 30 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 31 0.7333 0.7467 0.7733 0.7333 0.7867 0.7067 32 - - 0.7733 0.7333 0.7867 0.7067 33 - - 0.7733 0.7333 0.7867 0.7067 34 - - 0.7733 - - 0.7067 35 - - 0.7733 - - 0.7067 36 - - 0.7733 - - 0.7067 37 - - 0.7733 - - 0.7067 38 - - - - - 0.7067 39 - - - - - 0.7067 148

Outages: Disjunctive TREPAN (beam width 2)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6533 0.6533 0.6533 0.6533 0.6533 0.6533 1 0.7333 0.7333 0.7333 0.7333 0.6533 0.7200 2 0.7867 0.7867 0.7867 0.7867 0.5867 0.7200 3 0.7467 0.7467 0.7467 0.7467 0.7333 0.7200 4 0.7467 0.7467 0.7467 0.7467 0.7333 0.7467 5 0.7333 0.7333 0.7467 0.7467 0.7333 0.7467 6 0.7333 0.7333 0.7467 0.7600 0.7467 0.8000 7 0.7200 0.7200 0.7600 0.7333 0.7867 0.8000 8 0.7733 0.7733 0.7467 0.7333 0.7867 0.7867 9 0.7733 0.7733 0.7467 0.7733 0.7867 0.7867 10 0.7733 0.7733 0.7733 0.7733 0.7867 0.7867 11 0.7733 0.7733 0.7733 0.7733 0.7867 0.7867 12 0.7733 0.7733 0.7733 0.7733 0.7867 0.7600 13 0.7733 0.7733 0.7733 0.7733 0.7867 0.7600 14 0.7733 0.7733 0.7733 0.7733 0.7867 0.7600 15 0.7733 0.8000 0.7733 0.7733 0.7867 0.7600 16 0.7733 0.8000 0.7467 0.7733 0.7867 0.7600 17 0.7733 0.8000 0.7467 0.7733 0.7867 0.7600 18 0.7733 0.8000 0.7467 0.7733 0.7867 0.7600 19 0.7733 0.8000 0.7467 0.7733 0.7867 0.7600 20 0.7733 0.8000 0.7467 0.7733 0.7867 0.7600 21 0.7733 0.8000 0.7467 0.7733 0.8000 0.7333 22 0.7733 0.8000 0.7467 0.7733 0.7867 0.7333 23 0.7733 0.8000 0.7467 0.7733 0.7867 0.7333 24 0.7733 0.8000 0.7467 0.7733 0.7867 0.7333 25 0.7733 0.8000 0.7467 0.7733 0.7867 0.7467 26 0.7733 0.8000 0.7467 0.7733 0.7867 0.7467 27 0.7733 0.8000 0.7467 0.7733 0.7867 0.7467 28 0.7733 0.8000 0.7467 0.7733 0.8133 0.7467 29 0.7733 0.8000 0.7467 0.7733 0.8133 0.7467 30 0.7733 0.8000 0.7467 0.7733 0.8133 0.7467 31 0.7733 0.8000 0.7467 0.7733 0.8133 0.7467 32 0.7733 0.8000 0.7467 0.7733 0.8133 0.7467 33 0.7733 0.8000 0.7467 0.7733 0.8133 0.7467 34 0.7733 0.8000 - 0.7733 0.8133 0.7467 35 - 0.8000 - 0.7733 0.8133 0.7467 36 - - - 0.7733 0.8133 0.7467 37 - - - - 0.8133 0.7467 38 - - - - 0.8133 0.7467 39 - - - - 0.8133 0.7467 40 - - - - 0.8133 0.7467 41 - - - - 0.8133 - 42 - - - - 0.8133 - 149

Outages: Disjunctive TREPAN (beam width 3)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6533 0.6533 0.6533 0.6533 0.6533 0.6533 1 0.7333 0.7333 0.7333 0.7333 0.6533 0.7200 2 0.7867 0.7867 0.7867 0.7867 0.5733 0.7200 3 0.7467 0.7467 0.7467 0.7467 0.7200 0.7200 4 0.7467 0.7467 0.7467 0.7467 0.7200 0.7467 5 0.7333 0.7333 0.7467 0.7467 0.7200 0.7467 6 0.7333 0.7333 0.7467 0.7600 0.7333 0.8000 7 0.7200 0.7200 0.7600 0.7333 0.7600 0.8000 8 0.7733 0.7733 0.7733 0.7333 0.7467 0.7867 9 0.7733 0.7733 0.7733 0.7733 0.7467 0.7867 10 0.7733 0.7733 0.7733 0.7733 0.7467 0.7867 11 0.7733 0.7733 0.7733 0.7733 0.7467 0.7867 12 0.7733 0.7733 0.7467 0.7733 0.7467 0.7600 13 0.7733 0.7733 0.7333 0.7867 0.7467 0.7600 14 0.7733 0.7733 0.7467 0.7867 0.7467 0.7600 15 0.7733 0.8000 0.7467 0.7867 0.7467 0.7600 16 0.7733 0.8000 0.7467 0.7867 0.7467 0.7600 17 0.7733 0.8000 0.7467 0.7867 0.7200 0.7600 18 0.7733 0.8000 0.7467 0.7733 0.7200 0.7600 19 0.7733 0.8000 0.7467 0.7733 0.7200 0.7600 20 0.7733 0.8000 0.7467 0.7733 0.7067 0.7600 21 0.7733 0.8000 0.7467 0.7733 0.7067 0.7333 22 0.7733 0.8000 0.7467 0.7733 0.7200 0.7333 23 0.7733 0.8000 0.7467 0.7733 0.7333 0.7333 24 0.7733 0.8000 0.7467 0.7733 0.7467 0.7333 25 0.7733 0.8000 0.7467 0.7733 0.7467 0.7467 26 0.7733 0.8000 0.7467 0.7733 0.7600 0.7467 27 0.7733 0.8000 0.7467 0.7733 0.7600 0.7467 28 0.7733 0.8000 0.7467 0.7733 0.7600 0.7467 29 0.7733 0.8000 0.7467 0.7733 0.7600 0.7467 30 0.7733 0.8000 0.7467 0.7733 0.7600 0.7467 31 0.7733 0.8000 0.7467 0.7733 0.7600 0.7467 32 0.7733 0.8000 - 0.7733 0.7600 0.7467 33 0.7733 0.8000 - 0.7733 0.7600 0.7467 34 0.7733 0.8000 - 0.7733 0.7600 0.7467 35 0.7733 - - 0.7733 0.7600 0.7467 36 - - - 0.7733 0.7600 0.7467 37 - - - 0.7733 0.7600 0.7467 38 - - - - 0.7600 0.7467 39 - - - - 0.7600 0.7467 40 - - - - 0.7600 0.7467 150

Outages: Disjunctive TREPAN (beam width 5)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6533 0.6533 0.6533 0.6533 0.6533 0.6533 1 0.7200 0.7200 0.7200 0.7200 0.6533 0.7200 2 0.6800 0.6800 0.6800 0.6800 0.5733 0.7200 3 0.7333 0.7333 0.6800 0.6933 0.7200 0.7200 4 0.7333 0.7333 0.6800 0.6933 0.7200 0.7467 5 0.7200 0.7200 0.6667 0.6800 0.7200 0.7467 6 0.7733 0.7733 0.7200 0.7333 0.7333 0.8000 7 0.7733 0.7733 0.7200 0.7333 0.7600 0.8000 8 0.7733 0.7867 0.7467 0.7333 0.7467 0.7867 9 0.7867 0.7867 0.7600 0.7333 0.7467 0.7867 10 0.7867 0.7867 0.7467 0.7333 0.7467 0.7867 11 0.7867 0.7867 0.7600 0.7600 0.7467 0.7867 12 0.7867 0.7867 0.7600 0.7600 0.7467 0.7600 13 0.7867 0.7867 0.7867 0.7600 0.7467 0.7600 14 0.7867 0.7867 0.7867 0.7600 0.7467 0.7600 15 0.7867 0.7867 0.7867 0.7600 0.7467 0.7600 16 0.7867 0.7867 0.7867 0.7600 0.7467 0.7600 17 0.7867 0.7867 0.7867 0.7600 0.7200 0.7600 18 0.7867 0.8133 0.7867 0.7600 0.7200 0.7600 19 0.7867 0.8133 0.7867 0.7733 0.7200 0.7600 20 0.7867 0.8133 0.7867 0.7733 0.7067 0.7600 21 0.7867 0.8133 0.7867 0.7733 0.7067 0.7733 22 0.7867 0.8133 0.7867 0.7733 0.7200 0.7733 23 0.7867 0.8133 0.7867 0.7733 0.7333 0.7733 24 0.7867 0.8133 0.7867 0.7733 0.7467 0.7867 25 0.7867 0.8133 0.7867 0.7733 0.7467 0.7867 26 0.7867 0.8133 0.7867 0.7733 0.7600 0.7867 27 0.7867 0.8133 0.7867 0.7733 0.7600 0.7867 28 0.7867 0.8133 0.7867 0.7733 0.7600 0.7867 29 0.7867 0.8133 0.7867 0.7733 0.7600 0.7867 30 0.7867 0.8133 0.7867 0.7733 0.7600 0.7867 31 0.7867 0.8133 0.7867 0.7733 0.7600 0.7867 32 0.7867 0.8133 - 0.7733 0.7600 0.7867 33 - - - 0.7733 0.7600 0.7867 34 - - - 0.7733 0.7600 0.7867 35 - - - 0.7733 0.7600 0.7867 36 - - - 0.7733 0.7600 0.7867 37 - - - - 0.7600 0.7867 38 - - - - 0.7600 0.7867 39 - - - - 0.7600 0.7867 40 - - - - 0.7600 0.7867 151

Outages: Disjunctive TREPAN (beam width 7)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6533 0.6533 0.6533 0.6533 0.6533 0.6533 1 0.7200 0.7200 0.7200 0.7200 0.6533 0.7200 2 0.6800 0.6800 0.6800 0.6800 0.5733 0.7200 3 0.7333 0.7333 0.6800 0.6933 0.7200 0.7200 4 0.7333 0.7333 0.6800 0.6933 0.7200 0.7467 5 0.7200 0.7200 0.6667 0.6800 0.7200 0.7467 6 0.7733 0.7733 0.7200 0.7333 0.7333 0.8000 7 0.7733 0.7733 0.7200 0.7333 0.7600 0.8000 8 0.7733 0.7867 0.7467 0.7733 0.7467 0.7867 9 0.7867 0.7867 0.7600 0.7867 0.7467 0.7867 10 0.7867 0.7867 0.7467 0.8000 0.7467 0.7867 11 0.7867 0.7867 0.7600 0.8000 0.7467 0.7867 12 0.7867 0.7867 0.7600 0.8267 0.7467 0.7600 13 0.7867 0.7867 0.7867 0.8267 0.7467 0.7600 14 0.7867 0.7867 0.7867 0.8000 0.7467 0.7600 15 0.7867 0.7867 0.7867 0.8000 0.7467 0.7600 16 0.7867 0.7867 0.7867 0.7867 0.7467 0.7600 17 0.7867 0.7867 0.7867 0.7867 0.7467 0.7600 18 0.7867 0.8133 0.7867 0.7867 0.7467 0.7600 19 0.7867 0.8133 0.7867 0.7867 0.7467 0.7600 20 0.7867 0.8133 0.7867 0.7867 0.7467 0.7600 21 0.7867 0.8133 0.7867 0.7867 0.7467 0.7733 22 0.7867 0.8133 0.7867 0.7867 0.7467 0.7733 23 0.7867 0.8133 0.7867 0.7867 0.7333 0.7733 24 0.7867 0.8133 0.7867 0.7867 0.7467 0.7867 25 0.7867 0.8133 0.7867 0.7867 0.7600 0.7867 26 0.7867 0.8133 0.7867 0.7867 0.7600 0.7867 27 0.7867 0.8133 0.7867 0.7867 0.7600 0.7867 28 0.7867 0.8133 0.7867 0.7867 0.7600 0.7867 29 0.7867 0.8133 0.7867 0.7867 0.7600 0.7867 30 0.7867 0.8133 - 0.7867 0.7600 0.7867 31 0.7867 0.8133 - 0.7867 0.7600 0.7867 32 0.7867 0.8133 - 0.7867 0.7600 0.7867 33 - - - 0.7867 0.7600 0.7867 34 - - - 0.7867 0.7600 0.7867 35 - - - 0.7867 0.7600 0.7867 36 - - - 0.7867 0.7600 0.7867 37 - - - 0.7867 0.7600 0.7867 38 - - - 0.7867 - 0.7867 39 - - - - - 0.7867 40 - - - - - 0.7867 152

Outages: Disjunctive TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6533 0.6533 0.6533 0.6533 0.6533 0.6533 1 0.7200 0.7200 0.7200 0.7200 0.6533 0.7200 2 0.6800 0.6800 0.6800 0.6800 0.5733 0.7200 3 0.7333 0.7333 0.6800 0.6933 0.7200 0.7200 4 0.7333 0.7333 0.6800 0.6933 0.7200 0.7200 5 0.7200 0.7200 0.6667 0.6800 0.7200 0.7200 6 0.7733 0.7733 0.7200 0.7333 0.7333 0.7200 7 0.7733 0.7733 0.7200 0.7333 0.7600 0.7333 8 0.7733 0.7867 0.7467 0.7733 0.7467 0.7600 9 0.7867 0.7867 0.7600 0.7867 0.7467 0.7467 10 0.7867 0.7867 0.7467 0.7867 0.7467 0.7733 11 0.7867 0.7867 0.7600 0.7867 0.7467 0.7733 12 0.7867 0.7867 0.7600 0.7600 0.7467 0.7733 13 0.7867 0.7867 0.7867 0.7600 0.7467 0.7733 14 0.7867 0.7867 0.7867 0.7600 0.7467 0.8000 15 0.7867 0.7867 0.7867 0.7600 0.7467 0.8000 16 0.7867 0.7867 0.7867 0.7600 0.7467 0.8133 17 0.7867 0.7867 0.7867 0.7600 0.7467 0.8133 18 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 19 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 20 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 21 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 22 0.7867 0.8133 0.7867 0.7600 0.7333 0.8133 23 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 24 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 25 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 26 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 27 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 28 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 29 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 30 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 31 0.7867 0.8133 0.7867 0.7600 0.7467 0.8133 32 0.7867 0.8133 - 0.7600 0.7467 0.8133 33 - - - 0.7600 0.7467 0.8133 34 - - - 0.7600 0.7467 0.8133 35 - - - 0.7600 0.7467 0.8133 36 - - - 0.7600 0.7467 0.8133 37 - - - 0.7600 0.7467 0.8133 38 - - - 0.7600 0.7467 0.8133 39 - - - - 0.7467 - 40 - - - - 0.7467 - 153

APPENDIX D: IRIS RESULTS

Iris: TREPAN (beam width 2)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

0 0.2895 0.2895 0.2895 0.2895 0.2895 0.2895 1 0.6579 0.6579 0.6579 0.6579 0.6579 0.6579 2 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 3 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 4 0.8947 0.8947 0.8684 0.8947 0.8947 0.9474 5 - - - 0.8421 0.8947 0.9474 6 - - - 0.8421 0.8947 0.9474

Iris: TREPAN (beam width 3)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

Iris: TREPAN (beam width 5)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

0 0.2895 0.2895 0.2895 0.2895 0.2895 0.2895 1 0.6579 0.6579 0.6579 0.6579 0.6579 0.6579 2 0.8947 0.8947 0.8947 0.8947 0.8947 0.8947 3 0.8947 0.8947 0.8947 0.8947 0.8947 0.7895 4 0.8947 0.8947 0.8684 0.8947 0.7895 0.7895 5 - - - 0.8947 0.7895 0.7895 6 - - - 0.8947 0.7895 0.7895 154

Iris: TREPAN (beam width 7)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

0 0.2895 0.2895 0.2895 0.2895 0.2895 0.2895 1 0.6579 0.6579 0.6579 0.6579 0.7105 0.6579 2 0.8947 0.8947 0.8947 0.8947 0.8947 0.8947 3 0.8947 0.8947 0.8947 0.8947 0.7895 0.7895 4 0.8947 0.8947 0.8947 0.8947 0.7632 0.7895 5 - - 0.8947 0.8947 0.7632 0.7895 6 - - - 0.8947 - 0.7895 7 - - - 0.8947 - - 8 - - - 0.8947 - -

Iris: TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2895 0.2895 0.2895 0.2895 0.2895 0.2895 1 0.6579 0.6579 0.6579 0.6579 0.7105 0.6579 2 0.8947 0.8947 0.8947 0.8947 0.8947 0.8947 3 0.8947 0.8947 0.8947 0.8947 0.7895 0.7895 4 0.8947 0.8947 0.8947 0.8947 0.7632 0.7895 5 - - 0.8947 0.8947 0.7632 0.7895 6 - - - 0.8947 - 0.7895 7 - - - 0.8947 - - 8 - - - 0.8947 - -

Iris: Disjunctive TREPAN (beam width 2)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2895 0.2895 0.2895 0.2895 0.2895 0.2895 1 0.6579 0.6579 0.6579 0.6579 0.6579 0.6579 2 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 3 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 4 0.8947 0.8947 0.8684 0.8947 0.8947 0.9474 5 - - - 0.8421 0.8684 0.9474 6 - - - 0.8421 0.8684 0.9474 155

Iris: Disjunctive TREPAN (beam width 3)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2895 0.2895 0.2895 0.2895 0.2895 0.2895 1 0.6579 0.6579 0.6579 0.6579 0.6579 0.6579 2 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 3 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 4 0.8947 0.8947 0.8684 0.8947 0.8947 0.9474 5 - - - 0.8421 0.8684 0.9474 6 - - - 0.8421 0.8684 0.9474

Iris: Disjunctive TREPAN (beam width 5)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2895 0.2895 0.2895 0.2895 0.2895 0.2895 1 0.6579 0.6579 0.6579 0.6579 0.6579 0.6579 2 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 3 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 4 0.8947 0.8947 0.8684 0.8947 0.8947 0.9474 5 - - - 0.8947 0.8684 0.9474 6 - - - 0.8947 0.8684 0.9474 7 - - - 0.8947 - -

Iris: Disjunctive TREPAN (beam width 7)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2895 0.2895 0.2895 0.2895 0.2895 0.2895 1 0.6579 0.6579 0.6579 0.6579 0.6579 0.6579 2 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 3 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 4 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 5 - - 0.8947 0.8947 0.8684 0.9474 6 - - - 0.8947 0.8684 0.9474 7 - - - 0.8947 - - 156

Iris: Disjunctive TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2895 0.2895 0.2895 0.2895 0.2895 0.2895 1 0.6579 0.6579 0.6579 0.6579 0.6579 0.6579 2 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 3 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 4 0.8947 0.8947 0.8947 0.8947 0.8947 0.9474 5 - - 0.8947 0.8947 0.8684 0.9474 6 - - - 0.8947 0.8684 0.9474 7 - - - 0.8947 - - 157

APPENDIX E: BODY FAT RESULTS

Body fat: TREPAN (beam width 2)-Test accuracies at each node Classification Accuracy Node Min sample Min ample Min sample Min sample Min sample Min sample 1 10 50 100 500 1000 0 0.2222 0.2222 0.2222 0.2222 0.2222 0.2222 1 0.3651 0.3651 0.3651 0.5238 0.5238 0.5238 2 0.6508 0.6508 0.6508 0.6508 0.8413 0.8413 3 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 4 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 5 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 6 - - - 0.9683 - - 7 - - - 0.9683 - -

Body fat: TREPAN (beam width 3)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

0 0.2222 0.2222 0.2222 0.2222 0.2222 0.2222 1 0.3651 0.3651 0.3651 0.5238 0.5238 0.5238 2 0.6508 0.6508 0.6508 0.6508 0.8413 0.8413 3 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 4 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 5 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 6 - - 0.9683 0.9683 - - 7 - - 0.9683 0.9683 - -

Body fat: TREPAN (beam width 5)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

Body fat: TREPAN (beam width 7)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

Body fat: TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2222 0.2222 0.2222 0.2222 0.2222 0.2222 1 0.3651 0.3651 0.3651 0.5238 0.5238 0.5238 2 0.6508 0.6508 0.6508 0.6508 0.8413 0.8413 3 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 4 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 5 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 6 - - - 0.9683 - -

Body fat: Disjunctive TREPAN (beam width 2)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2222 0.2222 0.2222 0.2222 0.2222 0.2222 1 0.3651 0.3651 0.3651 0.5238 0.5238 0.5238 2 0.6508 0.6508 0.6508 0.6508 0.8413 0.8413 3 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 4 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 5 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 6 - - - 0.9683 - - 7 - - - 0.9683 - - 159

Body fat: Disjunctive TREPAN (beam width 3)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2222 0.2222 0.2222 0.2222 0.2222 0.2222 1 0.3651 0.3651 0.3651 0.5238 0.5238 0.5238 2 0.6508 0.6508 0.6508 0.6508 0.8413 0.8413 3 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 4 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 5 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 6 - - 0.9683 0.9683 - - 7 - - 0.9683 0.9683 - -

Body fat: Disjunctive TREPAN (beam width 5)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2222 0.2222 0.2222 0.2222 0.2222 0.2222 1 0.3651 0.3651 0.3651 0.5238 0.5238 0.5238 2 0.6508 0.6508 0.6508 0.6508 0.8413 0.8413 3 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 4 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 5 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 6 - - - 0.9683 - - 7 - - - 0.9683 - -

Body fat: Disjunctive TREPAN (beam width 7)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2222 0.2222 0.2222 0.2222 0.2222 0.2222 1 0.3651 0.3651 0.3651 0.5238 0.5238 0.5238 2 0.6508 0.6508 0.6508 0.6508 0.8413 0.8413 3 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 4 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 5 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 6 - - 0.9683 0.9683 - - 7 ------160

Body fat: Disjunctive TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.2222 0.2222 0.2222 0.2222 0.2222 0.2222 1 0.3651 0.3651 0.3651 0.5238 0.5238 0.5238 2 0.6508 0.6508 0.6508 0.6508 0.8413 0.8413 3 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 4 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 5 0.9683 0.9683 0.9683 0.9683 0.9683 0.9683 6 - - - 0.9683 - - 161

APPENDIX F: SAGINAW BAY RESULTS

Saginaw Bay: TREPAN (beam width 2)-Test accuracies at each node Classification Accuracy Node Min sample Min sample Min sample Min sample Min sample Min sample 1 10 50 100 500 1000 0 0.7828 0.7828 0.7828 0.7828 0.7828 0.7828 1 0.8648 0.8648 0.8648 0.8648 0.8648 0.8770 2 0.8279 0.8279 0.8279 0.8279 0.8238 0.8770 3 0.8279 0.8279 0.8279 0.8648 0.8443 0.8402 4 0.8279 0.8279 0.8279 0.8689 0.8484 0.8484 5 0.8484 0.8484 0.8484 0.8730 0.8648 0.8443 6 0.8443 0.8484 0.8484 0.8770 0.8648 0.8443 7 0.8443 0.8484 0.8484 0.8770 0.8648 0.8566 8 0.8566 0.8607 0.8484 0.8730 0.8607 0.8525 9 0.8566 0.8607 0.8484 0.8730 0.8648 0.8566 10 0.8607 0.8648 0.8484 0.8730 0.8648 0.8648 11 0.8525 0.8648 0.8484 0.8730 0.8648 0.8689 12 0.8525 0.8525 0.8484 0.8730 0.8648 0.8648 13 0.8525 0.8525 0.8484 0.8730 0.8648 0.8566 14 0.8525 0.8525 0.8443 0.8689 0.8607 0.8566 15 0.8525 0.8607 0.8443 0.8689 0.8607 0.8566 16 0.8525 0.8607 0.8443 0.8689 0.8607 0.8566 17 0.8525 0.8607 0.8443 0.8689 0.8607 0.8566 18 0.8525 0.8607 0.8443 0.8689 0.8607 0.8566 19 0.8525 0.8607 0.8525 0.8689 0.8525 0.8566 20 0.8525 0.8689 0.8525 0.8689 0.8484 0.8566 21 0.8525 0.8689 0.8525 0.8689 0.8484 0.8566 22 0.8525 0.8689 0.8525 0.8730 0.8484 0.8566 23 0.8525 0.8689 0.8525 0.8730 0.8484 0.8566 24 0.8525 0.8689 0.8525 0.8730 0.8484 0.8525 25 0.8525 0.8689 0.8525 0.8730 0.8484 0.8525 26 0.8525 0.8689 0.8566 0.8730 0.8484 0.8525 27 0.8525 0.8689 0.8607 0.8730 0.8484 0.8525 28 0.8525 0.8689 0.8607 0.8730 0.8484 0.8525 29 0.8525 0.8689 0.8607 0.8730 0.8484 0.8525 30 0.8525 0.8689 0.8566 0.8730 0.8484 0.8525 31 0.8525 0.8689 0.8566 0.8730 0.8484 0.8525 32 0.8525 0.8689 0.8566 0.8730 0.8484 0.8525 33 0.8525 0.8689 0.8566 0.8730 0.8484 0.8525 34 0.8525 0.8689 0.8566 0.8730 0.8484 0.8525 35 0.8525 0.8689 0.8566 0.8730 0.8484 0.8525 36 0.8525 0.8689 0.8566 0.8730 0.8484 0.8525 37 0.8525 0.8689 0.8566 0.8730 0.8484 0.8525 38 0.8525 0.8689 0.8566 0.8730 0.8484 - 39 0.8525 0.8689 0.8566 0.8730 0.8484 - 40 0.8525 0.8689 0.8566 0.8730 0.8484 - 162

Saginaw Bay: TREPAN (beam width 2)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

41 - 0.8689 0.8566 - 0.8484 - 42 - 0.8689 0.8566 - 0.8484 - 43 - - 0.8566 - 0.8484 - 44 - - - - 0.8484 - 163

Saginaw Bay: TREPAN (beam width 3)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.7828 0.7828 0.7828 0.7828 0.7828 0.7828 1 0.8893 0.8893 0.8893 0.8893 0.8893 0.8770 2 0.8934 0.8934 0.8934 0.8648 0.8852 0.8770 3 0.8811 0.8811 0.8811 0.8648 0.8852 0.8402 4 0.8811 0.8811 0.8811 0.8566 0.8852 0.8484 5 0.8811 0.8811 0.8811 0.8566 0.8770 0.8443 6 0.8852 0.8852 0.8811 0.8484 0.8770 0.8443 7 0.8852 0.8852 0.8607 0.8689 0.8852 0.8730 8 0.8852 0.8852 0.8770 0.8730 0.8852 0.8730 9 0.8730 0.8852 0.8770 0.8730 0.8811 0.8770 10 0.8730 0.8852 0.8811 0.8730 0.8811 0.8852 11 0.8730 0.8852 0.8770 0.8730 0.8770 0.8852 12 0.8689 0.8852 0.8770 0.8730 0.8770 0.8893 13 0.8689 0.8811 0.8730 0.8730 0.8770 0.8852 14 0.8648 0.8770 0.8730 0.8730 0.8648 0.8811 15 0.8648 0.8811 0.8730 0.8730 0.8648 0.8770 16 0.8648 0.8811 0.8730 0.8689 0.8648 0.8770 17 0.8648 0.8811 0.8730 0.8689 0.8648 0.8770 18 0.8648 0.8811 0.8770 0.8689 0.8566 0.8770 19 0.8648 0.8811 0.8770 0.8689 0.8566 0.8770 20 0.8648 0.8811 0.8770 0.8689 0.8648 0.8811 21 0.8648 0.8811 0.8811 0.8689 0.8648 0.8770 22 0.8648 0.8811 0.8648 0.8648 0.8648 0.8770 23 0.8648 0.8811 0.8648 0.8648 0.8689 0.8770 24 0.8648 0.8811 0.8648 0.8607 0.8689 0.8770 25 0.8648 0.8811 0.8689 0.8607 0.8689 0.8770 26 0.8648 0.8811 0.8689 0.8607 0.8689 0.8770 27 0.8648 0.8811 0.8689 0.8607 0.8689 0.8770 28 0.8648 0.8811 0.8689 0.8607 0.8648 0.8770 29 0.8648 0.8811 0.8689 0.8607 0.8689 0.8770 30 0.8648 0.8689 0.8648 0.8607 0.8689 0.8770 31 0.8648 0.8689 0.8648 0.8607 0.8689 0.8770 32 0.8648 0.8689 0.8648 0.8607 0.8689 0.8770 33 0.8648 0.8689 0.8648 0.8525 0.8689 0.8770 34 0.8648 0.8689 0.8648 0.8525 0.8689 0.8770 35 0.8648 0.8689 0.8648 0.8525 0.8689 0.8770 36 0.8648 0.8689 0.8648 0.8525 0.8770 0.8770 37 - 0.8689 0.8648 0.8443 0.8770 0.8770 38 - 0.8689 0.8648 0.8443 0.8770 0.8770 39 - 0.8689 0.8648 0.8443 0.8770 - 40 - 0.8689 0.8648 0.8443 0.8770 - 41 - - 0.8648 - 0.8770 - 42 - - 0.8648 - 0.8770 - 164

Saginaw Bay: TREPAN (beam width 5)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.7828 0.7828 0.7828 0.7828 0.7828 0.7828 1 0.8770 0.8770 0.8770 0.8770 0.8770 0.8770 2 0.8443 0.8443 0.8443 0.8402 0.8770 0.8566 3 0.8730 0.8730 0.8730 0.8770 0.8525 0.8566 4 0.8770 0.8770 0.8770 0.8770 0.8689 0.8566 5 0.8770 0.8770 0.8770 0.8770 0.8730 0.8484 6 0.8811 0.8811 0.8484 0.8770 0.8730 0.8484 7 0.8811 0.8811 0.8484 0.8730 0.8934 0.8648 8 0.8852 0.8852 0.8730 0.8607 0.8811 0.8607 9 0.8852 0.8852 0.8730 0.8607 0.8811 0.8607 10 0.8811 0.8811 0.8730 0.8607 0.8770 0.8607 11 0.8811 0.8811 0.8730 0.8607 0.8770 0.8689 12 0.8811 0.8811 0.8811 0.8607 0.8648 0.8689 13 0.8648 0.8811 0.8811 0.8689 0.8648 0.8689 14 0.8648 0.8811 0.8730 0.8689 0.8648 0.8689 15 0.8689 0.8811 0.8730 0.8689 0.8484 0.8689 16 0.8689 0.8811 0.8730 0.8689 0.8607 0.8648 17 0.8689 0.8811 0.8730 0.8730 0.8730 0.8648 18 0.8689 0.8852 0.8730 0.8730 0.8730 0.8648 19 0.8689 0.8770 0.8730 0.8730 0.8730 0.8402 20 0.8689 0.8770 0.8730 0.8730 0.8730 0.8361 21 0.8689 0.8770 0.8648 0.8730 0.8770 0.8361 22 0.8689 0.8770 0.8648 0.8852 0.8852 0.8361 23 0.8689 0.8770 0.8648 0.8852 0.8811 0.8361 24 0.8689 0.8770 0.8648 0.8852 0.8811 0.8361 25 0.8689 0.8770 0.8607 0.8852 0.8811 0.8361 26 0.8689 0.8770 0.8484 0.8730 0.8811 0.8402 27 0.8689 0.8770 0.8484 0.8852 0.8811 0.8402 28 0.8689 0.8770 0.8484 0.8770 0.8811 0.8402 29 0.8689 0.8770 0.8484 0.8689 0.8811 0.8402 30 0.8689 0.8770 0.8484 0.8689 0.8811 0.8402 31 0.8689 0.8770 0.8484 0.8689 0.8811 0.8402 32 0.8689 0.8689 0.8484 0.8689 0.8811 0.8402 33 0.8689 0.8689 0.8484 0.8689 0.8811 0.8402 34 0.8689 0.8689 0.8484 0.8689 0.8811 0.8402 35 0.8689 0.8689 0.8484 0.8689 0.8811 0.8402 36 - 0.8689 0.8484 0.8689 0.8811 0.8402 37 - 0.8648 0.8484 0.8689 0.8811 0.8402 38 - 0.8648 - 0.8689 0.8811 0.8402 39 - 0.8648 - 0.8689 0.8811 0.8402 40 - - - - 0.8811 0.8402 41 - - - - 0.8811 0.8402 42 - - - - - 0.8402 165

Saginaw Bay: TREPAN (beam width 5)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

43 - - - - - 0.8402 44 - - - - - 0.8402 45 - - - - - 0.8402 46 - - - - - 0.8402 47 - - - - - 0.8402 48 - - - - - 0.8402 49 - - - - - 0.8402 50 - - - - - 0.8402 166

Saginaw Bay: TREPAN (beam width 7)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

0 0.7828 0.7828 0.7828 0.7828 0.7828 0.7828 1 0.8893 0.8893 0.8893 0.8893 0.8893 0.8770 2 0.8893 0.8893 0.8893 0.8893 0.8893 0.8566 3 0.8893 0.8893 0.8525 0.8730 0.8893 0.8566 4 0.8689 0.8689 0.8689 0.8730 0.8852 0.8566 5 0.8689 0.8689 0.8689 0.8525 0.8893 0.8484 6 0.8730 0.8730 0.8607 0.8361 0.8893 0.8484 7 0.8566 0.8730 0.8607 0.8361 0.8852 0.8648 8 0.8607 0.8689 0.8607 0.8443 0.8852 0.8607 9 0.8566 0.8607 0.8607 0.8443 0.8852 0.8607 10 0.8566 0.8607 0.8607 0.8443 0.8852 0.8402 11 0.8607 0.8607 0.8648 0.8566 0.8852 0.8443 12 0.8607 0.8607 0.8648 0.8566 0.8893 0.8443 13 0.8525 0.8607 0.8689 0.8566 0.8893 0.8443 14 0.8525 0.8525 0.8689 0.8566 0.8893 0.8402 15 0.8525 0.8525 0.8730 0.8730 0.8893 0.8402 16 0.8525 0.8484 0.8730 0.8730 0.8893 0.8402 17 0.8525 0.8484 0.8730 0.8730 0.8893 0.8361 18 0.8525 0.8484 0.8730 0.8730 0.8893 0.8361 19 0.8607 0.8566 0.8730 0.8730 0.8893 0.8361 20 0.8607 0.8566 0.8730 0.8730 0.8893 0.8361 21 0.8607 0.8566 0.8689 0.8730 0.8893 0.8361 22 0.8607 0.8566 0.8689 0.8730 0.8893 0.8361 23 0.8607 0.8566 0.8689 0.8770 0.8893 0.8361 24 0.8607 0.8566 0.8689 0.8770 0.8852 0.8484 25 0.8607 0.8566 0.8607 0.8770 0.8852 0.8484 26 0.8607 0.8566 0.8443 0.8770 0.8852 0.8484 27 0.8607 0.8566 0.8443 0.8770 0.8852 0.8484 28 0.8607 0.8566 0.8443 0.8770 0.8852 0.8484 29 0.8607 0.8566 0.8443 0.8770 0.8852 0.8484 30 0.8607 0.8566 0.8443 0.8770 0.8852 0.8484 31 0.8607 0.8566 0.8443 0.8770 0.8852 0.8484 32 0.8607 0.8566 - 0.8770 0.8852 0.8484 33 - 0.8566 - 0.8770 0.8852 0.8484 34 - 0.8566 - - 0.8893 0.8484 35 - 0.8566 - - 0.8893 - 167

Saginaw Bay: TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.7828 0.7828 0.7828 0.7828 0.7828 0.7828 1 0.8770 0.8770 0.8770 0.8770 0.8770 0.8770 2 0.8443 0.8443 0.8443 0.8156 0.8770 0.8566 3 0.8730 0.8730 0.8730 0.8197 0.8525 0.8566 4 0.8770 0.8770 0.8730 0.8156 0.8689 0.8566 5 0.8770 0.8770 0.8607 0.8566 0.8730 0.8484 6 0.8770 0.8770 0.8607 0.8566 0.8730 0.8443 7 0.8811 0.8811 0.8607 0.8525 0.8770 0.8607 8 0.8811 0.8811 0.8525 0.8525 0.8648 0.8607 9 0.8770 0.8811 0.8525 0.8525 0.8607 0.8607 10 0.8811 0.8811 0.8525 0.8525 0.8566 0.8607 11 0.8811 0.8770 0.8525 0.8607 0.8484 0.8566 12 0.8811 0.8770 0.8566 0.8648 0.8320 0.8566 13 0.8648 0.8770 0.8566 0.8648 0.8279 0.8402 14 0.8689 0.8770 0.8566 0.8730 0.8320 0.8402 15 0.8689 0.8770 0.8566 0.8730 0.8320 0.8361 16 0.8689 0.8770 0.8566 0.8770 0.8320 0.8361 17 0.8689 0.8770 0.8566 0.8730 0.8197 0.8402 18 0.8689 0.8770 0.8566 0.8648 0.8197 0.8402 19 0.8689 0.8770 0.8566 0.8648 0.8361 0.8402 20 0.8689 0.8770 0.8484 0.8648 0.8402 0.8361 21 0.8689 0.8770 0.8484 0.8648 0.8402 0.8361 22 0.8689 0.8770 0.8484 0.8648 0.8402 0.8361 23 0.8689 0.8770 0.8566 0.8648 0.8402 0.8361 24 0.8689 0.8770 0.8566 0.8648 0.8402 0.8361 25 0.8689 0.8770 0.8566 0.8607 0.8402 0.8361 26 0.8689 0.8770 0.8566 0.8607 0.8402 0.8402 27 0.8689 0.8770 0.8566 0.8607 0.8402 0.8402 28 0.8689 0.8770 0.8525 0.8607 0.8402 0.8402 29 0.8689 0.8770 0.8525 0.8607 0.8402 0.8402 30 0.8689 0.8730 0.8525 0.8566 0.8443 0.8402 31 0.8689 0.8730 0.8525 0.8566 0.8443 0.8402 32 0.8689 0.8730 0.8525 0.8566 0.8443 0.8402 33 0.8689 0.8648 0.8525 0.8566 0.8443 0.8525 34 0.8689 0.8648 0.8525 0.8566 0.8443 0.8525 35 - 0.8607 0.8525 0.8566 0.8443 0.8525 36 - 0.8484 0.8525 0.8566 0.8443 0.8525 37 - 0.8402 - 0.8566 0.8443 0.8525 38 - - - 0.8566 0.8443 0.8525 39 - - - - 0.8443 0.8525 40 - - - - - 0.8525 168

Saginaw Bay: Disjunctive TREPAN (beam width 2)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.7828 0.7828 0.7828 0.7828 0.7828 0.7828 1 0.8648 0.8648 0.8648 0.8648 0.8648 0.8852 2 0.8279 0.8279 0.8279 0.8279 0.8238 0.8607 3 0.8279 0.8279 0.8279 0.8648 0.8525 0.8607 4 0.8279 0.8279 0.8279 0.8689 0.8566 0.8279 5 0.8484 0.8484 0.8484 0.8730 0.8566 0.8279 6 0.8443 0.8484 0.8484 0.8730 0.8566 0.8525 7 0.8279 0.8320 0.8484 0.8730 0.8566 0.8730 8 0.8402 0.8320 0.8484 0.8689 0.8566 0.8689 9 0.8402 0.8320 0.8484 0.8689 0.8566 0.8566 10 0.8443 0.8320 0.8484 0.8689 0.8525 0.8566 11 0.8607 0.8484 0.8484 0.8689 0.8525 0.8566 12 0.8525 0.8402 0.8443 0.8689 0.8566 0.8566 13 0.8525 0.8525 0.8443 0.8689 0.8566 0.8566 14 0.8525 0.8607 0.8402 0.8689 0.8566 0.8566 15 0.8525 0.8689 0.8443 0.8689 0.8566 0.8566 16 0.8525 0.8566 0.8443 0.8689 0.8566 0.8607 17 0.8525 0.8566 0.8443 0.8689 0.8648 0.8730 18 0.8525 0.8566 0.8443 0.8648 0.8730 0.8730 19 0.8525 0.8607 0.8566 0.8648 0.8730 0.8525 20 0.8525 0.8525 0.8566 0.8648 0.8689 0.8525 21 0.8525 0.8525 0.8566 0.8648 0.8689 0.8525 22 0.8525 0.8525 0.8566 0.8648 0.8689 0.8484 23 0.8525 0.8525 0.8566 0.8648 0.8689 0.8443 24 0.8525 0.8525 0.8566 0.8648 0.8689 0.8484 25 0.8525 0.8566 0.8566 0.8648 0.8689 0.8484 26 0.8525 0.8566 0.8566 0.8648 0.8730 0.8484 27 0.8525 0.8566 0.8566 0.8648 0.8730 0.8484 28 0.8525 0.8566 0.8566 0.8648 0.8730 0.8484 29 0.8525 0.8566 0.8566 0.8648 0.8730 0.8484 30 0.8525 0.8566 0.8566 0.8484 0.8730 0.8566 31 0.8525 0.8566 0.8566 0.8484 0.8730 0.8566 32 0.8525 0.8566 0.8566 0.8566 0.8770 0.8566 33 0.8525 0.8566 0.8566 0.8566 0.8770 0.8566 34 0.8525 0.8566 0.8566 0.8566 0.8811 0.8566 35 0.8525 0.8566 0.8566 0.8566 0.8811 0.8566 36 0.8525 0.8566 0.8525 0.8566 0.8770 0.8689 37 0.8525 0.8566 0.8525 0.8566 0.8770 0.8689 38 0.8525 0.8566 0.8525 0.8566 0.8770 0.8689 39 0.8525 0.8566 0.8525 0.8566 0.8730 0.8689 40 0.8525 0.8566 0.8525 0.8566 0.8730 0.8689 169

Saginaw Bay: Disjunctive TREPAN (beam width 2)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 41 0.8525 0.8566 0.8525 - 0.8730 0.8689 42 - 0.8566 0.8525 - - 0.8689 43 - 0.8566 0.8525 - - 0.8689 44 - - 0.8525 - - 0.8689 45 - - - - - 0.8689 170

Saginaw Bay: Disjunctive TREPAN (beam width 3)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.7828 0.7828 0.7828 0.7828 0.7828 0.7828 1 0.8648 0.8648 0.8648 0.8648 0.8648 0.8852 2 0.8279 0.8279 0.8279 0.8279 0.8238 0.8607 3 0.8279 0.8279 0.8279 0.8648 0.8525 0.8607 4 0.8279 0.8279 0.8279 0.8689 0.8566 0.8279 5 0.8484 0.8484 0.8484 0.8730 0.8566 0.8279 6 0.8443 0.8484 0.8484 0.8730 0.8566 0.8525 7 0.8279 0.8320 0.8484 0.8730 0.8566 0.8730 8 0.8402 0.8320 0.8484 0.8689 0.8566 0.8566 9 0.8402 0.8320 0.8484 0.8689 0.8566 0.8443 10 0.8443 0.8320 0.8484 0.8689 0.8566 0.8607 11 0.8607 0.8484 0.8484 0.8607 0.8525 0.8607 12 0.8525 0.8402 0.8443 0.8607 0.8525 0.8730 13 0.8525 0.8525 0.8443 0.8607 0.8484 0.8730 14 0.8525 0.8607 0.8402 0.8648 0.8484 0.8730 15 0.8525 0.8689 0.8443 0.8648 0.8443 0.8730 16 0.8525 0.8566 0.8443 0.8607 0.8443 0.8443 17 0.8525 0.8566 0.8443 0.8607 0.8443 0.8484 18 0.8525 0.8566 0.8443 0.8607 0.8443 0.8484 19 0.8525 0.8607 0.8566 0.8607 0.8443 0.8484 20 0.8525 0.8525 0.8566 0.8607 0.8443 0.8484 21 0.8525 0.8525 0.8566 0.8607 0.8443 0.8484 22 0.8525 0.8525 0.8566 0.8607 0.8443 0.8566 23 0.8525 0.8525 0.8566 0.8566 0.8484 0.8566 24 0.8525 0.8525 0.8566 0.8566 0.8361 0.8566 25 0.8525 0.8566 0.8566 0.8566 0.8361 0.8566 26 0.8525 0.8566 0.8566 0.8566 0.8361 0.8566 27 0.8525 0.8566 0.8566 0.8566 0.8361 0.8730 28 0.8525 0.8566 0.8566 0.8566 0.8361 0.8730 29 0.8525 0.8566 0.8566 0.8566 0.8361 0.8730 30 0.8525 0.8566 0.8566 0.8566 0.8361 0.8730 31 0.8525 0.8566 0.8566 0.8607 0.8361 0.8730 32 0.8525 0.8566 0.8566 0.8607 0.8361 0.8730 33 0.8525 0.8566 0.8566 0.8648 0.8361 0.8730 34 0.8525 0.8566 0.8566 0.8648 0.8361 0.8730 35 0.8525 0.8566 0.8566 0.8648 0.8361 0.8730 36 0.8525 0.8566 0.8525 0.8648 0.8361 0.8730 37 0.8525 0.8566 0.8525 0.8607 0.8361 0.8730 38 0.8525 0.8566 0.8525 0.8607 0.8361 0.8730 171

Saginaw Bay: Disjunctive TREPAN (beam width 3)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 39 0.8525 0.8566 0.8525 0.8607 0.8361 0.8730 40 0.8525 0.8566 0.8525 0.8607 0.8361 - 41 0.8525 0.8566 0.8525 - 0.8361 - 42 - 0.8566 0.8525 - 0.8361 - 43 - 0.8566 0.8525 - 0.8361 - 44 - - 0.8525 - 0.8361 - 172

Saginaw Bay: Disjunctive TREPAN (beam width 5)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.7828 0.7828 0.7828 0.7828 0.7828 0.7828 1 0.7828 0.7828 0.7828 0.7828 0.8934 0.8852 2 0.8197 0.8197 0.8197 0.8197 0.8934 0.8607 3 0.8402 0.8402 0.8402 0.8402 0.8566 0.8607 4 0.8402 0.8402 0.8402 0.8689 0.8689 0.8279 5 0.8566 0.8566 0.8566 0.8689 0.8648 0.8279 6 0.8566 0.8566 0.8566 0.8730 0.8566 0.8525 7 0.8566 0.8566 0.8484 0.8730 0.8566 0.8730 8 0.8525 0.8525 0.8484 0.8730 0.8689 0.8566 9 0.8525 0.8525 0.8484 0.8730 0.8689 0.8443 10 0.8566 0.8566 0.8484 0.8730 0.8689 0.8607 11 0.8607 0.8607 0.8484 0.8730 0.8689 0.8607 12 0.8443 0.8607 0.8484 0.8730 0.8689 0.8730 13 0.8443 0.8607 0.8525 0.8730 0.8689 0.8730 14 0.8402 0.8607 0.8525 0.8730 0.8689 0.8730 15 0.8402 0.8566 0.8525 0.8730 0.8607 0.8770 16 0.8402 0.8566 0.8566 0.8730 0.8607 0.8770 17 0.8402 0.8566 0.8525 0.8730 0.8689 0.8770 18 0.8402 0.8566 0.8525 0.8852 0.8770 0.8770 19 0.8443 0.8566 0.8525 0.8689 0.8770 0.8770 20 0.8443 0.8566 0.8525 0.8689 0.8770 0.8770 21 0.8443 0.8566 0.8525 0.8689 0.8770 0.8648 22 0.8443 0.8566 0.8525 0.8689 0.8770 0.8566 23 0.8443 0.8566 0.8525 0.8770 0.8770 0.8566 24 0.8443 0.8566 0.8566 0.8770 0.8648 0.8525 25 0.8443 0.8484 0.8566 0.8730 0.8648 0.8525 26 0.8443 0.8484 0.8566 0.8730 0.8648 0.8525 27 0.8443 0.8484 0.8566 0.8730 0.8648 0.8525 28 0.8443 0.8484 0.8566 0.8730 0.8648 0.8525 29 0.8443 0.8484 0.8566 0.8730 0.8648 0.8525 30 0.8443 0.8484 0.8566 0.8770 0.8648 0.8525 31 0.8443 0.8484 0.8566 0.8770 0.8689 0.8525 32 0.8443 0.8484 0.8566 0.8770 0.8689 0.8525 33 0.8443 0.8484 0.8566 0.8770 0.8689 0.8525 34 0.8443 0.8484 0.8566 0.8770 0.8689 0.8525 35 0.8443 0.8484 0.8566 0.8770 0.8689 0.8525 36 0.8443 0.8484 0.8566 0.8770 0.8730 0.8525 37 0.8443 0.8484 0.8566 0.8770 0.8730 0.8525 38 0.8443 0.8484 0.8566 0.8770 0.8689 - 173

Saginaw Bay: Disjunctive TREPAN (beam width 5)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 39 0.8443 0.8484 0.8566 0.8770 0.8689 - 40 0.8443 0.8484 0.8566 0.8770 0.8689 - 41 0.8443 0.8484 0.8566 0.8770 0.8689 - 42 0.8443 0.8484 0.8566 0.8770 0.8689 - 43 0.8443 0.8484 0.8566 0.8770 0.8689 - 44 - 0.8484 0.8566 0.8770 - - 45 - 0.8484 0.8566 0.8770 - - 46 - - 0.8566 0.8770 - - 47 - - - 0.8770 - - 174

Saginaw Bay: Disjunctive TREPAN (beam width 7)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.7828 0.7828 0.7828 0.7828 0.7828 0.7828 1 0.7828 0.7828 0.7828 0.7828 0.8934 0.8852 2 0.8197 0.8197 0.8197 0.8197 0.8934 0.8607 3 0.8402 0.8402 0.8402 0.8402 0.8566 0.8607 4 0.8811 0.8811 0.8811 0.8689 0.8689 0.8443 5 0.8607 0.8607 0.8730 0.8689 0.8648 0.8443 6 0.8607 0.8607 0.8770 0.8730 0.8566 0.8443 7 0.8607 0.8607 0.8770 0.8730 0.8566 0.8648 8 0.8607 0.8607 0.8770 0.8730 0.8689 0.8320 9 0.8607 0.8607 0.8770 0.8730 0.8689 0.8525 10 0.8566 0.8566 0.8852 0.8730 0.8689 0.8484 11 0.8402 0.8566 0.8852 0.8730 0.8689 0.8689 12 0.8320 0.8484 0.8811 0.8730 0.8689 0.8689 13 0.8320 0.8484 0.8811 0.8730 0.8689 0.8689 14 0.8402 0.8484 0.8852 0.8730 0.8689 0.8689 15 0.8484 0.8484 0.8852 0.8730 0.8566 0.8689 16 0.8484 0.8525 0.8852 0.8730 0.8566 0.8730 17 0.8443 0.8525 0.8852 0.8730 0.8648 0.8730 18 0.8484 0.8525 0.8852 0.8852 0.8648 0.8770 19 0.8484 0.8525 0.8852 0.8689 0.8607 0.8730 20 0.8484 0.8525 0.8852 0.8689 0.8607 0.8730 21 0.8525 0.8525 0.8852 0.8689 0.8607 0.8730 22 0.8525 0.8566 0.8852 0.8689 0.8607 0.8566 23 0.8525 0.8566 0.8893 0.8811 0.8607 0.8607 24 0.8525 0.8566 0.8893 0.8811 0.8566 0.8566 25 0.8525 0.8607 0.8893 0.8770 0.8566 0.8566 26 0.8525 0.8607 0.8893 0.8770 0.8566 0.8566 27 0.8525 0.8607 0.8893 0.8770 0.8566 0.8566 28 0.8525 0.8607 0.8893 0.8770 0.8566 0.8566 29 0.8525 0.8607 0.8893 0.8770 0.8566 0.8566 30 0.8525 0.8607 0.8730 0.8770 0.8525 0.8566 31 0.8525 0.8607 0.8730 0.8770 0.8525 0.8566 32 0.8525 0.8607 0.8730 0.8770 0.8525 0.8566 33 0.8525 0.8607 0.8730 0.8770 0.8525 0.8566 34 0.8525 0.8607 0.8730 0.8811 0.8525 0.8566 35 0.8525 0.8607 0.8730 0.8811 0.8525 0.8648 36 0.8525 0.8607 0.8730 0.8811 0.8566 0.8648 37 0.8525 0.8607 0.8730 0.8811 0.8566 0.8648 38 0.8525 0.8566 0.8730 0.8811 0.8566 0.8648 175

Saginaw Bay: Disjunctive TREPAN (beam width 7)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 39 0.8525 0.8566 0.8730 0.8811 0.8566 0.8648 40 0.8525 0.8566 0.8730 0.8811 0.8566 0.8648 41 0.8525 0.8566 0.8730 0.8811 0.8566 0.8648 42 0.8525 0.8566 0.8730 0.8811 0.8566 0.8648 43 0.8525 0.8566 0.8730 0.8811 - 0.8648 44 - 0.8566 0.8730 0.8811 - 0.8648 45 - 0.8566 0.8730 0.8811 - 0.8648 46 - 0.8566 0.8730 0.8811 - 0.8648 47 - - 0.8730 - - 0.8648 48 - - 0.8730 - - 0.8648 49 - - 0.8730 - - 0.8648 176

Saginaw Bay: Disjunctive TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.782787 0.782787 0.782787 0.782787 0.782787 0.782787 1 0.782787 0.782787 0.782787 0.782787 0.893443 0.885246 2 0.819672 0.819672 0.819672 0.819672 0.893443 0.860656 3 0.840164 0.840164 0.840164 0.840164 0.856557 0.860656 4 0.881148 0.881148 0.881148 0.868852 0.868852 0.836066 5 0.860656 0.860656 0.872951 0.868852 0.864754 0.836066 6 0.860656 0.860656 0.877049 0.872951 0.856557 0.848361 7 0.860656 0.860656 0.877049 0.872951 0.868852 0.836066 8 0.860656 0.860656 0.877049 0.872951 0.868852 0.856557 9 0.860656 0.860656 0.877049 0.872951 0.868852 0.852459 10 0.856557 0.856557 0.877049 0.872951 0.872951 0.852459 11 0.840164 0.856557 0.877049 0.872951 0.872951 0.852459 12 0.831967 0.848361 0.877049 0.872951 0.860656 0.852459 13 0.831967 0.848361 0.881148 0.872951 0.860656 0.852459 14 0.840164 0.848361 0.881148 0.872951 0.852459 0.860656 15 0.848361 0.848361 0.881148 0.872951 0.856557 0.860656 16 0.848361 0.852459 0.881148 0.872951 0.856557 0.868852 17 0.844262 0.852459 0.881148 0.872951 0.868852 0.868852 18 0.848361 0.852459 0.877049 0.885246 0.864754 0.868852 19 0.848361 0.852459 0.868852 0.868852 0.860656 0.877049 20 0.848361 0.852459 0.868852 0.868852 0.860656 0.877049 21 0.852459 0.852459 0.868852 0.868852 0.844262 0.864754 22 0.852459 0.856557 0.868852 0.868852 0.844262 0.864754 23 0.852459 0.856557 0.868852 0.881148 0.844262 0.864754 24 0.852459 0.856557 0.864754 0.881148 0.844262 0.864754 25 0.852459 0.860656 0.860656 0.877049 0.844262 0.860656 26 0.852459 0.860656 0.860656 0.877049 0.844262 0.860656 27 0.852459 0.860656 0.860656 0.877049 0.844262 0.864754 28 0.852459 0.860656 0.868852 0.877049 0.844262 0.864754 29 0.852459 0.860656 0.868852 0.877049 0.844262 0.860656 30 0.852459 0.860656 0.877049 0.877049 0.844262 0.860656 31 0.852459 0.860656 0.877049 0.877049 0.844262 0.860656 32 0.852459 0.860656 0.868852 0.877049 0.844262 0.860656 33 0.852459 0.860656 0.872951 0.877049 0.844262 0.856557 34 0.852459 0.860656 0.864754 0.881148 0.844262 0.856557 35 0.852459 0.860656 0.864754 0.881148 0.844262 0.856557 36 0.852459 0.860656 0.864754 0.881148 0.844262 0.856557 37 0.852459 0.860656 0.864754 0.881148 0.844262 0.856557 38 0.852459 0.856557 0.860656 0.881148 0.844262 0.856557 177

Saginaw Bay: Disjunctive TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 39 0.852459 0.856557 0.860656 0.881148 0.844262 0.860656 40 0.852459 0.856557 0.860656 0.881148 0.844262 0.860656 41 0.852459 0.856557 0.860656 0.881148 0.844262 0.860656 42 0.852459 0.856557 0.860656 0.881148 0.844262 0.860656 43 0.852459 0.856557 0.860656 0.881148 0.844262 0.860656 44 - 0.856557 0.860656 0.881148 0.844262 0.860656 45 - 0.856557 0.860656 0.881148 0.844262 - 46 - 0.856557 0.860656 - 0.844262 - 47 - - 0.860656 - - 48 - - 0.860656 - - 49 - - 0.860656 - - 50 - - 0.860656 - - 178

APPENDIX G: ADMISSIONS RESULTS

Admissions: TREPAN (beam width 2)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6448 0.6448 0.6448 0.6448 0.6448 0.6448 1 0.6929 0.6929 0.6929 0.6929 0.6929 0.6929 2 0.7004 0.7004 0.7004 0.7004 0.7004 0.7004 3 0.7004 0.7004 0.7004 0.7004 0.7004 0.7004 4 0.7004 0.7004 0.7004 0.7004 0.7004 0.7004 5 0.7016 0.7016 0.7016 0.7016 0.7004 0.7016 6 0.7016 0.7016 0.7016 0.7016 0.7054 0.7066 7 0.7016 0.7016 0.7016 0.7016 0.7054 0.7066 8 0.7060 0.7060 0.7060 0.7060 0.7097 0.7110 9 0.7060 0.7060 0.7060 0.7060 0.7097 0.7110 10 0.7066 0.7066 0.7066 0.7066 0.7097 0.7097 11 0.7066 0.7066 0.7066 0.7066 0.7104 0.7097 12 0.7085 0.7085 0.7085 0.7085 0.7122 0.7116 13 0.7110 0.7110 0.7110 0.7110 0.7116 0.7116 14 0.7110 0.7110 0.7110 0.7110 0.7116 0.7116 15 0.7122 0.7122 0.7122 0.7122 0.7072 0.7116 16 0.7122 0.7122 0.7122 0.7122 0.7072 0.7104 17 0.7122 0.7122 0.7122 0.7122 0.7097 0.7060 18 0.7122 0.7122 0.7122 0.7122 0.7116 0.7085 19 0.7122 0.7122 0.7122 0.7122 0.7116 0.7085 20 0.7122 0.7122 0.7122 0.7122 0.7116 0.7066 21 0.7122 0.7122 0.7122 0.7122 0.7116 0.7066 22 0.7122 0.7122 0.7122 0.7122 0.7116 0.7091 23 0.7104 0.7104 0.7104 0.7104 0.7116 0.7091 24 0.7085 0.7085 0.7085 0.7085 0.7116 0.7091 25 0.7066 0.7066 0.7066 0.7066 0.7097 0.7091 26 0.7066 0.7066 0.7066 0.7066 0.7091 0.7072 27 0.7066 0.7066 0.7066 0.7066 0.7091 0.7072 28 0.7066 0.7066 0.7066 0.7066 0.7091 0.7072 29 0.7072 0.7072 0.7072 0.7072 0.7091 0.7079 30 0.7072 0.7072 0.7072 0.7072 0.7072 0.7060 31 0.7072 0.7072 0.7072 0.7072 0.7072 0.7060 32 0.7072 0.7072 0.7072 0.7072 0.7072 0.7060 33 0.7072 0.7072 0.7072 0.7072 0.7066 0.7060 34 0.7072 0.7072 0.7072 0.7072 0.7066 0.7060 35 0.7079 0.7079 0.7079 0.7079 0.7060 0.7047 36 0.7079 0.7079 0.7079 0.7079 0.7060 0.7047 37 0.7079 0.7079 0.7079 0.7079 0.7060 0.7041 38 0.7079 0.7079 0.7079 0.7079 0.7060 0.7041 179

Admissions: TREPAN (beam width 2)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample Min sample Min sample Min sample Min sample Min sample 1 10 50 100 500 1000 39 0.7085 0.7085 0.7085 0.7085 0.7060 0.7022 40 0.7041 0.7041 0.7041 0.7085 0.7060 0.7029 41 0.7041 0.7041 0.7041 0.7085 0.7060 0.7029 42 0.7041 0.7041 0.7041 0.7097 0.7035 0.7022 43 0.7054 0.7054 0.7054 0.7110 0.7035 0.7022 44 0.7054 0.7054 0.7054 0.7110 0.7016 0.7041 45 0.7041 0.7041 0.7041 0.7097 0.7010 0.7022 46 0.7041 0.7041 0.7041 0.7097 0.7016 0.7010 47 0.7041 0.7041 0.7041 0.7097 0.6998 0.7010 48 0.7041 0.7041 0.7041 0.7097 0.6998 0.7022 49 0.7054 0.7054 0.7054 0.7110 0.6985 0.7004 50 0.7047 0.7047 0.7047 0.7104 0.6985 0.7004 180

Admissions: TREPAN (beam width 3)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

0 0.6448 0.6448 0.6448 0.6448 0.6448 0.6448 1 0.6929 0.6929 0.6929 0.6929 0.6929 0.6929 2 0.7004 0.7004 0.7004 0.7004 0.7004 0.7004 3 0.7004 0.7004 0.7004 0.7004 0.7004 0.7004 4 0.7004 0.7004 0.7004 0.7004 0.7004 0.7004 5 0.7016 0.7016 0.7016 0.7016 0.7004 0.7016 6 0.7016 0.7016 0.7016 0.7016 0.7054 0.7066 7 0.7016 0.7016 0.7016 0.7016 0.7054 0.7066 8 0.7060 0.7060 0.7060 0.7060 0.7097 0.7110 9 0.7060 0.7060 0.7060 0.7060 0.7097 0.7110 10 0.7066 0.7066 0.7066 0.7066 0.7097 0.7097 11 0.7066 0.7066 0.7066 0.7066 0.7104 0.7097 12 0.7085 0.7085 0.7085 0.7085 0.7122 0.7116 13 0.7110 0.7110 0.7110 0.7110 0.7116 0.7116 14 0.7110 0.7110 0.7110 0.7110 0.7116 0.7116 15 0.7122 0.7122 0.7122 0.7122 0.7072 0.7116 16 0.7122 0.7122 0.7122 0.7122 0.7072 0.7104 17 0.7122 0.7122 0.7122 0.7122 0.7097 0.7060 18 0.7122 0.7122 0.7122 0.7122 0.7116 0.7085 19 0.7122 0.7122 0.7122 0.7122 0.7116 0.7085 20 0.7122 0.7122 0.7122 0.7122 0.7116 0.7066 21 0.7122 0.7122 0.7122 0.7122 0.7116 0.7066 22 0.7122 0.7122 0.7122 0.7122 0.7116 0.7091 23 0.7104 0.7104 0.7104 0.7104 0.7116 0.7091 24 0.7085 0.7085 0.7085 0.7085 0.7116 0.7091 25 0.7066 0.7066 0.7066 0.7066 0.7097 0.7091 26 0.7066 0.7066 0.7066 0.7066 0.7091 0.7072 27 0.7066 0.7066 0.7066 0.7066 0.7091 0.7072 28 0.7066 0.7066 0.7066 0.7066 0.7091 0.7072 29 0.7072 0.7072 0.7072 0.7072 0.7091 0.7079 30 0.7072 0.7072 0.7072 0.7072 0.7072 0.7060 31 0.7072 0.7072 0.7072 0.7072 0.7072 0.7060 32 0.7072 0.7072 0.7072 0.7072 0.7072 0.7060 33 0.7072 0.7072 0.7072 0.7072 0.7066 0.7060 34 0.7072 0.7072 0.7072 0.7072 0.7066 0.7060 35 0.7079 0.7079 0.7079 0.7079 0.7060 0.7047 36 0.7079 0.7079 0.7079 0.7079 0.7060 0.7047 37 0.7079 0.7079 0.7079 0.7079 0.7060 0.7041 38 0.7079 0.7079 0.7079 0.7079 0.7060 0.7041 181

Admissions: TREPAN (beam width 3)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

39 0.7085 0.7085 0.7085 0.7085 0.7060 0.7022 40 0.7041 0.7041 0.7041 0.7085 0.7060 0.7029 41 0.7041 0.7041 0.7041 0.7085 0.7060 0.7029 42 0.7041 0.7041 0.7041 0.7097 0.7035 0.7022 43 0.7054 0.7054 0.7054 0.7110 0.7035 0.7022 44 0.7054 0.7054 0.7054 0.7110 0.7016 0.7041 45 0.7041 0.7041 0.7041 0.7097 0.7010 0.7022 46 0.7041 0.7041 0.7041 0.7097 0.7016 0.7010 47 0.7041 0.7041 0.7041 0.7097 0.6998 0.7010 48 0.7041 0.7041 0.7041 0.7097 0.6998 0.7022 49 0.7054 0.7054 0.7054 0.7110 0.6985 0.7004 50 0.7047 0.7047 0.7047 0.7104 0.6985 0.7004 182

Admissions: TREPAN (beam width 5)-Test accuracies at each node Classification Accuracy Node Min sample Min sample Min sample Min sample Min sample Min sample 1 10 50 100 500 1000 0 0.6448 0.6448 0.6448 0.6448 0.6448 0.6448 1 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 2 0.6991 0.6991 0.6991 0.6991 0.6991 0.6991 3 0.6991 0.6991 0.6991 0.6991 0.6991 0.6991 4 0.6991 0.6991 0.6991 0.6991 0.6991 0.6991 5 0.6991 0.6991 0.6991 0.6991 0.6991 0.6991 6 0.7060 0.7060 0.7060 0.7060 0.7060 0.7060 7 0.7060 0.7060 0.7060 0.7060 0.7060 0.7060 8 0.7066 0.7066 0.7066 0.7066 0.7066 0.7004 9 0.7066 0.7066 0.7066 0.7066 0.7066 0.7010 10 0.7122 0.7122 0.7122 0.7122 0.7122 0.7016 11 0.7141 0.7141 0.7141 0.7141 0.7129 0.7072 12 0.7147 0.7147 0.7147 0.7147 0.7147 0.7122 13 0.7147 0.7147 0.7147 0.7147 0.7147 0.7122 14 0.7147 0.7147 0.7147 0.7147 0.7147 0.7129 15 0.7141 0.7141 0.7141 0.7141 0.7141 0.7129 16 0.7141 0.7141 0.7141 0.7141 0.7141 0.7135 17 0.7172 0.7172 0.7172 0.7141 0.7197 0.7135 18 0.7172 0.7172 0.7172 0.7141 0.7197 0.7160 19 0.7166 0.7166 0.7166 0.7135 0.7191 0.7154 20 0.7160 0.7160 0.7160 0.7129 0.7191 0.7179 21 0.7160 0.7160 0.7160 0.7129 0.7191 0.7179 22 0.7160 0.7160 0.7160 0.7129 0.7191 0.7154 23 0.7141 0.7141 0.7141 0.7110 0.7141 0.7091 24 0.7141 0.7141 0.7141 0.7110 0.7104 0.7091 25 0.7141 0.7141 0.7141 0.7110 0.7104 0.7085 26 0.7141 0.7141 0.7141 0.7110 0.7079 0.7047 27 0.7141 0.7141 0.7141 0.7110 0.7079 0.7035 28 0.7072 0.7072 0.7072 0.7041 0.7079 0.7035 29 0.7072 0.7072 0.7072 0.7041 0.7085 0.7035 30 0.7072 0.7072 0.7072 0.7041 0.7085 0.7029 31 0.7072 0.7072 0.7072 0.7054 0.7129 0.7016 32 0.7116 0.7116 0.7116 0.7097 0.7104 0.7072 33 0.7110 0.7110 0.7110 0.7091 0.7104 0.7072 34 0.7072 0.7072 0.7072 0.7054 0.7104 0.7085 35 0.7085 0.7085 0.7085 0.7066 0.7104 0.7085 36 0.7085 0.7085 0.7085 0.7066 0.7104 0.7091 37 0.7085 0.7085 0.7085 0.7066 0.7104 0.7091 38 0.7085 0.7085 0.7085 0.7066 0.7104 0.7091 183

Admissions: TREPAN (beam width 5)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

39 0.7085 0.7085 0.7085 0.7066 0.7097 0.7104 40 0.7085 0.7085 0.7085 0.7066 0.7097 0.7104 41 0.7085 0.7085 0.7085 0.7066 0.7060 0.7110 42 0.7085 0.7085 0.7085 0.7066 0.7060 0.7110 43 0.7085 0.7085 0.7085 0.7066 0.7060 0.7110 44 0.7085 0.7085 0.7085 0.7079 0.7041 0.7091 45 0.7097 0.7097 0.7085 0.7079 0.7041 0.7091 46 0.7097 0.7097 0.7085 0.7079 0.7041 0.7091 47 0.7097 0.7097 0.7085 0.7079 0.7041 0.7091 48 0.7097 0.7097 0.7085 0.7079 0.7041 0.7091 49 0.7097 0.7097 0.7085 0.7079 0.7041 0.7091 50 0.7097 0.7097 0.7085 0.7079 0.7047 0.7091 184

Admissions: TREPAN (beam width 7)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

0 0.6448 0.6448 0.6448 0.6448 0.6448 0.6448 1 0.6854 0.6854 0.6854 0.6854 0.6854 0.6854 2 0.6973 0.6973 0.6973 0.6973 0.6973 0.6973 3 0.6973 0.6973 0.6973 0.6973 0.6973 0.6973 4 0.6973 0.6973 0.6973 0.6973 0.6973 0.6973 5 0.7104 0.7104 0.7104 0.7104 0.6973 0.7104 6 0.7104 0.7104 0.7104 0.7104 0.6973 0.7104 7 0.7104 0.7104 0.7104 0.7104 0.6973 0.7104 8 0.7135 0.7135 0.7135 0.7135 0.7004 0.7135 9 0.7141 0.7141 0.7141 0.7141 0.7010 0.7135 10 0.7141 0.7141 0.7141 0.7141 0.6954 0.7135 11 0.7160 0.7160 0.7160 0.7160 0.6954 0.7141 12 0.7179 0.7179 0.7179 0.7179 0.6954 0.7141 13 0.7179 0.7179 0.7179 0.7179 0.6954 0.7147 14 0.7179 0.7179 0.7179 0.7179 0.6954 0.7147 15 0.7122 0.7122 0.7122 0.7122 0.7029 0.7072 16 0.7154 0.7154 0.7154 0.7122 0.7054 0.7097 17 0.7154 0.7154 0.7154 0.7122 0.7054 0.7097 18 0.7154 0.7154 0.7154 0.7122 0.7047 0.7097 19 0.7147 0.7147 0.7147 0.7116 0.7054 0.7085 20 0.7147 0.7147 0.7147 0.7116 0.7072 0.7091 21 0.7147 0.7147 0.7147 0.7116 0.7072 0.7091 22 0.7154 0.7154 0.7154 0.7122 0.7072 0.7091 23 0.7154 0.7154 0.7154 0.7122 0.7072 0.7091 24 0.7154 0.7154 0.7154 0.7122 0.7079 0.7091 25 0.7135 0.7135 0.7135 0.7104 0.7060 0.7085 26 0.7135 0.7135 0.7135 0.7104 0.7060 0.7079 27 0.7135 0.7135 0.7135 0.7104 0.7035 0.7079 28 0.7135 0.7135 0.7135 0.7104 0.7035 0.7129 29 0.7066 0.7066 0.7066 0.7035 0.7022 0.7122 30 0.7066 0.7066 0.7066 0.7035 0.7022 0.7110 31 0.7066 0.7066 0.7066 0.7035 0.7022 0.7110 32 0.7066 0.7066 0.7066 0.7035 0.7022 0.7110 33 0.7060 0.7060 0.7060 0.7029 0.7010 0.7116 34 0.7066 0.7066 0.7066 0.7035 0.7010 0.7116 35 0.7110 0.7110 0.7110 0.7079 0.6998 0.7116 36 0.7110 0.7110 0.7110 0.7079 0.7041 0.7122 37 0.7110 0.7110 0.7110 0.7079 0.7041 0.7129 38 0.7110 0.7110 0.7110 0.7079 0.7035 0.7129 185

Admissions: TREPAN (beam width 7)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000

39 0.7110 0.7110 0.7110 0.7079 0.7047 0.7122 40 0.7110 0.7110 0.7110 0.7079 0.7085 0.7110 41 0.7110 0.7110 0.7110 0.7079 0.7085 0.7110 42 0.7110 0.7110 0.7072 0.7079 0.7085 0.7104 43 0.7122 0.7122 0.7085 0.7091 0.7091 0.7104 44 0.7154 0.7154 0.7116 0.7091 0.7091 0.7104 45 0.7160 0.7160 0.7122 0.7091 0.7091 0.7104 46 0.7160 0.7160 0.7122 0.7091 0.7091 0.7104 47 0.7160 0.7160 0.7122 0.7104 0.7091 0.7104 48 0.7160 0.7160 0.7122 0.7104 0.7091 0.7104 49 0.7160 0.7160 0.7122 0.7104 0.7091 0.7104 50 0.7141 0.7141 0.7122 0.7104 0.7091 0.7097 186

Admissions: TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6448 0.6448 0.6448 0.6448 0.6448 0.6448 1 0.6854 0.6929 0.6929 0.6929 0.6929 0.6929 2 0.6973 0.7004 0.7004 0.7004 0.7004 0.7004 3 0.6973 0.7004 0.7004 0.7004 0.7004 0.7004 4 0.6973 0.7004 0.7004 0.7004 0.7004 0.7004 5 0.7104 0.7016 0.7016 0.7016 0.7004 0.7016 6 0.7104 0.7016 0.7016 0.7016 0.7054 0.7066 7 0.7104 0.7016 0.7016 0.7016 0.7054 0.7066 8 0.7135 0.7060 0.7060 0.7060 0.7097 0.7110 9 0.7141 0.7060 0.7060 0.7060 0.7097 0.7110 10 0.7141 0.7066 0.7066 0.7066 0.7097 0.7097 11 0.7160 0.7066 0.7066 0.7066 0.7104 0.7097 12 0.7179 0.7085 0.7085 0.7085 0.7122 0.7116 13 0.7179 0.7110 0.7110 0.7110 0.7116 0.7116 14 0.7179 0.7110 0.7110 0.7110 0.7116 0.7116 15 0.7122 0.7122 0.7122 0.7122 0.7072 0.7116 16 0.7154 0.7122 0.7122 0.7122 0.7072 0.7104 17 0.7154 0.7122 0.7122 0.7122 0.7097 0.7060 18 0.7154 0.7122 0.7122 0.7122 0.7116 0.7085 19 0.7147 0.7122 0.7122 0.7122 0.7116 0.7085 20 0.7147 0.7122 0.7122 0.7122 0.7116 0.7066 21 0.7147 0.7122 0.7122 0.7122 0.7116 0.7066 22 0.7154 0.7122 0.7122 0.7122 0.7116 0.7091 23 0.7154 0.7104 0.7104 0.7104 0.7116 0.7091 24 0.7154 0.7085 0.7085 0.7085 0.7116 0.7091 25 0.7135 0.7066 0.7066 0.7066 0.7097 0.7091 26 0.7135 0.7066 0.7066 0.7066 0.7091 0.7072 27 0.7135 0.7066 0.7066 0.7066 0.7091 0.7072 28 0.7135 0.7066 0.7066 0.7066 0.7091 0.7072 29 0.7066 0.7072 0.7072 0.7072 0.7091 0.7079 30 0.7066 0.7072 0.7072 0.7072 0.7072 0.7060 31 0.7066 0.7072 0.7072 0.7072 0.7072 0.7060 32 0.7066 0.7072 0.7072 0.7072 0.7072 0.7060 33 0.7060 0.7072 0.7072 0.7072 0.7066 0.7060 34 0.7066 0.7072 0.7072 0.7072 0.7066 0.7060 35 0.7110 0.7079 0.7079 0.7079 0.7060 0.7047 36 0.7110 0.7079 0.7079 0.7079 0.7060 0.7047 37 0.7110 0.7079 0.7079 0.7079 0.7060 0.7041 38 0.7110 0.7079 0.7079 0.7079 0.7060 0.7041 187

Admissions: TREPAN (beam width 10)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 39 0.7110 0.7085 0.7085 0.7085 0.7060 0.7022 40 0.7110 0.7041 0.7041 0.7085 0.7060 0.7029 41 0.7110 0.7041 0.7041 0.7085 0.7060 0.7029 42 0.7110 0.7041 0.7041 0.7097 0.7035 0.7022 43 0.7122 0.7054 0.7054 0.7110 0.7035 0.7022 44 0.7154 0.7054 0.7054 0.7110 0.7016 0.7041 45 0.7160 0.7041 0.7041 0.7097 0.7010 0.7022 46 0.7160 0.7041 0.7041 0.7097 0.7016 0.7010 47 0.7160 0.7041 0.7041 0.7097 0.6998 0.7010 48 0.7160 0.7041 0.7041 0.7097 0.6998 0.7022 49 0.7160 0.7054 0.7054 0.7110 0.6985 0.7004 50 0.7141 0.7047 0.7047 0.7104 0.6985 0.7004 188

Admissions: Disjunctive TREPAN (beam width 2)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6448 0.6448 0.6448 0.6448 0.6448 0.6448 1 0.6779 0.6779 0.6779 0.6779 0.6779 0.6779 2 0.6966 0.6966 0.6966 0.6966 0.6966 0.6966 3 0.6985 0.6985 0.6985 0.6985 0.6985 0.6985 4 0.6985 0.6985 0.6985 0.6985 0.6985 0.6985 5 0.6841 0.6841 0.6841 0.6841 0.6841 0.6841 6 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 7 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 8 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 9 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 10 0.6910 0.6910 0.6910 0.6910 0.6910 0.6873 11 0.6966 0.6966 0.6966 0.6966 0.6966 0.7004 12 0.7022 0.7022 0.7022 0.7022 0.7022 0.6879 13 0.6985 0.6985 0.6985 0.6985 0.6985 0.6979 14 0.7016 0.7016 0.7016 0.7016 0.7016 0.7047 15 0.6954 0.6954 0.6954 0.6954 0.6954 0.7047 16 0.6954 0.6954 0.6954 0.6954 0.6954 0.6998 17 0.7016 0.7016 0.7016 0.7016 0.6941 0.6998 18 0.7116 0.7116 0.7116 0.7116 0.7041 0.6998 19 0.7116 0.7116 0.7116 0.7116 0.7041 0.6998 20 0.7072 0.7072 0.7072 0.7072 0.7054 0.7022 21 0.7060 0.7060 0.7060 0.7060 0.7054 0.7072 22 0.7004 0.7004 0.7004 0.7004 0.7054 0.7072 23 0.7004 0.7004 0.7004 0.7004 0.7054 0.7066 24 0.7010 0.7010 0.7010 0.7010 0.7054 0.7035 25 0.7010 0.7010 0.7010 0.7010 0.7072 0.7035 26 0.7047 0.7047 0.7047 0.7047 0.7110 0.7022 27 0.6991 0.6991 0.6991 0.6991 0.7110 0.6991 28 0.6991 0.6991 0.6991 0.6991 0.7160 0.6991 29 0.7029 0.7029 0.7029 0.7029 0.7122 0.6991 30 0.7035 0.7035 0.7035 0.7035 0.7097 0.6991 31 0.7054 0.7054 0.7054 0.7054 0.7097 0.6985 32 0.7104 0.7104 0.7104 0.7104 0.7041 0.7022 33 0.7079 0.7079 0.7079 0.7079 0.7010 0.7022 34 0.7110 0.7110 0.7110 0.7110 0.7054 0.7060 35 0.7104 0.7104 0.7104 0.7104 0.7085 0.7060 36 0.7104 0.7104 0.7104 0.7104 0.7085 0.7060 37 0.7104 0.7104 0.7104 0.7104 0.7054 0.7060 38 0.7066 0.7066 0.7066 0.7104 0.7054 0.7016 189

Admissions: Disjunctive TREPAN (beam width 2)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 39 0.7054 0.7054 0.7054 0.7091 0.7054 0.7029 40 0.7054 0.7054 0.7054 0.7091 0.7054 0.7029 41 0.7047 0.7047 0.7047 0.7091 0.7054 0.6998 42 0.7066 0.7066 0.7047 0.7091 0.7054 0.7004 43 0.7066 0.7066 0.7047 0.7091 0.7079 0.7004 44 0.7072 0.7072 0.7047 0.7097 0.7110 0.7004 45 0.7072 0.7072 0.7047 0.7097 0.7110 0.6998 46 0.7072 0.7072 0.7047 0.7097 0.7110 0.6998 47 0.7072 0.7072 0.7047 0.7097 0.7110 0.7004 48 0.7035 0.7035 0.7010 0.7060 0.7110 0.7004 49 0.7035 0.7035 0.7010 0.7035 0.7110 0.6998 50 0.7047 0.7047 0.7022 0.7035 0.7110 0.6998 190

Admissions: Disjunctive TREPAN (beam width 3)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6448 0.6448 0.6448 0.6448 0.6448 0.6448 1 0.6779 0.6779 0.6779 0.6779 0.6779 0.6779 2 0.6966 0.6966 0.6966 0.6966 0.6966 0.6966 3 0.6985 0.6985 0.6985 0.6985 0.6985 0.6985 4 0.6985 0.6985 0.6985 0.6985 0.6985 0.6985 5 0.6841 0.6841 0.6841 0.6841 0.6841 0.6841 6 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 7 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 8 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 9 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 10 0.6910 0.6910 0.6910 0.6910 0.6910 0.6873 11 0.6966 0.6966 0.6966 0.6966 0.6966 0.7004 12 0.7022 0.7022 0.7022 0.7022 0.7022 0.6879 13 0.6985 0.6985 0.6985 0.6985 0.6985 0.6979 14 0.7016 0.7016 0.7016 0.7016 0.7016 0.7047 15 0.6954 0.6954 0.6954 0.6954 0.6954 0.7047 16 0.6954 0.6954 0.6954 0.6954 0.6954 0.6998 17 0.7016 0.7016 0.7016 0.7016 0.6941 0.6998 18 0.7116 0.7116 0.7116 0.7116 0.7041 0.6998 19 0.7116 0.7116 0.7116 0.7116 0.7041 0.6998 20 0.7072 0.7072 0.7072 0.7072 0.7054 0.7022 21 0.7060 0.7060 0.7060 0.7060 0.7054 0.7072 22 0.7004 0.7004 0.7004 0.7004 0.7054 0.7072 23 0.7004 0.7004 0.7004 0.7004 0.7054 0.7066 24 0.7010 0.7010 0.7010 0.7010 0.7054 0.7035 25 0.7010 0.7010 0.7010 0.7010 0.7072 0.7035 26 0.7047 0.7047 0.7047 0.7047 0.7110 0.7022 27 0.6991 0.6991 0.6991 0.6991 0.7110 0.6991 28 0.6991 0.6991 0.6991 0.6991 0.7160 0.6991 29 0.7029 0.7029 0.7029 0.7029 0.7122 0.6991 30 0.7035 0.7035 0.7035 0.7035 0.7097 0.6991 31 0.7054 0.7054 0.7054 0.7054 0.7097 0.6985 32 0.7104 0.7104 0.7104 0.7104 0.7041 0.7022 33 0.7079 0.7079 0.7079 0.7079 0.7041 0.7022 34 0.7110 0.7110 0.7110 0.7110 0.7079 0.7060 35 0.7110 0.7110 0.7110 0.7110 0.7110 0.7060 36 0.7104 0.7104 0.7104 0.7104 0.7110 0.7060 37 0.7104 0.7104 0.7104 0.7104 0.7079 0.7054 38 0.7066 0.7066 0.7066 0.7066 0.7079 0.7010 191

Admissions: Disjunctive TREPAN (beam width 3)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 39 0.7054 0.7054 0.7054 0.7054 0.7079 0.7022 40 0.7054 0.7054 0.7054 0.7054 0.7079 0.7016 41 0.7047 0.7047 0.7047 0.7041 0.7072 0.6985 42 0.7066 0.7066 0.7047 0.7041 0.7104 0.6991 43 0.7066 0.7066 0.7047 0.7041 0.7141 0.6991 44 0.7072 0.7072 0.7054 0.7047 0.7141 0.6998 45 0.7072 0.7072 0.7054 0.7047 0.7141 0.6991 46 0.7072 0.7072 0.7054 0.7047 0.7141 0.6998 47 0.7072 0.7072 0.7054 0.7047 0.7141 0.6998 48 0.7035 0.7035 0.7016 0.7010 0.7141 0.6991 49 0.7035 0.7035 0.7016 0.7010 0.7147 0.6991 50 0.7047 0.7047 0.7029 0.7004 0.7147 0.6991 192

Admissions: Disjunctive TREPAN (beam width 5)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6448 0.6448 0.6448 0.6448 0.6448 0.6448 1 0.6779 0.6779 0.6779 0.6779 0.6779 0.6779 2 0.6966 0.6966 0.6966 0.6966 0.6966 0.6966 3 0.6985 0.6985 0.6985 0.6985 0.6985 0.6985 4 0.6985 0.6985 0.6985 0.6985 0.6985 0.6985 5 0.6841 0.6841 0.6841 0.6841 0.6841 0.6841 6 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 7 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 8 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 9 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 10 0.6910 0.6910 0.6910 0.6910 0.6910 0.6873 11 0.6966 0.6966 0.6966 0.6966 0.6966 0.7004 12 0.7022 0.7022 0.7022 0.7022 0.7022 0.6879 13 0.6985 0.6985 0.6985 0.6985 0.6985 0.6979 14 0.7016 0.7016 0.7016 0.7016 0.7016 0.7047 15 0.6954 0.6954 0.6954 0.6954 0.6954 0.7047 16 0.6954 0.6954 0.6954 0.6954 0.6954 0.6998 17 0.7016 0.7016 0.7016 0.7016 0.6941 0.6998 18 0.7116 0.7116 0.7116 0.7116 0.7041 0.6998 19 0.7116 0.7116 0.7116 0.7116 0.7041 0.6998 20 0.7072 0.7072 0.7072 0.7072 0.7054 0.7022 21 0.7060 0.7060 0.7060 0.7060 0.7054 0.7072 22 0.7004 0.7004 0.7004 0.7004 0.7054 0.7072 23 0.7004 0.7004 0.7004 0.7004 0.7054 0.7066 24 0.7010 0.7010 0.7010 0.7010 0.7054 0.7035 25 0.7010 0.7010 0.7010 0.7010 0.7072 0.7035 26 0.7047 0.7047 0.7047 0.7047 0.7110 0.7022 27 0.6991 0.6991 0.6991 0.6991 0.7110 0.6991 28 0.6991 0.6991 0.6991 0.6991 0.7160 0.6991 29 0.7029 0.7029 0.7029 0.7029 0.7122 0.6991 30 0.7035 0.7035 0.7035 0.7035 0.7097 0.6991 31 0.7054 0.7054 0.7054 0.7054 0.7097 0.6985 32 0.7104 0.7104 0.7104 0.7104 0.7041 0.7022 33 0.7079 0.7079 0.7079 0.7079 0.7041 0.7022 34 0.7110 0.7110 0.7110 0.7110 0.7079 0.7060 35 0.7110 0.7110 0.7110 0.7110 0.7110 0.7060 36 0.7104 0.7104 0.7104 0.7104 0.7110 0.7060 37 0.7104 0.7104 0.7104 0.7104 0.7079 0.7054 38 0.7066 0.7066 0.7066 0.7066 0.7079 0.7010 193

Admissions: Disjunctive TREPAN (beam width 5)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 39 0.7054 0.7054 0.7054 0.7054 0.7079 0.7022 40 0.7054 0.7054 0.7054 0.7054 0.7079 0.7016 41 0.7047 0.7047 0.7047 0.7041 0.7072 0.6985 42 0.7066 0.7066 0.7047 0.7041 0.7104 0.6991 43 0.7066 0.7066 0.7047 0.7041 0.7141 0.6991 44 0.7072 0.7072 0.7054 0.7047 0.7141 0.6998 45 0.7072 0.7072 0.7054 0.7047 0.7141 0.6991 46 0.7072 0.7072 0.7054 0.7047 0.7141 0.6998 47 0.7072 0.7072 0.7054 0.7047 0.7141 0.6998 48 0.7035 0.7035 0.7016 0.7010 0.7141 0.6991 49 0.7035 0.7035 0.7016 0.7010 0.7147 0.6991 50 0.7047 0.7047 0.7029 0.7004 0.7147 0.6991 194

Admissions: Disjunctive TREPAN (beam width 7)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6448 0.6448 0.6448 0.6448 0.6448 0.6448 1 0.6779 0.6779 0.6779 0.6779 0.6779 0.6779 2 0.6966 0.6966 0.6966 0.6966 0.6966 0.6966 3 0.6985 0.6985 0.6985 0.6985 0.6985 0.6985 4 0.6985 0.6985 0.6985 0.6985 0.6985 0.6985 5 0.6841 0.6841 0.6841 0.6841 0.6841 0.6841 6 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 7 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 8 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 9 0.6910 0.6910 0.6910 0.6910 0.6910 0.6910 10 0.6910 0.6910 0.6910 0.6910 0.6910 0.6873 11 0.6966 0.6966 0.6966 0.6966 0.6966 0.7004 12 0.7022 0.7022 0.7022 0.7022 0.7022 0.6879 13 0.6985 0.6985 0.6985 0.6985 0.6985 0.6979 14 0.7016 0.7016 0.7016 0.7016 0.7016 0.7047 15 0.6954 0.6954 0.6954 0.6954 0.6954 0.7047 16 0.6954 0.6954 0.6954 0.6954 0.6954 0.6998 17 0.7016 0.7016 0.7016 0.7016 0.6941 0.6998 18 0.7116 0.7116 0.7116 0.7116 0.7041 0.6998 19 0.7116 0.7116 0.7116 0.7116 0.7041 0.6998 20 0.7072 0.7072 0.7072 0.7072 0.7054 0.7022 21 0.7060 0.7060 0.7060 0.7060 0.7054 0.7072 22 0.7004 0.7004 0.7004 0.7004 0.7054 0.7072 23 0.7004 0.7004 0.7004 0.7004 0.7054 0.7066 24 0.7010 0.7010 0.7010 0.7010 0.7054 0.7035 25 0.7010 0.7010 0.7010 0.7010 0.7072 0.7035 26 0.7047 0.7047 0.7047 0.7047 0.7110 0.7022 27 0.6991 0.6991 0.6991 0.6991 0.7110 0.6991 28 0.6991 0.6991 0.6991 0.6991 0.7160 0.6991 29 0.7029 0.7029 0.7029 0.7029 0.7122 0.6991 30 0.7035 0.7035 0.7035 0.7035 0.7097 0.6991 31 0.7054 0.7054 0.7054 0.7054 0.7097 0.6985 32 0.7104 0.7104 0.7104 0.7104 0.7041 0.7022 33 0.7079 0.7079 0.7079 0.7079 0.7041 0.7022 34 0.7110 0.7110 0.7110 0.7110 0.7079 0.7060 35 0.7097 0.7097 0.7097 0.7097 0.7110 0.7060 36 0.7091 0.7091 0.7091 0.7091 0.7110 0.7060 37 0.7091 0.7091 0.7091 0.7091 0.7079 0.7054 38 0.7054 0.7054 0.7054 0.7054 0.7079 0.7010 195

Admissions: Disjunctive TREPAN (beam width 7)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 39 0.7041 0.7041 0.7041 0.7041 0.7079 0.7022 40 0.7041 0.7041 0.7041 0.7041 0.7079 0.7016 41 0.7035 0.7035 0.7035 0.7041 0.7072 0.6985 42 0.7054 0.7054 0.7035 0.7041 0.7104 0.6991 43 0.7054 0.7054 0.7035 0.7041 0.7141 0.6991 44 0.7060 0.7060 0.7041 0.7047 0.7141 0.6998 45 0.7060 0.7060 0.7041 0.7047 0.7141 0.6985 46 0.7060 0.7060 0.7041 0.7047 0.7141 0.7016 47 0.7060 0.7060 0.7041 0.7047 0.7141 0.7016 48 0.7022 0.7022 0.7004 0.7047 0.7141 0.7010 49 0.7022 0.7022 0.7004 0.7047 0.7147 0.7010 50 0.7035 0.7035 0.7016 0.7041 0.7147 0.7010 196

Admissions: Disjunctive TREPAN (beam width 10)-Test accuracies at each node Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 0 0.6448 0.6448 0.6448 0.6448 0.6448 0.6448 1 0.6779 0.6779 0.6779 0.6779 0.6779 0.6779 2 0.6991 0.6991 0.6991 0.6991 0.6991 0.6991 3 0.7022 0.7022 0.7022 0.7022 0.7022 0.7022 4 0.7022 0.7022 0.7022 0.7022 0.7022 0.7022 5 0.7022 0.7022 0.7022 0.7022 0.7022 0.7022 6 0.6941 0.6941 0.6941 0.6941 0.6941 0.6941 7 0.6941 0.6941 0.6941 0.6941 0.6941 0.6941 8 0.6941 0.6941 0.6941 0.6941 0.6941 0.6998 9 0.6929 0.6929 0.6929 0.6929 0.6929 0.7047 10 0.6966 0.6966 0.6966 0.6966 0.6966 0.7047 11 0.7085 0.7085 0.7085 0.7085 0.7085 0.7004 12 0.7085 0.7085 0.7085 0.7085 0.7085 0.7022 13 0.7135 0.7135 0.7135 0.7135 0.7135 0.7022 14 0.7129 0.7129 0.7129 0.7129 0.7129 0.7085 15 0.7129 0.7129 0.7129 0.7129 0.7129 0.7085 16 0.7129 0.7129 0.7129 0.7129 0.7129 0.7129 17 0.7091 0.7091 0.7091 0.7091 0.7054 0.7166 18 0.7091 0.7091 0.7091 0.7091 0.7054 0.7166 19 0.7054 0.7054 0.7054 0.7054 0.7047 0.7166 20 0.7079 0.7079 0.7079 0.7079 0.6973 0.7166 21 0.7072 0.7072 0.7072 0.7072 0.6973 0.7166 22 0.7041 0.7041 0.7041 0.7041 0.6973 0.7135 23 0.7010 0.7010 0.7010 0.7010 0.6960 0.7135 24 0.7010 0.7010 0.7010 0.7010 0.7016 0.7197 25 0.7047 0.7047 0.7047 0.7047 0.7016 0.7185 26 0.7029 0.7029 0.7029 0.7029 0.7079 0.7185 27 0.7029 0.7029 0.7029 0.7029 0.7079 0.7166 28 0.7010 0.7010 0.7010 0.7029 0.7079 0.7166 29 0.7010 0.7010 0.7010 0.7029 0.7072 0.7141 30 0.7010 0.7010 0.7010 0.7029 0.7047 0.7141 31 0.7022 0.7022 0.7022 0.7041 0.7047 0.7141 32 0.7029 0.7029 0.7029 0.7047 0.7060 0.7154 33 0.7035 0.7035 0.7035 0.7047 0.7079 0.7154 34 0.7047 0.7047 0.7047 0.7060 0.7079 0.7154 35 0.7047 0.7047 0.7047 0.7060 0.7079 0.7147 36 0.7047 0.7047 0.7047 0.7060 0.7079 0.7147 37 0.7029 0.7029 0.7029 0.7041 0.7060 0.7110 38 0.7029 0.7029 0.7029 0.7041 0.7060 0.7079 197

Admissions: Disjunctive TREPAN (beam width 10)-Test accuracies at each node (Continued) Classification Accuracy Node Min sample 1 Min sample 10 Min sample 50 Min sample 100 Min sample 500 Min sample 1000 39 0.7029 0.7029 0.7029 0.7041 0.7072 0.7079 40 0.7029 0.7029 0.7029 0.7041 0.7072 0.7072 41 0.7047 0.7047 0.7047 0.6998 0.7072 0.7047 42 0.7004 0.7004 0.7004 0.6998 0.7072 0.7035 43 0.7004 0.7004 0.7004 0.7010 0.7072 0.7035 44 0.7004 0.7004 0.7004 0.7010 0.7060 0.7035 45 0.7004 0.7004 0.7004 0.7010 0.7060 0.7041 46 0.7004 0.7004 0.7004 0.7010 0.7060 0.7041 47 0.7004 0.7004 0.7004 0.6998 0.7041 0.7041 48 0.6991 0.6991 0.6991 0.6998 0.7066 0.7041 49 0.6991 0.6991 0.6991 0.6998 0.7022 0.7060 50 0.6991 0.6991 0.6991 0.6998 0.7072 0.7060 198

Admissions database: C4.5 decision tree