WEKA: A Machine Machine Learning with Learning Toolkit WEKA The Explorer • Classification and Regression • Clustering Bernhard Pfahringer • Association Rules (based on material by Eibe Frank, Mark • Attribute Selection Hall, and Peter Reutemann) • Data Visualization Department of Computer Science, The Experimenter University of Waikato, New Zealand The Knowledge Flow GUI Other Utilities Conclusions WEKA: the bird The Weka or woodhen (Gallirallus australis) is an endemic bird of New Zealand. (Source: WikiPedia) Copyright: Martin Kramer ([email protected]) 5/14/2007 University of Waikato 2 WEKA: the software Machine learning/data mining software written in Java (distributed under the GNU Public License) Used for research, education, and applications Complements “Data Mining” by Witten & Frank Main features: Comprehensive set of data pre-processing tools, learning algorithms and evaluation methods Graphical user interfaces (incl. data visualization) Environment for comparing learning algorithms 5/14/2007 University of Waikato 3 History Project funded by the NZ government since 1993 Develop state-of-the art workbench of data mining tools Explore fielded applications Develop new fundamental methods 5/14/2007 University of Waikato 4 History (2) Late 1992 - funding was applied for by Ian Witten 1993 - development of the interface and infrastructure WEKA acronym coined by Geoff Holmes WEKA’s file format “ARFF” was created by Andrew Donkin ARFF was rumored to stand for Andrew’s Ridiculous File Format Sometime in 1994 - first internal release of WEKA TCL/TK user interface + learning algorithms written mostly in C Very much beta software Changes for the b1 release included (among others): “Ambiguous and Unsupported menu commands removed.” “Crashing processes handled (in most cases :-)” October 1996 - first public release: WEKA 2.1 5/14/2007 University of Waikato 5 History (3) July 1997 - WEKA 2.2 Schemes: 1R, T2, K*, M5, M5Class, IB1-4, FOIL, PEBLS, support for C5 Included a facility (based on Unix makefiles) for configuring and running large scale experiments Early 1997 - decision was made to rewrite WEKA in Java Originated from code written by Eibe Frank for his PhD Originally codenamed JAWS (JAva Weka System) May 1998 - WEKA 2.3 Last release of the TCL/TK-based system Mid 1999 - WEKA 3 (100% Java) released Version to complement the Data Mining book Development version (including GUI) 5/14/2007 University of Waikato 6 The GUI back then… TCL/TK interface of Weka 2.1 5/14/2007 University of Waikato 7 WEKA: versions There are several versions of WEKA: WEKA 3.4: “book version” compatible with description in data mining book WEKA 3.5.5: “development version” with lots of improvements This talk is based on a nightly snapshot of WEKA 3.5.5 (12-Feb-2007) 5/14/2007 University of Waikato 8 WEKA only deals with “flat” files @relation heart-disease-simplified @attribute age numeric @attribute sex { female, male} @attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina} @attribute cholesterol numeric @attribute exercise_induced_angina { no, yes} @attribute class { present, not_present} @data 63,male,typ_angina,233,no,not_present 67,male,asympt,286,yes,present 67,male,asympt,229,yes,present 38,female,non_anginal,?,no,not_present ... 5/14/2007 University of Waikato 9 WEKA only deals with “flat” files @relation heart-disease-simplified @attribute age numeric @attribute sex { female, male} @attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina} @attribute cholesterol numeric @attribute exercise_induced_angina { no, yes} @attribute class { present, not_present} @data 63,male,typ_angina,233,no,not_present 67,male,asympt,286,yes,present 67,male,asympt,229,yes,present 38,female,non_anginal,?,no,not_present ... 5/14/2007 University of Waikato 10 java weka.gui.GUIChooser 5/14/2007 University of Waikato 11 5/14/2007 University of Waikato 12 5/14/2007 University of Waikato 13 java -jar weka.jar 5/14/2007 University of Waikato 14 Explorer: pre-processing the data Data can be imported from a file in various formats: ARFF, CSV, C4.5, binary Data can also be read from a URL or from an SQL database (using JDBC) Pre-processing tools in WEKA are called “filters” WEKA contains filters for: Discretization, normalization, resampling, attribute selection, transforming and combining attributes, … 5/14/2007 University of Waikato 15 5/14/2007 University of Waikato 16 5/14/2007 University of Waikato 17 5/14/2007 University of Waikato 18 5/14/2007 University of Waikato 19 5/14/2007 University of Waikato 20 5/14/2007 University of Waikato 21 5/14/2007 University of Waikato 22 5/14/2007 University of Waikato 23 5/14/2007 University of Waikato 24 5/14/2007 University of Waikato 25 5/14/2007 University of Waikato 26 5/14/2007 University of Waikato 27 5/14/2007 University of Waikato 28 5/14/2007 University of Waikato 29 5/14/2007 University of Waikato 30 5/14/2007 University of Waikato 31 5/14/2007 University of Waikato 32 5/14/2007 University of Waikato 33 5/14/2007 University of Waikato 34 5/14/2007 University of Waikato 35 5/14/2007 University of Waikato 36 Explorer: building “classifiers” Classifiers in WEKA are models for predicting nominal or numeric quantities Implemented learning schemes include: Decision trees and lists, instance-based classifiers, support vector machines, multi-layer perceptrons, logistic regression, Bayes’ nets, … “Meta”-classifiers include: Bagging, boosting, stacking, error-correcting output codes, locally weighted learning, … 5/14/2007 University of Waikato 37 5/14/2007 University of Waikato 38 5/14/2007 University of Waikato 39 5/14/2007 University of Waikato 40 5/14/2007 University of Waikato 41 5/14/2007 University of Waikato 42 5/14/2007 University of Waikato 43 5/14/2007 University of Waikato 44 5/14/2007 University of Waikato 45 5/14/2007 University of Waikato 46 5/14/2007 University of Waikato 47 5/14/2007 University of Waikato 48 5/14/2007 University of Waikato 49 5/14/2007 University of Waikato 50 5/14/2007 University of Waikato 51 5/14/2007 University of Waikato 52 5/14/2007 University of Waikato 53 5/14/2007 University of Waikato 54 5/14/2007 University of Waikato 55 5/14/2007 University of Waikato 56 5/14/2007 University of Waikato 57 5/14/2007 University of Waikato 58 5/14/2007 University of Waikato 59 5/14/2007 University of Waikato 60 5/14/2007 University of Waikato 61 5/14/2007 University of Waikato 62 5/14/2007 University of Waikato 63 5/14/2007 University of Waikato 64 5/14/2007 University of Waikato 65 5/14/2007 University of Waikato 66 5/14/2007 University of Waikato 67 5/14/2007 University of Waikato 68 5/14/2007 University of Waikato 69 5/14/2007 University of Waikato 70 5/14/2007 University of Waikato 71 QuickTime?and5/14/2007 a TIFF (LZW) decompressor are needed to see this picture. University of Waikato 72 QuickTime?and5/14/2007 a TIFF (LZW) decompressor are needed to see this picture. University of Waikato 73 5/14/2007 University of Waikato 74 5/14/2007 University of Waikato 75 5/14/2007 University of Waikato 76 5/14/2007 University of Waikato 77 QuickTime?and a TIFF (LZW) decompressor are needed to see this pictu 5/14/2007 University of Waikato 78 5/14/2007 University of Waikato 79 5/14/2007 University of Waikato 80 5/14/2007 University of Waikato 81 5/14/2007 University of Waikato 82 5/14/2007 University of Waikato 83 5/14/2007 University of Waikato 84 5/14/2007 University of Waikato 85 5/14/2007 University of Waikato 86 5/14/2007 University of Waikato 87 5/14/2007 University of Waikato 88 5/14/2007 University of Waikato 89 5/14/2007 University of Waikato 90 5/14/2007 University of Waikato 91 5/14/2007 University of Waikato 92 Explorer: clustering data WEKA contains “clusterers” for finding groups of similar instances in a dataset Some implemented schemes are: k-Means, EM, Cobweb, X-means, FarthestFirst Clusters can be visualized and compared to “true” clusters (if given) Evaluation based on loglikelihood if clustering scheme produces a probability distribution 5/14/2007 University of Waikato 93 5/14/2007 University of Waikato 94 5/14/2007 University of Waikato 95 5/14/2007 University of Waikato 96 5/14/2007 University of Waikato 97 5/14/2007 University of Waikato 98 5/14/2007 University of Waikato 99 5/14/2007 University of Waikato 100 5/14/2007 University of Waikato 101 5/14/2007 University of Waikato 102 Explorer: finding associations WEKA contains the Apriori algorithm (among others) for learning association rules Works only with discrete data Can identify statistical dependencies between groups of attributes: milk, butter ⇒ bread, eggs (with confidence 0.9 and support 2000) Apriori can compute all rules that have a given minimum support and exceed a given confidence 5/14/2007 University of Waikato 103 5/14/2007 University of Waikato 104 5/14/2007 University of Waikato 105 5/14/2007 University of Waikato 106 Explorer: attribute selection Panel that can be used to investigate which (subsets of) attributes are the most predictive ones Attribute selection methods contain two parts: A search method: best-first, forward selection, random, exhaustive, genetic algorithm, ranking An evaluation method: correlation-based, wrapper, information gain, chi-squared, … Very flexible: WEKA allows (almost) arbitrary combinations of these two 5/14/2007 University of Waikato 107 5/14/2007 University of Waikato 108 5/14/2007 University of Waikato 109 5/14/2007 University of Waikato 110 5/14/2007 University of Waikato 111 5/14/2007 University of Waikato 112 5/14/2007 University of Waikato 113 Explorer: data visualization Visualization very useful in practice: e.g.