Support Vector Machines * The Interface to libsvm in package e1071

by David Meyer FH Technikum Wien, Austria David.Meyer@R-Project.org September 16, 2021

“Hype or Hallelujah?” is the provocative title used by Bennett & Campbell (2000) in an overview of Support Vector Machines (SVM). SVMs are currently a hot topic in the machine learning community, creating a similar enthusiasm at the moment as Artificial Neural Networks used to do before. Far from being a panacea, SVMs yet represent a powerful technique for general (nonlinear) classi- fication, regression and outlier detection with an intuitive model representation. The package e1071 offers an interface to the award-winning1 C++- implementation by Chih-Chung Chang and Chih-Jen Lin, libsvm (current ver- sion: 2.6), featuring: ˆ C- and ν-classification ˆ one-class-classification (novelty detection) ˆ - and ν-regression and includes: ˆ linear, polynomial, radial basis function, and sigmoidal kernels ˆ formula interface ˆ k-fold cross validation For further implementation details on libsvm, see Chang & Lin(2001).

Basic concept

SVMs were developed by Cortes & Vapnik(1995) for binary classification. Their approach may be roughly sketched as follows: Class separation: basically, we are looking for the optimal separating hyper- plane between the two classes by maximizing the margin between the classes’ closest points (see Figure1)—the points lying on the boundaries are called support vectors, and the middle of the margin is our optimal separating hyperplane; *A smaller version of this article appeared in R-News, Vol.1/3, 9.2001 1The library won the IJCNN 2001 Challenge by solving two of three problems: the Gener- alization Ability Challenge (GAC) and the Text Decoding Challenge (TDC). For more infor- mation, see: http://www.csie.ntu.edu.tw/~cjlin/papers/ijcnn.ps.gz.

1 Overlapping classes: data points on the “wrong” side of the discriminant margin are weighted down to reduce their influence (“soft margin”); Nonlinearity: when we cannot find a linear separator, data points are pro- jected into an (usually) higher-dimensional space where the data points effectively become linearly separable (this projection is realised via kernel techniques); Problem solution: the whole task can be formulated as a quadratic optimization problem which can be solved by known techniques.

A program able to perform all these tasks is called a Support Vector Machine.

Margin { Separating Hyperplane

Support Vectors

Figure 1: Classification (linear separable case)

Several extensions have been developed; the ones currently included in lib- svm are:

ν-classification: this model allows for more control over the number of support vectors (see Sch¨olkopf et al., 2000) by specifying an additional parameter ν which approximates the fraction of support vectors; One-class-classification: this model tries to find the support of a distribution and thus allows for outlier/novelty detection; Multi-class classification: basically, SVMs can only solve binary classification problems. To allow for multi-class classification, libsvm uses the one- against-one technique by fitting all binary subclassifiers and finding the correct class by a voting mechanism; -regression: here, the data points lie in between the two borders of the margin which is maximized under suitable conditions to avoid outlier inclusion;

2 ν-regression: with analogue modifications of the regression model as in the clas- sification case.

Usage in R

The R interface to libsvm in package e1071, svm(), was designed to be as intuitive as possible. Models are fitted and new data are predicted as usual, and both the vector/matrix and the formula interface are implemented. As expected for R’s statistical functions, the engine tries to be smart about the mode to be chosen, using the dependent variable’s type (y): if y is a factor, the engine switches to classification mode, otherwise, it behaves as a regression machine; if y is omitted, the engine assumes a novelty detection task.

Examples

In the following two examples, we demonstrate the practical use of svm() along with a comparison to classification and regression trees as implemented in rpart().

Classification In this example, we use the glass data from the UCI Repository of Machine Learning Databases for classification. The task is to predict the type of a glass on basis of its chemical analysis. We start by splitting the data into a train and test set:

> library(e1071) > library(rpart) > data(Glass, package="mlbench") > ## split data into a train and test set > index <- 1:nrow(Glass) > testindex <- sample(index, trunc(length(index)/3)) > testset <- Glass[testindex,] > trainset <- Glass[-testindex,] Both for the SVM and the partitioning tree (via rpart()), we fit the model and try to predict the test set values:

> ## svm > svm.model <- svm(Type ~ ., data = trainset, cost = 100, gamma = 1) > svm.pred <- predict(svm.model, testset[,-10]) (The dependent variable, Type, has column number 10. cost is a general penal- izing parameter for C-classification and gamma is the radial basis function-specific kernel parameter.)

> ## rpart > rpart.model <- rpart(Type ~ ., data = trainset) > rpart.pred <- predict(rpart.model, testset[,-10], type = "class") A cross-tabulation of the true versus the predicted values yields:

3 > ## compute svm confusion matrix > table(pred = svm.pred, true = testset[,10])

true pred 1 2 3 5 6 7 1 18 3 1 0 0 0 2 12 19 1 1 0 4 3 0 1 2 0 0 0 5 0 0 0 2 0 0 6 0 0 0 0 2 0 7 0 0 0 0 0 5

> ## compute rpart confusion matrix > table(pred = rpart.pred, true = testset[,10])

true pred 1 2 3 5 6 7 1 19 4 0 0 0 1 2 9 16 1 0 1 1 3 1 1 3 0 0 0 5 0 2 0 3 1 1 6 0 0 0 0 0 0 7 1 0 0 0 0 6

method Min. 1st Qu. Median Mean 3rd Qu. Max. Accuracy svm 0.58 0.6 0.61 0.62 0.63 0.68 rpart 0.37 0.42 0.45 0.45 0.48 0.55 Kappa svm 0.61 0.63 0.64 0.65 0.66 0.7 rpart 0.44 0.5 0.5 0.52 0.54 0.59

Table 1: Performance of svm() and rpart() for classification (10 replications)

Finally, we compare the performance of the two methods by computing the respective accuracy rates and the kappa indices (as computed by classAgree- ment() also contained in package e1071). In Table1, we summarize the results of 10 replications—Support Vector Machines show better results.

Non-linear -Regression The regression capabilities of SVMs are demonstrated on the ozone data. Again, we split the data into a train and test set.

> library(e1071) > library(rpart) > data(Ozone, package="mlbench") > ## split data into a train and test set > index <- 1:nrow(Ozone) > testindex <- sample(index, trunc(length(index)/3)) > testset <- na.omit(Ozone[testindex,-3]) > trainset <- na.omit(Ozone[-testindex,-3])

4 > ## svm > svm.model <- svm(V4 ~ ., data = trainset, cost = 1000, gamma = 0.0001) > svm.pred <- predict(svm.model, testset[,-3]) > crossprod(svm.pred - testset[,3]) / length(testindex)

[,1] [1,] 10.97535

> ## rpart > rpart.model <- rpart(V4 ~ ., data = trainset) > rpart.pred <- predict(rpart.model, testset[,-3]) > crossprod(rpart.pred - testset[,3]) / length(testindex)

[,1] [1,] 23.1046

Min. 1st Qu. Median Mean 3rd Qu. Max. svm 8.58 10.17 11.48 11.32 12.33 14.28 rpart 14.48 17.33 19.97 19.31 21.26 22.44

Table 2: Performance of svm() and rpart() for regression (Mean Squared Error, 10 replications)

We compare the two methods by the mean squared error (MSE)—see Table 2 for a summary of 10 replications. Again, as for classification, svm() does a better job than rpart()—in fact, much better.

Elements of the svm object

The function svm() returns an object of class “svm”, which partly includes the following components:

SV: matrix of support vectors found; labels: their labels in classification mode; index: index of the support vectors in the input data (could be used e.g., for their visualization as part of the data set).

If the cross-classification feature is enabled, the svm object will contain some additional information described below.

Other main features

Class Weighting: if one wishes to weight the classes differently (e.g., in case of asymmetric class sizes to avoid possibly overproportional influence of bigger classes on the margin), weights may be specified in a vector with named components. In case of two classes A and B, we could use something like: m <- svm(x, y, class.weights = c(A = 0.3, B = 0.7))

5 Cross-classification: to assess the quality of the training result, we can perform a k-fold cross-classification on the training data by setting the parameter cross to k (default: 0). The svm object will then contain some additional values, depending on whether classification or regression is performed. Values for classification: accuracies: vector of accuracy values for each of the k predictions tot.accuracy: total accuracy Values for regression: MSE: vector of mean squared errors for each of the k predictions tot.MSE: total mean squared error scorrcoef: Squared correlation coefficient (of the predicted and the true values of the dependent variable)

Tips on practical use

ˆ Note that SVMs may be very sensitive to the proper choice of parame- ters, so allways check a range of parameter combinations, at least on a reasonable subset of your data. ˆ For classification tasks, you will most likely use C-classification with the RBF kernel (default), because of its good general performance and the few number of parameters (only two: C and γ). The authors of libsvm suggest to try small and large values for C—like 1 to 1000—first, then to decide which are better for the data by cross validation, and finally to try several γ’s for the better C’s. ˆ However, better results are obtained by using a grid search over all pa- rameters. For this, we recommend to use the tune.svm() function in e1071. ˆ Be careful with large datasets as training times may increase rather fast. ˆ Scaling of the data usually drastically improves the results. Therefore, svm() scales the data by default.

Model Formulations and Kernels

Dual representation of models implemented: ˆ C-classification:

1 min α>Qα − e>α α 2 s.t. 0 ≤ αi ≤ C, i = 1, . . . , l, (1) y>α = 0 , where e is the unity vector, C is the upper bound, Q is an l by l positive > semidefinite matrix, Qij ≡ yiyjK(xi, xj), and K(xi, xj) ≡ φ(xi) φ(xj) is the kernel.

6 ˆ ν-classification:

1 min α>Qα α 2 s.t. 0 ≤ αi ≤ 1/l, i = 1, . . . , l, (2) e>α ≥ ν, y>α = 0 .

where ν ∈ (0, 1]. ˆ one-class classification:

1 min α>Qα α 2 s.t. 0 ≤ αi ≤ 1/(νl), i = 1, . . . , l, (3) e>α = 1 ,

ˆ -regression:

1 min (α − α∗)>Q(α − α∗) + α,α∗ 2 l l X ∗ X ∗  (αi + αi ) + yi(αi − αi ) i=1 i=1 ∗ s.t. 0 ≤ αi, αi ≤ C, i = 1, . . . , l, (4) l X ∗ (αi − αi ) = 0 . i=1

ˆ ν-regression:

1 ∗ > ∗ > ∗ min (α − α ) Q(α − α ) + z (αi − αi ) α,α∗ 2 ∗ s.t. 0 ≤ αi, αi ≤ C, i = 1, . . . , l, (5) e>(α − α∗) = 0 e>(α + α∗) = Cν .

Available kernels:

kernel formula parameters linear u>v (none) > d polynomial (γu v + c0) γ, d, c0 radial basis fct. exp{−γ|u − v|2} γ > sigmoid tanh{γu v + c0} γ, c0

7 Conclusion

We hope that svm provides an easy-to-use interface to the world of SVMs, which nowadays have become a popular technique in flexible modelling. There are some drawbacks, though: SVMs scale rather badly with the data size due to the quadratic optimization algorithm and the kernel transformation. Furthermore, the correct choice of kernel parameters is crucial for obtaining good results, which practically means that an extensive search must be conducted on the pa- rameter space before results can be trusted, and this often complicates the task (the authors of libsvm currently conduct some work on methods of efficient au- tomatic parameter selection). Finally, the current implementation is optimized for the radial basis function kernel only, which clearly might be suboptimal for your data.

References

Bennett, K. P. & Campbell, C. (2000). Support vector machines: Hype or hal- lelujah? SIGKDD Explorations, 2(2). http://www.acm.org/sigs/sigkdd/ explorations/issue2-2/bennett.pdf. Chang, C.-C. & Lin, C.-J. (2001). LIBSVM: a library for support vector ma- chines. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm, detailed documentation (algorithms, formulae, . . . ) can be found in http: //www.csie.ntu.edu.tw/~cjlin/papers/libsvm.ps.gz Cortes, C. & Vapnik, V. (1995). Support-vector network. Machine Learning, 20, 1–25.

Sch¨olkopf, B., Smola, A., Williamson, R. C., & Bartlett, P. (2000). New support vector algorithms. Neural Computation, 12, 1207–1245.

Vapnik, V. (1998). Statistical learning theory. New York: Wiley.

8