Machine Learning, 42, 287–320, 2001 c 2001 Kluwer Academic Publishers. Manufactured in The Netherlands.
Soft Margins for AdaBoost
G. RATSCH¨ raetsch@first.gmd.de GMD FIRST, Kekulestr.´ 7, 12489 Berlin, Germany
T. ONODA [email protected] CRIEPI,† Komae-shi, 2-11-1 Iwado Kita, Tokyo, Japan
K.-R. MULLER¨ klaus@first.gmd.de GMD FIRST, Kekulestr.´ 7, 12489 Berlin, Germany; University of Potsdam, Neues Palais 10, 14469 Potsdam, Germany
Editor: Robert Schapire
Abstract. Recently ensemble methods like ADABOOST have been applied successfully in many problems, while seemingly defying the problems of overfitting. ADABOOST rarely overfits in the low noise regime, however, we show that it clearly does so for higher noise levels. Central to the understanding of this fact is the margin distribution. ADABOOST can be viewed as a constraint gradient descent in an error function with respect to the margin. We find that ADABOOST asymptotically achieves a hard margin distribution, i.e. the algorithm concentrates its resources on a few hard-to-learn patterns that are interestingly very similar to Support Vectors. A hard margin is clearly a sub-optimal strategy in the noisy case, and regularization, in our case a “mistrust” in the data, must be introduced in the algorithm to alleviate the distortions that single difficult patterns (e.g. outliers) can cause to the margin distribution. We propose several regularization methods and generalizations of the original ADABOOST algorithm to achieve a soft margin. In particular we suggest (1) regularized ADABOOSTREG where the gradient decent is done directly with respect to the soft margin and (2) regularized linear and quadratic programming (LP/QP-) ADABOOST, where the soft margin is attained by introducing slack variables. Extensive simulations demonstrate that the proposed regularized ADABOOST-type algorithms are useful and yield competitive results for noisy data.
Keywords: ADABOOST, arcing, large margin, soft margin, classification, support vectors
1. Introduction
Boosting and other ensemble1 learning methods have been recently used with great success in applications like OCR (Schwenk & Bengio, 1997; LeCun et al., 1995). But so far the reduction of the generalization error by Boosting algorithms has not been fully understood. For low noise cases Boosting algorithms are performing well for good reasons (Schapire et al., 1997; Breiman, 1998). However, recent studies with highly noisy patterns (Quinlan, 1996; Grove & Schuurmans, 1998; R¨atsch et al., 1998) showed that it is clearly a myth that Boosting methods do not overfit.