Robust Regression Robust Data Mining Techniques By Boonyakorn Jantaranuson Outline ● Introduction ○ OLS and important terminology ● Least Median of Squares (LMedS) ● M-estimator ● Penalized least squares What is Regression? ● Fit a model to observed data ● Get minimum error between real data and predicted data https://en.wikipedia.org/wiki/Regression_analysis Outliers ● Noise: transmission error, measurement error ● Cause problem to resulting regression model Robust Regression ● More robust to outliers than normal regression ● Outliers are not removed but not strongly affect the model Problem formulation ● ● yi is called response variable ● xi is called explanatory variable with p dimensions ● ei is error term ● Goal: want to find the estimate of each parameter with minimum error Problem formulation (contd.) ● Estimates of parameter are called regression coefficients ● Residual ri is the difference between real and predicted value ● Formally, our goal is to find a model which can fit the data with smallest residuals Ordinary Least Squares (OLS) ● Most common regression model ● Also called sum of least squares or least squares (LS) ● Goal: find regression coefficients that minimize the sum of squared residuals Problem with OLS ● Regression model is sensitive to outlier Breakdown point ● Measure of robustness of regression method ● Ratio of the smallest number of outliers that causes the regression model to break down and total number of data points ● E.g. 1 outlier already corrupt OLS result ○ Its breakdown point is 1/n or 0% ● Highest possible breakdown point is 50% Leverage points ● Outliers can occur in both x- and y-directions ● Outliers in x-direction called leverage point ● Normally yields larger residual than outlier in y-axis Least Median of Squares (LMedS) ● Introduced by Hampel in 1975 ● Replace sum in OLS with median ● More robust because of median LMedS (contd.) ● Can achieve 50% breakdown point ● Computationally expensive for exact solution ○ O(np+1 log n) in p-dimension ● Need some approximation algorithm LMedS with randomization ● Calculate the approximation of LMedS ● Get a good running time of O(n log2 n) is 2-D with high probability and O(np-1 log n) in p-dimension in worst case LMedS with randomization (contd.) ● Goal: maintain the interval of slopes of lines to get minimum residual ● Set of line is defined by ● The interval of slopes (w.r.t. 2 points) is LMedS with randomization (contd.) ● In each iteration, n cones will be random from all possible (n-1)(n-2)/2 cones ● The median of residual will be tested and interval is shrinked ● Repeat until residual is small enough and find the optimal solution from the intersections in the remaining interval Reweighted Least Squares (RLS) ● One variant of LMedS ● Combines OLS with estimates from LMedS ○ S is scale estimate corresponding to LMedS RLS (contd.) From Robust Regression and Outlier Detection by Rousseuww M-estimator ● The name M is from Maximum Likelihood ● Replace squared residual in OLS with a symmetric, positive semi-definite function ρ M-estimator (contd.) ● To find regression coefficients that minimize the objective function, we need to find derivative of that function M-estimator (contd.) ● We can also reduce M-estimator to other types of regression ○ OLS: 2 ρ(ri) = r ○ Least absolute deviations (LAD): ρ(ri) = |r| ● LAD yields less residuals than OLS but in high-dimensional data OLS can perform slightly better ○ But still 0% breakdown points! ● Challenge: need to choose right ρ function to get a good result Penalized Least Squares ● OLS is equivalent to find maximum likelihood estimate (MLE) of data ● MLE only interested in training data, not in prior knowledge => Overfitting ● Solution: use maximum a posteriori (MAP) Penalized Least Squares ● With prior that the data is normally distributed (Gaussian), calculating MAP is equivalent to ● Intuitively, it is OLS with penalty term ● The above is called ridge regression or l2 regularization Penalized Least Squares (contd.) ● Different assumption on data and prior give different type of regularization From Machine Learning: A Probabilistic Perspective by Murphy Hard Thresholding (TORRENT) ● TORRENT = Thresholding Operator-based Robust RegrEssioN meThod ● Based on l1 regularized regression ● Iteratively maintain the active set St using hard thresholding operator ○ Active set is a set of clean points (not outliers) ● Keep updating weights (regression coeff.) until the residual less than some pre-specified error tolerance TORRENT (contd.) From paper Robust Regression via Hard Thresholding by Bhatia, Jain and Kar TORRENT (contd.) ● Offer several variants which are suitable in different situations ● Variants ○ TORRENT-FC: fully corrective LS, converge faster but expensive at each step ○ TORRENT-GD: using gradient descent, suitable for high dimensional data ○ TORRENT-HYB: hybrid version of above variants Self-Scaled Regularized Robust Regression ● Also based on l1 regularized regression ● Incorperate prior knowledge to make the penalty term able to scaled automatically ○ Prior e.g. Data occurrence Conclusion ● OLS is sensitive to outliers ● LMedS have high breakdown point but slow ● M-Estimate is flexible but hard to find the right function to make it robust ● Penalized Least Squares is also robust but require prior knowledge on data ○ Sometime need strong assumption and not always correct Remarks ● Old papers tend to talk more about high breakdown point i.e. try to reach 50% breakdown point ● More recent papers interested in computational speed instead ○ Effect of high dimensional data.