Lecture 12 Robust Estimation

Prof. Dr. Svetlozar Rachev Institute for Statistics and Mathematical Economics University of Karlsruhe

Financial Econometrics, Summer Semester 2007

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Copyright

These lecture-notes cannot be copied and/or distributed without permission. The material is based on the text-book: Financial Econometrics: From Basics to Advanced Modeling Techniques (Wiley-Finance, Frank J. Fabozzi Series) by Svetlozar T. Rachev, Stefan Mittnik, Frank Fabozzi, Sergio M. Focardi,Teo Jaˇsic`.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Outline

I Robust statistics.

I Robust estimators of regressions.

I Illustration: robustness of the corporate bond yield spread model.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics

I Robust statistics addresses the problem of making estimates that are insensitive to small changes in the basic assumptions of the statistical models employed.

I The concepts and methods of robust statistics originated in the 1950s. However, the concepts of robust statistics had been used much earlier.

I Robust statistics: 1. assesses the changes in estimates due to small changes in the basic assumptions; 2. creates new estimates that are insensitive to small changes in some of the assumptions.

I Robust statistics is also useful to separate the contribution of the tails from the contribution of the body of the data.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics

I Peter Huber observed, that robust, distribution-free, and nonparametrical actually are not closely related properties.

I Example: The sample mean and the sample median are nonparametric estimates of the mean and the median but the mean is not robust to outliers. In fact, changes of one single observation might have unbounded eﬀects on the mean while the median is insensitive to changes of up to half the sample.

I Robust methods assume that there are indeed parameters in the distributions under study and attempt to minimize the eﬀects of outliers as well as erroneous assumptions on the shape of the distribution.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Qualitative and Quantitative Robustness

I Estimators are functions of the sample data. 0 I Given an N-sample of data X = (x1,..., xN ) from a population with a cdf F (x), depending on parameter Θ∞, an estimator for Θ∞ is a function ϑˆ = ϑN (x1,..., xN )..

I Consider those estimators that can be written as functions of the cumulative empirical distribution function:

N −1 X FN (x) = N I (xi ≤ x) i=1 where I is the indicator function. For these estimators we can write ϑˆ = ϑN (FN )

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Qualitative and Quantitative Robustness

I Most estimators, in particular the ML estimators, can be written in this way with probability 1.

I In general, when N → ∞ then FN (x) → F (x) and ϑˆN → ϑ∞ in probability. The estimator ϑˆN is a random variable that depends on the sample.

I Under the distribution F , it will have a probability distribution LF (ϑN ).

I Statistics deﬁned as functionals of a distribution are robust if they are continuous with respect to the distribution.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Qualitative and Quantitative Robustness

I In 1968, Hampel introduced a technical deﬁnition of qualitative robustness based on metrics of the functional space of distributions.

I It states that an estimator is robust for a given distribution F if small deviations from F in the given metric result in small deviations from LF (ϑN ) in the same metric or eventually in some other metric for any sequence of samples of increasing size.

I The deﬁnition of robustness can be made quantitative by assessing quantitatively how changes in the distribution F aﬀect the distribution LF (ϑN ).

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Resistant Estimators

I An estimator is called resistant if it is insensitive to changes in one single observation.

I Given an estimator ϑˆ = ϑN (FN ),we want to understand what happens if we add a new observation of value x to a large sample. To this end we define the influence curve (IC), also called influence function.

I The IC is a function of x given ϑ, and F is deﬁned as follows:

ϑ((1 − s)F + sδx ) − ϑ(F ) ICϑ,F (x) = lim s→0 s

where δx denotes a point mass 1 at x.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Resistant Estimators

I As we can see from its previous deﬁnition, the IC is a function of the size of the single observation that is added. In other words, the IC measures the inﬂuence of a single observation x on a statistics ϑ for a given distribution F .

I In practice, the inﬂuence curve is generated by plotting the value of the computed statistic with a single point of X added to Y against that X value. Example: The IC of the mean is a straight line.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Resistant Estimators

Several aspects of the inﬂuence curve are of particular interest:

I Is the curve ”bounded” as the X values become extreme? Robust statistics should be bounded. That is, a robust statistic should not be unduly inﬂuenced by a single extreme point.

I What is the general behavior as the X observation becomes extreme? For example, does it becomes smoothly down-weighted as the values become extreme?

I What is the inﬂuence if the X point is in the ”center” of the Y points?.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Breakdown Bound

The breakdown (BD) bound or point is the largest possible fraction of observations for which there is a bound on the change of the estimate when that fraction of the sample is altered without restrictions. Example: We can change up to 50% of the sample points without provoking unbounded changes of the median. On the contrary, changes of one single observation might have unbounded eﬀects on the mean.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Rejection Point

I The rejection point is deﬁned as the point beyond which the IC becomes zero. Note: The observations beyond the rejection point make no contribution to the ﬁnal estimate except, possibly, through the auxiliary scale estimate.

I Estimators that have a ﬁnite rejection point are said to be redescending and are well protected against very large outliers. However, a ﬁnite rejection point usually results in the underestimation of scale.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Main concepts

I The gross error sensitivity expresses asymptotically the maximum eﬀect that a contaminated observation can have on the estimator. It is the maximum absolute value of the IC.

I The local shift sensitivity measures the eﬀect of the removal of a mass at y and its reintroduction at x. For continuous and diﬀerentiable IC, the local shift sensitivity is given by the maximum absolute value of the slope of IC at any point.

I Winsor’s principle states that all distributions are normal in the middle.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: M-Estimators

I M-estimators are those estimators that are obtained by minimizing a function of the sample data.

I Suppose that we are given an N-sample of data 0 X= (x1,..., xN ) . The estimator T (x1,..., xN ) is called an M-estimator if it is obtained by solving the following minimum problem: ( N ) X T = arg mint J = ρ(xi , t) i=1

where ρ(xi , t) is an arbitrary function.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: M-Estimators

Alternatively, if ρ(xi , t) is a smooth function, we can say that T is an M-estimator if it is determined by solving the equations:

N X ψ(xi , t) = 0 i=1 where ∂ρ(x , t) ψ(x , t) = i i ∂t

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: M-Estimators

I When the M-estimator is equivariant, that is T (x1 + a,..., xN + a) = T (x1,..., xN ) + a, ∀a ∈ R, we can write ψ and ρ in terms of the residuals x − t.

I Also, in general, an auxiliary scale estimate, S, is used to obtain the scaled residuals r = (x − t)/S. If the estimator is also equivariant to changes of scale, we can write x − t ψ(x, t) = ψ = ψ(r) S x − t ρ(x, t) = ρ = ρ(r) S

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: M-Estimators

I ML estimators are M-estimators with ρ = − log f , where f is the probability density.

I The name M-estimators means maximum likelihood-type estimators. LS estimators are also M-estimators.

I The IC of M-estimators has a particularly simple form. In fact, it can be demonstrated that the IC is proportional to the function ψ: IC = Constant × ψ

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: L-Estimators

0 I Consider an N-sample (x1,..., xN ) . Order the samples so that x(1) ≤ x(2) ≤ · · · ≤ x(N). The i-th element X = x(i) of the ordered sample is called the i-th order statistic.

I L-estimators are estimators obtained as a linear combination of order statistics: N X L = ai x(i) i=1

where the ai are ﬁxed constants. Constants are typically normalized so that N X ai = 1 i=1

I An important example of an L-estimator is the trimmed mean. It is a mean formed excluding a fraction of the highest and/or lowest samples.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: R-Estimators

R-estimators are obtained by minimizing the sum of residuals weighted by functions of the rank of each residual. The functional to be minimized is the following:

( N ) X arg min J = a(Ri )ri i=1

where Ri is the rank of the i-th residual ri and a is a nondecreasing score function that satisﬁes the condition

N X a(Ri ) = 0 i=1

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: The Least Median of Squares Estimator

I The least median of squares (LMedS) estimator referred to minimizing the median of squared residuals, proposed by Rousseuw.

I This estimator eﬀectively trims the N/2 observations having the largest residuals, and uses the maximal residual value in the remaining set as the criterion to be minimized. It is hence equivalent to assuming that the noise proportion is 50%.

I LMedS is unwieldy from a computational point of view because of its nondiﬀerentiable form. This means that a quasi-exhaustive search on all possible parameter values needs to be done to ﬁnd the global minimum.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: The Least Trimmed of Squares Estimator

I The least trimmed of squares (LTS) estimator offers an efficient way to find robust estimates by minimizing the objective function given by

( h ) X 2 J = r(i) i=1

2 where r(i) is the i-th smallest residual or distance when the residuals are ordered in ascending order, that is: 2 2 2 r(1) ≤ r(2) ≤ · · · ≤ r(N) and h is the number of data points whose residuals we want to include in the sum.

I This estimator basically ﬁnds a robust estimate by identifying the N − h points having the largest residuals as outliers, and discarding (trimming) them from the dataset.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Reweighted Least Squares Estimator

I Some algorithms explicitly cast their objective functions in terms of a set of weights that distinguish between inliers and outliers.

I These weights usually depend on a scale measure that is also diﬃcult to estimate. Example: The reweighted least squares (RLS) estimator uses the following objective function:

( N ) X 2 arg min J = ωi ri i=1

where ri are robust residuals resulting from an approximate LMedS or LTS procedure. The weights ωi trim outliers from the data used in LS minimization, and can be computed after a preliminary approximate step of LMedS or LTS.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Robust Estimators of the Center

The mean estimates the center of a distribution but it is not resistant. Resistant estimators of the center are the following:

I Trimmed mean. Suppose x(1) ≤ x(2) ≤ · · · ≤ x(N) are the sample order statistics (that is, the sample sorted). The trimmed mean TN (δ, 1 − γ) is deﬁned as follows:

U 1 XN TN (δ, 1 − γ) = xj UN − LN j=LN +1

δ, γ ∈ (0, 0.5), LN = ﬂoor[Nδ], UN = ﬂoor[Nγ]

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Robust Estimators of the Center

I Winsorized mean. The Winsorized mean is the mean X¯W of Winsorized data:  x j ≤ L  IN +1 N yj = xj LN + 1 ≤ j ≤ UN  xj = xUN +1 j ≥ UN + 1

X¯W = Y¯

I Median. The median Med(X) is deﬁned as that value that occupies a central position in a sample order statistics:

x if N is odd Med(X ) = ((N+1)/2) ((x(N/2) + x(N/2+1))/2) if N is even

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Robust Estimators of the Spread

The variance is a classical estimator of the spread but it is not robust. Robust estimators of the spread are the following:

I Median absolute deviation. The median absolute deviation (MAD) is deﬁned as the median of the absolute value of the diﬀerence between a variable and its median, that is,

MAD = MED|X − MED(X )|

I Interquartile range. The interquartile range (IQR) is deﬁned as the diﬀerence between the highest and lowest quartile:

IQR = Q(0.75) − Q(0.25)

where Q(0.75) and Q(0.25) are the 75th and 25th percentiles of the data.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Robust Estimators of the Spread

I Mean absolute deviation. The mean absolute deviation (MeanAD) is deﬁned as follows:

N 1 X |x − MED(X )| N j j=1

I Winsorized standard deviation. The Winsorized standard deviation is the standard deviation of Winsorized data, that is,

σN σW = (UN − LN )/N

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Illustration of Robust Statistics

I To illustrate the eﬀect of robust statistics, consider the series of daily returns of Nippon Oil in the period 1986 through 2005. The following was computed: Mean = 3.8396e-005 Trimmed mean (20%) = -4.5636e-004 Median = 0

I In order to show the robustness properties of these estimators, lets multiply the 10% highest/lowest returns by 2. Then Mean = 4.4756e-004 Trimmed mean (20%) = -4.4936e-004 Median = 0

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Illustration of Robust Statistics

I We can perform the same exercise for measures of the spread. We obtain the following results: Standard deviation = 0.0229 IQR = 0.0237 MAD = 0.0164

I Lets multiply the 10% highest/lowest returns by 2. The new values are: Standard deviation = 0.0415 IQR = 0.0237 MAD = 0.0248

I If we multiply the 25% highest/lowest returns by 2, then Standard deviation = 0.0450 IQR = 0.0237 MAD = 0.0299

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Estimators of Regressions

I Identifying robust estimators of regressions is a rather diﬃcult problem.

I In fact, diﬀerent choices of estimators, robust or not, might lead to radically diﬀerent estimates of slopes and intercepts.

I Consider the following linear regression model:

N X Y = β0 + βi Xi + ε i=1

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Estimators of Regressions

If data are organized in matrix form as usual,     Y1 1 X11 ··· XN1  .   . . .. .  Y =  .  , X =  . . . .  , YT 1 X1T ··· XNT     β1 ε1  .   .  β =  .  , ε =  .  βN εT then the regression equation takes the form,

Y = X β + ε

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Statistics: Illustration of Robust Statistics

The standard nonrobust LS estimation of regression parameters minimizes the sum of squared residuals,

2 T T  N  X 2 X X εt = Yi − βij Xij  i=1 i=1 j=0

or, equivalently solves the system of N + 1 equations,

T  N  X X Yi − βij Xij  Xij = 0 i=1 j=0

or, in matrix notation, X 0X β = X 0Y . The solution of this system is

βˆ = (X 0X )−1X 0Y

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Estimators of Regressions

I The ﬁtted values (i.e, the LS estimates of the expectations) of the Y are Yˆ = X (X 0X )−1X 0Y = HY

I The H matrix is called the hat matrix because it puts a hat on, that is, it computes the expectation Yˆ of the Y .

I The hat matrix H is a symmetric T × T projection matrix; that is, the following relationship holds: HH = H .

I The matrix H has N eigenvalues equal to 1 and T − N eigenvalues equal to 0. Its diagonal elements, hi ≡ hii satisfy:

0 ≤ hi ≤ 1

and its trace (i.e., the sum of its diagonal elements) is equal to N: tr(H) = N

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Estimators of Regressions

I Under the assumption that the errors are independent and identically distributed with mean zero and variance σ2, it can be demonstrated that the Yˆ are consistent, that is, Yˆ → E(Y ) in probability when the sample becomes inﬁnite if and only if h = max(hi ) → 0.

I Points where the hi have large values are called leverage points. It can be demonstrated that the presence of leverage points signals that there are observations that might have a decisive inﬂuence on the estimation of the regression parameters.

I A rule of thumb suggests that values hi ≤ 0.2 are safe, values 0.2 ≤ hi ≤ 0.5 require careful attention, and higher values are to be avoided.

Prof. Dr. Svetlozar Rachev Institute for Statistics and MathematicalLecture Economics 12 Robust University Estimation of Karlsruhe Robust Estimators of Regressions: Robust Regressions Based on M-Estimators

The LS estimators βˆ = (X 0X )−1X 0Y are M-estimators but are not robust. We can generalize LS seeking to minimize

T  N  X X J = ρ Yi − βij Xij  i=1 j=0

by solving the set of N + 1 simultaneous equations

T  N  X X ψ Yi − βij Xij  Xij = 0 i=1 j=0

where ∂ρ ψ = ∂β

W-estimators oﬀer an alternative form of M-estimators. They are obtained by rewriting M-estimators as follows:

 N   N   N  X X X ψ Yi − βij Xij  = w Yi − βij Xij  Yi − βij Xij  j=0 j=0 j=0

Hence the N + 1 simultaneous equations become

 N   N  X X w Yi − βij Xij  Yi − βij Xij  = 0 j=0 j=0

or, in matrix form X 0WX β = X 0WY where W is a diagonal matrix.

I The above is not a linear system because the weighting function is in general a nonlinear function of the data.

I A typical approach is to determine iteratively the weights through an iterative reweighted least squares (RLS) procedure. Clearly the iterative procedure depends numerically on the choice of the weighting functions.

I Two commonly used choices are the Huber weighting function wH (e) and the Tukey bisquare weighting function wT (e).

I The Huber weighting function deﬁned as

1 for |e| ≤ k w (e) = H k/|e| for |e| > k

I The Tukey bisquare weighting function deﬁned as

(1 − (e/k)2)2 for |e| ≤ k w (e) = T 0 for |e| > k

where k is a tuning constant often set at 1.345 × (standard deviation of errors) for the Huber function and k = 4.6853 ×(standard deviation of errors) for the Tukey function.