168 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 1, JANUARY 2005 Robust Mean-Squared Error Estimation in the Presence of Model Uncertainties Yonina C. Eldar, Member, IEEE, Aharon Ben-Tal, and Arkadi Nemirovski Abstract—We consider the problem of estimating an unknown so that is chosen to minimize the Euclidian norm of the data parameter vector x in a linear model that may be subject to error . However, in an estimation context, the objective x uncertainties, where the vector is known to satisfy a weighted typically is to minimize the size of the estimation error , norm constraint. We first assume that the model is known ex- actly and seek the linear estimator that minimizes the worst-case rather than that of the data error . mean-squared error (MSE) across all possible values of x. We show To develop an estimation method that is based directly on the that for an arbitrary choice of weighting, the optimal minimax estimation error, we may seek the estimator that minimizes MSE estimator can be formulated as a solution to a semidefinite the mean-squared error (MSE), where the MSE of an estimate programming problem (SDP), which can be solved very efficiently. of is the expected value of the squared norm of the esti- We then develop a closed form expression for the minimax MSE mation error and is equal to the sum of the variance and the estimator for a broad class of weighting matrices and show that it coincides with the shrunken estimator of Mayer and Willke, with squared norm of the bias. Since the bias generally depends on a specific choice of shrinkage factor that explicitly takes the prior the unknown parameters , we cannot choose an estimator to information into account. directly minimize the MSE. A common approach is to restrict Next, we consider the case in which the model matrix is subject to the estimator to be linear and unbiased and then seek the esti- uncertainties and seek the robust linear estimator that minimizes mator of this form that minimizes the variance or the MSE. It is the worst-case MSE across all possible values of x and all possible values of the model matrix. As we show, the robust minimax MSE well known that the LS estimator minimizes the variance in the estimator can also be formulated as a solution to an SDP. estimate among all unbiased linear estimators. However, this Finally, we demonstrate through several examples that the min- does not imply that the LS estimator leads to a small variance or imax MSE estimator can significantly increase the performance a small mean-squared error (MSE). A difficulty often encoun- over the conventional least-squares estimator, and when the model tered in this estimation problem is that the resulting variance can matrix is subject to uncertainties, the robust minimax MSE esti- be very large, particularly in nonorthogonal and ill-conditioned mator can lead to a considerable improvement in performance over the minimax MSE estimator. problems. Various modifications of the LS estimator for the case in Index Terms—Data uncertainty, linear estimation, mean squared error estimation, minimax estimation, robust estimation. which the data model holds i.e., with and known exactly, have been proposed. Among the alternatives are Tikhonov regularization [5], which is also known in the I. INTRODUCTION statistical literature as the ridge estimator [6], the shrunken HE problem of estimating a set of unknown deterministic estimator [7], and the covariance shaping LS estimator [8], [9]. T parameters observed through a linear transformation In general, these LS alternatives attempt to reduce the MSE , and corrupted by additive noise , arises in a large variety in estimating by allowing for a bias. However, each of the of areas in science and engineering, e.g., communication, estimators above is designed to optimize an objective which economics, signal processing, seismology, and control. does not depend directly on the MSE, but rather depends on the Owing to the lack of statistical information about the pa- data error . rameters , often, the estimated parameters are chosen to opti- In many engineering applications, the model matrix is also mize a criterion based on the observed signal . The celebrated subject to uncertainties. For example, the matrix may be es- least-squares (LS) estimator [1]–[3], which was first used by timated from noisy data, in which case, may not be known Gauss to predict movements of planets [4], seeks the linear esti- exactly. If the actual data matrix deviates from the one assumed, mate of that results in an estimated data vector that then the performance of an estimator designed based on alone is closest, in a LS sense, to the given data vector may deteriorate considerably. Various methods have been pro- posed to account for uncertainties in . The Total LS method [10], [11] seeks the parameters and the minimum perturbation Manuscript received January 30, 2003; revised January 29, 2004. The work of Y. C. Eldar was supported in part by the Taub Foundation and by a Technion to the model matrix that minimize the data error. Although V.P.R. Fund. The associate editor coordinating the review of this manuscript and the total LS method allows for uncertainties in , in many cases, approving it for publication was Dr. Jean Pierre Delmas. it results in correction terms that are unnecessarily large. In par- Y. C. Eldar is with the Department of Electrical Engineering, Technion—Is- ticular, when the model matrix is square, the total LS method rael Institute of Technology, Haifa 32000, Israel (e-mail: [email protected] nion.ac.il). recuse to the conventional LS method, which does not take the A. Ben-Tal and A. Nemirovski are with the MINERVA Optimization uncertainties into account. Recently, several methods [12]–[14] Center, Department of Industrial Engineering, Technion—Israel Institute of have been developed to treat the case in which the perturba- Technology, Haifa 32000, Israel (e-mail: [email protected]; nemirovs@ ie.technion.ac.il). tion to the model matrix is bounded. These methods seek Digital Object Identifier 10.1109/TSP.2004.838933 the parameters that minimize the worst-case data error across 1053-587X/$20.00 © 2005 IEEE Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on February 16,2010 at 17:11:22 EST from IEEE Xplore. Restrictions apply. ELDAR et al.: ROBUST MEAN-SQUARED ERROR ESTIMATION IN THE PRESENCE OF MODEL UNCERTAINTIES 169 all bounded perturbations of and possibly bounded perturba- have the same eigenvector matrix. In Section IV, we tions of the data vector. In [15], the authors seek the estimator consider the case in which both and the model matrix are that minimizes the best possible data error over all possible per- subject to uncertainties and show that the minimax MSE esti- turbations of . Here again, the above objectives depend on the mator that minimizes the worst-case MSE in the region of un- data error and not on the estimation error or the MSE. certainty can again be formulated as an SDP. We then consider, In this paper, we consider the case in which the (possibly in Section V, the special case in which and have weighted) norm of the unknown vector is bounded and de- the same eigenvector matrix and and have the same velop robust estimators of , whose performance is reason- eigenvector matrix. Examples illustrating the performance ad- ably good across all possible values of in the region of uncer- vantage of the minimax MSE estimator over the LS estimator, tainty, by minimizing objectives that depend explicitly on the and the advantage of the robust minimax MSE estimator over MSE. the minimax MSE estimator in the presence of model uncer- We first consider the case in which is known exactly and tainties, are discussed in Section VI. develop a minimax linear robust estimator that minimizes the worst-case MSE across all possible bounded values of , i.e., II. MINIMAX MSE ESTIMATION WITH KNOWN over all values of such that for some constant We denote vectors in by boldface lowercase letters and and weighting matrix . The minimax MSE estimator for the matrices in by boldface uppercase letters. denotes the special case in which and the covariance matrix of identity matrix of appropriate dimension, denotes the Her- the noise vector is given by for some mitian conjugate of the corresponding matrix, and denotes has been developed in [16]. Here, we extend the results to ar- an estimated vector or matrix. The notation means that bitrary and and show that the minimax MSE estimator can be formulated as the solution to a semidefinite programming is positive semidefinite. problem (SDP) [17]–[19], which is a tractable convex optimiza- Consider the problem of estimating the unknown determin- tion problem that can be solved efficiently, e.g., using interior istic parameters in the linear model point methods [19], [20]. We then develop a closed-form solu- (1) tion to the minimax estimation problem for the case in which the weighting and have the same eigenvector matrix. where is a known matrix with full rank , and is a In particular, when , we show that the optimal estimator is zero-mean random vector with covariance . We assume that a shrunken estimator proposed by Mayer and Willke [7], with a is known to satisfy the weighted norm constraint specific choice of shrinkage factor, that explicitly takes the prior for some positive definite matrix and scalar , where information into account. We demonstrate through simulations, . that the minimax MSE estimator can increase the performance We estimate using a linear estimator so that for over the conventional LS approach. some matrix . The MSE of the estimator is We then consider the case in which the model matrix is given by not known exactly, but is rather given by , where is known, and is a bounded perturbation matrix.