Bayesian inference for the Errors-In-Variables model
XING FANG1,2, BOFENG LI3, HAMZA ALKHATIB4, WENXIAN ZENG1* AND YIBIN YAO1
1 School of Geodesy and Geomatics, Wuhan University, China ([email protected]) 2 School of Earth Science, Ohio State University, USA 3 College of Surveying and Geo-Informatics, Tongji University, Shanghai, China 4 Geodetic Institute, Leibniz University Hannover, Germany * Corresponding author
Received: December 26, 2015; Revised: June 5, 2016; Accepted: June 28, 2016
ABSTRACT
We discuss the Bayesian inference based on the Errors-In-Variables (EIV) model. The proposed estimators are developed not only for the unknown parameters but also for the variance factor with or without prior information. The proposed Total Least-Squares (TLS) estimators of the unknown parameter are deemed as the quasi Least-Squares (LS) and quasi maximum a posterior (MAP) solution. In addition, the variance factor of the EIV model is proven to be always smaller than the variance factor of the traditional linear model. A numerical example demonstrates the performance of the proposed solutions.
Ke ywo rd s: Errors-In-Variables, Total Least-Squares, Bayesian inference, quasi solution, Maximum Likelihood, noninformative prior, informative prior
1. INTRODUCTION
The method of least-squares (LS), which was developed by C.F. Gauss and A.M. Legendre in nineteenth century (Stigler, 1986), has been widely applied to solve an overdetermined system. In spite of its wide use, however, the principle hypothesis of the coefficient matrix within the mathematical model is not necessarily true in all cases in geodesy. A popular type of models with an uncertain coefficient matrix is known in the literature as Errors-In-Variables (EIV) models. In 1980 Total Least-Squares (TLS) is introduced by Golub and van Loan (1980) in the field of numerical analysis. TLS is nowadays frequently utilized as a standard terminology of the estimation method sets for adjusting the EIV model in science and engineering fields, respectively. The EIV model is now attached the high importance in the geodetic data processing, since within the model the random errors of all measured data are rigorously regarded. In typical geodetic problems, e.g., regression and transformation, the random errors of the measured data contained in the design matrix should be properly taken into consideration. Therefore, the Gauss-Markov model (GMM) is improper for treating such cases any more rigorously.
Stud. Geophys. Geod., 61 (2017), 3552, DOI: 10.1007/s11200-015-6107-9 35 © 2017 Inst. Geophys. CAS, Prague X. Fang et al.
Recently, the investigation on adjusting the EIV model has been also exhibited by quite a number of publications in geodesy. Usually, the adjustment of EIV model without linearization is called TLS. The most frequent approaches include the closed form solution in terms of the singular value decomposition (SVD) of the data matrix (e.g., Teunissen, 1988; Felus, 2004; Akyilmaz, 2007; Schaffrin and Felus, 2008; Grafarend and Awange, 2012), reformulation of the TLS problem as a constrained minimization optimization problem (e.g., Schaffrin and Wieser, 2008; Xu et al., 2012; Snow, 2012; Li et al., 2013; Fang, 2014a), the iterative LS solution by properly treating the weight matrix (e.g., Amiri-Simkooei and Jazaeri, 2012, 2013) and transformation of the TLS problem into an unconstrained optimization problem (Xu et al., 2012; Fang, 2011, 2013, 2014b, 2015). All these methods provide the identical TLS solution and guarantee the (weighted) orthogonality when, and only when the design matrix within the EIV model contains only linear terms of random errors. Being an alternative to these TLS methods, the iteratively linearized Gauss Helmert model (GHM) method proposed by Pope (1972) can also solve the (weighted) TLS problem. Xu (2016) analyzed how random errors in the design matrix influence the variance components within the EIV model. Although a significant number of publications as mentioned above has been presented to adjust the EIV model, they are all based on the assumption that only the first and second moments of errors are available. In fact, most of the methods are indeed optimal/proper in case of normal distribution although the distribution information is not explicitly used. However, when the errors are not normally distributed, these methods are not proper anymore. The earliest studies in Bayesian EIV models can be found in Lindley and El Sayyad (1968) and Zellner (1971). Later, the investigation on adjusting the EIV model from the Bayesian perspective instead of from the frequentist point of view has been presented by a number of publications in different disciplines: e.g., Bauwens and Lubrano (1999) in economics, Florens et al. (1974), Polasek (1995), Reilly and Patino-Lea (1981), Bolfarine and Rodrigues (2007) and Huang (2010) in statistics, Dellaportas and Stephens (1995) in biometrics. However, parameter estimation methods proposed in the aforementioned publications were based on a simplified functional model (e.g, regression model) and the special stochastic information, i.e. the dispersion matrix is given in a special structure. Therefore, we will propose the Bayesian approach to handle the EIV model with general stochastic information. As a result, the prior information about the parameter vector as well as the variance factor can be fully considered in the Bayesian inference for suitable solution of the EIV model. Furthermore, the estimated variance component will be proven to be always smaller than that in the LS estimation. The objectives of this contribution are as follows. Firstly, the EIV model is formulated with the known probability density function (PDF); as a case study, the normal distribution is assumed. Secondly, the formulae of the maximum likelihood estimator (MLE) and maximum a posteriori estimator (MAPE) are derived to solve the model with the noninformative and informative prior density function of parameters under the conditions of known and unknown variance factor, respectively. Furthermore, it is proven that the estimate of the variance factor estimated by the (marginal) maximum likelihood method proposed is always smaller than the traditional estimate of the variance factor in the framework of the LS method. At the next step, a simulated example is presented. Finally, some concluding remarks and further work are given.
36 Stud. Geophys. Geod., 61 (2017)
Bayesian inference for the Errors-In-Variables model
2. THE FORMULATION OF THE ERRORS-IN-VARIABLES MODEL
It is well-known that the LS estimation is the best linear unbiased estimation when the design matrix is free of noise and the expectation of the random errors in the traditional observation vector equals zero. This kind of estimation has frequently been applied in the GMM for the error adjustment. In contrast, an EIV model is a model similar to GMM except for the elements of the design matrix that are contaminated by random errors. Consequently, the observation equation of the EIV model can be expressed as
yv=y AVA ξ , (1) where the column full rank n m matrix A affected by random errors and the conventional observation vector y have the correction matrix VA and vector vy. Vector is a vector of unknowns. With all error-affected variables as observations, the observations can be expressed as an extended vector vec A l , (2) y where ‘vec’ denotes the operator that stacks one column of a matrix underneath the previous one. The corresponding extended uncertainty vector v and the stochastic properties of the uncertainties can be characterized as follows:
vecVA vA 0 v , v , Σll , (3) vy vy 0 where
1 22Q0AA P0AA Σll 00= 0Qyy 0Pyy is the dispersion matrix of the extended observation vector, matrices QAA and Qyy are the 2 symmetric and positive definite cofactor matrices for vA and vy, vA vecVA , and 0 is the variance factor. PAA and Pyy are the symmetric and positive definite weight matrices for vA and vy. In this paper, the vectors vA and vy are assumed independent. In order to adjust the model, the objective function in the sense of weighted TLS criterion reads
TT vvvvAAAAPP y yyy min . (4)
Stud. Geophys. Geod., 61 (2017) 37
X. Fang et al.
3. THE MAXIMUM LIKELIHOOD ESTIMATOR FOR THE EIV MODEL
As the common assumption, the (vectorized) model matrix A and observation vector y 2 are here assumed to be normally distributed. Provided that 0 is known, the distributions are given as follows
2 2 y UUQAAy,,ξξ N 0 y , vecAUAAA N vec U , 0 QA , (5)
2 where U A denotes the expectation of the coefficient matrix. Here, the variance factor 0 for all measured quantities is modeled to be identical (see Schaffrin and Felus, 2008; Shen et al., 2010). Modeling of the different variance components of design matrix A and the conventional observation vector y leads to the issue of the variance components estimation within the EIV model (Amiri-Simkooei, 2013; Xu and Liu, 2014). With the distribution in Eq. (5), the likelihood function is given as
T 1vecUUP0AA vec exp llAA 2 2 UUξξ0Pyy 0 AA L l |,ξ U . (6) A n 2 12 2det Σll Conveniently, working with the logarithm of the likelihood function (6) instead of the exponential function yields
lnL l |ξ , UA T 11vecUU vec n (7) ll AAΣΣ1 ln 2 ln det . ll ll 222UUAAξξ In order to obtain the maximum of the log-likelihood function of Eq. (7), two partial derivatives with respect to the unknown parameter vector and the (vectorized) model matrix UA should be analytically derived. The first-order derivative of Eq. (7) with respect to the parameter vector reads:
lnL l |ξ , U A ξ T vecUU vec l AAΣ1 l ll UUAAξξ (8) ξ T 12y UPAyyAξ y Uξ UPT Uξ y . 22Ayy A 00 ξ
38 Stud. Geophys. Geod., 61 (2017)
Bayesian inference for the Errors-In-Variables model
The first derivative of Eq. (7) with respect to the vectorized coefficient matrix vecUA is
T vecUU vec llAAΣ1 ll lnLl |ξ , UA UUAAξξ vecUUAAvec T IImn II mn llvecU Σ1 vecU ξξTTIIAll A nn (9) vecU A T IImnII mn 2 Σ1 l vecU ξξTTIIll A nn vecU 2,IIξ Σ1 A l mnll U Aξ where the operator is the ‘Kronecker-Zehfuss product’ (e.g., Grafarend and Schaffrin, 1993, p.409), and In is the n n identity matrix. Let the first-order partial derivative of the log-likelihood function with respect to the parameter vector be equal to zero, the normal equations are derived to be ˆˆT ˆ UPAyy y U Aξ 0 . (10)
Accordingly, the MLE solution (with noninformative prior) reads
1 ˆ ˆˆˆTT ξMLE UPAyyA U UP Ayyy . (11) In this case, no prior information of parameters is applied. In the Bayesian inference, the posterior PDF (the prior density function multiplied with the likelihood function) is maximized where the noninformative prior is implemented with a prior density function of parameters proportional to a constant (see Koch, 2007). In such case, the MLE is equivalent to Bayesian inference, because the constant PDF of parameters does not influence the partial derivative with respect to parameters and then the solution. The MLE of the EIV model (11) is identical to the LS solution if the estimated model ˆ matrix UA is substituted by the deterministic model matrix A. The MLE solution of the heteroscedastic EIV model could be deemed as the quasi-LS solution with the following reasoning: 1. Model definition: Schaffrin and Felus (2008) claimed the EIV model can be regarded as the quasi linear GMM.
Stud. Geophys. Geod., 61 (2017) 39
X. Fang et al.
2. Data processing method category: the TLS adjustment belongs to the quasi indirect error adjustment category (Fang, 2014c) whereas the GMM is well known as the indirect error adjustment. 3. Ththe form of the estimator: identical to the LS solution when the deterministic model matrix A in LS sense is replaced by the estimated stochastic model matrix ˆ UA in the TLS sense (see Eq. (11)). 4. Geometrical interpretation: The traditional observation vector y is (orthogonally or ˆ P-orthogonally) projected on the column space of the estimated matrix UA , in ˆ ˆ which UAξ has to be estimated. Therefore, the argument that the TLS approach is not a new adjustment method but the LS method for adjusting the EIV model should be again emphasized due to the strong similarity of the obtained solution and the LS solution based on the above discussion. ˆ Equation (9) represents the relationship between the residual matrix UAA and ˆ ˆ vector UAξ y . The relationship can be straightforwardly formulated after setting Eq. (9) to zero as 11ˆˆˆˆ ΣAA vecUA A vec ξξ In Σ yy U A y , (12) which is equivalent to ˆ ˆ 1 vec VAAAnyyyΣ ξ I Σ vˆ . (13)
With the observation equations and vecABC CT A vec B , we have
ˆˆT ˆˆ ˆ ξ IVnAyAy vecvˆˆ Vξ v y Aξ . (14) ˆ Combining Eqs (13) and (14) the estimated residual matrix VA and vector vˆy can be derived as
1 vec Vˆ Σ ξξˆˆIIT Σ ξ ˆ I Σ y Aξˆ, AAAnnAAnyy (15) 1 vˆ Σ ξξˆˆT I Σ I Σ y Aξ ˆ. yyynAAnyy According to Eq. (11) we provide the first-order approximation of the cofactor matrix ˆ of the parameter vector by neglecting the randomness of UA as follows
1 QUPU ˆˆT . (16) ˆˆNON AyyA
40 Stud. Geophys. Geod., 61 (2017)
Bayesian inference for the Errors-In-Variables model
Being a non-linear problem, the iterative algorithm based on Eq. (11) for solving the EIV model is designed as Algorithm 1: MLE for the EIV model Step 1: We obtain an initial value for from the LS solution
ˆ0T1 T ξ APyyy A APyy . Step 2: Using the initial value we start the following iterative process:
1 iii1 ˆˆT ˆ i ˆ i vˆAAAnnAAnΣ ξξII Σ ξ IAy ξ , ˆ ii11 VAnuA Invec vˆ .
The operator Invecnu is the opposite of the ‘vec’ operator and reshapes the vector as the assigned matrix form
TT1 ˆii11ˆˆˆ i 1 i 1 ξ AVAyyA P AV AV Ayy Py .
Step 3: Until Step 2 ξξiii1 ξ ; the computed WTLS estimator with the ˆˆi1 noninformative prior is then ξξWTLS MLE : .
The quantity ξξiii1 ξ (c.f. Markovsky et al., 2006) for terminating the iteration procedure can be also substituted by e.g., ξξii1 (e.g., Schaffrin and Felus, 2008). is a sufficiently small threshold to terminate the iteration procedure ( > 0).
4. THE MAXIMUM A POSTERIORI ESTIMATOR FOR THE EIV MODEL
In the previous part, the noninformative prior density for the parameter vector is applied. Bayesian inference works now under the assumption that prior information on the parameter vector is available. If first and second central moments of the parameter vector are given as E ξμ and D ξ Σ , the prior distribution supposed as standard normal distribution follows
ξμ N , Σ . (17)
In terms of the Bayesian theorem that the posteriori density function is proportional to the product of the likelihood function and prior density function (Koch, 2007, p. 32), it follows
Stud. Geophys. Geod., 61 (2017) 41
X. Fang et al.
fLfξ ,,,UUUAAAll ξξ , (18) where the symbol denotes the relation of proportionality. Of course, with the fair assumption that the parameter is independent from the model matrix U A , we obtain fffξξ, UUAA . Here, it is meaningful that only the informative prior f ξ of the conventional parameters is available whereas the noninformative prior of the design matrix is given. Considering f U A as a constant, the posterior function reads
f ξ , UA l
LffLfl ξξ,,UUAA l ξξ U A (19) T vecUUAA11 vec T expllΣΣ ξμ ξμ . UUξξll AA Because the exponential function is a monotone increasing function, the maximum of the exponential function is equivalent to the maximum of the quadratic term
maxf ξ , UA l T (20) vecUUAA11 vec T maxllΣΣξμ ξμ . UUξξll AA In order to maximize the quadratic form, the first-order derivatives with respect to the unknown parameter vector and the (vectorized) model matrix UA should be zero. Thus, the first-order derivative with respect to is
T vecUU vec T llAAΣΣ11 ξμ ξμ ll UUAAξξ ξ T T 1 1 y UAllAξ Σ y U ξ ξμΣ ξμ (21) ξξ T1 22,UPAyy U Aξ y Σξμ which leads to the solution
1 ˆ ˆˆT1T1 ˆ ξMAP UPAyyA U Σ UPAyyy Σ μ . (22)
42 Stud. Geophys. Geod., 61 (2017)
Bayesian inference for the Errors-In-Variables model
The Bayesian inference is conducted for solving the weighted TLS problem incorporating the prior stochastic information of the parameters. In this case, the posterior function formed by the informative prior multiplied with the likelihood function is maximized. Within the quasi linear model the solution (22) can be termed as the quasi MAPE estimator in the contribution. The argument has similar reasons as given regarding the terminology of the quasi LS solution. ˆ The analytical formulation of the estimated residual matrix VA and vector vˆy is identical with Eq. (15). According to Eq. (22) we provide the first-order approximation of ˆ the cofactor matrix of parameter vector neglecting the randomness of UA as follows
1 QUPUˆˆT1Σ . (23) ˆˆIN AyyA
Consequently, the Algorithm 2 is designed as follows: Algorithm 2: MAPE for the EIV model Step 1: We obtain an initial value for from the MAPE solution (no EIV model)
ˆ0T 11 T 1 ξ APyy A Σ APyy y Σμ . Step 2: Using the initial value we start the following iterative process:
1 iii1 ˆˆT ˆ i ˆ i vˆAAAnΣ ξξII nAAnΣ ξ IAy ξ , ˆ ii11 VAnuA Invec vˆ ,
TT1 ˆii11ˆˆ i 111 ˆ i 1 ξ AVAyyA P AV Σ AV Ayy Py Σμ .
Step 3: Until Step 2 ξξiii1 ξ ; the computed WTLS estimator with the ˆˆi1 informative prior is then ξξWTLS MAP : . The Algorithm 2 presents the estimator for the EIV model with available informative prior of unknown parameters, whereas the Algorithm 1 provides the solution with noninformative prior for unknown parameters. Both solutions are similar to the well- known solutions with the deterministic design matrix. The differences are presented by ˆ the estimated model matrix UA and the iterative process due to the stochastic model matrix.
Stud. Geophys. Geod., 61 (2017) 43
X. Fang et al.
5. DISCUSSION ABOUT THE BAYESIAN INFERENCE FOR THE VARIANCE FACTOR
If the variance factor is assumed unknown, the likelihood function considering the variance factor as an unknown quantity can be described by
1 T P0AA exp vv 2 0P 20 yy L l ξ ,,U 2 . (24) A 0 n 2 n 2 12 2 2det Qll 0 According to Bayes’s theorem the posteriori function reads
2T11 P0AA flξ ,,U exp vv (25) A 0 n 21 2 0P 2 20 yy 0 in the case of noninformative prior. It is rather easy to prove that in such case, both MLE and MAPE solutions are not changed. Many publications (e.g., Felus, 2004; Schaffrin, 2006; Schaffrin and Felus, 2008; Kwon et al., 2009) claim that the variance factor can be estimated by the sum of the weighted squared residuals divided by the redundancy (or degrees of freedom) r = n m, namely P0AA vvˆˆT 0P 2 yy ˆ0 r (26) 1 ˆˆT T ˆ ˆ AIQIQAξ y ξξnAAn yy ξy . r The biasedness of the estimator was presented by Shen et al (2010), where a numerical method is applied to obtain the unbiased variance factor. Xu et al. (2012) also provide a variance component estimator based on the bias-corrected residuals. However, no one gives the analytical solution and discussion of the variance factor from the Bayesian perspective. It is well-known that the variance factor of the LS problem can be estimated by T T 2 yyIRPIRnyyn ˆ0 , (27) LS r where the matrix R is known as the ‘hat matrix’ since it projects the observation vector y into the vector of adjusted observation yˆ (Schaffrin, 1997). The matrix 1 TT IRIAAPAAPnn yyyy is called the well-known projection matrix (Koch,
44 Stud. Geophys. Geod., 61 (2017)
Bayesian inference for the Errors-In-Variables model
1999), the reliability matrix (Shan, 1989) or redundancy matrix (Schaffrin, 1997). In the 2 2 LS framework the estimated variance factor ˆ0 can be expressed as ˆ0 y without the parameter vector ξˆ . In contrast to the variance estimate in the LS problem, the variance estimate in the 2 TLS problem cannot be exempted from the parameter estimates. The estimator ˆ0 could 2 ˆ be recognized as ˆ0 ξ , l which is under the condition of the parameter estimates besides the observations l. Losing the primitive physical meaning denotes that the variance factor 2 ˆ should originally relate to the observations, and the estimated variance factor ˆ0 ξ , l is under the condition of ξˆ , which represents that the solution is obtained only in the mathematical sense. The unbiased estimator of the variance factor (or factors) in the LS sense is derived with the (marginal or restricted) maximum likelihood method (e.g., Xu et al., 2007) or orthogonal complement likelihood function (e.g., Koch, 1986). Each of the two methods uses the separation of the parameter vector and variance factor in the estimation process, i.e.
222 fff y 01020,,ξ yy ξ . (28)
2 With the function f10 y one can obtain the estimate of the variance factor T vvˆˆyyyyP r . Ambiguously using the redundancy r utilized in the above equation must be 2 ˆ an issue of concern in the case where ˆ0 and ξ cannot be separated. Furthermore, the variance factor estimated by vvˆˆT P r is a biased estimator in the framework of the TLS problem, which was already recognized in Schaffrin and Felus (2008) and Shen et al (2010). Therefore, the estimated variance factor should be expressed as vvˆˆT P n instead of vvˆˆT P r in the aspect of likelihood function, which is expressed in the weighted case as follows
1 ˆˆT T ˆ ˆ T AIQIQAξ y ξξnAAn yy ξy 2 vvˆˆP ˆ0 . (29) nn The bias of the variance component estimator above for the nonlinear model needs to be investigated in the future. According to the posterior density function (25) the estimate of the variance component can analogously be obtained as vvˆˆT P n 2 with the noninformative prior density. The difference between the MLE and MAPE for the variance components in the case is due to the noniformative prior density about the variance component, see Koch (1999) for detail.
Stud. Geophys. Geod., 61 (2017) 45
X. Fang et al.
If the informative prior about the parameters and the variance factor are simultaneously available, the conjugate normal-gamma distribution should be introduced in this case according Eq. (2.212) in Koch (2007). However, the solution of the parameter vector ξˆ and the residual vector are not influenced by this distribution. Furthermore, the estimated variance factor is obtained with the help of the inverted Gamma distribution (c.f. Koch, 1999) as well as the partial derivatives of the posterior density function, and omitted in this presentation.
6. COMPARISON OF THE VARIANCE FACTOR ESTIMATES BETWEEN LS AND TLS
The fact that the variance factor estimated by the LS approach is larger and even significantly larger than the variance factor estimated by the TLS technique is presented in almost all studies of TLS in geodesy (e.g., Schaffrin and Felus, 2008; Felus and Burtch, 2009). However, no one has analytically proved that the variance factor in the TLS sense is smaller than the variance factor in the LS sense. Using the matrix identity (Koch, 1999), the cofactor matrix is decomposed as
1 ˆˆT QIQIyyξξ n AA n (30) 1 11ˆˆˆˆTT 1 1 QQyyyynξξξξ I Q AAny IQ I n IQ nyy.
1 111ˆˆˆˆTT In order to establish QIQyynAAnynξξξξ IQI IQ nyy as a positive definite matrix, we use some properties presented in Koch (1999) about positive definite matrices: An inverse and a quadratic form of a positive definite matrix are a positive matrix and a semidefinite definite matrix. The sum of a positive definite matrix A and a positive semidefinite matrix B is also a positive definite matrix, since xxxxxxTTTAB A B 0 if x 0 . On this basis we have the positive definite 1 111ˆˆˆˆTT matrix QIQyynAAnynξξξξ IQI IQ nyy. Thus, we can have the inequality as follows
1 ˆˆTT11 ˆˆ T AQIQIQIξLS y yyξξξ n AA n yy n ˆˆ1 ξξIQnyy ALS y 0, where the condition for the equality sign is ξˆ 0 .
46 Stud. Geophys. Geod., 61 (2017)
Bayesian inference for the Errors-In-Variables model
Here follows the comparison between the estimated variance components for the weighted TLS and LS problem
1 ˆˆT ˆˆT min ABQBAξ y ll ξ y 2 ˆ0 n 1 ˆˆT ˆˆT min ABQBAξ y ll ξ y r 1 ˆˆT ˆˆT ABξTLSy QBAll ξ TLS y r (31) 1 ˆˆT ˆˆT ABQBAξLSy ll ξ LS y r ˆˆT 1 AQAξLSy yy ξ LS y r T AQAξˆˆy 1 ξ y LSyy LS 2 ˆLS , r ˆˆTTˆˆ where BQll Bξξ In Q AA I n Q yy . 2 Based on the analytical comparison, the conclusion can be drawn that ˆ0 is smaller 2 than ˆLS for the heteroscedastic case irrespective of whether the variance component is expressed with n or r. Note that either the variance component in the TLS sense or the variance components in LS sense do not consider the biases raised by the nonlinear system.
7. EXPERIMENT AND ANALYSIS
In this part the numerical example is designed to examine the proposed solutions. First, the MLE solution is tested if it is identical to the result of the current methods, then the MAPE solution is applied to show that it varies between the MLE solution and the prior parameter mean according to the different prior stochastic matrices of the parameter vector. We compute a straight-line fit problem where both variables have been observed. We use the data presented in Neri et al. (1989), which is also applied in Schaffrin and Wieser (2008) and Shen et al. (2010), and estimate the intercept and slope regression parameter 1 and 2 in the following regression line
Stud. Geophys. Geod., 61 (2017) 47
X. Fang et al.
1 y+vyn =1 , x+v x , 2 where 1n is the vector of ones with dimension n 1. In Table 1, the observations are in columns 2 and 3, and their corresponding weights in 1 columns 4 and 5. We arrange the cofactor matrices as Q yy diag wyi and 00 1 QAA diagwxi , where ‘diag’ denotes the operator that converts a vector 01 into a diagonal matrix with the vector’s elements representing the diagonal entries of the matrix. This model formulation coincides with the model used by Schaffrin and Wieser (2008) and Shen et al. (2010). First of all, we compute the MLE solution using the general LS (GLS) solution as the initial value for the iterative process proposed (Algorithm 1). The GLS represents the ordinary LS algorithm when only considering the weight matrix of the y data. Choosing the same threshold 1010, it is seen in Table 2 that our MLE solution is identical to the exact solution. In order to examine Algorithm 2, we simulate the stochastic prior information of the parameter vector as ξξ GLS, I 2 , where ξGLS is the GLS solution without the
Table 1. Observed data (xi, yi) and corresponding weights (wxi, wyi), taken from Neri et al.(1989).
Point Number xi wxi yi wyi 1 0.0 1000 5.9 1.0 2 0.9 1000 5.4 1.8 3 1.8 500 4.4 4.0 4 2.6 800 4.6 8.0 5 3.3 200 3.5 20 6 4.4 80 3.7 20 7 5.2 60 2.8 70 8 6.1 20 2.8 70 9 6.5 1.8 2.4 100 10 7.4 1.0 1.5 500
ˆ ˆ Table 2. Results of straight-line fit of the observed data of Table 1. 1 : intercept, 2 : slope of the fit line.
MLE Solution Parameter Estimate Exact Solution (WTLS) GLS Solution (Algorithm 1) ˆ 1 5.47991022 5.479910224 6.100 ˆ 2 0.480533407 0.480533407 0.611
48 Stud. Geophys. Geod., 61 (2017)
Bayesian inference for the Errors-In-Variables model
Fig. 1. The maximum a posteriori estimator (MAPE) solution of straight-line fit using the different prior variance component of parameters. GLS: general least squares, MLE: maximum likelihood estimator, LS: least squares, TLS: total least squares. randomness (the numerical value) and is the variance component. In the example we change the value of from 105 until 104 via multiplying by 10 in one time. According to different values the distinct parameter estimates are obtained by means of Algorithm 2. The results of MAPE are presented in Fig. 1, where the x axis presents the changing values of variance component in logarithmic scale and the y axis shows the parameter estimates with the symbol ‘circle’. The symbol ‘cross’ denotes the GLS estimate whereas the symbol ‘plus’ denotes the MLE result. The phenomenon is clearly illustrated in Fig. 1 that the results of the MAPE are much closer to GLS solution (also prior expectation of the parameter vector) if the variance component is small whereas they match exactly the MLE solution if the variance component is large. The phenomenon demonstrated in Fig. 1 identifies to the fact that the MAPE is the weighted mean of the prior expectation of parameters and the MLE. Thus, the MAPE result varies between the GLS and TLS solution.
8. CONCLUSIONS AND FURTHER WORK
We have derived a procedure to compute the WTLS for an EIV model with a noninformative or informative prior. In particular, the formulation of the proposed WTLS solution with a noninformative or informative prior is allowed to be regarded as a quasi LS solution or a quasi MAPE estimation. The variance component is also estimated from the Bayesian inference, where the sum of the weighted squared residuals should be divided by n instead of the redundancy number due to the EIV model’s nonlinear nature and the variance component’s inseparability to the unknown parameter vector. Furthermore, the variance component of the EIV model is analytically proved to be smaller than the variance component in the LS adjustment. A numerical example is demonstrated to support the mathematical development in this contribution.
Stud. Geophys. Geod., 61 (2017) 49
X. Fang et al.
The model may be adjusted by some numeric methods such as Markov Chain Monte Carlo (MCMC) type methods (Metropolis algorithm or Gibbs sampler) if the (vectorized) model matrix and observation vector belong to another distribution, which is not analytically tractable. However, it needs a systematic investigation in the future.
Acknowledgments: This research was supported by the National Natural Science Foundation of China(41474006; 41304005) and the Fundamental Research Funds for the Central Universities (2042016kf0175)
References Akyilmaz O., 2007. Total least squares solution of coordinate transformation. Surv. Rev., 39, 6880. Amiri-Simkooei A., 2013. Application of least squares variance component estimation to errors-in- variables models. J. Geodesy, 87, 935944. Amiri-Simkooei A. and Jazaeri S., 2012. Weighted total least squares formulated by standard least squares theory. J. Geod. Sci., 2, 113124. Amiri-Simkooei A. and Jazaeri S., 2013. Data-snooping procedure applied to errors-in-variables models. Stud. Geophys. Geod., 57, 426441. Bauwens L., Lubrano M. and Richard J.-F., 1999. Bayesian Inference in Dynamic Econometric Models. Oxford University Press, Oxford, U.K. Bolfarine H. and Rodrigues J., 2007. Bayesian inference for an extended simple regression measurement error model using skewed priors. Bayesian Anal., 2, 349364. Dellaportas P. and Stephens D.A., 1995. Bayesian analysis of errors-in-variables regression models. Biometrics, 51, 10851095. Fang X., 2011. Weighted Total Least Squares Solutions for Applications in Geodesy. PhD Thesis. Leibniz University, Hannover, Germany. Fang X., 2013. Weighted Total Least Squares: necessary and sufficient conditions, fixed and random parameters. J. Geodesy, 87, 733749. Fang X., 2014a. A structured and constrained Total Least-Squares solution with cross-covariances. Stud. Geophys. Geod., 58, 116. Fang X., 2014b. On non-combinatorial weighted Total Least Squares with inequality constraints. J. Geodesy, 88, 805816. Fang X., 2014c. A total least squares solution for geodetic datum transformations. Acta Geod. Geophys., 49, 189207. Fang X., 2015. Weighted total least-squares with constraints: a universal formula for geodetic symmetrical transformations. J. Geodesy, 89, 459469. Felus F., 2004. Application of Total Least Squares for Spatial Point Process Analysis. J. Surv. Eng., 130, 126133. Felus F. and Burtch R., 2009. On symmetrical three-dimensional datum conversion. GPS Solut., 13, 6574.
50 Stud. Geophys. Geod., 61 (2017)
Bayesian inference for the Errors-In-Variables model
Florens J.-P., Mouchart M. and Richard J.-F., 1974. Bayesian inference in error-in-variables models. J. Multivar. Anal., 4, 419452. Golub G. and Van Loan C., 1980. An analysis of the Total least-squares problem. SIAM J. Numer. Anal., 17, 883893. Grafarend E. and Awange J.L., 2012. Applications of Linear and Nonlinear Models. Fixed Effects, Random Effects, and Total Least Squares. Springer-Verlag, Berlin, Germany. Grafarend E.W. and Schaffrin B., 1993. Ausgleichungsrechnung in linearen Modellen. BI- Wissenschaftsverlag, Mannheim, Germany (in German). Huang H.-J., 2010. Bayesian Analysis of Errors-in-Variables Growth Curve Models. PhD Thesis. University of California, Riverside, CA. Koch K., 1986. Maximum likelihood estimate of variance components. Bull. Geod., 60, 329338. Koch K., 1999. Parameter Estimation and Hypothesis Testing in Linear Models. 2nd Edition. Springer-Verlag, Berlin, Germany. Koch K., 2007. Introduction to Bayesian Statistics. 2nd Edition. Springer-Verlag, Berlin, Germany. Markovsky I. and Van Huffel S., 2007. Overview of total least-squares methods. Signal Process., 87, 22832302. Kwon J.H., Lee J.K., Schaffrin B., Yun S.C. and Lee I., 2009. New affine transformation parameters for the horizontal network of Seoul/Korea by multivariat TLS-adjustment. Surv. Rev., 41, 279291 Li B., Shen Y., Zhang X., Li C. and Lou L., 2013. Seamless multivariate affine error-in-variables transformation and its application to map rectification. Int. J. Geogr. Inf. Sci., 27, 15721592. Lindley D.V. and El Sayyad G.M., 1968. The Bayesian estimation of a linear functional relationship. J. R. Stat. Soc. Ser. B-Stat. Methodol., 30, 190202. Neitzel F., 2010. Generalization of total least-squares on example of unweighted and weighted 2D similarity transformation. J. Geodesy, 84, 751762. Neri F., Saitta G. and Chiofalo S., 1989. An accurate and straightforward approach to line regression analysis of error-affected experimental data. J. Phys. E-Sci. Instr., 22, 215217. Pope A., 1972. Some pitfalls to be avoided in the iterative adjustment of nonlinear problems. In: Proceedings of the 38th Annual Meeting, American Society of Photogrammetry, Washington, D.C., Mar. 1217, 1972. American Society of Photogrammetry, Bethesda, MD, 449477. Polasek W., 1995. Bayesian generalized errors in variables (GEIV) models for censored regrssions. In: Mammitzsch V. and Schneeweiss H. (Eds), Symposia Gaussiana. Conference B: Statistical Sciences. De Gruyter, Berlin, Germany, 261279. Reilly P.M. and Patino-Lea H., 1981. A Bayesian study of the error-in-variables model. Technometrics, 23, 221231. Schaffrin B., 1997. Reliability measures for correlated observations. J. Surv. Eng.-ASCE, 123, 126137. Schaffrin B., 2006. A note on Constrained Total Least-Squares estimation. Linear Alg. Appl., 417, 245258. Schaffrin B. and Felus Y., 2008. On the multivariate total least-squares approach to empirical coordinate transformations. Three algorithms. J. Geodesy, 82, 373383.
Stud. Geophys. Geod., 61 (2017) 51
X. Fang et al.
Schaffrin B. and Wieser A., 2008. On weighted total least-squares adjustment for linear regression. J. Geodesy, 82, 415421. Shan J., 1989. A fast recursive method for repeated computation of the reliability matrix QvvP. Photogrammetria, 43, 337346. Shen Y., Li B.F. and Chen Y., 2010. An iterative solution of weighted total least-squares adjustment. J. Geodesy, 85, 229238. Snow K., 2012. Topics in Total Least-Squares Adjustment within the Errors-In-Variables Model: Singular Cofactor Matrices and Priori Information. PhD Thesis. Report 502. Geodetic Science Program, School of Earth Sciences, The Ohio State University, Columbus, OH. Stigler S.M., 1986. The History of Statistics: The Measurement of Uncertainty before 1900. Belknap Press of Harvard University Press, Cambridge, MA, ISBN 0-674-40340-1. Teunissen P.J.G., 1988. The nonlinear 2D symmetric Helmert transformation: an exact nonlinear least squares solution, Bull. Geod., 62, 115. Xu P., Liu Y., Shen Y. and Fukuda Y., 2007. Estimability analysis of variance and covariance components. J. Geodesy, 81, 593602. Xu P., Liu J. and Shi C., 2012. Total least squares adjustment in partial errors-in-variables models: algorithm and statistical analysis. J. Geodesy, 86, 661675. Xu P., 2016. The effect of errors-in-variables on variance component estimation. J. Geodesy, DOI: 10.1007/s00190-016-0902-0 Zellner A., 1971. An Introduction to Bayesian Inference in Econometrics. John Wiley & Sons.
52 Stud. Geophys. Geod., 61 (2017)