DOCSLIB.ORG

Tikhonov Regularization for Weighted Total Least Squares Problems✩

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Tikhonov Regularization for Weighted Total Least Squares Problems✩ View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Elsevier - Publisher Connector Applied Mathematics Letters 20 (2007) 82–87 www.elsevier.com/locate/aml Tikhonov regularization for weighted total least squares problems✩ Yimin Wei a,b,∗, Naimin Zhangc, Michael K. Ngd, Wei Xue a School of Mathematical Sciences, Fudan University, Shanghai 200433, PR China b Key Laboratory of Mathematics for Nonlinear Sciences (Fudan University), Ministry of Education, PR China c Mathematics and Information Science College, Wenzhou University, Wenzhou 325035, PR China d Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong e Department of Computing and Software, McMaster University, Hamilton, Ont., Canada L8S 4L7 Received 15 August 2003; received in revised form 15 January 2006; accepted 1 March 2006 Abstract In this work, we study and analyze the regularized weighted total least squares (RWTLS) formulation. Our regularization of the weighted total least squares problem is based on the Tikhonov regularization. Numerical examples are presented to demonstrate the effectiveness of the RWTLS method. c 2006 Elsevier Ltd. All rights reserved. Keywords: Tikhonov regularization; Total least squares; Weighted regularized total least squares; Matrix differentiation; Lagrange multiplier 1. Introduction In this work, we study the regularized weighted total least squares (RWTLS) formulation. Our regularization of the weighted total least squares problem is based on the Tikhonov regularization [1]. For the total least squares (TLS) problem [2], the truncation approach has already been studied by Fierro et al. [3]. In [4], Golub et al. has considered the Tikhonov regularization approach for TLS problems. They derived a new regularization method in which stabilization enters the formulation in a natural way, and that is able to produce regularized solutions with superior properties for certain problems in which the perturbations are large. In the present work, we focus on RWTLS problems. We show that the RWTLS solution is closely related to the Tikhonov solution to the weighted least squares solution. Our work is organized as follows. In Section 2, we introduce the RWTLS formulation and study its regularizing properties. Computational methods are described in Section 3. In Section 4, numerical examples are presented to demonstrate the RWTLS method. ✩ Research supported in part by the National Natural Science Foundation of China under grant 10471027 and Shanghai Education Committee, and Hong Kong RGC Grants Nos 7046/03P, 7035/04P and 7035/05P, and HKBU FRGs. ∗ Corresponding author at: School of Mathematical Sciences, Fudan University, Shanghai 200433, PR China. E-mail addresses: [email protected], ymwei [email protected] (Y. Wei), [email protected] (N. Zhang), [email protected] (M.K. Ng). 0893-9659/$ - see front matter c 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.aml.2006.03.004 Y. Wei et al. / Applied Mathematics Letters 20 (2007) 82–87 83 2. The regularized weighted total least squares A general version of Tikhonov’s formulation for the linear weighted total least squares (WTLS) problem takes the form [5] ˜ ˜ ˜ ˜ min U[(A, b) − (A, b)]V F subject to b = Ax, DxS ≤ δ, (1) x where D is the regularization matrix, V = diag(W,γ)with γ being a non-zero constant, U and W are nonsingular 2 = T δ matrices, S is a symmetric positive definite matrix with y S y Sy,and is a positive constant. By using the Lagrange multiplier formulation, this problem can be rewritten as follows: L( ˜, ,μ)= [( , ) − ( ˜, ˜)] 2 + μ( 2 − δ2), ˜ = ˜ , A x U A b A b V F Dx S subject to b Ax (2) where μ is the Lagrange multiplier, and μ is equal to zero if the inequality constraint becomes equality. The solution x¯δ to this problem is different from the solution xWTLS to ˜ ˜ ˜ ˜ min U[(A, b) − (A, b)]V F subject to b = Ax, (3) x for δ less than DxWTLS2. Before we show the properties of the solution to (2), we have the following results about the matrix differentiation for the matrices A, A˜, W and U. Lemma 1. ∂ tr(W T ATU TU AW˜ ) ∂ tr(W T A˜TU TUAW) (i) = U TUAWWT (ii) = U TUAWWT ∂ A˜ ∂ A˜ ∂ tr(W T A˜TU TU AW˜ ) ∂(bTU TU Ax˜ ) (iii) = 2U TU AWW˜ T (iv) = U TUbxT ∂ A˜ ∂ A˜ ∂(xT A˜TU TUb) ∂(xT A˜TU TU Ax˜ ) (v) = U TUbxT (vi) = 2U TU Axx˜ T. ∂ A˜ ∂ A˜ Proof. Since (i) is equivalent to (ii), (iv) is equivalent to (v), and (vi) is a special case of (iii), we only give the proofs of (i) and (iii). Let Z be a p × q matrix of differentiable functions of the m × n matrix X.If ∂ Z = (mn) + ( (mn))T , = ,..., , = ,..., GEij H C Eij F i 1 m j 1 n ∂xij then ∂zij ( ) ( ) = GT E pq H T + F(E pq )TC, i = 1,...,p, j = 1,...,q, ∂ X ij ij and the converse is also true (see p. 57, Theorem 7.1 in [6]), where G = (gij) is a p ×m matrix, H = (hij) is an n ×q = ( ) × = ( ) × (kl) ( , ) matrix, C cij is a p n matrix, F fij is an m q matrix Eij is a k-by-l zero matrix except the i j -entry being equal to one. ˜ ˜ T T T ∂ tr(Y AW) ∂ ˜ ∂(Y AW)ii For (i), we consider Y = W A U U and we have = (Y AW)ii = .Since ∂ A˜ ∂ A˜ i i ∂ A˜ ˜ ∂( ˜ ) ˜ ∂(Y AW) = Y AW ij = T T ∂ tr(Y AW) = T T = T T ˜ YEijW and ˜ Y EijW , we obtain ˜ i Y Eii W Y W . The result follows. ∂ Aij ∂ A ∂ A ˜ ˜ ∂[(U AW)T(U AW)] T T T ˜ ˜ T For (iii), we find that ˜ = W E U U AW + (U AW) UEijW, and therefore we have ∂ Aij ij ˜ T ˜ ∂[(U AW) (U AW)]ii T ˜ T T T ˜ T = U U AW E W + U U AW Eii W . It follows that ∂ A˜ ii 84 Y. Wei et al. / Applied Mathematics Letters 20 (2007) 82–87 ∂ tr[(U AW˜ )T(U AW˜ )] ∂[(U AW˜ )T(U AW˜ )] = ii = U TU AW˜ ET W T + U TU AW˜ E W T ∂ ˜ ∂ ˜ ii ii A i A i = 2U TU AWW˜ T. Theorem 1. The RWTLS solution to (1) with the inequality constraint replaced by equality is a solution to the problem − − (ATU TUA+ αW T W 1 + β DT SD)x = ATU TUb, (4) where the parameters α and β are given by −γ 2b − Ax2 μ α = U TU ,β= ( + γ 2 2 ) 1 x −T −1 (5) + γ 2x2 γ 2 W W 1 W −T W −1 and μ is the Lagrange multiplier in (2). The two parameters are related by 1 βδ2 = (Ub)TU(b − Ax) + α, (6) γ 2 and the weighted TLS residual satisfies [( , ) − ( ˜, ˜)] 2 =−α. U A b A b V F (7) Proof. We characterize the solution to (1) by setting the partial derivatives of L(A˜, x,μ)to zero. Using Lemma 1,the differentiation of L(A˜, x,μ)with respect to A˜ yields U AWW˜ T − UAWWT − γ rx˜ T = 0, (8) where r˜ = γ U(b − Ax˜ ) = γ U(b − b˜). Moreover, the differentiation of L(A˜, x,μ) with respect to the entries in x yields −γ A˜TU Tr˜ + μDT SDx = 0or(γ 2 A˜TU TU A˜ + μDT SD)x = γ 2 A˜TU TUb. (9) By using (8) and (9),wehave ATU TUA = (U A˜ − γ rx˜ TW −T W −1)T(U A˜ − γ rx˜ TW −T W −1) = ˜T T ˜ + γ 2˜2 −T −1 T −T −1 A U U A r 2W W xx W W − − − − − μDT SDxxTW T W 1 − μW T W 1xxT DT SD ˜T T T T −T −1 T and A U Ub = A U Ub + γ W W xr˜ Ub. By using the assumption that DxS = δ and gathering the above terms, we obtain (5) with μ 2 2 2 2 T 2 2 α = μδ − γ ˜r x − − − γ r˜ Ub and β = (1 + γ x − − ). 2 W T W 1 γ 2 W T W 1 In order to obtain the expression for α, we first rewrite r˜ as ˜ = γ ( − ˜ ) = γ ( − − γ −1 ˜ T −T −1 ) = γ ( − ) − γ 2 ˜ 2 r U b Ax U b Ax U rx W W x U b Ax r x W −T W −1 from which we obtain the relation γ U(b − Ax) r˜ = . (10) 1 + γ 2x2 W −T W −1 From (9),wehave γ xT A˜TU Tr˜ (γ Ub−˜r)Tr˜ μ = = . (11) xT DT SDx δ2 By inserting (10) and (11) into the expression for α, we obtain (5).Eq.(6) is proved by multiplying β by δ2 and inserting (10) and (11). Y. Wei et al. / Applied Mathematics Letters 20 (2007) 82–87 85 Finally, we note from (8) that UAW − U AW˜ =−γ rx˜ TW −T and therefore we have (UAW,γUb) − (U AW˜ ,γU Ax˜ ) = (−γ rx˜ TW −T , r˜). It follows that [( , ) − ( ˜, ˜)] 2 =γ ˜ T −T 2 +˜2 U A b A b V F rx W F r 2 γ 2b − Ax2 = ( + γ 2 2 )˜2 = U TU =−α. 1 x −T −1 r W W 2 1 + γ 2x2 W −T W −1 The next theorem tells us the relationship between the RWTLS solution and the WTLS solution in (3) without the regularization. Theorem 2. For a given value of δ, the RWTLS solution x RWTLS(δ) is related to the solution xWTLS to the weighted total least squares problem without the regularization as follows: δ solution αβ ∂α δ<DxWTLSS xRWT LS(δ) = xWTLS α<0and ∂δ > 0 β>0 2 δ ≥DxWTLSS xRWT LS(δ) = xWTLS α =−σmin((UAW,γUb)) β = 0 Here σmin((UAW,γUb)) is the smallest singular value of the matrix (UAW,γUb).
Load more

Recommended publications
  • Least-Squares Fitting of Circles and Ellipses∗
    Least-Squares Fitting of Circles and Ellipses∗ Walter Gander Gene H. Golub Rolf Strebel Dedicated to Ake˚ Bj¨orck on the occasion of his 60th birthday. Abstract Fitting circles and ellipses to given points in the plane is a problem that arises in many application areas, e.g. computer graphics [1], coordinate metrol- ogy [2], petroleum engineering [11], statistics [7]. In the past, algorithms have been given which fit circles and ellipses in some least squares sense without minimizing the geometric distance to the given points [1], [6]. In this paper we present several algorithms which compute the ellipse for which the sum of the squares of the distances to the given points is minimal. These algorithms are compared with classical simple and iterative methods. Circles and ellipses may be represented algebraically i.e. by an equation of the form F(x) = 0. If a point is on the curve then its coordinates x are a zero of the function F. Alternatively, curves may be represented in parametric form, which is well suited for minimizing the sum of the squares of the distances. 1 Preliminaries and Introduction Ellipses, for which the sum of the squares of the distances to the given points is minimal will be referred to as \best fit” or \geometric fit", and the algorithms will be called \geometric". Determining the parameters of the algebraic equation F (x)=0intheleast squares sense will be denoted by \algebraic fit" and the algorithms will be called \algebraic". We will use the well known Gauss-Newton method to solve the nonlinear least T squares problem (cf.
    [Show full text]
  • Lesson 4 – Instrumental Variables
    Ph.D. IN ENGINEERING AND Advanced methods for system APPLIED SCIENCES identification Ph.D. course Lesson 4: Instrumental variables TEACHER Mirko Mazzoleni PLACE University of Bergamo Outline 1. Introduction to error-in-variables problems 2. Least Squares revised 3. The instrumental variable method 4. Estimate an ARX model from ARMAX generated data 5. Application study: the VRFT approach for direct data driven control 6. Conclusions 2 /44 Outline 1. Introduction to error-in-variables problems 2. Least Squares revised 3. The instrumental variable method 4. Estimate an ARX model from ARMAX generated data 5. Application study: the VRFT approach for direct data driven control 6. Conclusions 3 /44 Introduction Many different solutions have been presented for system identification of linear dynamic systems from noise-corrupted output measurements On the other hand, estimation of the parameters for linear dynamic systems when also the input is affected by noise is recognized as a more difficult problem Representations where errors or measurement noises are present on both inputs and outputs are usually called Errors-in-Variables (EIV) models In case of static systems, errors-in-variables representations are closely related to other well-known topics such as latent variables models and factor models 4 /44 Introduction Errors-in-variables models can be motivated in several situations: • modeling of the dynamics between the noise-free input and noise-free output. The reason can be to have a better understanding of the underlying relations (e.g. in econometrics), rather than to make a good prediction from noisy data • when the user lacks enough information to classify the available signals into inputs and outputs and prefer to use a “symmetric” system model.
    [Show full text]
  • Overview of Total Least Squares Methods
    Overview of total least squares methods ,1 2 Ivan Markovsky∗ and Sabine Van Huffel 1 — School of Electronics and Computer Science, University of Southampton, SO17 1BJ, UK, [email protected] 2 — Katholieke Universiteit Leuven, ESAT-SCD/SISTA, Kasteelpark Arenberg 10 B–3001 Leuven, Belgium, [email protected] Abstract We review the development and extensions of the classical total least squares method and describe algorithms for its generalization to weighted and structured approximation problems. In the generic case, the classical total least squares problem has a unique solution, which is given in analytic form in terms of the singular value de- composition of the data matrix. The weighted and structured total least squares problems have no such analytic solution and are currently solved numerically by local optimization methods. We explain how special structure of the weight matrix and the data matrix can be exploited for efficient cost function and first derivative computation. This allows to obtain computationally efficient solution methods. The total least squares family of methods has a wide range of applications in system theory, signal processing, and computer algebra. We describe the applications for deconvolution, linear prediction, and errors-in-variables system identification. Keywords: Total least squares; Orthogonal regression; Errors-in-variables model; Deconvolution; Linear pre- diction; System identification. 1 Introduction The total least squares method was introduced by Golub and Van Loan [25, 27] as a solution technique for an overde- termined system of equations AX B, where A Rm n and B Rm d are the given data and X Rn d is unknown.
    [Show full text]
  • Uncorrected Proof
    Annals of Operations Research 124, 69–79, 2003 2003 Kluwer Academic Publishers. Manufactured in The Netherlands. 1 1 2 2 3 3 4 Multiple Neutral Data Fitting 4 5 5 6 6 CHRIS TOFALLIS [email protected] 7 7 Deptament of Stats, Econ, Accounting and Management Systems, Business School, 8 University of Hertfordshire, UK 8 9 9 10 10 11 Abstract. A method is proposed for estimating the relationship between a number of variables; this differs 11 12 from regression where the emphasis is on predicting one of the variables. Regression assumes that only 12 13 one of the variables has error or natural variability, whereas our technique does not make this assumption; 13 instead, it treats all variables in the same way and produces models which are units invariant – this is 14 important for ensuring physically meaningful relationships. It is thus superior to orthogonal regression in 14 15 that it does not suffer from being scale-dependent. We show that the solution to the estimation problem is 15 16 a unique and global optimum. For two variables the method has appeared under different names in various 16 17 disciplines, with two Nobel laureates having published work on it. 17 18 18 Keywords: functional relation, data fitting, regression, model fitting 19 19 20 20 21 Introduction 21 22 22 23 23 24 The method of regression is undoubtedly one of the most widely used quantitative meth- 24 25 ods in both the natural and social sciences. For most users of the technique the term is 25 26 taken to refer to the fitting of a function to data by means of the least squares criterion.
    [Show full text]
  • ROBUST SOLUTIONS to LEAST-SQUARES PROBLEMS with UNCERTAIN DATA ∗ ´ LAURENT EL GHAOUI† and HERVE LEBRET† Abstract
    SIAM J. MATRIX ANAL. APPL. c 1997 Society for Industrial and Applied Mathematics Vol. 18, No. 4, pp. 1035–1064, October 1997 015 ROBUST SOLUTIONS TO LEAST-SQUARES PROBLEMS WITH UNCERTAIN DATA ∗ ´ LAURENT EL GHAOUI† AND HERVE LEBRET† Abstract. We consider least-squares problems where the coefficient matrices A, b are unknown but bounded. We minimize the worst-case residual error using (convex) second-order cone program- ming, yielding an algorithm with complexity similar to one singular value decomposition of A. The method can be interpreted as a Tikhonov regularization procedure, with the advantage that it pro- vides an exact bound on the robustness of solution and a rigorous way to compute the regularization parameter. When the perturbation has a known (e.g., Toeplitz) structure, the same problem can be solved in polynomial-time using semidefinite programming (SDP). We also consider the case when A, b are rational functions of an unknown-but-bounded perturbation vector. We show how to mini- mize (via SDP) upper bounds on the optimal worst-case residual. We provide numerical examples, including one from robust identification and one from robust interpolation. Key words. least-squares problems, uncertainty, robustness, second-order cone programming, semidefinite programming, ill-conditioned problem, regularization, robust identification, robust in- terpolation AMS subject classifications. 15A06, 65F10, 65F35, 65K10, 65Y20 PII. S0895479896298130 Notation. For a matrix X, X denotes the largest singular value and X F k k k k the Frobenius norm. If x is a vector, maxi xi is denoted by x . For a matrix A, | | k k∞ A† denotes the Moore–Penrose pseudoinverse of A.
    [Show full text]
  • Thesis Would Not Be Possible
    SYMMETRIC AND TRIMMED SOLUTIONS OF SIMPLE LINEAR REGRESSION by Chao Cheng A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree MASTER OF SCIENCE (STATISTICS) December 2006 Copyright 2006 Chao Cheng Dedication to my grandmother to my parents ii Acknowledgments First and foremost, I would like to thank my advisor, Dr. Lei Li, for his patience and guidance during my graduate study. Without his constant support and encouragement, the completion of the thesis would not be possible. I feel privileged to have had the opportunity to work closely and learn from him. I am very grateful to Dr. Larry Goldstein and Dr. Jianfeng Zhang, in the Depart- ment of Mathematics, for their most valuable help in writing this thesis. Most of my knowledge about linear regression is learned from Dr. Larry Goldstein’s classes. I would also like to thank Dr. Xiaotu Ma, Dr. Rui Jiang, Dr. Ming Li, Huanying Ge, and all the past and current members in our research group for their suggestions and recommendations. I greatly benefit from the discussion and collaboration with them. Finally, I would like to thank my little sister and my parents for their love, support and encouragement. I would also like to thank my grandmother, who passed away in 2002, soon after I began my study in USC. I feel sorry that I did not accompany her in her last minute. This thesis is dedicated to her. iii Table of Contents Dedication ii Acknowledgments iii List of Tables vi List of Figures vii Abstract 1 Chapter 1: INTRODUCTION 2 1.1 Simple linear regression model .
    [Show full text]
  • Total Least Squares Regression in Input Sparsity Time
    Total Least Squares Regression in Input Sparsity Time Huaian Diao Zhao Song Northeast Normal University & KLAS of MOE University of Washington [email protected] [email protected] David P. Woodruff Xin Yang Carnegie Mellon University University of Washington [email protected] [email protected] Abstract In the total least squares problem, one is given an m n matrix A, and an m d × × matrix B, and one seeks to “correct” both A and B, obtaining matrices A and B, so that there exists an X satisfying the equation AX = B. Typically the problem is overconstrained, meaning that m max(n, d). The cost of the solutionb A,bB 2 2 b b is given by A A F + B B F . We give an algorithm for finding a solution k − k k − k 2 b2 b X to the linear system AX = B for which the cost A A F + B B F is at most a multiplicativeb (1 + )bfactor times the optimalk − cost,k up tok an− additivek error η that may be an arbitrarilyb b small function of n. Importantly,b our runningb time is O(nnz(A) + nnz(B)) + poly(n/) d, where for a matrix C, nnz(C) denotes its number of non-zero entries. Importantly,· our running time does not directlye depend on the large parameter m. As total least squares regression is known to be solvable via low rank approximation, a natural approach is to invoke fast algorithms for approximate low rank approximation, obtaining matrices A and B from this low rank approximation, and then solving for X so that AX = B.
    [Show full text]
  • A Bayesian Approach to Nonlinear Parameter Identification for Rigid
    A Bayesian Approach to Nonlinear Parameter Identification for Rigid Body Dynamics Jo-Anne Ting∗, Michael Mistry∗, Jan Peters∗, Stefan Schaal∗† and Jun Nakanishi†‡ ∗Computer Science, University of Southern California, Los Angeles, CA, 90089-2520 †ATR Computational Neuroscience Labs, Kyoto 619-0288, Japan ‡ICORP, Japan Science and Technology Agency Kyoto 619-0288, Japan Email: {joanneti, mmistry, jrpeters, sschaal}@usc.edu, [email protected] Abstract— For robots of increasing complexity such as hu- robots such as humanoid robots have significant additional manoid robots, conventional identification of rigid body dynamics nonlinear dynamics beyond the rigid body dynamics model, models based on CAD data and actuator models becomes due to actuator dynamics, routing of cables, use of protective difficult and inaccurate due to the large number of additional nonlinear effects in these systems, e.g., stemming from stiff shells and other sources. In such cases, instead of trying to wires, hydraulic hoses, protective shells, skin, etc. Data driven explicitly model all possible nonlinear effects in the robot, parameter estimation offers an alternative model identification empirical system identification methods appear to be more method, but it is often burdened by various other problems, useful. Under the assumption that a rigid body dynamics such as significant noise in all measured or inferred variables (RBD) model is sufficient to capture the entire robot dynamics, of the robot. The danger of physically inconsistent results also exists due to unmodeled nonlinearities or insufficiently rich data. this problem is theoretically straightforward as all unknown In this paper, we address all these problems by developing a parameters of the robot such as mass, center of mass and Bayesian parameter identification method that can automatically inertial parameters appear linearly in the rigid body dynamics detect noise in both input and output data for the regression equations [3].
    [Show full text]
  • Orthogonal Decomposition of Left Ventricular Remodeling in Myocardial Infarction
    Touro Scholar Office of the esidentPr Publications and Research Office of the esidentPr 2017 Orthogonal Decomposition of Left Ventricular Remodeling in Myocardial Infarction Xingyu Zhang Pau Medrano-Gracia Bharath Ambale-Venkatesh David A. Bluemke Brett R. Cowan See next page for additional authors Follow this and additional works at: https://touroscholar.touro.edu/president_pubs Part of the Cardiovascular Diseases Commons, and the Disease Modeling Commons Recommended Citation Zhang, X., Medrano-Gracia, P., Ambale-Venkatesh, B., Bluemke, D. A., Cowan, B. R., Finn, J. P., . Kadish, A. H. (2017). Orthogonal decomposition of left ventricular remodeling in myocardial infarction. GigaScience, 6(3), 1-15. This Article is brought to you for free and open access by the Office of the President at Touro Scholar. It has been accepted for inclusion in Office of the President Publications and Research by an authorized administrator of Touro Scholar. For more information, please contact [email protected]. Authors Xingyu Zhang, Pau Medrano-Gracia, Bharath Ambale-Venkatesh, David A. Bluemke, Brett R. Cowan, J. Paul Finn, and Alan H. Kadish This article is available at Touro Scholar: https://touroscholar.touro.edu/president_pubs/162 GigaScience, 6, 2017, 1–15 doi: 10.1093/gigascience/gix005 Advance Access Publication Date: 6 February 2017 Research RESEARCH Orthogonal decomposition of left ventricular remodeling in myocardial infarction Xingyu Zhang1, Pau Medrano-Gracia1, Bharath Ambale-Venkatesh2, David A. Bluemke3,BrettRCowan1,J.PaulFinn4, Alan
    [Show full text]
  • A Gauss--Newton Iteration for Total Least Squares Problems
    A GAUSS–NEWTON ITERATION FOR TOTAL LEAST SQUARES PROBLEMS ∗ DARIO FASINO† AND ANTONIO FAZZI‡ Abstract. The Total Least Squares solution of an overdetermined, approximate linear equation Ax ≈ b minimizes a nonlinear function which characterizes the backward error. We show that a globally convergent variant of the Gauss–Newton iteration can be tailored to compute that solution. At each iteration, the proposed method requires the solution of an ordinary least squares problem where the matrix A is perturbed by a rank-one term. Key words. Total Least Squares, Gauss–Newton method AMS subject classifications. 65F20 1. Introduction. The Total Least Squares (TLS) problem is a well known tech- nique for solving overdetermined linear systems of equations Ax b, A Rm×n, b Rm (m>n), ≈ ∈ ∈ in which both the matrix A and the right hand side b are affected by errors. We consider the following classical definition of TLS problem, see e.g., [4, 13]. Definition 1.1 (TLS problem). The Total Least Squares problem with data A Rm×n and b Rm, with m n, is ∈ ∈ ≥ min (E f) F, subject to b + f Im(A + E), (1.1) E,f k | k ∈ where E Rm×n and f Rm. Given a matrix (E¯ f¯) that attains the minimum in (1.1), any∈x Rn such that∈ | ∈ (A + E¯)x = b + f¯ is called a solution of the Total Least Squares problem (1.1). Here and in what follows, F denotes the Frobenius matrix norm, while (E f) denotes the m (n +1) matrixk·k whose first n columns are the ones of E, and the last| column is the vector× f.
    [Show full text]
  • Appendix D: Nonlinear Least Squares for Data Fitting
    Appendix D Nonlinear Least Squares Data Fitting D.1 Introduction A nonlinear least squares problem is an unconstrained minimization problem of the form m 2 minimize f(x)= fi(x) , x i=1 where the objective function is defined in terms of auxiliary functions { fi }.It is called “least squares” because we are minimizing the sum of squares of these functions. Looked at in this way, it is just another example of unconstrained min- imization, leading one to ask why it should be studied as a separate topic. There are several reasons. In the context of data fitting, the auxiliary functions { fi } are not arbitrary nonlinear functions. They correspond to the residuals in a data fitting problem (see m Chapter 1). For example, suppose that we had collected data { (ti,yi) }i=1 consist- ing of the size of a population of antelope at various times. Here ti corresponds to the time at which the population yi was counted. Suppose we had the data ti :12458 yi :3461120 where the times are measured in years and the populations are measured in hun- dreds. It is common to model populations using exponential models, and so we might hope that x2ti yi ≈ x1e for appropriate choices of the parameters x1 and x2. A model of this type is illus- trated in Figure D.1. If least squares were used to select the parameters (see Section 1.5) then we would solve 5 x2ti 2 minimize f(x1,x2)= (x1e − yi) x1,x2 i=1 743 744 Appendix D. Nonlinear Least Squares Data Fitting 20 y 18 16 14 12 10 8 6 4 2 0 0 1 2 3 4 5 6 7 8 9 10 t Figure D.1.
    [Show full text]
  • A Fast Algorithm for Solving Regularized Total Least Squares Problems∗
    Electronic Transactions on Numerical Analysis. ETNA Volume 31, pp. 12-24, 2008. Kent State University Copyright 2008, Kent State University. [email protected] ISSN 1068-9613. A FAST ALGORITHM FOR SOLVING REGULARIZED TOTAL LEAST SQUARES PROBLEMS∗ JORG¨ LAMPE† AND HEINRICH VOSS† Abstract. The total least squares (TLS) method is a successful approach for linear problems if both the system matrix and the right hand side are contaminated by some noise. For ill-posed TLS problems Renaut and Guo [SIAM J. Matrix Anal. Appl., 26 (2005), pp. 457–476] suggested an iterative method based on a sequence of linear eigenvalue problems. Here we analyze this method carefully, and we accelerate it substantially by solving the linear eigenproblems by the Nonlinear Arnoldi method (which reuses information from the previous iteration step considerably) and by a modified root finding method based on rational interpolation. Key words. Total least squares, regularization, ill-posedness, Nonlinear Arnoldi method. AMS subject classifications. 15A18, 65F15, 65F20, 65F22. 1. Introduction. Many problems in data estimation are governed by overdetermined linear systems Ax b, A Rm×n, b Rm,m n, (1.1) ≈ ∈ ∈ ≥ where both the matrix A and the right hand side b are contaminated by some noise. An appropriate approach to this problem is the total least squares (TLS) method which determines perturbations ∆A Rm×n to the coefficient matrix and ∆b Rm to the vector b such that ∈ ∈ [∆A, ∆b] 2 = min! subject to (A + ∆A)x = b + ∆b, (1.2) k kF where F denotes the Frobenius norm of a matrix; see, e.g., [7, 18].
    [Show full text]