Numerical Analysis in Nonlinear Least Squares Methods and Applications
Total Page:16
File Type:pdf, Size:1020Kb
Department of Electrical and Computer Engineering Numerical Analysis in Nonlinear Least Squares Methods and Applications Christina Nguk Ling Eu This thesis is presented for the Degree of Master of Philosophy (Electrical and Computer Engineering) of Curtin University December 2017 DECLARATION To the best of my knowledge and belief this thesis contains no material previously published by any other person except where due acknowledgment has been made. This thesis contains no material which has been accepted for the award of any other degree or diploma in any university. i ACKNOWLEDGEMENTS I would like to express my special thanks to my main supervisor, Professor Bean San Goh, for his quality guidance, continual encouragement and endless support through- out my MPhil study. During my research, he has provided mewith invaluable academic advice and insightful suggestions and steered me into the right direction whenever I am heading towards the wrong path in my research. The door to his office was always open whenever I ran into a trouble spot or had a problem on my research or writing. He is not only a good supervisor but as a good friend as well. I would also like to thank my co-supervisor, Dr. Hendra Gunawan Harno, for his continual patience, guidance and support throughout my research. He has always been helpful and supportive whenever I faced any academic and administrative issues. He always tried his best to help me by providing any possible assistance. My sincere thanks also goes to my second co-supervisor, Dr. Garenth King Hann Lim, for his support and cooperation throughout my years of study. Without their passionate input, this research project could not have been successfully conducted. I would also like to express my appreciation to the chairperson of my thesis com- mittee, Associate Professor Zhuquan Zang, for his encouragement and continual sup- port throughout these years. He has really done a great job in ensuring that my studies were carried out well. I would also like to acknowledge Professor Clem Kuek, the Dean of Research and Development, for his continual support in improving the quality of my research. He ii has always tried to organize research related seminars and talks that are beneficial to improve the quality of the research output. It is also a pleasure to thank my colleagues for their friendly support and friend- ship that blossomed throughout the duration of my studies. They have enriched and broadened my knowledge and perspectives on various aspects of life. I must express my very profound gratitude to my parents, especially my mother, for providing me with unfailing support and continuous encouragement throughout my years of study and through the process of researching and writing this thesis. This accomplishment would not have been possible without them. Last, but not least, I appreciate with gratitude the fee waiver award I received from Faculty of Engineering and Science for granting me an opportunity to conduct MPhil study in Electrical and Computer Engineering, Curtin University Malaysia. Thank you. iii ABSTRACT A nonlinear least squares (NLS) problem commonly arises in nonlinear data-fitting when a nonlinear mathematical model with n unknown parameters is used to fit a set of m observed data with m>n. The best fit to the m observed data is achieved when the residuals between the observed data and its corresponding fitted modeled data are minimized. This is made possible by minimizing an objective function formulated as the sum of squares residual functions of all the m observed data. This procedure is also known as parameter estimation in NLS data-fitting. As a result, the NLS prob- lem is a special class of unconstrained minimization problem and the solution of this minimization problem yields the minimum point which gives the minimal value of the objective function of the NLS problem. Various numerical methods have been developed to solve the NLS problem as un- constrained optimization. These methods can be classified into line search methods or trust region methods. In this thesis, four of the most well-known numerical methods in the NLS literature are considered. The line search methods considered are the steepest descent (SD) method, the Newton’s method and the Gauss-Newton (GN) method while the only trust region method considered is the Levenberg-Marquardt (LM) method. In order to avoid expensive computations of the Hessian matrix in each iteration, the GN and the LM methods use, without justification, a truncated Hessian matrix of the objective function of the NLS problem. However, this truncated Hessian matrix may not be valid especially when the iterations result in large residuals. In addition, iv the computation of the derivatives of the objective function may be prone to analyti- cal mistakes. This is especially true when computing derivatives for high-dimensional NLS problem. To address these issues, numerical differentiation, which uses finite difference approximations, is incorporated into numerical algorithms so that numeri- cal derivatives can be performed by just providing the original objective function of the NLS problem. This saves time and effort while preventing analytical mistakes. Thus, the use of the truncated Hessian matrix can be avoided when developing new numerical methods for solving the NLS problem. With the incorporation of numerical differentiation, all the numerical methods can be implemented in practical problems. The convergence analyses of the numerical methods follow from the Lyapunov function theorem where a sufficient decrease in the objective function is required at every iteration. The Lyapunov function theorem provides a feedback-type analysis which is robust against small numerical errors in the current iteration. If the level sets of the objective function are properly nested, all trajectories will converge to a mini- mum point x provided that the iterations stay within the properly nested region con- taining x. In order to implement the Lyapunov function theorem, all the line search numerical methods perform a backtracking line search so as to ensure a sufficient de- crease of the objective function value of the NLS problem at every iteration. On the other hand, this sufficient decrease requirement of the Lyapunov function theorem is also ensured implicitly in the trust region LM method through the ratio test. When using the MATLAB software to plot the level sets of a two-variable objective function, the level curves near stationary points may not appear in the plot. Hence, a stiff ordinary differential equation (ODE) method, which gives great control to the user, is used as a technique to plot a missing level curve around the stationary points of the objective function through a specific point. This is particularly useful when the objective function has multiple stationary points that are close to each other. The approximate greatest descent (AGD) method has been developed to solve an unconstrained optimization problem. Unlike other methods, the AGD method uses the actual objective function to construct its iterations instead of an approximate linear or v quadratic model. Furthermore, the AGD method is constructed for long-term subopti- mal outcomes to generate the next iterative point on the boundary of the current search region. However, it has not been applied to solve the NLS problem. In addition, a two-phase AGD method (abbreviated as AGDN) is also proposed as a new numerical approach to solve the NLS problem. It consists of two explicitly defined phases with AGD method in Phase-I where the current iterations are far away from the minimum point and then switches to the Newton’s method in Phase-II when the gradient is suf- ficiently small (i.e. near the minimum point). This method is motivated by the fast quadratic convergence rate of the Newton’s method near the minimum point. In order to demonstrate the efficiency, reliability and robustness of all the numerical methods, a standard set of two-variable and multi-variable test problems are selected from Moré et al. (1981) and Adorio (2005) and available in the constrained and un- constrained testing environment, revisited/safe threads (CUTEr/CUTEst) are used to perform numerical experiments. Furthermore, a performance profile is also used as a tool to provide an overall comparison of all the numerical methods in terms of num- ber of iterations and the CPU time used to achieve convergence. When the numerical methods are applied to solve the two-variable and the multi-variable test problems, the numerical results indicate that both the AGD and the AGDN methods have shown en- couraging results in terms of number of iterations and convergence rates as compared to the other methods. For the two-variable test problems, the AGD and the AGDN methods show similar results. However, the outcomes of these methods may differ when they are applied to solve the multi-variable test problems. The results prove that the AGDN method outperforms the AGD method since it has a faster convergence rate with less number of iterations. Nonetheless, the AGDN method may fail to converge if the Hessian matrix is singular near the minimum point. In this case, the AGD method should be used instead. vi ABBREVIATIONS Throughout the report, we use the following abbreviations: NLS: Nonlinear least squares SD: Steepest descent GN: Gauss-Newton LM: Levenberg-Marquardt AGD: Approximate greatest descent AGDN: Two-phase approximate greatest descent TP: Test Problem w.r.t: with respect to vii NOTATIONS Throughout the report, we use the following