V. Linear & Nonlinear Least-Squares V. Linear & Nonlinear Least-Squares V.0.Examples, linear/nonlinear least-squares V.1.Linear least-squares V.1.1.Normal equations V.1.2.The orthogonal transformation method V.2.Nonlinear least-squares V.2.1.Newton method V.2.2.Gauss-Newton method 1 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of a given function (from natural laws or model assumptions) through a series of measurements. Usually the number of measurements m is much bigger than the number of parameters n, i.e. Measurement Time Quantity 1 Model: Goal: Find 2 2 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares In practice, one has often to determine unknown parameters of a given function (from natural laws or model assumptions) through a series of measurements. Usually the number of measurements m is much bigger than the number of parameters n, i.e. Measurement Time Quantity 1 Model: Goal: Find 2 3 Problem: more equations than unknowns! … overdetermined V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares Data: Model: 1 Goal: Find Linear in model parameters! 4 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares Data: Model: 1 Goal: Find Linear in model parameters! JUST 5 NOTATION! V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares ● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best. 6 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares ● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best. Model: 1 7 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares ● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best. MINIMIZE! 8 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares ● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best. ● Least squares solution: Euclidean distance, Domain of valid parameters a.k.a. 2-norm (from application!) 9 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares Data: Model: 2 Goal: Find Nonlinear in model parameters! JUST 10 NOTATION! V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares ● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best. 11 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares ● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best. MINIMIZE! 12 V. Linear&NonlinearLeast-Squares V. In parameters! In Nonlinear Linear ● General problem: V.0. Examples, V.0. linear measurements /nonlinear least-squares Overdetermined! . usually.. parameters 13 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares ● Least squares solution: Linear Nonlinear ● Define scalar-valued function Linear Nonlinear ● Least squares solution: 14 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares ● Quiz: linear or nonlinear model? 1) Model parameters: 2) Model parameters: 15 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares ● Quiz: linear or nonlinear model? 1) Model parameters: 2) Model parameters: 16 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares ● General problem: measurements parameters Overdetermined! Assumption: has full column rank (linearly indep.) Least squares solution: 17 V. Linear & Nonlinear Least-Squares V.1.1. Normal equations ● Least squares solution: ● Rewrite: Transpose! ~ ~ Scalar! 18 V. Linear & Nonlinear Least-Squares V.1.1. Normal equations ● Gradient of must vanish at extremum (min/max): Gradient (Gauss') Normal equations Nothing difficult… ~ 19 V. Linear & Nonlinear Least-Squares V.1.1. Normal equations ● Necessary condition: Not sufficient! (Gauss') Normal equations ● Is it a minimum? We have to make sure that the matrix is positive definite 1 Norm (length!) ~ 2 Unless Because of our assumption of rank n 20 Columns linearly independent! V. Linear & Nonlinear Least-Squares V.1.1. Normal equations ● Geometric interpretation of the normal equations . ● It turns out that they lead to a worser conditioned problem, i.e. difficult to solve the normal equations numerically… Therefore the next method is preferred 21 V. Linear & Nonlinear Least-Squares V.1.2. The orthogonal decomposition method ● Definition: An orthogonal matrix is a real square matrix whose columns and rows are orthogonal unit vectors This means: ● Key property: Orthogonal matrices leave the Euclidean length invariant 22 V. Linear & Nonlinear Least-Squares V.1.2. The orthogonal decomposition method Fact: Every matrix with full column rank (the columns are linearly independent) has a so-called QR-decomposition where is an orthogonal matrix and an upper triangular matrix Matlab: [Q,R]=qr(A) !23 Non zero diagonal! V. Linear & Nonlinear Least-Squares V.1.2. The orthogonal decomposition method Fact: Every matrix with full column rank (the columns are linearly independent) has a so-called QR-decomposition where is an orthogonal matrix and an upper triangular matrix Upper triangular Matlab: [Q,R]=qr(A) 24 V. Linear & Nonlinear Least-Squares V.1.2. The orthogonal decomposition method Minimize residuum: Orthogonal! We still solve the same minimization problem!!! 25 V. Linear & Nonlinear Least-Squares V.1.2. The orthogonal decomposition method Minimize residuum: Orthogonal! We still solve the same minimization problem!!! 26 V. Linear & Nonlinear Least-Squares V.1. Example: linear least-squares Data: Model: 1 Goal: Find Linear in model parameters! 27 V. Linear & Nonlinear Least-Squares V.1. Example: linear least-squares Data: Model: 1 Goal: Find Normal equations: Orthogonal transformation: analogue… 28 V. Linear & Nonlinear Least-Squares V.2. Nonlinear least-squares ● General problem: measurements parameters Overdetermined! usually.. Least squares solution: 29 V. Linear & Nonlinear Least-Squares V.2.1 Newton method ● Gradient of must vanish at extremum (min/max): Gradient equations ! unknowns 30 V. Linear & Nonlinear Least-Squares V.2.1 Newton method Apply Newton's method to solve Gradient NOT Hessian 31 V. Linear & Nonlinear Least-Squares V.2.2 Gauss-Newton method Linearize the residuum/error equations Linearize at some Linear least squares problem! 32 V. Linear & Nonlinear Least-Squares V.2.2 Gauss-Newton method Problem: Given an initial guess: Solve a sequence of linear least squares problems . Until convergence Small enough33 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares Data: Model: 2 Goal: Find Nonlinear in model parameters! 34 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares Data: Model: 2 Goal: Find Newton method: 35 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares Data: Model: 2 Goal: Find Newton method: Gradient Two equations in two unknowns! 36 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares Data: Model: 2 Goal: Find Newton method: Hessian 37 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares Data: Model: 2 Goal: Find Newton method: Initial guess: Iterate! 38 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares Data: Model: 2 Goal: Find Gauss-Newton method: Linearize residuum/error equations: Linear in parameters now! 39 V. Linear & Nonlinear Least-Squares V. Summary ● Linear/Nonlinear in parameters ● Overdetermined system of equations! Determine parameters in the least-squares sense... ● Linear: – Normal equations – Orthogonal transformation method – Example ● Nonlinear: – Newton method – Gauss-Newton method – Example 40.