V. Linear & Nonlinear Least-Squares

V. Linear & Nonlinear Least-Squares V. Linear & Nonlinear Least-Squares

V.0.Examples, linear/nonlinear least-squares V.1.Linear least-squares V.1.1.Normal equations V.1.2.The orthogonal transformation method V.2.Nonlinear least-squares V.2.1.Newton method V.2.2.Gauss-Newton method

1 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

In practice, one has often to determine unknown parameters of a given function (from natural laws or model assumptions) through a series of measurements. Usually the number of measurements m is much bigger than the number of parameters n, i.e.

Measurement

Time

Quantity 1 Model: Goal: Find 2 2 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

Measurement

Time

Quantity 1 Model: Goal: Find 2 3 Problem: more equations than unknowns! … overdetermined V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

Data:

Model: 1 Goal: Find Linear in model parameters!

4 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

Data:

Model: 1 Goal: Find Linear in model parameters!

JUST 5 NOTATION! V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best.

6 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best.

Model: 1 7 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best.

MINIMIZE!

8 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best.

● Least squares solution:

Euclidean distance, Domain of valid parameters a.k.a. 2-norm (from application!)

9 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

Data:

Model: 2 Goal: Find Nonlinear in model parameters!

JUST 10 NOTATION! V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best.

11 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

● Idea: choose the parameters such that the distance between the data and the curve is minimal, i.e. the curve that fits best.

MINIMIZE!

12 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

● General problem: measurements parameters Overdetermined! r a e n i L r a e n i l n o N

13 In parameters! usually.. . V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

● Least squares solution: Linear

Nonlinear

● Define scalar-valued function

Linear

Nonlinear

● Least squares solution: 14 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

● Quiz: linear or nonlinear model? 1) Model parameters:

2) Model parameters:

15 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

● Quiz: linear or nonlinear model? 1) Model parameters:

2) Model parameters:

16 V. Linear & Nonlinear Least-Squares V.0. Examples, linear/nonlinear least-squares

● General problem: measurements parameters Overdetermined!

Assumption: has full column rank (linearly indep.)

Least squares solution: 17 V. Linear & Nonlinear Least-Squares V.1.1. Normal equations

● Least squares solution:

● Rewrite: Transpose!

~ Scalar!

18 V. Linear & Nonlinear Least-Squares V.1.1. Normal equations

● Gradient of

must vanish at extremum (min/max):

Gradient

(Gauss') Normal equations

Nothing difficult…

~ 19 V. Linear & Nonlinear Least-Squares V.1.1. Normal equations

● Necessary condition: Not sufficient! (Gauss') Normal equations

● Is it a minimum? We have to make sure that the matrix is positive definite

Norm (length!)

~ 2

Unless Because of our assumption of rank n 20 Columns linearly independent! V. Linear & Nonlinear Least-Squares V.1.1. Normal equations

● Geometric interpretation of the normal equations

● It turns out that they lead to a worser conditioned problem, i.e. difficult to solve the normal equations numerically…

Therefore the next method is preferred 21 V. Linear & Nonlinear Least-Squares V.1.2. The orthogonal decomposition method

● Definition: An orthogonal matrix is a real square matrix whose columns and rows are orthogonal unit vectors

This means:

● Key property: Orthogonal matrices leave the Euclidean length invariant

22 V. Linear & Nonlinear Least-Squares V.1.2. The orthogonal decomposition method

Fact: Every matrix with full column rank (the columns are linearly independent) has a so-called QR-decomposition

where is an orthogonal matrix and an upper triangular matrix

Matlab: [Q,R]=qr(A) !23 Non zero diagonal! V. Linear & Nonlinear Least-Squares V.1.2. The orthogonal decomposition method

Fact: Every matrix with full column rank (the columns are linearly independent) has a so-called QR-decomposition

where is an orthogonal matrix and an upper triangular matrix

Upper triangular

Matlab: [Q,R]=qr(A) 24 V. Linear & Nonlinear Least-Squares V.1.2. The orthogonal decomposition method

Minimize residuum:

Orthogonal! We still solve the same minimization problem!!!

25 V. Linear & Nonlinear Least-Squares V.1.2. The orthogonal decomposition method

Minimize residuum:

Orthogonal! We still solve the same minimization problem!!!

26 V. Linear & Nonlinear Least-Squares V.1. Example: linear least-squares

Data:

Model: 1 Goal: Find Linear in model parameters!

27 V. Linear & Nonlinear Least-Squares V.1. Example: linear least-squares

Data:

Model: 1 Goal: Find

Normal equations:

Orthogonal transformation: analogue… 28 V. Linear & Nonlinear Least-Squares V.2. Nonlinear least-squares

● General problem: measurements parameters Overdetermined!

usually.. .

Least squares solution: 29 V. Linear & Nonlinear Least-Squares V.2.1 Newton method

● Gradient of

must vanish at extremum (min/max):

Gradient equations ! unknowns

30 V. Linear & Nonlinear Least-Squares V.2.1 Newton method Apply Newton's method to solve

Gradient

NOT

Hessian

31 V. Linear & Nonlinear Least-Squares V.2.2 Gauss-Newton method

Linearize the residuum/error equations

Linearize at some

Linear least squares problem!

32 V. Linear & Nonlinear Least-Squares V.2.2 Gauss-Newton method

Problem:

Given an initial guess:

Solve a sequence of linear least squares problems . . .

Until convergence

Small enough33 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares

Data:

Model: 2 Goal: Find Nonlinear in model parameters!

34 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares

Data:

Model: 2 Goal: Find Newton method:

35 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares

Data:

Model: 2 Goal: Find Newton method:

Gradient

Two equations in two unknowns! 36 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares

Data:

Model: 2 Goal: Find Newton method:

Hessian

37 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares

Data:

Model: 2 Goal: Find Newton method:

Initial guess:

Iterate!

38 V. Linear & Nonlinear Least-Squares V.2. Example: nonlinear least-squares

Data:

Model: 2 Goal: Find Gauss-Newton method:

Linearize residuum/error equations:

Linear in parameters now! 39 V. Linear & Nonlinear Least-Squares V. Summary

● Linear/Nonlinear in parameters

● Overdetermined system of equations! Determine parameters in the least-squares sense...

● Linear: – Normal equations – Orthogonal transformation method – Example

● Nonlinear: – Newton method – Gauss-Newton method – Example