SECTION 7: CURVE FITTING
MAE 4020/5020 –Numerical Methods with MATLAB 2 Introduction
K. Webb MAE 4020/5020 Curve Fitting
3
Often have data, , that is a function of some independent variable, , but the underlying relationship is unknown Know ’s and ’s (perhaps only approximately), but don’t know Measured data Tabulated data Determine a function (i.e., a curve) that “best” describes relationship between and An approximation to (the unknown) This is curve fitting
K. Webb MAE 4020/5020 Regression vs. Interpolation
4 We’ll look at two categories of curve fitting:
Least‐squares regression Noisy data – uncertainty in value for a given value Want “good” agreement between and data points Curve (i.e., ) may not pass through any data points
Polynomial interpolation Data points are known exactly – noiseless data Resulting curve passes through all data points
K. Webb MAE 4020/5020 5 Review of Basic Statistics
Before moving on to discuss least‐squares regression, we’ll first review a few basic concepts from statistics.
K. Webb MAE 4020/5020 Basic Statistical Quantities
6
Arithmetic mean –the average or expected value ∑
Standard deviation (unbiased) –a measure of the spread of the data about the mean
1
where is the total sum of the squares of the residuals
K. Webb MAE 4020/5020 Basic Statistical Quantities
7
Variance –another measure of spread The square of the standard deviation Useful measure due to relationship with power and power spectral density of a signal or data set
or
K. Webb MAE 4020/5020 Normal (Gaussian) Distribution
8
Many naturally‐occurring random process are normally‐distributed Measurement noise Very often assume noise in our data is Gaussian Probability density function (pdf):
where is the variance, and is the mean of the random variable,
K. Webb MAE 4020/5020 Statistics in MATLAB
9
Generation of random values Very useful for simulations including noise Uniformly‐distributed: rand(m,n) E.g., generate an m n matrix of uniformly‐distributed points between xmin and xmax: x = xmin + (xmax – xmin)*rand(m,n)
Normally‐distributed: randn(m,n) E.g., generate an m n matrix of normally‐distributed points with mean of mu and standard deviation of sigma: x = mu + sigma*randn(m,n)
K. Webb MAE 4020/5020 Statistics in MATLAB
10
Histogram plots: hist(x,binx,Nbins) Graphical depiction of the variation of random quantities Plots the frequency of occurrence of ranges (bins) of values Provides insight into the nature of the distribution E.g., generate a histogram of the values in the vector x with 20 bins: hist(x,20)
Optional input argument, binx, specifies x values at the centers of the bins Built‐in statistical functions: min.m, max.m, mean.m, var.m, std.m, median.m, mode.m
K. Webb MAE 4020/5020 Statistics in MATLAB
11
K. Webb MAE 4020/5020 12 Linear Least Squares Regression
K. Webb MAE 4020/5020 Linear Regression
13
Noisy data, ,values at known values Suspect relationship between and is linear i.e., assume Determine and that define the “best‐ How do we define the fit” line for the data “best fit”?
K. Webb MAE 4020/5020 Measured Data
14
Assumed a linear relationship between and :
Due to noise, can’t measure exactly at each Can only approximate values
Measured values are approximations True value of plus some random error or residual
K. Webb MAE 4020/5020 Best Fit Criteria
15
Noisy data do not all line on a single line – discrepancy between each point and the line fit to the data The error, or residual:
Minimize some measure of this residual: Minimize the sum of the residuals Positive and negative errors can cancel Non‐unique fit Minimize the sum of the absolute values of the residuals Effect of sign of error eliminated, but still not a unique fit Minimize the maximum error – minimax criterion Excessive influence given to single outlying points
K. Webb MAE 4020/5020 Least‐Squares Criterion
16
Better fitting criterion is to minimize the sum of the squares of the residuals
Yields a unique best‐fit line for a given set of data The sum of the squares of the residuals is a function of the two fitting parameters, and , Minimize by setting its partial derivatives to zero and solving for and
K. Webb MAE 4020/5020 Least‐Squares Criterion
17
At its minimum point, partial derivatives of with respect to and will be zero
2 0
2 0
Breaking up the summation:
0
0
K. Webb MAE 4020/5020 Normal Equations
18
and form a system of two equations with two unknowns, and
1
2
In matrix form:
3
These are the normal equations
K. Webb MAE 4020/5020 Normal Equations
19
Normal equations can be solved for and :
∑ ∑ ∑ ∑ ∑ ∑ ∑ ̅
Or solve the matrix form of the normal equations, (3), in MATLAB using mldivide.m (\)
\
K. Webb MAE 4020/5020 Linear Least‐Squares ‐ Example
20
Noisy data with suspected linear relationship Calculate summation terms in the normal equations: , , ,
K. Webb MAE 4020/5020 Linear Least‐Squares ‐ Example
21
Assemble normal equation matrices Solve normal equations for vector of coefficients, , using mldivide.m
K. Webb MAE 4020/5020 Goodness of Fit
22
How well does a function fit the data? Is a linear fit best? A quadratic, higher‐order polynomial, or other non‐linear function? Want a way to be able to quantify goodness of fit
Quantify spread of data about the mean prior to regression:
Following regression, quantify spread of data about the regression line (or curve):
K. Webb MAE 4020/5020 Goodness of Fit
23
quantifies the spread of the data about the mean quantifies spread about the best‐fit line (curve) The spread that remains after the trend is explained The unexplained sum of the squares
represents the reduction in data spread after regression explains the underlying trend
Normalize to ‐ the coefficient of determination
K. Webb MAE 4020/5020 Coefficient of Determination
24
For a perfect fit: No variation in data about the regression line 0 → 1
If the fit provides no improvement over simply characterizing data by its mean value: → 0
If the fit is worse at explaining the data than their mean value: → 0
K. Webb MAE 4020/5020 Coefficient of Determination
25
Calculate for previous example:
K. Webb MAE 4020/5020 Coefficient of Determination
26 Don’t rely too heavily on the value of Anscombe’s famous data sets:
Chapra
Same line fit to all four data sets in each case
K. Webb MAE 4020/5020 27 Linearization of Nonlinear Relationships
K. Webb MAE 4020/5020 Nonlinear functions
28
Not all data can be explained by a linear relationship to an independent variable, e.g.
Exponential model
Power equation
Saturation‐growth‐rate equation
K. Webb MAE 4020/5020 Nonlinear functions
29 Methods for nonlinear curve fitting:
Linearization of the nonlinear relationship Transform the dependent and/or independent data values Apply linear least‐squares regression Inverse transform the determined coefficients back to those that define the nonlinear functional relationship
Nonlinear regression Treat as an optimization problem –more later…
K. Webb MAE 4020/5020 Linearizing an Exponential Relationship
30
Have noisy data that is believed to be best described by an exponential relationship
Linearize the fitting equation:
or
where
,
K. Webb MAE 4020/5020 Linearizing an Exponential Relationship
31
Fit a line to the transformed data using linear least‐ squares regression
Determine and :
Can calculate for the line fit to the transformed data Note that original data must be positive
K. Webb MAE 4020/5020 Linearizing an Exponential Relationship
32
Transform the linear fitting parameters, and , back to the parameters defining the exponential relationship
Exponential fit:
where , Note that is different than that for the line fit to the transformed data
K. Webb MAE 4020/5020 Linearizing an Exponential Relationship
33
K. Webb MAE 4020/5020 Linearizing a Power Equation
34
Have noisy data that is believed to be best described by an power equation
Linearize the fitting equation: log log log or
log log where
log ,
K. Webb MAE 4020/5020 Linearizing a Power Equation
35
Fit a line to the transformed data using linear least‐ squares regression
Determine and :
Can calculate for the line fit to the transformed data Note that original data – both and –must be positive
K. Webb MAE 4020/5020 Linearizing a Power Equation
36
Transform the linear fitting parameters, and , back to the parameters defining the power equation
Power equation: where , Note that is different than that for the line fit to the transformed data
K. Webb MAE 4020/5020 Linearizing a Power Equation
37
K. Webb MAE 4020/5020 Linearizing a Saturation Growth‐Rate Equation
38
Have noisy data that is believed to be best described by a saturation growth‐rate equation
Linearize the fitting equation: 1 1 1 or 1 1 where
,
K. Webb MAE 4020/5020 Linearizing a Saturation Growth‐Rate Equation
39
Fit a line to the transformed data using linear least‐ squares regression
Determine and :
Can calculate for the line fit to the transformed data
K. Webb MAE 4020/5020 Linearizing a Saturation Growth‐Rate Equation
40
Transform the linear fitting parameters, and , back to the parameters defining the saturation growth‐rate equation
Saturation growth‐rate equation: where , Note that is different than that for the line fit to the transformed data
K. Webb MAE 4020/5020 Linearizing a Saturation Growth‐Rate Equation
41
K. Webb MAE 4020/5020 42 Polynomial Regression
K. Webb MAE 4020/5020 Polynomial Regression
43
So far we’ve looked at fitting straight lines to linear and linearized data sets Can also fit mth‐order polynomials directly to data using polynomial regression
Same fitting criterion as linear regression: Minimize the sum of the squares of the residuals m+1 fitting parameters for an mth‐order polynomial m+1 normal equations
K. Webb MAE 4020/5020 Polynomial Regression
44
Assume, for example, that we have data we believe to be quadratic in nature 2nd‐order polynomial regression Fitting equation:
Best fit will minimize the sum of the squares of the residuals:
K. Webb MAE 4020/5020 Polynomial Regression –Normal Equations
45
Best‐fit polynomial coefficients will minimize Differentiate w.r.t. each coefficient and set to zero
2 0
2 0
2 0
K. Webb MAE 4020/5020 Polynomial Regression –Normal Equations
46
Rearranging the normal equations yields Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ
Which can be put into matrix form: Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ
This system of equations can be solved for the vector of unknown coefficients using MATLAB’s mldivide.m (\)
K. Webb MAE 4020/5020 Polynomial Regression –Normal Equations
47
For mth‐order polynomial regression the normal equations are: Σ ⋯Σ Σ Σ Σ ⋯Σ Σ ⋮⋮⋱⋮⋮ ⋮ Σ Σ ⋯Σ Σ
Again, this system of 1equations can be solved for the vector of 1unknown polynomial coefficients using MATLAB’s mldivide.m (\)
K. Webb MAE 4020/5020 Polynomial Regression –Example
48
K. Webb MAE 4020/5020 Polynomial Regression – polyfit.m
49 p = polyfit(x,y,m)
x: ‐vector of independent variable data values y: ‐vector of dependent variable data values m: order of the polynomial to be fit to the data p: 1 ‐vector of best‐fit polynomial coefficients
Least‐squares polynomial regression if: 1 i.e., for over‐determined systems Polynomial interpolation if: 1 Resulting fit passes through all (x,y) points –more later
K. Webb MAE 4020/5020 Polynomial Regression – polyfit.m
50
Note that the result matches that obtained by solving normal equations
K. Webb MAE 4020/5020 Polynomial Regression Using polyfit.m
51
Determine the 4th‐order polynomial with roots at
Generate noiseless data points by evaluating this polynomial at integer values of from to
Add Gaussian white noise with a standard deviation of to your data points
Use polyfit.m to fit a 4th‐order polynomial to the
Exercise noisy data Calculate the coefficient of determination,
Plot the noisy data points, along with the best‐fit polynomial
K. Webb MAE 4020/5020 52 Multiple Linear Regression
K. Webb MAE 4020/5020 Multiple Linear Regression
53
We have so far fit lines or curves to data described by functions of a single variable For functions of multiple variables, fit planes or surfaces to data Linear function of two independent variables: multiple linear regression
Sum of the squares of the residuals is now
, ,
K. Webb MAE 4020/5020 Multiple Linear Regression – Normal Equations
54
Differentiate w.r.t. fitting coefficients and equate to zero The normal equations:
, , , , , , , , , , , ,
Solve as before –now fitting coefficients, , define a plane
K. Webb MAE 4020/5020 55 General Linear Least Squares Regression
K. Webb MAE 4020/5020 General Linear Least Squares
56
We’ve seen three types of least‐squares regression Linear regression Polynomial regression Multiple linear regression All are special cases of general linear least‐squares regression
The ’s are basis functions Basis functions may be nonlinear This is linear regression, because dependence on fitting coefficients, , is linear
K. Webb MAE 4020/5020 General Linear Least Squares
57
For linear regression – simple or multiple:
, , , …
For polynomial regression: , , , …
In all cases, this is a linear combination of basis function, which may, themselves, be nonlinear
K. Webb MAE 4020/5020 General Linear Least Squares
58
The general linear least‐squares model:
⋯
Can be expressed in matrix form: where is an 1 matrix, the design matrix, whose entries are the 1 basis functions evaluated at the independent variable values corresponding to the measurements:
⋯ ⋯ ⋮⋮⋱⋮ ⋯
where is the basis function evaluated at the independent variable value. (Note: is not the row index and is not the column index, here.)
K. Webb MAE 4020/5020 General Linear Least Squares
59
The least‐squares model is:
More measurements than coefficients is not square – tall and narrow Over‐determined system does not exist
K. Webb MAE 4020/5020 General Linear Least Squares –Design Matrix Example
60
For example, consider fitting a quadratic to five measured values, , at Model is: Basis functions are , , and Least‐squares equation is
K. Webb MAE 4020/5020 General Linear Least Squares – Residuals
61
Linear least‐squares model is:
(1)
Residual: (2)
Sum of the squares or the residuals: (3)
Expanding, (4)
K. Webb MAE 4020/5020 Deriving the Normal Equations
62
Best fit will minimize the sum of the squares of the residuals Differentiate with respect to the coefficient vector, , and set to zero (5) We’ll need to use some matrix calculus identities:
(6)
K. Webb MAE 4020/5020 Deriving the Normal Equations
63
Using the matrix derivative relationships, (6), (7)
Equation (7) is the matrix form of the normal equations: (8) Solution to (8) is the vector of least‐squares fitting coefficients: (9)
K. Webb MAE 4020/5020 Solving the Normal Equations
64 (9)
Remember, our starting point was the linear least‐ squares model: (10)
Couldn’t we have solved (10) for fitting coefficients as
(11)
No, must solve using (9), because: Don’t have , only noisy approximations, We have an over‐determined system is not square does not exist
K. Webb MAE 4020/5020 Solving the Normal Equations
65
Solution to the linear least‐squares problem is:
(12) where (13) is the Moore‐Penrose pseudo‐inverse of
Use the pseudo‐inverse to find the least‐squares solutions to an over‐determined system
K. Webb MAE 4020/5020 Coefficient of Determination
66
Goodness of fit characterized by the coefficient of determination:
where is given by (3) (14) and (15)
K. Webb MAE 4020/5020 General Least Squares in MATLAB
67
Have measurements ⋯
at known independent variable values ⋯
and a model, defined by 1basis functions
⋯
Generate design matrix by evaluating 1basis functions at all values of ⋯ ⋯ ⋮⋮⋱⋮ ⋯
K. Webb MAE 4020/5020 General Least Squares in MATLAB
68
Solve for vector of fitting coefficients as the solution to the normal equations
or by using mldivide.m (\)
Result is the same, though the methods are different mldivide.m uses QR factorization to find the solution to over‐determined systems
K. Webb MAE 4020/5020 69 Nonlinear Regression
K. Webb MAE 4020/5020 Nonlinear Regression – fminsearch.m
70
Nonlinear models: Have nonlinear dependence on fitting parameters E.g., Two options for fitting nonlinear models to data Linearize the model first, then use linear regression Fit a nonlinear model directly by treating as an optimization problem Want to minimize a cost function Cost function is the sum of the squares of the residuals
Find the minimum of –a multi‐dimensional optimization Use MATLAB’s fminsearch.m
K. Webb MAE 4020/5020 Nonlinear Regression – fminsearch.m
71
Have noisy data that is believed to be best described by an exponential relationship
Cost function:
Find and to minimize Use fminsearch.m
K. Webb MAE 4020/5020 Nonlinear Regression – fminsearch.m
72
K. Webb MAE 4020/5020 Nonlinear Regression – lsqcurvefit.m
73
An alternative to minimizing a cost function using fminsearch.m, if the optimization toolbox is available: a = lsqcurvefit(f,a0,x,y)
f: handle to the fitting function e.g., f = @(a,x) a(1)*exp(a(2)*x);
a0: initial guess for fitting parameters x: independent variable data y: dependent variable data a: best‐fit parameters
K. Webb MAE 4020/5020 Nonlinear Regression – lsqcurvefit.m
74
K. Webb MAE 4020/5020 75 Polynomial Interpolation
K. Webb MAE 4020/5020 Polynomial Interpolation
76
Sometimes we know both and values exactly Want a function that describes Allows for interpolation between know data points Fit an ‐order polynomial to data points
Polynomial will pass through all points We’ll look at polynomial interpolation using
Newton’s polynomial The Lagrange polynomial
K. Webb MAE 4020/5020 Polynomial Interpolation
77
Can approach similar to linear least‐squares regression where , , … For an ‐order polynomial, we have equations with unknowns
In matrix form
K. Webb MAE 4020/5020 Polynomial Interpolation
78
Now, unlike for linear regression All 1values in are known exactly 1equations with 1unknown coefficients is square 1 1 ⋯1 ⋯1 ⋮ ⋮⋮⋱⋮⋮ ⋯1
Could solve in MATLAB using mldivide.m \
is a Vandermonde matrix Tend to be ill‐conditioned The techniques that follow are more numerically robust
K. Webb MAE 4020/5020 79 Newton Interpolating Polynomial
K. Webb MAE 4020/5020 Linear Interpolation
80
Fit a line (1st‐order polynomial) to two data points using a truncated Taylor series (or simple trigonometry):
where is the function for the line fit to the data, and are the known data values
This is the Newton linear‐interpolation formula
K. Webb MAE 4020/5020 Quadratic Interpolation
81
To fit a 2nd‐order polynomial to three data points, consider the following form
Evaluate at to find
Back‐substitution and evaluation at and at will yield the other coefficients and
K. Webb MAE 4020/5020 Quadratic Interpolation
82
Can still view this as a Taylor series approximation
represents an offset is slope is curvature
Choice of initial quadratic form (Newton interpolating polynomial) was made to facilitate the development Resulting polynomial would be the same for any initial form of an ‐order polynomial Solution is unique
K. Webb MAE 4020/5020 ‐Order Newton Interpolating Polynomial
83 Extending the quadratic example to ‐order
⋯ ⋯
Solve for coefficients as before with back‐substitution and evaluation of
denotes a finite divided difference
K. Webb MAE 4020/5020 Finite Divided Differences
84
First finite divided difference
,
Second finite divided difference
, , , , finite divided difference
,…, ,…, , ,…, ,
Calculate recursively
K. Webb MAE 4020/5020 ‐Order Newton Interpolating Polynomial
85 ‐order Newton interpolating polynomial in terms of divided differences:
, ⋯
, ,…, , ⋯
Divided difference table for calculation of coefficients:
Chapra
K. Webb MAE 4020/5020 Newton Interpolating Polynomial – Example
86
K. Webb MAE 4020/5020 87 Lagrange Interpolating Polynomial
K. Webb MAE 4020/5020 Linear Lagrange Interpolation
88
Fit a first‐order polynomial (a line) to two known data points: and
∙ ∙
and are weighting functions, where
1, 0,
1, 0,
The interpolating polynomial is a weighted sum of the individual data point values
K. Webb MAE 4020/5020 Linear Lagrange Interpolation
89
For linear (1st‐order) interpolation, the weighting functions are:
The linear Lagrange interpolating polynomial is:
K. Webb MAE 4020/5020 ‐Order Lagrange Interpolation
90
Lagrange interpolation technique can be extended to ‐order polynomials
∙ where
K. Webb MAE 4020/5020 Lagrange Interpolating Polynomial – Example
91
K. Webb MAE 4020/5020 92 MATLAB’s Curve‐Fitting Tool
K. Webb MAE 4020/5020 MATLAB’s Curve‐Fitting Tool GUI
93
Type cftool at the command line Launches curve fitting tool GUI in a new window Import data from the workspace Quickly try fitting different types of functions to the data Polynomial Exponential Power Fourier Custom equations, and more…
Interpolation for Least‐squares regression for
K. Webb MAE 4020/5020 MATLAB’s Curve‐Fitting Tool GUI
94
K. Webb MAE 4020/5020 Curve Fitting with the cftool GUI
95
Go to the course website and download the following data file: cftool_ex_data.mat
Load data file into the workspace At the command line, type: load(‘cftool_ex_data’)
Launch the Curve Fitting Tool
Exercise Type cftool at the command line
Try fitting different functions to each of the three data sets Can open a new tab in the GUI for each fit
K. Webb MAE 4020/5020