ECE 3040 Lecture 18: Curve Fitting by Least-Squares-Error Regression © Prof. Mohamad Hassoun

This lecture covers the following topics:

 Introduction  Linear least-squares-Error (LSE) regression: The straight-line model  Linearization of nonlinear models  General linear LSE regression and the polynomial model  Polynomial regression with Matlab: polyfit  Non-linear LSE regression  Numerical solution of the non-linear LSE optimization problem: Gradient search and Matlab’s fminsearch function  Solution of differential equations based on LSE minimization  Appendix: Explicit matrix formulation for the quadratic regression problem

Introduction

In the previous lecture, polynomial and cubic spline interpolation methods were introduced for estimating a value between a given set of precise data points. The idea was to (interpolate) “fit” a function to the data points so as to perfectly pass through all data points. Many engineering and scientific observations are made by conducting experiments in which physical quantities are measured and recorded as inexact (noisy) data points. In this case, the objective would be to find the best-fit analytic curve (model) that approximates the underlying functional relationship present in the data set. Here, the best-fit curve is not required to pass through the data points, but it is required to capture the shape (general trend) of the data. This curve fitting problem is referred to as regression. The following sections present formulations for the regression problem and provide solutions.

The following figure compares two polynomials that attempt to fit the shown data points. The blue curve is the solution to the interpolation problem. The green curve is the solution (we seek) to the linear regression problem.

Linear Least-Squares-Error (LSE) Regression: The Straight-Line Model

The regression problem will first be illustrated for fitting the linear model (straight- line), 푦(푥) = 푎1푥 + 푎0, to a set of 푛 paired experimental observations: (푥1, 푦1), (푥2, 푦2), … , (푥푛, 푦푛). So, the idea here is to position the straight-line (i.e., to determine the regression coefficients 푎0 and 푎1) so that some error measure of fit is minimized. A common error measure is the sum-of-the-squares (SSE) of the residual errors 푒푖 = 푦푖 − 푦(푥푖), 푛 푛 푛 2 2 2 퐸(푎0, 푎1) = ∑ 푒푖 = ∑[푦푖 − 푦(푥푖)] = ∑[푦푖 − (푎1푥푖 + 푎0)] 푖=1 푖=1 푖=1

The residual error 푒푖 is the discrepancy between the measured value, 푦푖, and the approximate value 푦(푥푖) = 푎0 + 푎1푥푖, predicted by the straight-line regression model. The residual error for the 푖th data point is depicted in the following figure.

A solution can be obtained for the regression coefficients, {푎0, 푎1}, that minimizes 퐸(푎0, 푎1). This criterion, 퐸, which is called least-squares-error (LSE) criterion, has a number of advantages, including that it yields a unique line for a given data set. Differentiating 퐸(푎0, 푎1) with respect to each of the unknown regression model coefficients, and setting the result to zero lead to a system of two linear equations, 푛 휕 퐸(푎0, 푎1) = 2 ∑(푦푖 − 푎1푥푖 − 푎0)(−1) = 0 휕푎0 푖=1 푛 휕 퐸(푎0, 푎1) = 2 ∑(푦푖 − 푎1푥푖 − 푎0)(−푥푖) = 0 휕푎1 푖=1 After expanding the sums, we obtain 푛 푛 푛

− ∑ 푦푖 + ∑ 푎0 + ∑ 푎1푥푖 = 0 푖=1 푖=1 푖=1 푛 푛 푛 2 − ∑ 푥푖푦푖 + ∑ 푎0푥푖 + ∑ 푎1푥푖 = 0 푖=1 푖=1 푖=1

푛 Now, realizing that ∑푖=1 푎0 = 푛푎0, and that multiplicative quantities that do not depend on the summation index 푖 can be brought outside the summation (i.e., 푛 푛 ∑푖=1 푎푥푖 = 푎 ∑푖=1 푥푖), we may rewrite the above equations as

These are called the normal equations. We can solve for 푎1 using Cramer’s rule and for 푎0 by substitution (Your turn: Perform the algebra) to arrive at the following LSE solution:

푛 푛 푛 ∗ 푛 ∑푖=1 푥푖푦푖 − ∑푖=1 푥푖 ∑푖=1 푦푖 푎1 = 푛 2 푛 2 푛 ∑푖=1 푥푖 − (∑푖=1 푥푖) ∑푛 푦 ∑푛 푥 푎∗ = 푖=1 푖 − 푎∗ 푖=1 푖 0 푛 1 푛 ∗ ∗ Therefore, the regression model is the straight-line 푦(푥) = 푎1푥 + 푎0. ∗ ∗ The value 퐸(푎0, 푎1) represents the LSE value and will be referred to as 퐸퐿푆퐸 and expressed as 푛 ∗ ∗ 2 퐸퐿푆퐸 = ∑(푦푖 − 푎1푥푖 − 푎0) 푖=1

Any other straight-line will lead to an error 퐸(푎0, 푎1) > 퐸퐿푆퐸.

Let the value of the sum-of-the-square of the difference between the 푦푖 values and ∑푛 푦 their average value, 푦̅ = 푖=1 푖, be 푛 푛 2 퐸푀 = ∑(푦푖 − 푦̅) 푖=1

Then, the (positive) difference 퐸푀 − 퐸퐿푆퐸 represents the improvement (where the smaller 퐸퐿푆퐸 is, the better) due to describing the data in terms of a straight-line, rather than as an average value (a straight-line with zero slope and 푦-intercept equals to 푦̅). The coefficient of determination, 푟2, is defined as the relative error between 퐸푀 and 퐸퐿푆퐸, 퐸 − 퐸 퐸 푟2 = 푀 퐿푆퐸 = 1 − 퐿푆퐸 퐸푀 퐸푀

For perfect fit, where the regression line goes through all data points, 퐸퐿푆퐸 = 0 and 푟2 = 1, signifying that the line explains 100% of the variability in the data. On the 2 other hand for 퐸푀 = 퐸퐿푆퐸 we obtain 푟 = 0, and the fit represents no improvement over a simple average. A value of 푟2 between 0 and 1 represents the extent of improvement. So, 푟2 = 0.8 indicates that 80% of the original uncertainty has been ∗ ∗ explained by the linear model. Using the above expressions for 퐸퐿푆퐸, 퐸푀, 푎0 and 푎1 one may derive the following formula for the correlation coefficient, 푟, (your turn: Perform the algebra)

퐸 − 퐸 푛 ∑(푥 푦 ) − (∑ 푥 )(∑ 푦 ) 푟 = √ 푀 퐿푆퐸 = 푖 푖 푖 푖 퐸 푀 2 2 2 2 √푛 ∑ 푥푖 − (∑ 푥푖) √푛 ∑ 푦푖 − (∑ 푦푖) where all sums are performed from 1 to 푛. Example. Fit a straight-line to the data provided in the following table. Find 푟2. x 1 2 3 4 5 6 7 y 2.5 7 38 55 61 122 110

Solution. The following Matlab script computes the linear regression coefficients, ∗ ∗ 푎0 and 푎1, for a straight-line employing the LSE solution. x=[1 2 3 4 5 6 7]; y=[2.5 7 38 55 61 122 110]; n=length(x); a1=(n*sum(x.*y)-sum(x)*sum(y))/(n*sum(x.^2)-(sum(x)).^2) a0=sum(y)/n-a1*sum(x)/n

∗ ∗ The solution is 푎1 = 20.5536 and 푎0 = −25.7143. The following plot displays the data and the regression model, 푦(푥) = 20.5536푥 − 25.7143.

The following script computes the correlation coefficient, 푟. x=[1 2 3 4 5 6 7]; y=[2.5 7 38 55 61 122 110]; n=length(x); r=(n*sum(x.*y)-sum(x)*sum(y))/((sqrt(n*sum(x.^2)- ... (sum(x))^2))*(sqrt(n*sum(y.^2)-(sum(y))^2)))

The script returns 푟 = 0.9582 (so, 푟2 = 0.9181). These results indicate that about 92% of the variability in the data has been explained by the linear model.

A word of caution: Although the coefficient of determination provides a convenient measure of the quality of fit, you should be careful not to rely on it completely. For, it is possible to construct data sets that will similar 푟2 values, while the regression line is not well positioned for some sets. A good practice would be to visually inspect the plot of the data along with the regression curve. The following example illustrates these ideas.

Example. Anscombe's quartet comprises four datasets that have 푟2 ≅ 0.666, yet appear very different when graphed. Each dataset consists of eleven (푥푖, 푦푖) points. They were constructed in 1973 by the statistician Francis Anscombe to demonstrate both the importance of graphing data before analyzing it and the effect of outliers on statistical properties. Notice that if we are to ignore the outlier point in the third data set, then the regression line would be perfect, with 푟2 = 1.

Your turn: Employ linear regression to generate the above plots and determine 푟2 for each of the Anscombe’s data sets. Linearization of Nonlinear Models

The straight-line regression model is not always suitable for curve fitting. The choice of regression model is often guided by the plot of the available data, or can be guided by the knowledge of the physical behavior of the system that generated the data. In general, polynomial or other nonlinear models are more suitable. A nonlinear regression technique (introduced later) is available to fit complicated nonlinear equations to data. However, some basic nonlinear functions can be readily transformed into linear functions in their regression coefficients (we will refer to such functions as transformable or linearizable). Here, we can take advantage of the LSE regression formulas, which we have just derived, to fit the transformed equations to the data.

One example of a linearizable linear model is the exponential model, 푦(푥) = 훼푒훽푥, where 훼 and 훽 are constants. This equation is very common in engineering (e.g., capacitor transient voltage) and science (e.g., population growth or radioactive decay). We can linearize this equation by simply taking its natural logarithm to yield: ln(푦) = ln(훼) + 훽푥. Thus, if we transform the 푦푖 values in our data, by taking their natural logarithms, and define 푎0 = ln(훼) and 푎1 = 훽 we arrive at the equation of a straight-line (of the form 푌 = 푎0 + 푎1푥). Then, we can readily use the ∗ ∗ formulas for the LSE solution, (푎0, 푎1), derived earlier. The final step would be to ∗ 푎0 ∗ set 훼 = 푒 and 훽 = 푎1 and arrive at the regression solution, ∗ ∗ 푦(푥) = 훼푒훽푥 = (푒푎0)푒푎1푥 A second common linearizable nonlinear regression model is the power model, 푦(푥) = 훼푥훽. We can linearize this equation by simply taking its natural logarithm to yield: ln(푦) = ln(훼) + 훽ln (푥) (which is a linear model of the form 푌 = 푎0 + 푎1푋). In this case, we need to first transform the 푦푖 and 푥푖 values into ln(푦푖) and ln(푥푖), respectively, and then apply the LSE solution to the transformed data.

Other useful linearizable models include: 푦(푥) = 훼 ln(푥) + 훽 (logarithmic 1 훼푥 function), 푦(푥) = (reciprocal function), and 푦(푥) = (saturation-growth- 훼푥+훽 훽+푥 rate function). The following two tables serve list twelve nonlinear models that are linearizable and provide their corresponding linearized form and their change of variable formulas, respectively.

Example. Fit the exponential model and the power model to the data in the following table. Compare the fit quality to that of the straight-line model. 푥 1 2 3 4 5 6 7 8 푦 2.5 7 38 55 61 122 83 143

Solution. Matlab script (linear.m) for the linear model, 푦 = 푎0 + 푎1푥: x=[1 2 3 4 5 6 7 8]; y=[2.5 7 38 55 61 122 83 143]; n=length(x); a1=(n*sum(x.*y)-sum(x)*sum(y))/(n*sum(x.^2)-(sum(x)).^2); a0=sum(y)/n-a1*sum(x)/n; r=(n*sum(x.*y)-sum(x)*sum(y))/((sqrt(n*sum(x.^2)-(sum(x))^2))*(... sqrt(n*sum(y.^2)-(sum(y))^2))); a1, a0, r^2

Result 1. Linear model solution: 푦 = 19.3036푥 − 22.9286, 푟2 = 0.8811. Matlab script (exponential.m) for the exponential model, 푦 = 훼푒훽푥: x=[1 2 3 4 5 6 7 8]; y=[2.5 7 38 55 61 122 83 143]; ye=log(y); n=length(x); a1=(n*sum(x.*ye)-sum(x)*sum(ye))/(n*sum(x.^2)-(sum(x)).^2); a0=sum(ye)/n-a1*sum(x)/n; r=(n*sum(x.*ye)-sum(x)*sum(ye))/((sqrt(n*sum(x.^2)-(sum(x))^2))... *(sqrt(n*sum(ye.^2)-(sum(ye))^2))); alpha=exp(a0), beta=a1, r^2

Result 2. Exponential model solution: 푦 = 3.4130푒0.5273푥, 푟2 = 0.8141. Matlab script (power_eq.m) for the power model, 푦 = 훼푥훽: x=[1 2 3 4 5 6 7 8]; y=[2.5 7 38 55 61 122 83 143]; xe=log(x); ye=log(y); n=length(x); a1=(n*sum(xe.*ye)-sum(xe)*sum(ye))/(n*sum(xe.^2)-(sum(xe)).^2); a0=sum(ye)/n-a1*sum(xe)/n; r=(n*sum(xe.*ye)-sum(xe)*sum(ye))/((sqrt(n*sum(xe.^2)- … (sum(xe))^2))*(sqrt(n*sum(ye.^2)-(sum(ye))^2))); alpha=exp(a0), beta=a1, r^2

Result 3. Power model solution: 푦 = 2.6493푥1.9812, 푟2 = 0.9477

From the results for 푟2, the power model has the best fit. The following graph compares the three models. By visually inspecting the plot we see that, indeed, the power model (red; 푟2 = 0.9477) is a better fit compared to the linear model (blue; 푟2 = 0.8811) and to the exponential model (green; 푟2 = 0.8141). Also, note that the straight-line fits the data better than the exponential model.

Your turn: Repeat the above regression problem employing: (a) The logarithmic 1 function, 푦 = 훼 ln(푥) + 훽; (b) The reciprocal function, 푦 = ; (c) The 훼푥+훽 훼푥 saturation-growth-rate function, 푦(푥) = . Compare the results by plotting. 훽+푥 General Linear LSE Regression and the Polynomial Model

For some data sets, the underlying model cannot be captured accurately with a straight-line, exponential, logarithmic or power models. A model with a higher degree of nonlinearity (i.e., with added flexibility) is required. There are a number of higher order functions that can be used as regression models. One important regression model would be a polynomial. A general LSE formulation is presented next. It extends the earlier linear regression analysis to a wider class of nonlinear functions, including polynomials. (Note: when we say linear regression, we are referring to a model that is linear in its regression parameters, 푎푖, not 푥.) Consider the general function in z,

푦 = 푎푚푧푚 + 푎푚−1푧푚−1 + ⋯ + 푎1푧1 + 푎0 (1) where the 푧푖 represents a basis function in 푥. It can be easily shown that if the basis 푖 functions are chosen as 푧푖 = 푥 , then the above model is that of an 푚-degree polynomial, 푚 푚−1 푦 = 푎푚푥 + 푎푚−1푥 + ⋯ + 푎1푥 + 푎0

There are many classes of functions that can be described by the above general function in Eqn. (1). Examples include:

−푥2 푦 = 푎0 + 푎1푥, 푦 = 푎0 + 푎1 cos(푥) + 푎2sin (2푥) and 푦 = 푎0 + 푎1푥 + 푎2푒

One example of a function that can’t be represented by the above general function is the radial-basis-function (RBF) 2 푎2(푥−푎3) 푦 = 푎0 + 푎1푒

In other words, this later function is not transformable into a linear regression model, as was the case (say) for the exponential function, 푦 = 훼푒훽푥. The regression with such non-transformable functions is known as nonlinear regression and is considered later in this lecture. In the following formulation of the LSE regression problem we restrict the model of the regression function to the polynomial 푚 푚−1 푦 = 푎푚푥 + 푎푚−1푥 + ⋯ + 푎1푥 + 푎0

Given 푛 data points {(푥1, 푦1), (푥2, 푦2), … , (푥푛, 푦푛)} we want to determine the regression coefficients by solving the system of 푛 equations with 푚 + 1 unknowns: 푚 푚−1 푎푚푥1 + 푎푚−1푥1 + ⋯ + 푎1푥1 + 푎0 = 푦1 푚 푚−1 푎푚푥2 + 푎푚−1푥2 + ⋯ + 푎1푥2 + 푎0 = 푦2 . 푚 푚−1 푎푚푥푛 + 푎푚−1푥푛 + ⋯ + 푎1푥푛 + 푎0 = 푦푛

The above system can be written in matrix form as Za = y,

푥푚 푥푚−1 ⋯ 푥 1 푎 푦 1 1 1 푚 1 푚 푚−1 푥 푥 ⋯ 푥 1 푎푚−1 푦2 2 2 2 ⋮ ⋮ ⋱ ⋮ ⋮ ⋮ = ⋮ 푚 푚−1 푎 푦 푥푛−1 푥푛−1 ⋯ 푥푛−1 1 1 푛−1 푚 푚−1 [ 푎0 ] [ 푦푛 ] [ 푥푛 푥푛 ⋯ 푥푛 1]

where Z is an 푛x(푚 + 1) rectangular matrix formulated from the 푥푖 data values 푚−푗+1 with 푧푖푗 = 푥푖 (푖 = 1, 2, … , 푛; 푗 = 1, 2, … , 푚 + 1), a is an (푚 + 1) column vector of unknown regression coefficients and y is an 푛 column vector of 푦푖 data values. For regression problems, the above system is over-determined (푛 > 푚 + 1) and, therefore, there is generally no solution ‘a’ that satisfies Za = y. So, we seek the LSE solution, a*; the solution which minimizes the sum-of-squared-error (SSE) criterion

2 푛 푚+1 2 E(a) = ||y - Za|| = ∑푖=1(푦푖 − ∑푗=1 푧푖푗 푎푚−푗+1) 2 푛 푚+1 푚−푗+1 = ∑푖=1(푦푖 − ∑푗=1 푥푖 푎푚−푗+1) where ||.|| denotes the vector norm. As we did earlier in deriving the straight-line 휕 regression coefficients 푎0 and 푎1, we set all partial derivatives 퐸(a) to zero and 휕푎푖 solve the resulting system of (푚 + 1) equations:

푛 푚+1 휕 푚−푗+1 0 퐸(푎0, 푎1, … , 푎푚) = −2 ∑ (푦푖 − ∑ 푥푖 푎푚−푗+1) 푥푖 = 0 휕푎0 푖=1 푗=1

푛 푚+1 휕 푚−푗+1 1 퐸(푎0, 푎1, … , 푎푚) = −2 ∑ (푦푖 − ∑ 푥푖 푎푚−푗+1) 푥푖 = 0 휕푎1 푖=1 푗=1

푛 푚+1 휕 푚−푗+1 2 퐸(푎0, 푎1, … , 푎푚) = −2 ∑ (푦푖 − ∑ 푥푖 푎푚−푗+1) 푥푖 = 0 휕푎2 푖=1 푗=1 . .

푛 푚+1 휕 푚−푗+1 푚 퐸(푎0, 푎1, … , 푎푚) = −2 ∑ (푦푖 − ∑ 푥푖 푎푚−푗+1) 푥푖 = 0 휕푎푚 푖=1 푗=1

0 This system can be rearranged as (note: 푥푖 = 1)

푛 푚+1 푚−푗+1 ∑ (푦푖 − ∑ 푥푖 푎푚−푗+1) = 0 푖=1 푗=1

푛 푚+1 푚−푗+1 ∑ (푥푖푦푖 − 푥푖 ∑ 푥푖 푎푚−푗+1) = 0 푖=1 푗=1

푛 푚+1 2 2 푚−푗+1 ∑ (푥푖 푦푖 − 푥푖 ∑ 푥푖 푎푚−푗+1) = 0 푖=1 푗=1 . .

푛 푚+1 푚 푚 푚−푗+1 ∑ (푥푖 푦푖 − 푥푖 ∑ 푥푖 푎푚−푘+1) = 0 푖=1 푗=1 or,

푛 푚+1 푛 푚−푗+1 ∑ ( ∑ 푥푖 푎푚−푗+1) = ∑ 푦푖 푖=1 푗=1 푖=1

푛 푚+1 푛 푚−푗+1 ∑ 푥푖 ( ∑ 푥푖 푎푚−푗+1) = ∑ 푥푖푦푖 푖=1 푗=1 푖=1

푛 푚+1 푛 2 푚−푗+1 2 ∑ 푥푖 ( ∑ 푥푖 푎푚−푗+1) = ∑ 푥푖 푦푖 푖=1 푗=1 푖=1 . .

푛 푚+1 푛 푚 푚−푗+1 푚 ∑ 푥푖 ( ∑ 푥푖 푎푚−푗+1) = ∑ 푥푖 푦푖 푖=1 푗=1 푖=1

푚−푗+1 which by setting 푧푖푗 = 푥푖 (푖 = 1, 2, … , 푛; 푗 = 1, 2, … , 푚 + 1) can then be expressed in matrix form as (Your turn: derive it) ZT(Za) = ZTy or (ZTZ)a= ZTy (Refer to the Appendix for an explicit representation of the above equation for the case of a quadratic regression polynomial.) The matrix ZTZ is a (푚 + 1)x(푚 + 1) square matrix (recall that 푚 is the degree of the polynomial model being used). Generally speaking, the inverse of ZTZ does exist for the above regression formulation. Multiplying both sides of the equation by (ZTZ)-1 leads to the LSE solution for the regression coefficient vector a, Ia* = a* = [(ZTZ)-1 ZT]y Where I is the identity matrix. Matlab offers two ways for solving the above system of linear equations: (1) using the left-division operator a = (Z′*Z)\(Z′*y), where ‘′’ is the Matlab transpose operator, or (2) using a = pinv(Z)*y, where pinv is the built-in Matlab pseudo-inverse function that computes the matrix (ZTZ)-1 ZT. The coefficient of determination, 푟2, for the above polynomial regression formulation is given by (for 푛 ≫ 푚)

∑푛 ( )2 2 푖=1 푦푖 − 푦̂푖 푟 = 1 − 푛 2 ∑푖=1(푦푖 − 푦̅)

where 푦̂푖 is the 푖th component of the prediction vector Za*, and 푦̅ is the mean of the 2 푦푖 values. Matlab can conveniently compute 푟 as,

1-sum((y-Z*a).^2)/sum((y-mean(y)).^2)

Example. Employ the polynomial LSE regression formulation to solve for a cubic curve fit for the following data set. Also, compute 푟2.

푥 1 2 3 4 5 6 7 8 푦 2.5 7 38 55 61 122 83 145

Solution. A cubic function is a third-order polynomial (푚 = 3) with the four coefficients 푎0, 푎1, 푎2 and 푎3. The number of data points is 8, therefore, 푛 = 8 and the Z matrix is 푛x(푚 + 1) = 8x4. The matrix formulation (Za = y) for this linear regression problem is 1 1 1 1 2.5 8 4 2 1 7

27 9 3 1 푎3 38

64 16 4 1 푎2 55 [ ] = 125 25 5 1 푎1 61 216 36 6 1 푎0 122 343 49 7 1 83 [512 64 8 1] [145] Solving using the pinv function, a = pinv(Z)*y, (y must be a column vector)

Alternatively, we may use the left-division operator and obtain the same result:

Therefore, the cubic fit solution is 푦 = 0.029푥3 − 0.02푥2 + 17.6176푥 − 19.2857 whose plot is shown below. Note that the contribution of the cubic and quadratic terms is very small compared to the linear part of the solution, for 0 ≤ 푥 ≤ 8. That is why the plot of the cubic fit model is close to linear.

The coefficient of determination 푟2 is computed as,

which indicates that the cubic regression model explains 88% of the variability in the data. This result has a similar quality to that of the straight-line regression model (computed in an earlier example).

The following is a snapshot of a session with the “Basic Fitting tool” (introduced in the previous lecture) applied to data in the above example. It computes and compares a cubic fit to a 5th-degree and a 6th-degree polynomial fit.

Your turn. Employ the linear LSE regression formulation to fit the following data 2 set employing the model: 푦(푥) = 푎0 + 푎1 cos(푥) + 푎2sin (2푥). Also, determine 푟 and plot the data along with 푦(푥). Hint: First determine the 10x3 matrix Z required for the Za = y formulation.

푥 1 2 3 4 5 6 7 8 9 10 푦 1.17 0.93 −0.71 −1.31 2.01 3.42 1.53 1.02 −0.08 −1.51 Polynomial Regression with Matlab: Polyfit

The Matlab polyfit function was introduced in the previous lecture for solving polynomial interpolation problems (푚 + 1 = 푛, same number of equations as unknowns). This function can also be used for solving 푚-degree polynomial regression given 푛 data points (푚 + 1 < 푛, more equations than unknowns). The syntax of polyfit call is p=polyfit(x,y,m), where x and y are the vectors of the independent and dependent variables, respectively, and m is the order of the regression polynomial. The function returns a row vector, p, that contains the polynomial coefficients.

Example. Here is a solution to the straight-line regression problem (first example encountered in this lecture):

Example. Use polyfit to solve for the cubic regression model encountered in the example from the previous section. Solution:

Note that this solution is identical to the one obtained using the pseudo-inverse- based solution. Non-Linear LSE Regression

In some engineering applications, nonlinear models are required to be used to fit a given data set. The above general linear regression formulation can handle such regression problems as long as the nonlinear model is transformable into an equivalent linear function in the unknown coefficients. However, in some cases, the models are not transformable. In this case, we have to come up with an appropriate set of equations whose solution leads to the LSE solution.

푎1푥 As an example, consider the nonlinear model 푦(푥) = 푎0(1 − 푒 ). This equation can’t be manipulated into a linear regression formulation in the 푎0 and 푎1 coefficients. The LSE formulation (for this model with 푛 data points) takes the form 푛 푛 2 2 푎1푥푖 퐸(푎0, 푎1) = ∑(푦푖 − 푦(푥푖)) = ∑(푦푖 − 푎0(1 − 푒 )) 푖=1 푖=1 푛 휕 푎1푥푖 푎1푥푖 퐸(푎0, 푎1) = 2 ∑(푦푖 − 푎0(1 − 푒 )) (−1 + 푒 ) = 0 휕푎0 푖=1 푛 휕 푎1푥푖 푎1푥푖 퐸(푎0, 푎1) = 2 ∑(푦푖 − 푎0(1 − 푒 )) (푎0푥푖푒 ) = 0 휕푎1 푖=1

This set of two nonlinear equations need to be solved for the two coefficients, 푎0 and 푎1. Numerical algorithms such as Newton’s iterative method for solving a set of two nonlinear equations, or Matlab’s built-in fsolve and solve functions can be used to solve this system of equations, as shown in the next example.

푎1푥 Example. Employ the regression function 푦(푥) = 푎0(1 − 푒 ) to fit the following data. x −2 0 2 4 y 1 0 −4 −12

Here, we have 푛 = 4, and the system of nonlinear equations to be solved is given by 4

푎1푥푖 푎1푥푖 ∑(푦푖 − 푎0(1 − 푒 )) (−1 + 푒 ) = 0 푖=1 4

푎1푥푖 푎1푥푖 ∑(푦푖 − 푎0(1 − 푒 )) (푎0푥푖푒 ) = 0 푖=1 Substituting the data point values in the above equations leads to

−2푎1 −2푎1 2푎1 2푎1 4푎1 4푎1 (1 − 푎0(1 − 푒 ))(−1 + 푒 ) + (−4 − 푎0(1 − 푒 ))(−1 + 푒 ) + (−12 − 푎0(1 − 푒 ))(−1 + 푒 ) = 0

−2푎1 −2푎1 2푎1 2푎1 4푎1 4푎1 (1 − 푎0(1 − 푒 ))(−2푎0푒 ) + (−4 − 푎0(1 − 푒 ))(2푎0푒 ) + (−12 − 푎0(1 − 푒 ))(4푎0푒 ) = 0 After expansion and combining terms, we get

−2푎1 2푎1 4푎1 −4푎1 −2푎1 2푎1 4푎1 8푎1 15 + (푒 − 4푒 − 12푒 ) + 푎0(3 + 푒 − 2푒 − 2푒 − 푒 + 푒 ) = 0

−4푎1 −2푎1 2푎1 4푎1 8푎1 2푎0[−푎0푒 + (푎0 − 1)푒 − (푎0 + 4)푒 − (푎0 + 24)푒 + 2푎0푒 ] = 0 Matlab solution using function solve: syms a0 a1 f1=15+(exp(-2*a1)-4*exp(2*a1)-12*exp(4*a1))+a0*(3+exp(-4*a1)... -2*exp(-2*a1)-2*exp(2*a1)-exp(4*a1)+exp(8*a1)); f2=2*a0*(-a0*exp(-4*a1)+(a0-1)*exp(-2*a1)-(a0+4)*exp(2*a1)... -(a0+24)*exp(4*a1)+2*a0*exp(8*a1));

Matlab returns a set of four solutions to the above minimization problem. The first thing we notice is that for nonlinear regression, minimizing LSE may lead to multiple solutions (multiple minima). The solutions for this particular problem are: 0 1. 푎0 = 푎1 = 0, which leads to: 푦 = 0(1 − 푒 ) = 0, or 푦 = 0 (the 푥-axis). 2. 푎0 = 0 and 푎1 ≅ −1.3610 + 1.5708푖, which leads to: 푦 = 0 3. 푎0 = 0 and 푎1 ≅ 0.1186 + 1.5708푖, which leads to: 푦 = 0. 0.441푥 4. 푎0 ≅ 2.4979 and 푎1 ≅ 0.4410, which leads to 푦(푥) = 2.4979(1 − 푒 ).

The solutions 푦(푥) = 0 and 푦(푥) = 2.4979(1 − 푒0.441푥) are plotted below. It is obvious that the optimal solution (in the LSE sense) is 푦(푥) = 2.4979(1 − 푒0.441푥).

Your turn: Solve the above system of two nonlinear equations employing Matlab’s fsolve. Your turn: Fit the exponential model, 푦 = 훼푒훽푥, to the data in the following table employing nonlinear least squares regression. Then, linearize the model and determine the model coefficients by employing linear least squares regression (i.e., use the formulas derived in the first section or polyfit). Plot the solutions. 푥 0 1 2 3 4 푦 1.5 2.5 3.5 5.0 7.5

Ans. Nonlinear least squares fit: 푦 = 1.61087푒0.38358푥 Fit by linearization: 푦 = 1.57991푒0.39120푥 Numerical Solution of the Non-Linear LSE Optimization Problem: Gradient Search and Matlab’s fminsearch Function

In the above example, we were lucky in the sense that the (symbolic-based) solve function returned the optimal solution for the optimization problem at hand. In more general non-linear LSE regression problems the models employed are complex and normally have more than two unknown coefficients. Here, solving (symbolically) for the partial derivatives of the error function becomes tedious and impractical. Therefore, one would use numerically-based multi-variable optimization algorithms to minimize 퐸(풂) = 퐸(푎0, 푎1, 푎2, … ), which are extensions of the ones considered in Lectures 13 and 14.

One method would be to extend the gradient-search minimization function grad_optm2 to handle a function with two variables. Recall that this version of the function approximates the gradients numerically, therefore there is no need to determine the analytical expressions for the derivatives. For the case of two variables 푎0 and 푎1, the gradient-descent equations are 휕 푎0(푘 + 1) = 푎0(푘) + 푟 퐸(푎0, 푎1) 휕푎0 휕 푎1(푘 + 1) = 푎1(푘) + 푟 퐸(푎0, 푎1) 휕푎1 where −1 < 푟 < 0. Upon using the simple backward finite-difference approximation for the derivatives, we obtain

퐸[푎0(푘), 푎1(푘)] − 퐸[푎0(푘 − 1), 푎1(푘)] 푎0(푘 + 1) = 푎0(푘) + 푟 푎0(푘) − 푎0(푘 − 1)

퐸[푎0(푘), 푎1(푘)] − 퐸[푎0(푘), 푎1(푘 − 1)] 푎1(푘 + 1) = 푎1(푘) + 푟 푎1(푘) − 푎1(푘 − 1)

The following is a Matlab implementation (function grad_optm2d) of these iterative formulas.

−4 The above function [with 푎0(0) = 2, 푎1(0) = 0.5 and 푟 = −10 ] returns the same solution that was obtained above with solve [with, a minimum error value of 퐸(2.4976, 0.4410) = 0.4364]:

Matlab has an important built-in function for numerical minimization of nonlinear multivariable functions. The function name is fminsearch. The (basic) function call syntax is [a,fa] = fminsearch(f,a0), where f is an anonymous function, a0 is a vector of initial values. The function returns a solution vector ‘a’ and the value of the function at that solution, fa. Here is an application of fminsearch to solve the above non-linear regression problem [note how the unknown coefficients are represented as the elements of the vector, 푎0 = a(1), 푎1 = a(2), and are initialized at [0 0] for this problem].

A more proper way to select the initial search vector a = [푎0 푎1] for the above optimization problem is to solve a set of 푘 nonlinear equations that is obtained from forcing the model to go through 푘 points (selected randomly from the data set). Here, 푘 is the number of unknown model parameters. For example, for the above problem, we solve the set of two nonlinear equations

푎1푥푖 푦푖 − 푎0(1 − 푒 ) = 0

푎1푥푗 푦푗 − 푎0(1 − 푒 ) = 0 where (푥푖, 푦푖) and (푥푗, 푦푗) are two distinct points selected randomly from the set of points being fitted. A numerical nonlinear equation solver can be used, say Matlab’s fsolve, as shown below [here, the end points (−2,1) and (4, −12) were selected].

Your turn: The height of a person at different ages is reported in the following table. x (age) 0 5 8 12 16 18 y (in) 20 36.2 52 60 69.2 70

Determine the parameters 푎, 푏 and 푐 so that the following regression model is optimal in the LSE sense. 푎 푦(푥) = 1 + 푏푒−푐푥 74.321 Ans. 푦(푥) = 1+2.823푒−0.217푥

Your turn: Employ nonlinear LSE regression to fit the function 퐾 푦(푥) = √푥4 + (푎2 − 2푏)푥2 + 푏2 to the data 푥 0 0.5 1 2 3 푦 0.95 1.139 0.94 0.298 0.087

Plot the data points and your solution for 푥 ∈ [0 6].

0.888 Ans. 푦(푥) = √푥4−0.956푥2+0.819

As mentioned earlier, different initial conditions may lead to different local minima of the nonlinear function being minimized. For example, consider the function of two variables that exhibits multiple minima (refer to the plot): 푓(푥, 푦) = −0.02 sin(푥 + 4푦) − 0.2 cos(2푥 + 3푦) − 0.3 sin(2푥 − 푦) + 0.4cos (푥 − 2푦)

The following script generates a plot of 푓(푥, 푦), for −1 ≤ 푥 ≤ 3 and −4 ≤ 푦 ≤ 4.

A contour plot can be generated as follows (the local minima are located at the center of the blue contour lines):

The following are the local minima discovered by function grad_optm2d for the indicated initial conditions:

The same local minima are discovered by fminsearch when starting from the same initial conditions:

Note that for this limited set of searches, the solution with the smallest SSE value is (푥∗, 푦∗) = (0.0441, −1.7618), which represents a more optimal solution.

Your turn (Email your solution to your instructor one day before Test 3). Fit the following data 푥 1 2 3 4 5 6 7 8 9 10 푦 1.17 0.93 −0.71 −1.31 2.01 3.42 1.53 1.02 −0.08 −1.51 employing the model

푦(푥) = 푎0 + 푎1 cos(푥 + 푏1) + 푎2cos (2푥 + 푏2) This problem can be solved employing nonlinear regression (think solution via fminsearch), or it can be linearized which allows you to use linear regression (think solution via pseudo-inverse). Hint: cos(푥 + 푏) = cos(푥) cos(푏) − sin(푥) sin(푏). Plot the data points and 푦(푥) on the same graph.

In practice, a nonlinear regression model can have hundreds or thousands of coefficients. Examples of such models are neural networks and radial-basis-function models that often involve fitting multidimensional data sets, where each 푦 value depends on many variables, 푦(푥1, 푥2, 푥3 …). Numerical methods such as gradient-based optimization methods are often used to solve for the regression coefficients associated with those high-dimensional models. For a reference, check Chapters 5 and 6 in the following textbook: Fundamentals of Artificial Neural Networks, Mohamad Hassoun (MIT Press, 1995). Solution of Differential Equations Based on LSE Minimization

Consider the second-order time-varying coefficient differential equation

1 푦̈(푥) + 푦̇(푥) + 9푥2푦(푥) = 0 with 푦(0) = 1 and 푦̇(0) = 2 5 defined over the interval 푥 ∈ [0 1].

2 3 4 We seek a polynomial 푦̃(푥) = 푎0 + 푎1푥 + 푎2푥 + 푎3푥 + 푎4푥 that approximates the solution, 푦(푥). In general, 푦̃(푥) does not have to be a polynomial. By applying the initial conditions to 푦̃(푥) we can solve for 푎0 and 푎1

푑푦̃(0) 푦̃(0) = 푎 = 푦(0) = 1 and = 푎 = 푦̇(0) = 2 0 푑푥 1

Now, we are left with the problem of estimating the remaining polynomial coefficients 푎2, 푎3, 푎4 such that the residual

푑2푦̃(푥) 1 푑푦̃(푥) 푓(푥, 푎 , 푎 , 푎 ) = + + 9푥2푦̃(푥) 2 3 4 푑푥2 5 푑푥 is as close to zero as possible for all 푥 ∈ [0 1]. We will choose to minimize the integral of the squared residual,

1 2 퐼(푎2, 푎3, 푎4) = ∫ [푓(푥, 푎2, 푎3, 푎4)] 푑푥 0

푑푦̃(푥) 푑2푦̃(푥) First, we compute the derivatives, and , 푑푥 푑푥2

푑푦̃(푥) = 2 + 2푎 푥 + 3푎 푥2 + 4푎 푥3 푑푥 2 3 4

푑2푦̃(푥) = 2푎 + 6푎 푥 + 12푎 푥2 푑푥2 2 3 4 which lead to (recall that 푎0 = 1 and 푎1 = 2)

푑2푦̃(푥) 1 푑푦̃(푥) 푓(푥, 푎 , 푎 , 푎 ) = + + 9푥2푦̃(푥) 2 3 4 푑푥2 5 푑푥 1 = 2푎 + 6푎 푥 + 12푎 푥2 + (2 + 2푎 푥 + 3푎 푥2 + 4푎 푥3) 2 3 4 5 2 3 4 2 2 3 4 + 9푥 (1 + 2푥 + 푎2푥 + 푎3푥 + 푎4푥 ) 2 2 3 = + 9푥2 + 18푥3 + 푎 (2 + 푥 + 9푥4) + 푎 (6푥 + 푥2 + 9푥5) 5 2 5 3 5 4 + 푎 (12푥2 + 푥3 + 9푥6) 4 5 or,

2 2 푓(푥, 푎 , 푎 , 푎 ) = + 9푥2 + 18푥3 + 푎 (2 + 푥 + 9푥4) + 2 3 4 5 2 5 3 4 푎 (6푥 + 푥2 + 9푥5) + 푎 (12푥2 + 푥3 + 9푥6) 3 5 4 5

The following Matlab session shows the results of using fminsearch to solve for the coefficients 푎2, 푎3, 푎4 that minimize the error function

1 2 퐼(푎2, 푎3, 푎4) = ∫ [푓(푥, 푎2, 푎3, 푎4)] 푑푥 0

Note: the first component a(1) in the solution vector ’a’ is redundant; it is not used in function 퐼.

Therefore, the optimal solution is

푦̃(푥) = 1 + 2푥 + 0.1347푥2 − 0.8170푥3 − 0.8345푥4 The following plot compares the “direct” numerical solution (red trace) to the minimum residual solution (blue trace). We will study the very important topic of numerical solution of differential equations in Lecture 22 (e.g., employing ode45).

Your turn: Consider the first-order, nonlinear, homogeneous differential equation with varying coefficient 푦̇(푥) + (2푥 − 1)푦2(푥) = 0, with 푦(0) = 1 and 푥 ∈ [0 1]. Employ the method of minimizing the squared residual to solve for the approximate solution 2 3 4 푦̃(푥) = 푎0 + 푎1푥 + 푎2푥 + 푎3푥 + 푎4푥 over the interval 푥 ∈ [0 1]. Plot 푦̃(푥) and the exact solution 푦(푥) given by

1 푦(푥) = 푥2 − 푥 + 1

Ans. 푦̃(푥) = 1 + 0.964푥 + 0.487푥2 − 2.903푥3 + 1.452푥4

Your turn: Determine the parabola 푦(푥) = 푎푥2 + 푏푥 + 푐 that approximates the cubic 푔(푥) = 2푥3 − 푥2 + 푥 + 1 (over the interval 푥 ∈ [0 2]) in the LSE sense. In other words, determine the coefficients 푎, 푏 and 푐 such that the following error function is minimized,

2 퐸(푎, 푏, 푐) = ∫ [푔(푥) − 푦(푥)]2푑푥 0 Solve the problem in two ways: (1) analytically; and (2) Employing fminsearch after evaluating the integral. Plot 푔(푥) and 푦(푥) on the same set of axis. Appendix: Explicit Matrix Formulation for the Quadratic Regression Problem Earlier in this lecture we have derived the 푚-degree polynomial LSE regression formulation as follows,

푛 푚+1 푛 푚−푗+1 ∑ ( ∑ 푥푖 푎푚−푗+1) = ∑ 푦푖 푖=1 푗=1 푖=1

푛 푚+1 푛 푚−푗+1 ∑ 푥푖 ( ∑ 푥푖 푎푚−푗+1) = ∑ 푥푖푦푖 푖=1 푗=1 푖=1

푛 푚+1 푛 2 푚−푗+1 2 ∑ 푥푖 ( ∑ 푥푖 푎푚−푗+1) = ∑ 푥푖 푦푖 푖=1 푗=1 푖=1 . .

푛 푚+1 푛 푚 푚−푗+1 푚 ∑ 푥푖 ( ∑ 푥푖 푎푚−푗+1) = ∑ 푥푖 푦푖 푖=1 푗=1 푖=1

Now, setting 푚 = 2 (designating a quadratic regression model) leads to three equations:

푛 3 푛 3−푗 ∑ (∑ 푥푖 푎3−푗) = ∑ 푦푖 푖=1 푗=1 푖=1

푛 3 푛 3−푗 ∑ 푥푖 (∑ 푥푖 푎3−푗) = ∑ 푥푖푦푖 푖=1 푗=1 푖=1

푛 3 푛 2 3−푗 2 ∑ 푥푖 (∑ 푥푖 푎3−푗) = ∑ 푥푖 푦푖 푖=1 푗=1 푖=1 It can be shown that the above equations (Your Turn) can be cast in matrix form as, 푛 푛 푛 푛 ∑ 푥 ∑ 푥2 ∑ 푦 푖 푖 푖 푖=1 푖=1 푖=1 푛 푛 푛 푎 푛 0 2 3 푎 ∑ 푥푖 ∑ 푥푖 ∑ 푥푖 [ 1] = ∑ 푥푖푦푖 푖=1 푖=1 푖=1 푎3 푖=1 푛 푛 푛 푛 2 3 4 2 ∑ 푥푖 ∑ 푥푖 ∑ 푥푖 ∑ 푥푖 푦푖 [푖=1 푖=1 푖=1 ] [푖=1 ]

This is a 3x3 linear system that can be solved using the methods of Lectures 15 & 16. With this type of formulation, care must be taken as to employ numerical solution methods that can handle ill-condition coefficient matrices; note the dominance of the (all positive) coefficients in the last row of the matrix. The following two-part video (part1, part2) derives the above result (directly) from basic principles. Here is an example of quadratic regression: Part 1 Part 2.

Example. Employ the above formulation to fit a parabola to the following data. 푥 0 5 8 12 16 18 푦 20 36.2 52 60 69.2 70

Matlab solution:

The code that generated the above result is shown below.

Your turn: Verify the above solution employing Polyfit. Repeat employing the pseudo-inverse solution a = pinv(Z)*y applied to the formulation, 푥푚 푥푚−1 ⋯ 푥 1 푎 푦 1 1 1 푚 1 푚 푚−1 푥 푥 ⋯ 푥 1 푎푚−1 푦2 2 2 2 퐙퐚 = ⋮ ⋮ ⋱ ⋮ ⋮ ⋮ = ⋮ 푚 푚−1 푎 푦 푥푛−1 푥푛−1 ⋯ 푥푛−1 1 1 푛−1 푚 푚−1 [ 푎0 ] [ 푦푛 ] [ 푥푛 푥푛 ⋯ 푥푛 1] Your turn: Extend the explicit matrix formulation of this appendix to a cubic function 2 3 푓(푥) = 푎0 + 푎1푥 + 푎2푥 + 푎3푥 and use it to determine the polynomial coefficients for the data set from the last example. Compare (by plotting) your solution to the following solution that was obtained using nonlinear regression, 74.321 푦(푥) = . 1+2.823푒−0.217푥 Verify your solution employing Polyfit.

Ans. Your solution should look like the one depicted below.

Solution Strategies for LSE Regression Problem