sign in sign up
<<
Home , Interpolation

Regression and

CS 740 Ajit Rajwade

1 of 20 Problem statements

• Consider a set of N points {(xi,yi)}, 1 ≤ i ≤ N. • Suppose we know that these points actually lie on a of the form y = f(x;a) where f(.) represents a function family and a represents a set of parameters. • For example: f(x) is a of x, i.e. of the form y = f(x) = mx+c. In this case, a = (m,c).

2 Problem statements

• Example 2: f(x) is a quadratic function of x, i.e. of the form y = f(x) = px2+qx+r. In this case, a = (p,q,r).

• Example 3: f(x) is a trigonometric function of x, i.e. of the form f(x) = p sin(qx+r). In this case, a = (p,q,r).

• In each case, we assume knowledge of the function family. But we do not know the function parameters, and would like to estimate them from {(xi,yi)}.

• This is the problem of fitting a function (of known family) to a set of points.

3 Problem statements

• In function regression (or approximation), we

want to find a such that for all i, f(xi;a)≈yi.

• In function interpolation, we want to fit some

function such that f(xi;a)=yi.

4 Regression

• A polynomial of degree n is a function of the form:

n i 2 n y  ai x  a0  a1x  a2 x ... an x i0

• Polynomial regression is the task of fitting a polynomial function to a set of points.

5 Polynomial regression

• Let us assume that x is the independent variable, and y is the dependent variable.

• In the point set {(xi,yi)} containing N points, we will assume that the x-coordinates of the points are available accurately, whereas the y- coordinates are affected by measurement error – called as noise.

6 7 Polynomial regression

• If we assume the polynomial is linear, we have

yi = mxi + c + ei, where ei is the noise in yi. We want to estimate m and c. • We will do so by minimizing the following w.r.t. m and c: N 2 ( yi  mxi  c) i0 • This is called as least squares regression.

8 Least Squares Polynomial regression

• In matrix, form we can write the following  y   x 1  e   1   1   1  The solution will exist provided there are at  y2   x2 1  e2     m   least 2 non-coincident .  . .    . points. Basically, XTX     c     .   . .  .  must be non-singular.        yN   xN 1 eN  y  Xa a (X T X)1 X T y  X y 9 Least Squares Polynomial regression • For higher order , we have: The solution an   y   xn . . x2 x 1   e  will exist  1   1 1 1   1  n 2  .  provided  y2   x . . x x 1  e2  2 2 2  .  there are at  .    . . . .    .  least n non-       a2  coincident,  .   . . . .   .  non-collinear     a1   points. y xn . . x2 x 1   e  N   N N N a   N  Basically, XTX  0  must be non- y  Xa singular. T 1 T  a (X X)X y  X y 10 Least Squares Polynomial regression • Regardless of the order of the polynomial, we are dealing with a linear form of regression – because in each case, we are solving an equation of the form y ≈ Xa.

• There are some assumptions made on the errors in the measurement of y. We will not go into those details.

11 Least Squares Polynomial Regression: Geometric Interpretation • The least squares solution seeks a set of polynomial coefficients that minimize the sum of the squared difference between the measured y coordinate of each point and the y coordinate if the assumed polynomial model was strictly obeyed, i.e.

N n min (y  a x j )2 {a j }  i  j i i0 j0  2 min y  Xa 12 {a} 2 Least Squares Polynomial Regression: Geometric Interpretation • The least squares solution seeks a set of polynomial coefficients that minimize the sum of the squared difference between the measured y coordinate of each point and the y coordinate if the assumed polynomial model was strictly obeyed, i.e. we are trying to find a model that will minimize vertical distances (distances along the Y axis).

13 Weighted least squares: Row weighting

N n min (y  a x j )2 {a j }  i  j i i0 j0  min y  Xa 2 {a} 2 Different instances of y get different weights (higher the weight the more the importance to that instance!)

N n j 2 Wy  WXa min{a } wi (yi  a jxi ) j   T 1 T i1 j0 a  (WX) WX (WX) Wy  min W(y  Xa )2 {a} 2

W  diag(w1, w2 ,..., wN ) 14 Regularization

• In some cases, the system of equations may be ill- conditioned. • In such cases, there may be more than one solution. • It is common practice to choose a “simple” or “smooth” solution. • This practice of choosing a “smooth” solution is called as regularization. • Regularization is a common method used in several problems in machine learning, computer vision and .

15 Regularization: Ridge regression

Penalize (or discourage) solutions in which the min y  Xa 2   a 2 {a} 2 2 magnitude of vector a is too high. The parameter λ is called the regularization parameter. Larger the value of λ, the greater the encouragement to find a smooth solution.

T T Taking w.r.t. a, we get this regularized X X  Ia  X y solution for a. X  USVT VS2V T  Ia  VSUTy VS2  IV Ta  VSUTy S2  IV Ta  S UTy r S UT y V Ta ii i   2 i1 Sii   16 Regularization: Tikhonov regularization

min y  Xa 2   Ba 2 T T T {a} 2 2 X X  B Ba  X y

Penalize (or discourage) solutions in which the magnitude of vector Ba is too high. The parameter λ is called the regularization parameter. Larger the value of λ, the greater the encouragement to find a smooth solution. The matrix B can be chosen in different ways – very commonly, one penalizes large values of the gradient of vector a, given as follows:

a  a1 a2  a1 a3  a2 . . an1  an2 an  an1

In such a case, B is given as follows:  1 0 0 0 0 0 0   1 1 0 . . . 0  . 1 1 . . . 0   B   ......   ......     0 . . 0 1 1 0    0 . . . 0 1 1 17 Total Least Squares

• There are situations when there are errors in not just the yi values, but in the xi values as well. • In such a situations, least squares is not applicable – instead a method called total least squares is used. • Total least squares solutions heavily use the SVD – so this is one more application of the SVD for you!

18 Total Least Squares

• Consider the equation y ≈ Xa for polynomial regression. • We want to find a, in the case where both y and X have errors (in the earlier case, only y had errors). • So we seek to find a new version of y (denoted y*) close to y, and a new version of X (denoted X*) close to X, such that y*=X*a. • Thus we want (see next slide):

19 Find (y*,X*) to minimize y  y * 2  X  X * 2 such that y* X * a. 2 F Define Z* (X* | y*),Z (X | y).

2  a  Minimize Z  Z * such that Z *    0. F   1 Size of X * is N  (n 1). Size of Z * is N  (n  2).  a  As the vector   is not all zeros, 1 the column rank of Z * is less than n  2. To minimize Z  Z * 2 , Z * is the best rank n 1 approximation to Z, given by F n1 t Z*   iuivi from SVD of Z (Eckhart - Young Theorem). i1

 a  v1:n1,n2 Also    vn2 ,i.e. a   . 1 vn2,n2 20 Interpolation

• In the case of interpolation, we want the function being fit to pass through the given data-points exactly. • The type of interpolation is determined by whether it is a function of one or more variables, as well as the type of function. • Let us consider lower-order in 1D.

21 Interpolation • Given a set of N points lying on a curve, we can perform by simply joining consecutive points with a line.

22 Interpolation

• The problem with this approach is that the slopes of adjacent lines change abruptly at each data- point – called as C1 discontinuity. • This can be resolved by stitching piecewise quadratic polynomials between consecutive point pairs and requiring that  Adjacent polynomials actually pass through the point (interpolatory condition)  Adjacent polynomials have the same slope at that point (C1 continuity condition)

23 Interpolation

• We could go one step further and require that  Adjacent polynomials have the same second derivative at that point (C2 continuity condition) • This requires that the polynomials be piecewise cubic (degree 3) at least. • This type of interpolation is called cubic- interpolation and is very popular in graphics and image processing. • The data-points (where the continuity conditions) are imposed are called as knots or control points. • A cubic spline is a piecewise polynomial that is twice continuously differentiable (including at the knots).

24 Cubic

• A curve is parameterized as y = f(t) where ‘t’ is the independent variable.

• Consider N points (ti,yi) where i ranges from 1 to N. • We will have N-1 cubic polynomials, each of 2 3 the form yi(t) = ai + bi t + ci t + di t where i ranges from 1 to N-1.

• Thus the overall function y(t) is equal to yi(t) where t lies in between ti and ti+1.

25 Cubic spline interpolation • The total number of unknowns here is 4(N-1) (why?). • To determine them, we need to specify 4(N-1) equations. • As the curve must pass through the N points, we get N equations as follows: 2 3 y1  a1  b1t1  c1t1  d1t1 2 3 y2  a1  b1t2  c1t2  d1t2 . 2 3 yi  ai1  bi1ti  ci1ti  di1ti . 2 3 yN  aN 1  bN 1tN  cN 1tN  d N 1tN 26 Cubic spline interpolation

• The interpolatory conditions at the N-2 interior points yield N-2 equations of the form:

2 3 2 3 y2  a1  b1t2  c1t2  d1t2  a2  b2t2  c2t2  d2t2 . 2 3 2 3 yi  ai1  bi1ti  ci1ti  di1ti  ai  biti  citi  diti . 2 3 2 3 yN 1  aN 2  bN 2tN 1  cN 2tN 1  d N 2tN 1  aN 1  bN 1tN 1  cN 1tN 1  d N 1tN 1

27 Cubic spline interpolation

• The C1 continuity conditions at the N-2 interior points yield N-2 equations of the form: 2 2 b1  2c1t2  3d1t2  b2  2c2t2  3d2t2 . 2 2 bi1  2ci1ti  3di1ti  bi  2citi  3diti . 2 2 bN 2  2cN 2tN 1  3d N 2tN 1  bN 1  2cN 1tN 1  3d N 1tN 1

28 Cubic spline interpolation

• The C2 continuity conditions at the N-2 interior points yield N-2 equations of the form:

2c1  6d1t2  2c2  6d2t2 .

2ci1  6di1ti  2ci  6diti .

2cN 2  6d N 2tN 1  2cN 1  6d N 1tN 1 • The total number of equations is N+3(N-2) = 4N-6 whereas there are 4N-4 unknowns. 29 Cubic spline interpolation

• We need two more equations. • One possible set of conditions is to impose that the second derivative of the curve at the two end-points be zero. This gives two more equations of the form:

2c1  6d1t1  0

2cN 1  6dN 1tN  0 • A cubic spline with these two conditions is called a natural cubic spline and these conditions are called natural boundary conditions. • We now have 4(N-1) linear equations and 4(N-1) unknowns. • Solving the linear system gives us the cubic spline!

30 Cubic spline interpolation

• For the case of N = 3 points, the system of equations looks as follows: 1 t t2 t3 0 0 0 0  a   y   1 1 1  1 1 2 3     1 t2 t2 t2 0 0 0 0  b1   y2  0 0 0 0 1 t t2 t3  c   y   2 2 2  1   2  2 3 0 0 0 0 1 t3 t3 t3  d1   y3   2 2 0  0 1 2t 3t 0 1  2t  3t a   0   2 2 2 2  2    0 0 2 6t2 0 0  2  6t2  b2   0       0 0 2 6t 0 0 0 0 c 0  1  2         0 0 0 0 0 0 2 6t3 d2   0 

31 32 19

18

17

16

15

14

13

12

11

10 1 1.5 2 2.5 3

A cubic-spline fit to three points (1,10), (2,18) and (3,11) – marked with red asterisks.

33 References

• Scientific Computing, Michael Heath • http://en.wikipedia.org/wiki/Total_least_squa res

34