Ilnumerics Optimization Toolbox
ILNumerics Optimization Toolbox
1 INTRODUCTION
Optimization deals with the minimization or maximization of functions. The ILNumerics Optimization Toolbox consists of functions that perform minimization (or maximization) on general nonlinear functions and problems. An optimization problem is the problem of finding the best solution from all feasible solutions. Optimization problems can be divided into two categories depending on whether the variables are continuous or discrete. Here, we are focussing on continuous optimization problems.
The standard form of a continuous optimization problem is
min
Subject to ℎ ≤0, =1,…, , , ∈ Ω =0, =1,…, Where is the objective function to be minimized over the variable : ℝ → ℝ are called the inequality constraints , ℎ: ℝ → ℝ are called the equality constraints , and : ℝ → ℝ is a convex set in , called bound constraints. Ω ℝ By convention, the standard form defines a minimization problem . A maximization problem can be handled by negating the objective function. Based on the description of the function f and the feasible set , the problem can be classified as linear, quadratic, non- Ω linear, semi-infinite, semi-definite, multiple-objective, or discrete optimization problem. However, in the current status of the ILNumerics optimization toolbox, only nonlinear unconstraint and constraint optimization functions are provided.
2 UNCONSTRAINED OPTIMIZATION
The program available for unconstrained optimization problem in ILNumerics is called optimUnconst . The op timUnconst function solves optimization problems with nonlinear objectives, without bound constraints on the unknown variables . A quasi-Newton method using a Broyden-Fletcher-Goldfarb-Shanno (BFGS) formula to update the approximate Hessian matrix is implemented. The quasi-Newton method has a O(n²) memory requirement. For the moment, only the BFGS and the classical adaptive Newton Method algorithm is available for unconstrained optimization problem.
optimUnconst gives the option to provide user defined functions for the computation of the hessian or gradient. By default, the gradient is computed using finite differences based on an optimal step size.
The optimUnconst function is essentially an unconstrained nonlinear optimization solver:
• xopt = optimUnconst(objfunc,x0); • xopt = optimUnconst(objfunc,x0, gradfunc: gradient); • xopt = optimUnconst(objfunc,x0, hessianFunc: hessian); • xopt = optimUnconst(objfunc,x0, gradfunc: gradient, hessianFunc: hessian);
where • objfunc is the objective function, • x0 is the initial guess, • gradient is the gradient of the objective function, • xopt is the optimal point, or the minimizer, • hessianFunc is the hessian function giving the explicit expression of the hessian matrix.
2.12.12.1 TTTHE COST FUNCTION The cost function is directly passed to optimUnconst as a function parameter. In most cases, this will be an ILNumerics function. But it is also fine, to give the cost function as an anonymous function. Requirements on the cost function: The cost function has to be “smooth enough”, i.e. the 2nd derivative of the cost function (Hessian matrix) is expected to exist and to be non-zero on the whole definition set.
2.2 GETTING STARTED WITH UNCONSTRAINED OPTIMIZATION The simplest use of the optimUnconst algorithm is as follows: xopt = optimUnconst(objfunc,x0) ; where • objfunc is the objective function, • x0 is the initial guess, • xopt is the optimal point, or the minimizer.
2.2.1 Example In the following example, we compute the unconstrained minimum of the Rosenbrock 1 function. The function is given by
= 100 − + − 1
1 http://en.wikipedia.org/wiki/Rosenbrock_function
In C# the algorithm looks as follows: public static ILRetArray
The minimum of the Rosenbrock function is known to lay at [1,1]. In order to find this minimum programmatically, we give it to the optimUnconst function, together with a point to start looking for the minimum, let’s say: x0 = (-5,-5):
ILArray
>xopt >
The value returned from optimUnconst is called the minimizer of the objective function. It corresponds to the point x, where the objective function reaches a minimum. For non- convex problems, that minimum will be a local minimum with respect to the given starting point or initial guess. For convex functions f, the minimizer corresponds to the global minimum. The minimum, i.e. the value of the objective function at the minimizer can be found by simply evaluating the objective function:
>Rosenbrock(xopt) >
2.3 OPTIM UNCONST WITH GRADIENT AND /OR HESSIAN FOR UNCONSTRAINED OPTIMIZATION By default, the gradient and the hessian approximations (i.e. first and 2 nd derivatives) of the objective function are computed automatically by optimUnconst . In order to speed up this computation, user defined functions for the hessian and/or the gradient can be provided:
Xopt=optimUnconst(objfunc, x0, gradfunc: userGradientFunction); xopt=optimUnconst(objfunc, x0, hessianFunc:userDefinedHessian); xopt=optimUnconst(objfunc, x0, gradfunc: userGradientFunction, hessianFunc: userDefinedHessian); where userGradientFunction and userDefinedHessian are the user provided hessian and gradient functions. When the hessian is provided, the algorithm will become a classical Newton method 2 algorithm, with exact hessian approximation instead of the BFGS approximation of the hessian.
2 For an introduction, see: http://en.wikipedia.org/wiki/Newton_method_in_optimization
2.3.1 Example We would like to minimize the function = − 1 + − 2 + + 4 .
The derivative of the function is
. = − 1, − 2, + 4 / − 1 + − 2 + + 4
The C# algorithm of the function is as follow. public ILRetArray
The gradient is given by public ILRetArray
Giving the starting point (10.0, 100.0, 1000.0), the minimum of the function is found by:
ILArray
>xopt >
The value of the function at the minimizer is found by
> NewExampleFunction(xopt) >
In a similar way, the hessian function can be provided for the computation. In this case, the BFGS approximation of the hessian is replaced by the user defined function, resulting in a faster convergence of the optimization algorithm.
Note that the hessian function must return symmetric matrices of size n x n, where n is the dimensionality of the given starting point. 2.3.2 Example We would like to minimize the function = 2 , The derivative of the function is = 4 ,
And the hessian is . ′ = 4
The C# algorithm for the computation is as follows. public ILRetArray
The gradient function is defined by: public ILRetArray
The hessian function is defined by: public ILRetArray
Giving the starting point 1000.0, the method can be called as follows:
ILArray
From the immediate window, it appears
>
2.4 FEATURES : optimUnconst provides an efficient optimization solver based on the robust BFGS algorithm. The BFGS is a Quasi-Newton algorithm based on updates using a Gauss C-G algorithm , and a line-search of optimUnconst is based on the Golden Section Search algorithm. Even with “big” number, the BFGS will manage to adapt. However, it is recommend to normalize / scale functions with high values to a better floating point precision range (i.e.: values ‘near 1.0’) .
3 CONSTRAINED NONLINEAR OPTIMIZATION ALGORITHMS
ILNumerics provides method optim to obtain the value of x that minimizes a nonlinear objective functional f(x) allowing different kind of constraints. The simplest call of optim is: xopt=optim(objfunc, x0); where xopt is the cost function minimizer and x0 the initial guess and objfunc the cost function. In fact, the previous call is equivalent to the optimUnconst function call. However, the hessian and the gradient cannot be provided using this call, but the Newton method can still be used through approximation of the hessian matrix instead of BFGS approximation.
3.13.13.1 NNNONLINEAR OPTIMIZATION WITH BOUNDARY CONSTRAINTS optim solves bounded constraints optimization problems defined by
min Subject to . ≤ ≤ The calling sequence for such problem is : xopt=optim(objfunc, x0,lowerBound:a,upperBound:b);
• objfunc is the objective function, • x0 is the initial guess, • lowerBound is the lower bound constraint, • upperBound is the lower bound constraint.
3.1.1 Example Let us consider the minimization problem
min = − 1 ,
− 5 ≤ ≤ 0.
The C# function to solve the problem can be written as following:
public ILRetArray
ILArray
By printing the solution, it appears in the console:
>xopt >
It is worth noticing that both constraints are independent. It means a problem with only a lower bound or the upper bound can be solve as well. The order is not important and additionally, if a bound constraint is not provided, then the double .MaxValue() is used for the upper bound and double .Minvalue() for the lower bound. For example, public ILRetArray
// Minimizer computation ILArray
Will return:
>xopt >
3.23.23.2 NNNONLINEAR OPTIMIZATIOOPTIMIZATIONNNN WITH EQUALITY CONSTRCONSTRAINTAINTAINTAINT optim can be used to solve optimization problem with equality constraint as defined by
min Subject to . = 0 The simplest calling sequence for such problem is
ILArray
• x0 is the initial guess, • eqConst is the equality constraint.
3.2.1 Example We would like to minimize the function
= − 1 + + 2 − 4,
= 2 − = 0.
The IlNumerics code to solve the problem can be written as following:
//Cost function definition public ILRetArray
//Equality constraint function definition public ILRetArray
//Initial guess ILArray
//Minimizer computation ILArray
And the value of the minimizer is given by
>xopt >
Besides equality constraints, bounded constraints can be combined. The calling sequence will therefore be:
ILArray
In this case, the Lagrangian will be automatically computed and the result will be projected on the corresponding interval.
3.33.33.3 NNNONLINEAR OPTIMIZATIOOPTIMIZATIONNNN WITH INEQUALITY CONSCONSTRAINTTRAINT optim can also be used to solve inequality constraint optimization problems defined by
min . Subject to ≤ 0 The simplest calling sequence for such problem is : xopt=optim(objfunc,x0,InequalityConstraint: ineqConst); where • objfunc is the objective function, • x0 is the initial guess, • IneqConst is the equality constraint.
3.3.1 Example Let us consider the following minimization problem
min = − 1 + + 2 − 4, = 2 − ≤ 0.
The C# function to solve the problem can be written as following:
//Cost function definition public ILRetArray
//Inequality constraint definition public ILRetArray
//Initial guess ILArray
//Minimizer computation ILArray
And on the value of the minimizer is given by
>xopt >
Besides inequality constraints, bounded constraints can be included. The calling sequence will therefore be: xopt=optim(objfunc,x0,InequalityConstraint:ineqConst, lowerBound:a, upperBound:b);
The Lagrangian will be automatically computed and the result will be projected on the corresponding interval.
In addition to inequality constraints, equality constraints and bounded constraints can be combined together.
3.43.43.4 NNNONLINEAR OPTIMIZATIOOPTIMIZATIONN WITH BOUNDED ,,, IIINEQUALITY AND EQUALIEQUALITYTY CONSTRAINTS optim can also be used to solve the standard nonlinear constraint optimization problems defined by
min
Subject to ℎ ≤0, =1,…, , , ≤ ≤ =0, =1,…, where is the objective function to be minimized over the variable : ℝ → ℝ is called inequality constraints , and ℎ: ℝ → ℝ is called equality constraints . : ℝ → ℝ The simplest calling sequence for such problem is as follow: xopt=optim(f,x0, InequalityConstraint: h, EqualityConstraint: g, lowerBound: a, upperBound: b);
3.4.1 Example We would like to minimize the function
= − 1 + + 2 − 4,
= 2 − ,
= − .
The c# function to solve the problem can be written as following:
//Cost function definition public ILRetArray
//Equality constraint function definition public ILRetArray
//Inequality constraint function definition public ILRetArray
// Lower bound constraint ILArray
// Upper bound constraint ILArray
//Initial guess ILArray
//Minimizer computation ILArray
The value of the minimizer is given by
>xopt >
3.5 FEATURES : optim provides an efficient optimization solver based on the robust BFGS algorithm, a Quasi-Newton algorithm, based on updates using a Gauss C-G algorithm , and line-search using the Golden Section Search algorithm and the classical Newton method with computation of the hessian (second derivative). With “big” numbers, optim will manage to
adapt. However, it is recommend to scale or normalise high values of functions to a low floating point precision (around 1.0) . In low dimensions, the classical Newton method is faster than the BFGS approximation the hessian. In both cases, the same line search algorithm and the Newton C-G direction search algorithm are used. In high dimension, the classical Newton method is slower than the BFGS algorithm because the computation of the hessian matrix is too expensive.
444 LLLEAST SQUARE MINIMIZAMINIMIZATIONTION PROBLEM
ILNumerics includes methods for solving nonlinear equations and nonlinear least squares problems. The algorithms proceed either from an analytic specification of the Jacobian matrix or directly from the problem functions. The paths include facilities for systems of equations with a banded Jacobian matrix, for least squares problems with a large amount of data, and for checking the consistency of the Jacobian matrix with the functions.
Given a set of n nonlinear equations in n unknowns, F(x) = 0 , Powell's dog leg method is used to seek a minimizer x. Given a set of m nonlinear functions in n unknowns, the Powell’s dog leg method is used to seek an x which minimizes the L 2 norm of the residual || F(x) || 2.
The user supplies a subroutine to evaluate the nonlinear function; the Jacobian matrix dFi(x)/dx j may also be supplied by the user in a subroutine, or approximated by finite differences.
The Powell’s dog leg method uses the Newton step if it is in the trust region. Otherwise, it finds the Cauchy point which minimizes the objective function along the steepest descent direction. If the Cauchy point is outside the trust region, the truncated Cauchy step is taken as an approximate solution. When the Cauchy point is inside the trust region, the approximate solution is chosen as the intersection of the trust region boundary and the straight line joining the Cauchy point and the Newton step. The piecewise linear curve passing through the origin, the Cauchy point and the Newton step looks like a dog-leg. Therefore the method is called the dog-leg method. optimpdl can be used to solve the standard nonlinear least square optimization problems defined by
min ‖ , − ‖ or more generally
2 min ‖ ‖ where is the vector of objective functions to be minimized over the variable x. : ℝ → ℝ The simplest calling sequence for such problem is as follow:
• xopt=optim(F,x0); • xopt=optim(F,x0,ydata); where F is the function to be minimized, x0 is the starting guess and ydata the measurable data points when not directly included in the objective functional.
4.1.1 Example We would like to minimize the function
= 10 − 1 + + 2
The c# function to solve the problem can be written as follows:
//Cost function definition public ILRetArray
The execution is performed as follows.
//Minimizer computation ILArray
The value of the minimizer is given by
>xopt >
4.1.2 Data fitting
Curve fitting is the process of constructing a curve, or mathematical function that has the best fit to a series of data points, possibly subject to constraints. Curve fitting can involve either interpolation where an exact fit to the data is required, or smoothing, in which a "smooth" function is constructed that approximately fits the data. optimpdl can be used for model fitting as in the previous paragraph. Following, an example of use will be provided.
4.1.2.1 Example Let us consider the example of the Meyer function defined by
= With unknown parameters . , , Data points available are given in the following table:
0 34780.0 8 8261.0 1 28610.0 9 7030.0 2 23650.0 10 6005.0 3 19630.0 11 5147.0 4 16370.0 12 4427.0 5 13720.0 13 3820.0 6 11540.0 14 3307.0 7 9744.0 15 2872.0
The function which gives values of the f(x) is written as follows: public static ILRetArray
The computation of the minimum is done as follows:
// Minimum computation:
ILArray
ILArray
Func
ILArray
//solution
> xm >
555 CCCONCLUSION
Several optimization features are missing in ILNumerics. Here is a list of features which are not available in ILNumerics, but may be included in a later release:
• Integer parameter with linear objective solver and sparse matrices; • Linear objective with or without sparse matrices ; • Quadratic objective solver with sparse objective matrix; • Simplex programming method; • Non-linear objective with nonlinear constraints problems based on sparse linear algebra