Optimization Methods: Optimization using Calculus - Unconstrained Optimization 1

Module – 2 Lecture Notes – 3

Optimization of Functions of Multiple Variables: Unconstrained Optimization

Introduction In the previous lectures we learnt how to determine the convexity and concavity of functions of single and two variables. For functions of single and two variables we also learnt determining stationary points and examining higher derivatives to check for convexity and concavity, and tests were recommended to evaluate stationary points as local minima, local maxima or points of inflection. In this lecture functions of multiple variables, which are more difficult to be analyzed owing to the difficulty in graphical representation and tedious calculations involved in mathematical analysis, will be studied for unconstrained optimization. This is done with the aid of the gradient vector and the Hessian matrix. Examples are discussed to show the implementation of the technique.

Unconstrained optimization If a convex function is to be minimized, the stationary point is the global minimum and analysis is relatively straightforward as discussed earlier. A similar situation exists for maximizing a concave variable function. The necessary and sufficient conditions for the optimization of unconstrained function of several variables are given below. Necessary condition In case of multivariable functions a necessary condition for a stationary point of the function f(X) is that each partial derivative is equal to zero. In other words, each element of the gradient vector defined below must be equal to zero. i.e. the gradient vector of f(X), Δ x f at X=X*, defined as follows, must be equal to zero:

⎡ ∂f ⎤ (*)Χ ⎢ ∂x ⎥ ⎢ 1 ⎥ ⎢ ∂f ⎥ (*)Χ ⎢ ∂x ⎥ ⎢ 2 ⎥ Δ=x f = 0 ⎢ M ⎥ ⎢ ⎥ ⎢ M ⎥ ⎢ ∂f ⎥ ⎢ (*)Χ ⎥ ⎣∂dxn ⎦

D Nagesh Kumar, IISc, Bangalore M2L3

Optimization Methods: Optimization using Calculus - Unconstrained Optimization 2

The proof given for the theorem on necessary condition for single variable optimization can be easily extended to prove the present condition.

Sufficient condition For a stationary point X* to be an extreme point, the matrix of second partial derivatives (Hessian matrix) of f(X) evaluated at X* must be: (i) positive definite when X* is a point of relative minimum, and (ii) negative definite when X* is a relative maximum point. Proof (Formulation of the Hessian matrix) The Taylor’s theorem with reminder after two terms gives us

nndf 1 n∂2 f fhf(*Χ+ ) = (*) Χ +∑∑∑ hii (*) Χ + hhj ii==11dx 2! j=1∂∂x x iijΧ=Χ* +θh 0<θ <1 Since X* is a stationary point, the necessary condition gives df = 0 , i = 1,2,…,n dxi Thus

1 nn ∂2 f fhf(*Χ+ ) − (*) Χ = ∑∑ hhij 2! ij==11 ∂∂xx ijΧ=Χ* +θh For a minimization problem the left hand side of the above expression must be positive. Since the second partial derivative is continuous in the neighborhood of X* the sign of

2 2 ∂∂∂Χ=Χ+f xxij * θ h is the same as the sign of ∂ fxx∂∂ij Χ=Χ* . And hence fhf(*Χ+ ) − (*) Χ will be a relative minimum, if

1 nn ∂2 f fhf(*Χ+ ) − (*) Χ = ∑∑ hhij 2! ij==11 ∂∂xx ijΧ=Χ* is positive. This can also be written in the matrix form as Q = hHhT X=X* where

⎡ ∂2 f ⎤ H = ⎢⎥ X=X* ⎢∂∂xxij ⎥ ⎣ Χ=Χ* ⎦ is the matrix of second partial derivatives and is called the Hessian matrix of f(X).

D Nagesh Kumar, IISc, Bangalore M2L3

Optimization Methods: Optimization using Calculus - Unconstrained Optimization 3

Q will be positive for all h if and only if H is positive definite at X=X*. i.e. the sufficient condition for X* to be a relative minimum is that the Hessian matrix evaluated at the same point is positive definite, which completes the proof for the minimization case. In a similar manner, it can be proved that the Hessian matrix will be negative definite if X* is a point of relative maximum. A matrix A will be positive definite if all its eigenvalues are positive. i.e. all values of λ that satisfy the equation AI− λ = 0

should be positive. Similarly, the matrix A will be negative definite if its eigenvalues are negative. When some eigenvalues are positive and some are negative then matrix A is neither positive definite or negative definite. When all eigenvalues are negative for all possible values of X, then X* is a global maximum, and when all eigenvalues are positive for all possible values of X, then X* is a global minimum. If some of the eigenvalues of the Hessian at X* are positive and some negative, or if some are zero, the stationary point, X*, is neither a local maximum nor a local minimum.

Example

222 Analyze the function f() x=−−−+ x123 x x2 xx 12131 + 2 xx + 4 x − 5 x 3 + 2 and classify the stationary points as maxima, minima and points of inflection. Solution ⎡⎤∂f ⎢⎥(*)Χ ∂x ⎢⎥1 ⎡−+2224xxx + +⎤⎡⎤ 0 ⎢⎥∂f 123 Δ=fx(*) Χ =⎢ − 2 + 2x⎥⎢⎥ = 0 x ⎢⎥⎢ 21⎥⎢⎥ ⎢⎥∂x2 ⎣⎢ −+−225xx31 ⎦⎣⎦⎥⎢⎥ 0 ⎢⎥∂f ⎢⎥(*)Χ ⎣⎦∂x3 Solving these simultaneous equations we get X*=[1/2, 1/2, -2] ∂∂∂222fff = −=−=2; 2;− 2 ∂∂∂xxx222 123 ∂∂22ff ==2 ∂∂xx12 ∂∂ xx 21

D Nagesh Kumar, IISc, Bangalore M2L3

Optimization Methods: Optimization using Calculus - Unconstrained Optimization 4

∂∂22ff = = 0 ∂∂xx ∂∂ xx 23 32 ∂∂22ff = = 2 ∂∂xx31 ∂∂ xx 13 Hessian of f(X) is

⎡ ∂2 f ⎤ H = ⎢ ⎥ ⎣⎢∂xij∂x ⎦⎥ ⎡−22 2⎤ ⎢ ⎥ H =−⎢ 220⎥ ⎣⎢ 20− 2⎦⎥ λ +−22 − 2 λλI-H = −+220 =0 −+20λ 2

or (λ +++−++ 2)(λλ 2)( 2) 2( λ 2)(2) 2(2)( λ += 2) 0

(2)[4444]λλλ+++−+=2 0

(2)λ + 3 = 0

or λ123= λλ==−2 Since all eigenvalues are negative the function attains a maximum at the point X*=[1/2, 1/2, -2]

D Nagesh Kumar, IISc, Bangalore M2L3