Contents 1 Definiteness of a Matrix

Constantin B¨urgi Optimization 1

Contents

1 Definiteness of a Matrix 1 1.1 Definitions ...... 1 1.2 Finding the definiteness of a symmetric Matrix ...... 3

2 Concave and Convex functions 4

3 Unconstrained optimization 7 3.1 Deﬁnitions ...... 7 3.2 Existence of maxima ...... 7 3.3 Necessary condition ...... 8 3.4 Suﬃcient condition ...... 8

4 Constrained optimization 10 4.1 Equality constraints ...... 10 4.1.1 Marginal Rate of Substitution ...... 11 4.1.2 The Lagrangian ...... 13 4.1.3 Constraint qualification ...... 13 4.1.4 Sufficiency ...... 15 4.1.5 Linear optimization ...... 16 4.1.6 Monotonic increasing transformations ...... 16 4.1.7 Non-differentiability ...... 16 4.2 Inequality constraints ...... 17 4.2.1 Kuhn-Tucker Conditions ...... 17

1 Deﬁniteness of a Matrix

1.1 Deﬁnitions

Previously, it was shown, that any linear system has a matrix representation. This is also true for any polynomial of degree two. The representation is called quadratic form, as any variable is at most quadratic. The general quadratic forms is

n X 0 q(x1, ..., xn) = aijxixj = x Ax i,j=1 Constantin B¨urgi Optimization 2

2 2 Example 1. a11x1 + a12x1x2 + a22x2 has the corresponding quadratic form

" a12 #" # h i a11 2 x1 x1 x2 a12 2 a22 x2

Remark 1. For x0Ax to be a quadratic form, A needs to be symmetric.

These quadratic forms can always be (strictly) positive, negative or changing their sign depending on the symmetric matrix A. Based on this, it can be deﬁned that

Deﬁnition 1. A symmetric matrix A is positive (negative) deﬁnite, if the quadratic form x0Ax > 0 (x0Ax < 0) ∀x 6= 0.

Pn 2 Example 2. In is positive deﬁnite, as it corresponds to the quadratic form i=1 xi .

Similarly, −In is negative deﬁnite.

Deﬁnition 2. A symmetric matrix A is positive (negative) semi-deﬁnite, if the quadratic form x0Ax ≥ 0 (x0Ax ≤ 0) ∀x and ∃x for which it holds with strict inequality.

Remark 2. Every positive (negative) deﬁnite matrix is also positive (negative) semi- deﬁnite.

Example 3. The matrix " # 1 1 B = 1 1

2 is positive semi-deﬁnite, as it corresponds to the quadratic form (x1 + x2) and it is 0 for x1 = −x2. Similarly, −B is negative semi-deﬁnite.

Deﬁnition 3. A symmetric matrix A is indeﬁnite, if ∃x such that the quadratic form x0Ax takes positive and negative values, or if A = 0.

Example 4. The matrix " # 1 0 0 −1

2 2 is indeﬁnite, as it corresponds to the quadratic form x1 − x2, which is positive for large x1 and negative for large x2.

The deﬁnitions above can be easily expanded to non-symmetric matrices, by noting that the quadratic form underlying any matrix B, is equivalent to the quadratic form 1 0 underlying the symmetric matrix A = 2 (B + B ). Constantin B¨urgi Optimization 3

1.2 Finding the deﬁniteness of a symmetric Matrix

The two approaches to determine the deﬁniteness of a symmetric matrix discussed here are using eigenvalues and principal minors. Looking at eigenvalues, a matrix is positive

(negative) deﬁnite, if all its eigenvalues λi > (<)0. Similarly, a matrix is positive (negative) semi-deﬁnite, if all its eigenvalues λi ≥ (≤)0. Often, calculating eigenvalues might be tedious and using principal minors is easier.

Deﬁnition 4. A kth order principal minor denoted PMk(A) is the determinant of a k ×k submatrix of the n × n matrix A, where (n − k) rows and corresponding columns were deleted.

Example 5. A 3 × 3 matrix has 7 principal minors. The three diagonal elements are ﬁrst order principal minors and the determinant of the entire matrix is the third order principal minor. Deleting the ﬁrst, second and third row and column individually and calculating the determinant of the remaining matrix give the three second order principal minors.

Deﬁnition 5. A kth order leading principal minor denoted LP Mk(A) is given by the determinant of the k × k submatrix of a n × n matrix A, where the last (n − k) rows and columns were deleted.

Remark 3. Every n × n matrix has n leading principal minors.

Example 6. The three leading principal minors of a 3 × 3 matrix A are

a11 a12 a13 a11 a12 a11; ; a21 a22 a23 a a 21 22 a31 a32 a33 Theorem 1.1. A matrix A is positive (semi-)deﬁnite, if for all its leading principle minors

(principle minors) LP Mk(A) > 0 (PMk(A) ≥ 0).

Theorem 1.2. A matrix A is negative (semi-)deﬁnite, if for all its leading principle k k minors (principle minors) (−1) LP Mk(A) > 0 ((−1) PMk(A) ≥ 0).

Remark 4. As with the deﬁnitions for semi-deﬁniteness, the inequalities have to hold with strict inequality for at least some principle minors.

Example 7. The leading principal minors of the matrix

1 2 1

2 4 2

1 2 1 Constantin B¨urgi Optimization 4

are LP M1 = 1, LP M2 = 0 and LP M3 = 0, hence the other principal minors need to be checked. Because all other ﬁrst order principle minors are positive and all other principle minors 0, the matrix is positive semi-deﬁnite.

2 Concave and Convex functions

Deﬁnition 6. A function f : S → R deﬁned over the convex set S is concave (convex) if for all vectors x, y ∈ S,

f(tx + (1 − t)y) ≥ (≤)tf(x) + (1 − t)f(y), ∀t ∈ [0, 1].

The function is strictly concave (convex), if the condition holds with strict inequality. Intuitively, this means that a concave (convex) function is weakly above (below) the line of two points of that function. A linear function is both concave and convex.

f(x) f(x)

f(y) f(x)

x x x y x y Concave Convex

Theorem 2.1. A twice continuously differentiable function f : S → R defined over the convex set S is concave (convex), if and only if the Hessian is negative (positive) semi- definite ∀x ∈ S.

Theorem 2.2. A twice continuously differentiable function f : S → R defined over the convex set S is strictly concave (convex), if the Hessian is negative (positive) definite ∀x ∈ S.

Example 8. The function f(x) = x2 is strictly convex, as f 00(x) = 2 > 0.

Remark 5. The converse for the second theorem is not true however, as the example of x4 (−x4) shows. This function is strictly convex (concave), however its Hessian is only positive (negative) semi-deﬁnite as it becomes 0 at x = 0. Constantin B¨urgi Optimization 5

A useful property of concave and convex function is given by the next theorem.

Theorem 2.3. If f(x) is a concave function, f(x) ≤ f(x) + 5f(x)0[x − x], ∀x in the domain of the function.

Proof. Deﬁne z = x − x. Then the deﬁnition of a concave function can be rewritten as

f(tx + (1 − t)x) ≥ tf(x) + (1 − t)f(x) f(x + tz) ≥ tf(x) + (1 − t)f(x) f(x + tz) − f(x) ≥ tf(x) − tf(x) f(x + tz) − f(x) ≥ f(x) − f(x) | lim t t→0 5f(x)0[z] ≥ f(x) − f(x) f(x) ≤ f(x) + 5f(x)0[x − x]

f(x) f(x)

x x

Quasi-Concave Quasi-Convex

Deﬁnition 7. A function f : S → R deﬁned over the convex set S is quasi-concave if for all vectors x, y ∈ S,

f(tx + (1 − t)y) ≥ min[f(x), f(y)], ∀t ∈ [0, 1].

Deﬁnition 8. A function f : S → R deﬁned over the convex set S is quasi-convex if for all vectors x, y ∈ S,

f(tx + (1 − t)y) ≤ max[f(x), f(y)], ∀t ∈ [0, 1]. Constantin B¨urgi Optimization 6

f(x)

Neither

Remark 6. There are equivalent deﬁnitions for quasi-concave (quasi-convex) functions using subsets of the function. If the upper (lower) contour set of a function is a convex set, the function is quasi-concave (quasi-convex). More formally, a function is quasi-concave (quasi-convex), if the set

Sa = {x|f(x) ≥ (≤)a} is convex for all a ∈ R. Similarly as before, for strict quasi-convex/quasi-concave functions the conditions have to hold with strict inequality. Functions that are (weakly) increasing/decreasing are both quasi-convex and quasi-concave (quasi-linear).

Example 9. The function f(x) = x3 is both quasi-convex and quasi-concave, as it is an increasing function but neither concave or convex.

Aside from these quasi-linear functions, quasi-convex or quasi-concave functions intuitively have some point where the function reaches a global maximum or minimum. There are suﬃcient conditions for quasi-concave (quasi-convex) functions based on the bordered Hessian. The bordered Hessian has the Hessian matrix in the bottom right corner, the ﬁrst order conditions on both sides of the Hessian and a 0 in the top left corner.

" # 0 5f(x)0 BH(f(·)) = 5f(x) H(f(x)) Theorem 2.4. A function is quasi-concave, if the leading principle minors of the bordered ( Hessian alternate their sign according to (−1) k + 1)LP Mk(BH(f(·))) > 0 for k > 1.

Theorem 2.5. A function is quasi-convex, if all but the ﬁrst leading principle minors of the bordered Hessian are negative LP Mk(BH(f(·))) < 0 for k > 1. Constantin B¨urgi Optimization 7

Remark 7. If the above conditions hold with weak inequality, they are only necessary but not suﬃcient.

3 Unconstrained optimization

Unconstrained optimization assumes that there is no constraint to the optimization problem aside from simple restrictions on the domain.

3.1 Deﬁnitions

∗ n Deﬁnition 9.x ∈ R is a local maximizer of f(x), if there exists an open ε-ball Oε ∗ ∗ ∗ around x , f(x ) ≥ f(x), ∀x ∈ Oε. If f(x ) ≥ f(x) holds for the entire domain, it is a global maximizer

Remark 8.x ∗ is a strict local/global maximizer, if the above condition holds with strict inequality (>).

Remark 9. Any minimization problem can be turned into a maximization problem by changing the objective function f(x) to −f(x).

Example 10. The strict global maximum of f(x) = 2 − x2 is x = 0 ⇒ f(0) = 2.

3.2 Existence of maxima

There is a set of conditions that guarantees that a function has a maximum and a minimum. These conditions are also called the extreme value theorem.

Theorem 3.1. Suppose f(x) is a continuous function deﬁned over the compact (closed and bounded) set C ⊆ Rn, then there exists a point x ∈ C for which f(·) attains a maximum and a point x ∈ C for which f(·) attains a minimum.

Remark 10. If some of these conditions are not satisﬁed, functions might only have a supremum and inﬁmum, but no maximum or minimum.

Example 11. The function f(x) = x2 has a minimum for x ∈ R, but no maximum (but still a supremum). Does this change with restricting the domain to x ∈ (0, 1)?

Example 12. The function f(x) is deﬁned over x ∈ [−1, 2] and takes the form  x2 x 6= 0 1 x = 0 Does it have a minimum or maximum? Constantin B¨urgi Optimization 8

Remark 11. Any minimization problem can be transformed into a maximization problem by multiplying the objective function by -1.

3.3 Necessary condition

There are two possibilities for a maximum. Either the maximum lies at the boundary of the set C or it is at an interior point. Assuming the (local) maximizer is an interior point and the function is twice continuously diﬀerentiable, the slope at this point must be 0 from every direction. More formally, this gives the necessary ﬁrst order condition (FOC) that for any interior maximizer x∗

 ∂f(x∗)  ∂x1 ∗  .  5f(x ) =  .  = 0   ∂f(x∗) ∂xn Example 13. Assume a function f(x) where x ∈ R as shown in the graphs below. If

f(x) f(x) f 0(x0) < 0 f 0(x0) > 0

x x x0 x0 x0 was a maximizer, the previous condition required either a slope of 0 or boundary solution. Excluding the boundary solution, it is easy to see from the graph, that x0 is not a maximizer, because to the left (right) of that point, the function is still increasing, as f 0(x0) < 0 (f 0(x0) > 0).

3.4 Suﬃcient condition

While every interior maximizer x∗ will require that the gradient at x∗ is 0, the converse does not have to hold. There are interior points, where the gradient is 0, but they are not maximizers.

Example 14. Let f(x) = x3 for x ∈ R. f 0(0) = 0 however, the point is a saddle point and hence neither a maximizer nor a minimizer. Constantin B¨urgi Optimization 9

Also, even if all non-extrema points can be excluded, it is required to know if it is a maximum or minimum. This leads to the converse or suﬃcient second order condition, which puts restrictions on the Hessian denoted Hf(x).

Theorem 3.2. If 5f(x∗) = 0 and Hf(x∗) is negative (positive) deﬁnite at some x∗, f(·) has a strict local maximum (minimum) at x∗.

Proof. A second degree Taylor approximation around x, where ε ∈ Rn is

1 f(x∗ + ε) − f(x∗) = 5f(x∗)0ε + ε0Hf(x∗)ε + Remainder 2 1 ≈ ε0Hf(x∗)ε 2 for small kεk and due to the first order condition. But since ε0Hf(x∗)ε is a quadratic form, restrictions on the Hessian are sufficient to determine the sign of the right hand side. If the Hessian is positive definite, the expression to the right is positive and f(x∗ + ε) > f(x∗) and makes x∗ a strict local minimizer. Conversely, if the Hessian is negative definite, f(x∗ + ε) < f(x∗) which makes x∗ a strict local maximizer. It is not a global minimizer/maximizer, because kεk is small.

Theorem 3.3. If 5f(x∗) = 0 and Hf(x) is negative (positive) semi-deﬁnite ∀x over the convex domain of f(·), f(·) has a global maximum (minimum) at x∗.

Proof. The semi-deﬁniteness of the Hessian ensures a concave (convex) function. Due to this, f(x) ≤ (≥)f(x∗) + 5f(x∗)0[x − x∗], ∀x. But since 5f(x∗) = 0, the expression simpliﬁes to f(x) ≤ (≥)f(x∗)

As seen in the previous section, this implies that the function is concave/convex, hence this property ensures that it is suﬃcient to check the ﬁrst order (necessary) condition.

Remark 12. The sufficiency theorems only go in one direction, but not in the other. Thus there might be extrema that are not classified as such, because the Hessian is indefinite at that point (and hence failing conditions of both theorems). An example for this would be f(x) = x4, which has a minimum at x = 0, but the Hessian at that point is indefinite (0).

Example 15. Let f(x, y) = x3 − y3 + 12xy, R2 → R be the function to be maximized. The first order condition (necessary) condition giving critical points is ! 3x2 + 12y 5f(x, y) = , −3y2 + 12x Constantin Bürgi Optimization 10 which is 0 for the points (0,0) and (4,-4). The Hessian is given by " # 6x 12 Hf(x, y) = , 12 −6y which is indefinite, as LP M1 ≶ 0 and hence the function is neither concave nor convex.

LP M2(Hf(0, 0)) = −144 and (0,0) cannot be classiﬁed as a maximum or minimum, while 2 2 LP M1(Hf(4, −4)) = 24 > 0 and LP M2(Hf(4, −4)) = 24 − 12 > 0 making Hf(4, −4) positive deﬁnite and (4,-4) a minimizer.

4 Constrained optimization

In general, constrained optimization takes the form

max f(x) x∈Rn

s.t. g1(x) = b1 . .

gm(x) = bm

h1(x) ≤ c1 . .

hl(x) ≤ cl

Where f(x) is the objective function to be maximized, gi(x) = bi; i = 1, ..., m are the equality constraints and hj(x) = cj; j = 1, ..., l the inequality constraints.

4.1 Equality constraints

Many constrained optimization problems with few variables and few equality constraints (but no inequality constraints) can be treated in a similar way as unconstrained optimization. If the constraints are linear, one can eliminate variables using the constraints and turn the constrained maximization problem into an unconstrained problem.

Example 16. Let the constrained maximization problem be f(x1, x2) = x1x2 and the only constraint be g(x1, x2) = x1 + x2 = 3. Then it is easy to solve the constraint for x1 and replacing it in the objective function to turn this constrained optimization problem into an unconstrained one: max f(x) = x(3 − x) = 3x − x2 x∈R Constantin B¨urgi Optimization 11

Taking ﬁrst order (necessary) conditions, this function becomes

f 0(x) = 3 − 2x = 0 and there is one critical point x = 3/2. Checking the second order (suﬃcient) condition,

f 00(x) = −2 < 0 and this point is a maximum. Note that it is a strict global maximum, because the function is concave, and corner solutions can be excluded due to the multiplicative objective function. With the help of the constraint, x1 can be recovered and the solution is x1 = x2 = 3/2.

4.1.1 Marginal Rate of Substitution

If the constraint is not linear, it might not be possible to turn a constrained optimization problem into an unconstrained problem this easily. Due to this, a more general solution method needs to be introduced. Consider the utility maximization problem with a constraint

max u(x) x∈Rn s.t. g(x) = b

In the two goods case, where the constraint is the budget line and excluding corner solutions, it is known that the highest indiﬀerence line must be tangent at the budget line for utility maximization. Graphically, this point corresponds to

∗ x2 ∗ ∗ u(x1, x2)

x1 ∗ x1

The slope of these indifference curves called marginal rate of substitution (MRS) must be set equal to the slope of the budget line, which is the price ratio. There are multiple indifference curves that have that slope, some outside the budget set and some inside. To ensure that the correct indifference curve on the budget line is chosen, the constraint must hold with equality as well. Constantin Bürgi Optimization 12

In the n dimensional case with general constraint, bilateral marginal rates of substitution can be constructed by total diﬀerentiation of the objective function

5u(x)0dx = 0 which leads to the marginal rate of substitution between good i and j ∂u(x) dxi ∂xj MUj − = ∂u(x) = = MRSji dxj MUi ∂xi Similarly for the constraint, total diﬀerentiation gives

5g(x)0dx = 0 which leads to ∂g(x) dxi ∂xj − = ∂g(x) dxj ∂xi setting them equal, ∂u(x) ∂g(x) ∂xj ∂xj ∂u(x) = ∂g(x) ∂xi ∂xi rearranging ∂u(x) ∂u(x) ∂xj ∂xi ∂g(x) = ∂g(x) = λ, ∂xj ∂xi which has to hold for all i, j and some constant λ. Separating this expression into n equations, and adding the constraint, the full set of conditions for an interior optimum become ∂u(x) ∂g(x) − λ = 0 ∂x1 ∂x1 . . ∂u(x) ∂g(x) − λ = 0 ∂xn ∂xn g(x) − b = 0

Note that it is still possible to have corner solutions as the following example shows

Example 17. Let the objective function with positive domain be f(x1, x2) = 2x1 + x2 and the constraint g(x1, x2) = x1 + x2 = 3. Then the optimal interior solution would need to satisfy ∂f(x) ∂g(x) 2 1 ∂x1 = = = ∂x1 , ∂f(x) 1 1 ∂g(x) ∂x2 ∂x2 plus the constraint, which is impossible. Hence it is necessary to check corner solutions as well, which easily show that x1 = 3; x2 = 0 is the maximum. Constantin B¨urgi Optimization 13

4.1.2 The Lagrangian

As seen in the previous section, every constrained maximization problem with interior solution and with equality constraints can be transformed into a system of n+m equations, where n involve ﬁrst order conditions of the constraints and the objective function and the m constraints. The Lagrangian transforms all n + m equations into an unconstrained optimization problem with n + m unknowns by adding 0 to the objective function. In particular, it takes the form

L(x, λ1, ..., λm) = f(x) − λ1(g1(x) − b1) − ... − λm(gm(x) − bm).

By construction, 0 was added to the objective function, as all constraints are binding. At the same time, the ﬁrst order conditions of this function with respect to x and the

Lagrange multipliers λ1, ..., λm will lead to the set of equations necessary for an interior solution.

Example 18. Assume Cobb-Douglas utility u(x, y) = xy and the budget constraint pxx + pyy = m. Due to the form of utility, corner solutions can be ruled out an only interior solutions are optimal. Then the Lagrangian becomes

L(x, y, λ) = xy − λ(pxx + pyy − m)

The corresponding ﬁrst order conditions are ∂L(x, y, λ) = y − λp = 0 ∂x x ∂L(x, y, λ) = x − λp = 0 ∂y y ∂L(x, y, λ) = p x + p y − m = 0 ∂λ x y Solving for the variables and assuming only positive x,y gives x = m , y = m and λ = 2px 2py m . 2pxpy

4.1.3 Constraint qualiﬁcation

There is a regularity condition necessary, such that the first order conditions are all equal to 0 for interior solutions. The constraint qualification ensures that the Lagrangian works in those cases. For the constraint qualification to hold, the matrix with the first order conditions of all m binding constraints with respect to all n variables must have rank min[m, n] at all points. This ensures that the gradients at the optimal point are the same for the constraints and the objective function and thus all FOCs are equal to 0. Constantin Bürgi Optimization 14

If the constraint qualiﬁcation fails for a given point, that point might be a maximum or minimum of the optimization problem as well and needs to be checked separately.

Example 19. Assume one wants to maximize the objective function f(x, y) = x with the constraint g(x, y) = y2 + (1 − x)2 = 0. This leads to the Lagrangian

L(x, y, λ) = x − λ(y2 + (1 − x)2)

Taking ﬁrst order conditions ∂L(x, y, λ) = 1 − 2λ(1 − x) = 0 ∂x ∂L(x, y, λ) = −2λy = 0 ∂y ∂L(x, y, λ) = y2 + (1 − x)2 = 0 ∂λ From the constraint, it is clear that x = 1 and y = 0 should be the unique solutions to the problem, which also satisfy the last two FOCs. However, @λ such that the ﬁrst FOC = 0! Looking at the gradient of the constraint

5g(x, y) = [2(1 − x), 2y] which has rank 0 at the optimal point (1, 0). Thus at least 2 more independent constraints are required for all ﬁrst order conditions to be equal to 0.

A satisfied constraint qualification does not imply that boundary solutions will result in all first order conditions being equal to 0.

Example 20. Assume one wants to maximize the objective function f(x, y) = x2 + y2 with the constraint g(x, y) = 2x + 3y = 5 and x, y ∈ R+. This leads to the Lagrangian

L(x, y, λ) = x2 + y2 − λ(2x + 3y − 5)

Taking first order conditions ∂L(x, y, λ) = 2x − 2λ = 0 ∂x ∂L(x, y, λ) = 2y − 3λ = 0 ∂y ∂L(x, y, λ) = 2x + 3y − 5 = 0 ∂λ 10 15 This leads to the unique solution of x = λ = 13 and y = 13 , which is a minimum and not the maximum. Due to linearity, the constraint qualification trivially holds, and the 5 maximum is x = 2 and y = 0. Constantin Bürgi Optimization 15

4.1.4 Suﬃciency

Similar to unconstrained optimization, the Lagrangian needs to satisfy several conditions to ensure the optimality of solutions.

Theorem 4.1. If the objective function f(x) is concave (convex) and the constraints gi(x) − bi; i = 1, ..., m convex (concave), then the Lagrangian

m X L(x, λ1, ..., λm) = f(x) − λi(gi(x) − bi) i=1 is concave (convex).

As the critical point of a concave (convex) function is a maximum (minimum), this is suﬃcient (see Theorem 3.3) A more complicated condition is necessary, if just the critical point is to be checked. Firstly, the bordered Hessian needs to be constructed. It has the Hessian of the Lagrangian with respect to the variables of the objective function in the bottom right corner, 0s in the top left and the ﬁrst order conditions of the binding constraints in the other corners. " # 0 5g (x∗)0 BH(L(·)) = E ∗ ∗ 5gE(x ) D2L(x )

Theorem 4.2. Let m denote the number of binding constraints and n the size of the vector x. Then n + m > 2m is the size of the bordered Hessian at x∗. Then x∗ is a minimum (maximum), if the last n − m leading principal minors all have the sign of (−1)m (alter their sign, where the smallest to check has the sign (−1)m+1).

Remark 13. If m = 0, the theorem reduces to the rule for the normal Hessian. Also, it cannot be determined, if a point is a maximum or minimum in a two variable case with two binding constraints using this method.

If there is one constraint and two variables, it is suﬃcient to check the determinant of the bordered Hessian. If the determinant is positive, it is a maximum, if it is negative, it is a minimum.

Remark 14. Similarly to unconstrained optimization, sufficiency conditions might determine a point to be indefinite, even if it is a maximum or minimum. Constantin Bürgi Optimization 16

4.1.5 Linear optimization

If both the constraint and the objective function are linear, there is an easier solution method than the Lagrangian. In particular, this always leads to corner solutions, provided the gradient of the constraints and the gradient of the objective function are linearly independent. If the gradients are linearly dependent, there is an inﬁnite number of solutions to the optimization problem.

4.1.6 Monotonic increasing transformations

An important property of optimization is that any monotonic and strictly increasing transformation of the objective function will lead to the same solution. This can simplify the constrained optimization problem by rendering the objective function additive. In particular, optimizing (h ◦ f)(x) = h(f(x)) or f(x) will lead to the same optimal value, provided h(·) is a strictly monotonic increasing function.

c d Example 21. Assume a general Cobb-Douglas function of the form u(x1, x2) = x1x2. In this case, it might be more convenient to use the log of the objective function ⇒ u(x1, x2) = cln(x1) + dln(x2). Note that this is a concave transformation of a concave function and hence will give the same equilibrium points.

4.1.7 Non-diﬀerentiability

If the objective function or the constraint are not diﬀerentiable, other methods need to be found to solve the constrained optimization problem. The most common case is some variation of the example below.

Example 22. Leontief preferences show perfect complementarity among goods. They take the form u(x1, x2) = min[ax1, bx2] and are not diﬀerentiable. They were used before Cobb-

Douglas preferences became prevalent. To maximize, one simply sets ax1 = bx2 and makes sure that the constraint is binding. Constantin B¨urgi Optimization 17

4.2 Inequality constraints

Consider the utility maximization problem with the constraints

2 2 max u(x1, x2) = x1x2 x1,x2

s.t. p1x1 + p2x2 ≤ m

x1 ≥ 0

x2 ≥ 0

+ The objective function is quasi-linear for x1, x2 ∈ R , but not over the entire real line. If negative values of were allowed, one could make x1, x2 < 0 and arbitrarily small, satisfying the budget constraint. The Lagrangian can be used to solve this problem as well, but some further restrictions are necessary.

4.2.1 Kuhn-Tucker Conditions

The Lagrangian for both equality and inequality constraints becomes

m l X X L(x, λ1, ..., λm, µ1, ..., µl) = f(x) − λi(gi(x) − bi) − µj(hj(x) − cj) i=1 i=1 given that still 0 should be added to the objective function, there is an issue with the inequality constraints, as they might not hold with equality. To circumvent this issue,

µj = 0 is deﬁned to hold, if the constraint is not binding. This leads to the Kuhn-Tucker conditions

m l X X 5L(·) = 5f(x) − λi 5 gi(x) − µj 5 hj(x) = 0 i=1 i=1

gi(x) − bi = 0; i = 1, ..., m

µj(hj(x) − cj) = 0; j = 1, ..., l

µj ≥ 0; j = 1, ..., l

Where the last two conditions are referred to as complementary slackness and dual feasi- bility respectively as well.

Remark 15. As it might not always be clear, which constraint is binding and which one is not, all possible combinations of binding and non-binding constraints need to be checked, unless some combinations can be excluded. Constantin B¨urgi Optimization 18

Example 23. Looking at the problem from the beginning of this section

2 2 max u(x1, x2) = x1x2 x1,x2

s.t. p1x1 + p2x2 ≤ m ⇒ µ1

x1 ≥ 0 ⇒ µ2

x2 ≥ 0 ⇒ µ3

The conditions become

∂L(·) 2 = 2x1x2 − µ1p1 + µ2 = 0 ∂x1

∂L(·) 2 = 2x1x2 − µ1p2 + µ3 = 0 ∂x2

µ1(p1x1 + p2x2 − m) = 0

−µ2x1 = 0

−µ3x2 = 0

due to the objective function, it is known that corner solutions are not maxima, as having both x1 and x2 non-zero gives a higher value for the objective function than having either or both equal to 0. Hence it is known that the solution will be an interior point and

µ2, µ3 = 0. Similarly, it is known that the objective function is strictly increasing in both x1 and x2, and thus the budget constraint has to be binding and µ1 6= 0. From the other three conditions, one can solve for x1, x2 and µ1 to get the standard Cobb-Douglas solution m x1 = 2p1 m x2 = 2p2 m3 µ1 = 2 2 4p1p2 Creating the bordered Hessian and evaluating at x∗

0 p p 0 p1 p2 1 2 2 2 2 m2 m2 2 m m 2 BH(L(·)) = p1 2x2 4x1x2 = p1 2p2 p p = 2m − − = m > 0 2 1 2 2 2 2 m2 m2 p2 4x1x2 2x1 p2 2 p1p2 2p1 and it is a maximum.