Numericaloptimization

Home , AMPL, CPLEX, FICO Xpress, Gurobi, IPOPT, Pyomo

Numerical Optimization

Alberto Bemporad http://cse.lab.imtlucca.it/~bemporad/teaching/numopt

Academic year 2020-2021 Course objectives

Solve complex decision problems by using numerical optimization

Application domains:

• Finance, management science, economics (portfolio optimization, business analytics, investment plans, resource allocation, logistics, ...)

• Engineering (engineering design, process optimization, embedded control, ...)

• Artificial intelligence (machine learning, data science, autonomous driving, ...)

• Myriads of other applications (transportation, smart grids, water networks, sports scheduling, health-care, oil & gas, space, ...)

What this course is about:

• How to formulate a decision problem as a numerical optimization problem? (modeling)

• Which numerical algorithm is most appropriate to solve the problem? (algorithms)

• What’s the theory behind the algorithm? (theory)

• Optimization modeling – Linear models

– Convex models

• Optimization theory – Optimality conditions, sensitivity analysis

– Duality

• Optimization algorithms – Basics of numerical linear algebra

– Convex programming

– Nonlinear programming

• Stephen Boyd’s “Convex Optimization” courses at Stanford: http://ee364a.stanford.edu http://ee364b.stanford.edu

• Lieven Vandenberghe’s courses at UCLA: http://www.seas.ucla.edu/~vandenbe/

• For more tutorials/books see http://plato.asu.edu/sub/tutorials.html

• Optimization = assign values to a set of decision variables so to optimize a certain objective function

• Example: Which is the best velocity to minimize fuel consumption ?

fuel [ℓ/km]

velocity [km/h] 0 30 60 90 120 160

• Optimization = assign values to a set of decision variables so to optimize a certain objective function

• Example: Which is the best velocity to minimize fuel consumption ?

best fuel fuel consumption [ℓ/km] optimal velocity velocity [km/h] 0 30 60 90 120 160

optimization variable: velocity cost function to minimize: fuel consumption parameters of the decision problem: engine type, chassis shape, gear, …

min f(x) x f(x*) x ∗ x* f = minx f(x) = optimal value ∗ n n x = arg minx f(x) = optimizer x ∈ R , f : R → R   x  1  maxx f(x)  .  x =  .  , f(x) = f(x1, x2, . . . , xn)

Most often the problem is difficult to solve by inspection use a numerical solver implementing an optimization algorithm

min f(x) x

• The objective function f : Rn → R models our goal: minimize (or maximize) some quantity.

For example fuel, money, distance from a target, etc.

• The optimization vector x ∈ Rn is the vector of optimization variables

(or unknowns) xi to be decided optimally.

For example velocity, number of assets in a portfolio, voltage applied to a motor, etc.

©2021 A. Bemporad - Numerical Optimization 10/102 Constrained optimization problem • The optimization vector x may not be completely free, but rather restricted to a feasible set X ⊆ Rn • Example: the velocity must be smaller than 60 km/h

fuel [ℓ/km]

velocity [km/h] 0 30 60 90 120 160

best fuel fuel consumption [ℓ/km] optimal velocity velocity [km/h] 0 30 60 90 120 160

f(x) minx f(x) s.t. g(x) ≤ 0 h(x) = 0 g(x) 0  x

• The (in)equalities define the feasible set X of admissible variables g : Rn → Rm, h : Rn → Rp " # g (x ,x ,...,x ) X = {x ∈ Rn : g(x) ≤ 0, h(x) = 0} 1 1 2 n g(x) = . . • Further constraints may restrict X , gm(x1,x2,...,xn) " # for example: h1(x1,x2,...,xn) n h(x) = . x ∈ {0, 1} (x = binary vector) . x ∈ Zn (x = integer vector) hp(x1,x2,...,xn)

max f(x) = − min{−f(x)} x∈X x∈X

• Similarly, an inequality gi(x) ≥ 0 is equivalent to −gi(x) ≤ 0

• An equality h(x) = 0 is equivalent to the double inequalities h(x) ≤ 0, −h(x) ≤ 0 (often this is only good in theory, but not numerically)

• Scaling f(x) to αf(x) and/or gi(x) to βigi(x), or shifting to f(x) + γ, does not change the optimizer, for all α, βi > 0 and γ. Same if hj(x) is scaled to γjhj(x), ∀γj ≠ 0

• Adding constraints makes the objective worse or equal: min f(x) ≤ min f(x) x∈X1 x∈X1, x∈X2

• Strict inequalities gi(x) < 0 can be approximated by gi(x) ≤ −ϵ (0 < ϵ ≪ 1)

• A vector x ∈ Rn is feasible if x ∈ X , i.e., it satisfies the given constraints

• A problem is infeasible if X = ∅ (the constraints are too tight)

• A problem is unbounded if ∀M > 0 ∃x ∈ X such that f(x) < −M. In this case we write inf f(x) = −∞ x∈X

• A vector x∗ ∈ Rn is a global optimizer if x ∈ X and f(x) ≥ f(x∗), ∀x ∈ X

• A vector x∗ ∈ Rn is a strict global optimizer if x∗ ∈ X and f(x) > f(x∗), ∀x ∈ X , x ≠ x∗

• A vector x∗ ∈ Rn is a (strict) local optimizer if x∗ ∈ X and there exists a neighborhood1 N of x∗ such that f(x) ≥ f(x∗), ∀x ∈ X ∩ N (f(x) > f(x∗), ∀x ∈ X ∩ N , x ≠ x∗)

1Neighborhood of x = open set containing x

• We have a dataset (uk, yk), uk, yk ∈ R, k = 1,...N

• We want to fit a line yˆ = au + b to the dataset that minimizes   " y # 2 XN XN u1 1 1 ′ . 2 uk 2  . .  . f(x) = (yk − auk − b) = ([ ] x − yk) = . . x − . 1 . . k=1 k=1 uN 1 yN 2 a with respect to x = [ b ] a∗ a ∗ ∗ • The problem b∗ = arg min f([ b ]) is a least-squares problem: yˆ = a u + b In MATLAB: 1.5 1

x=[u ones(size(u))]\y 0.5

y -0.5 In Python: -1

import numpy as np -1.5 -1 -0.5 0 0.5 1 A=np.hstack((u,np.ones(u.shape))) u x=np.linalg.lstsq(A,y,rcond=0)[0]

©2021 A. Bemporad - Numerical Optimization 16/102 Least Squares using Basis Functions • More generally: we can fit nonlinear functions y = f(u) expressed as the sum Xn of basis functions yk ≈ xiϕi(uk) using least squares i=1 2 3 4 • Example: fit polynomial function y = x1 + x2u1 + x3u1 + x4u1 + x5u1

XN h i 2 2 3 4 min yk − 1 uk u u u x least squares x k k k k=1 | {z } linear with respect to x   1 2.5    u1    2  2  ϕ(u) =  u1   u3  1 1.5 4 u1

1 0 0.5 1 1.5 2

©2021 A. Bemporad - Numerical Optimization 17/102 Least Squares - Fitting a circle • Example: fit a circle to a set of data2 XN 2 2 2 2 min (r − (xk − x0) − (yk − y0) ) x0,y0,r k=1 x0 • Let x = y0 be the optimization vector (note the change of variables!) 2− 2− 2 r x0 y0

• The problem becomes the least squares problem

2 N h i X 2 1 − 2 2 min 2xk 2yk 1 x (xk + yk) x 0 k=1

-1

-2

-3 -2 -1 0 1 2 3

2http://www.utc.fr/~mottelet/mt94/leastSquares.pdf

Definition n A set S ⊆ R is convex if for all x1, x2 ∈ S

λx1 + (1 − λ)x2 ∈ S, ∀λ ∈ [0, 1]

convex set nonconvex set S x1 x2 x1 x2

• f : S → R is a convex function if S is convex and

f(λx1 + (1 − λ)x2) ≤ λf(x1) + (1 − λ)f(x2)

∀x1, x2 ∈ S, λ ∈ [0, 1]

Jensen’s inequality (Jensen, 1906)

3 • If f is convex and differentiable at x2, take the limit λ → 0 and get

′ f(x1) ≥ f(x2) + ∇f(x2) (x1 − x2) Johan Jensen (1859–1925)

• A function f is strictly convex if f(λx1 + (1 − λ)x2) < λf(x1) + (1 − λ)f(x2), ∀x1 ≠ x2 ∈ S, ∀λ ∈ (0, 1)

3 ′ f(x1) − f(x2) ≥ limλ→0(f(x2 + λ(x1 − x2)) − f(x2))/λ = ∇f (x2)(x1 − x2)

• A function f : S → R is strongly convex with parameter m ≥ 0 if

mλ(1 − λ) f(λx + (1 − λ)x ) ≤ λf(x ) + (1 − λ)f(x ) − ∥x − x ∥2 1 2 1 2 2 1 2 2

• If f strongly convex with parameter m ≥ 0 and differentiable then m f(y) ≥ f(x) + ∇f(x)′(y − x) + ∥y − x∥2 2 2 • Equivalently, f is strongly convex with parameter m ≥ 0 if and only if − m ′ f(x) 2 x x convex • Moreover, if f is differentiable twice this is equivalent to ∇2f(x) ≽ mI (i.e., matrix ∇2f(x) − mI is positive semidefinite), ∀x ∈ Rn

• A function f is (strictly/strongly) concave if −f is (strictly/strongly) convex

The optimization problem min f(x) s.t. x ∈ S

is a convex optimization problem if S is a convex set S x x and f : S → R is a convex function 1 2

• Often S is defined by linear equality constraints Ax = b and convex inequality constraints g(x) ≤ 0, g : Rn → Rm convex

• Every local solution is also a global one (we will see this later)

• Efficient solution algorithms exist (we will see many later)

• Often occurring in many problems in engineering, economics, and science

Excellent textbook: “Convex Optimization” (Boyd, Vandenberghe, 2002)

• Hyperplane (H-)representation: b 2 x= A 2 v2 A A 2 1 P = {x ∈ Rn : Ax ≤ b} v3

v1 A 1 x= b A x=b3 A3 1 • Vertex (V-)representation: 3 Convex hull = transformation Xq Xp n from V- to H-representation P = {x ∈ R : x = αivi + βjrj} i=1 i=1 Vertex enumeration =

∑q transformation from H- to n αi, βj ≥ 0, αi = 1, vi, rj ∈ R V-representation i=1 when q = 0 the polyhedron is a cone vi = vertex, rj = extreme ray ©2021 A. Bemporad - Numerical Optimization 23/102 Linear programming

• Linear programming (LP) problem: -c ′ min c x x* s.t. Ax ≤ b, x ∈ Rn Ex = f George Dantzig (1914–2005)

′ • LP in standard form: min c x s.t. Ax = b x ≥ 0, x ∈ Rn • Conversion to standard form:

1. introduce slack variables ∑n ∑n aij xj ≤ bi ⇒ aij xj + si = bi, si ≥ 0 j=1 j=1 2. split positive and negative part of x   ∑n ∑n   + − a x + s = b a (x − x ) + s = b ij j i i ⇒ ij j j i i  j=1  j=1 ≥ + − ≥ xj free, si 0 xj , xj , si 0

• Quadratic programming (QP) problem:

Ax b 1  min x′Qx + c′x 2 x* ≤ ∈ Rn s.t. Ax b, x 1 2x0Qx + c0x = constant Ex = f

• Convex optimization problem if Q ≽ 0 (Q = positive semidefinite matrix) 4 • Without loss of generality, we can assume Q = Q′:

′ ′ ′ 1 x′Qx = 1 x′( Q+Q + Q−Q )x = 1 x′( Q+Q )x + 1 x′Qx − 1 (x′Q′x)′ 2 2 2 ′ 2 2 2 4 4 1 ′ Q+Q = 2 x ( 2 )x • Hard problem if Q ̸≽ 0 (Q = indefinite matrix) 4A matrix P ∈ Rn×n is positive semidefinite (P ≽ 0) if x′P x ≥ 0 for all x. It is positive definite (P ≻ 0) if in addition x′P x > 0 for all x ≠ 0. It is negative (semi)definite (P ≺ 0, P ≼ 0) if −P is positive (semi)definite. It is indefinite otherwise.

• In some problems the optimization variables can only take integer values. We call x ∈ Z an integrality constraint

• A special case is x ∈ {0, 1} (binary constraint)

• When all variables are integer (or binary) the problem is an integer programming problem (a special case of discrete optimization)

• In a mixed integer programming (MIP) problem some of the variables are real

(xi ∈ R), some are discrete/binary (xi ∈ Z or xi ∈ {0, 1})

Optimization problems with integer variables are more difficult to solve

1 min c′x min x′Qx + c′x xc 2 ≤ xc s.t. Ax b, x = [ xb ] ≤ s.t. Ax b, x = [ xb ] nc nb x ∈ R , x ∈ {0, 1} nc nb c b xc ∈ R , xb ∈ {0, 1}

mixed-integer linear program (MILP) mixed-integer quadratic program (MIQP)

• Some variables are real, some are binary (0/1)

• MILP and MIQP are NP-hard problems, in general

• Many good solvers are available (CPLEX, Gurobi, GLPK, FICO Xpress, CBC, ...) For comparisons see http://plato.la.asu.edu/bench.html

• Relations affected by random numbers lead to stochastic models

min Ew[f(x, w)] x

• The model is enriched by the information about the probability distribution of w

• Other stochastic measures can be minimized (Var, conditional value-at-risk, ...)

• The deterministic version minx f(x, Ew[w]) of the problem only considers the expected value of w, not its entire distribution

If f is convex w.r.t. w then f(x, Ew[w]) ≤ Ew[f(x, w)]

• chance constraints are constraints enforced only in probability: prob(g(x, w) ≤ 0) ≥ 99%

• robust constraints are constraints that must be always satisfied: g(x, w) ≤ 0, ∀w

• Dynamic optimization involves decision variables that evolve over time

Example: For a given a value of x0 we want to optimize NX−1 2 2 minx,u xt + ut t=0 s.t. xt+1 = axt + but

where ut is the control value (to be decided) and xt the state at time t.

The decision variables are " # " # u0 x1 . . u = . , x = . uN−1 xN

• Heavily used to solve optimal control problems, such as in model predictive control

• An optimization algorithm is a procedure to find an optimizer x∗ of a given

optimization problem minx∈X f(x)

• It is usually iterative: starting from an initial guess x0 of x it generates a sequence xk of “iterates”, with hopefully xN ≈ x∗ after N iterations

• Good optimization algorithms should possess the following properties:

– Robustness = perform well on a wide variety of problems in their class, for all reasonable values of the initial guess x0

– Efficiency = do not require excessive CPU time/flops and memory allocation

– Accuracy = find a solution close to the optimal one, without being affected by rounding errors due to the finite precision arithmetic of the CPU

• The above are often conflicting properties

Case Studies Optimization Guide Server Resources Server Information NEOS Server

Optimization Taxonomy

Back to Types of Optimization Problems

It is difficult to provide a taxonomy of optimization because many of the subfields have multiple links. ShownOptimization here is one perspective, taxonomy focused mainly on the subfields of deterministic optimization with a single objective function.

Optimization

Uncertainty Deterministic Multiobjective Optimization

Stochastic Programming Robust Optimization Continuous Discrete

Integer Combinatorial Unconstrained Constrained Programming Optimization

Nonlinear Nondifferentiable Nonlinear Equations Global Optimization Nonlinear Programming Network Optimization Bound Constrained Linearly Constrained Least Squares Optimization

Semiinﬁnite Mathematical Programs Mixed Integer Derivative-Free Quadratic Linear Semideﬁnite Programming Programming with Equilibrium Constraints Nonlinear Programming Optimization Programming Programming

Second-Order Complementarity Cone Programming Problems

Quadratically-Constrained Quadratic Programming

Optimizationhttps://neos-guide.org/content/optimization-taxonomy Guide

Footer

• Comparison on benchmark problems: http://plato.la.asu.edu/bench.html

• Taxonomy of many solvers for different classes of optimization problems: http://www.neos-guide.org

• NEOS server for remotely solving optimization problems:

http://www.neos-server.org

• Good open-source optimization software:

http://www.coin-or.org/

• GitHub , MATLAB Central , Google , ...

• An optimization model is a mathematical model that captures the objective function to minimize and the constraints imposed on the optimization variables

• It is a quantitative model, the decision problem must be formulated as a set of mathematical relations involving the optimization variables

Steps required to formulate an optimization model that solves a given decision problem:

1. Talk to the domain expert to understand the problem we want to solve

2. Single out the optimization variables xi (what are we able to decide?) and their domain (real, binary, integer)

3. Treat the remaining variables as parameters (=data that affect the problem but are not part of the decision process)

4. Translate the objective(s) into a cost function of x to minimize (or maximize)

5. Are there constraints on the decision variables ? If yes, translate them into (in)equalities involving x

6. Make sure we have all the required data available

solution optimization model 1.1 modeling minx f(x) solver real problem x⇤ = 2 3 3 6 7 s.t. g(x) 0 6 25 7 6 7  4 5

x 0c +

Qx b 0 1 x 2  Ax

min s.t.

analysis of the solution

• It may take several iterations to formulate the optimization model properly, as:

– A solution does not exist (anything wrong in the constraints?)

– The solution does not make sense (is any constraint missing or wrong?)

– The optimal value does not make sense (is the cost function properly defined?)

– It takes too long to find the solution (can we simplify the model?)

(Guerét et al., Applications of Optimization with XpressMP, 1999)

A small joinery makes two different sizes of boxwood chess sets. The small set requires 3 hours of machining ona lathe, and the large set requires 2 hours. There are four lathes with skilled operators who each work a 40 hour week, so we have 160 lathe-hours per week. The small chess set requires 1 kg of boxwood, and the large set requires 3 kg. Unfortunately, boxwood is scarce and only 200 kg per week can be obtained. When sold, each of the large chess sets yields a profit of $20, and one of the small chess set has a profit of$5.

The problem is to decide how many sets of each kind should be made each week so as to maximize profit.

• A small joinery makes two different sizes of boxwood chess sets. The small set requires 3 hours of machining on a lathe, and the large set requires 2 hours. • There are four lathes with skilled operators who each work a 40 hour week, so we have 160 lathe-hours per week. • The small chess set requires 1 kg of boxwood, and the large set requires 3 kg. Unfortunately, boxwood is scarce and only 200 kg per week can be obtained. • When sold, each of the large chess sets yields a profit of $20, and one of the small chess set has a profit of $5.

• The problem is to decide how many sets of each kind should be made each week so as to maximize profit.

• Optimization variables: xs, xℓ = produced quantities of small/large chess sets

• Cost function: f(x) = 5xs + 20xℓ (profit)

• Constraints:

– 3xs + 2xℓ ≤ 4 · 40 (maximum lathe-hours)

– xs + 3xℓ ≤ 200 (available kg of boxwood)

– xs, xℓ ≥ 0 (produced quantities cannot be negative)

max 5xs + 20xℓ 3 2 xs ≤ 160 s.t. [ 1 3 ][ xℓ ] [ 200 ] xs, xℓ ≥ 0

• What is the best decision ? Let us make some guesses: Table 1.1: Values for xs and xl xs xl Lathe-hours Boxwood OK? Proﬁt Notes A 0 0 0 0 Yes 0 Unproﬁtable! B 10 10 50 40 Yes 250 We won’t get rich doing this. C -10 10 -10 20 No 150 Planning to make a negative number of small sets. D 53 0 159 53 Yes 265 Uses all the lathe-hours. There is spare boxwood. E 50 20 190 110 No 650 Uses too many lathe-hours. F 25 30 135 115 Yes 725 There are spare lathe-hours and spare boxwood. G 12 62 160 198 Yes 1300 Uses all the resources H 0 66 130 198 Yes 1320 Looks good. There are spare resources.

1.2 •LinearWhat is Programming the best solution ? A numerical solver provides the following solution

We have just built a model for the decision process that the joinery owner has to make. We have isolated ∗ ∗ ⇒ ∗ the decisions he has to makexs = (how 0, xmanyℓ = of 66 each.6666 type of chessf( setx to) manufacture), = 1333.3 $ and taken his objective of maximizing proﬁt. The constraints acting on the decision variables have been analyzed. We have given names to his variables and then written down the constraints and the objective function in terms of these variable names. ©2021At A. theBemporad same - Numerical time as Optimization doing this we have made, explicitly or implicitly, various assumptions. The explicit39/102 assumptions that we noted were:

For each size of chess set, manufacturing time was proportional to the number of sets made. • There was no down-time because of changeovers between sizes of sets. • We could sell all the chess sets we made. • But we made many implicit assumptions too. For instance, we assumed that no lathe will ever break or get jammed; that all the lathe operators will turn up for work every day; that we never find any flaws in the boxwood that lead to some being unusable or a chess set being unacceptable; that we never have to discount the sale price (and hence the per unit profit) to get an order. And so on. We have even avoided a discussion of what is the worth of a fraction of a chess set — is it a meaningless concept, or can we just carry the fraction that we have made over into next week’s production? All mathematical models necessarily contain some degree of simplification of the real world that we are attempting to describe. Some assumptions and simplifications seem eminently reasonable (for instance, that we can get the total profit by summing the contributions of the individual profits from the two sizes); others may in some circumstances be very hopeful (no changeover time lost when we swap between sizes); whilst others may just be cavalier (all the lathe operators will arrive for work the day after the World Cup finals). Modeling is an art, not a precise science. Different modelers will make different assumptions, and come up with different models of more or less precision, and certainly of different sizes, having different numbers of decision variables. And at the same time as doing the modeling, the modeler has to be thinking about whether he will be able to solve the resulting model, that is find the maximum or minimum value of the objective function and the values to be given to the decision variables to achieve that value. It turns out that many models can be cast in the form of Linear Programming models, and it is fortunate that Linear Programming (LP) models of very large size can be solved in reasonable time on relatively inexpensive computers. It is not the purpose of this book to discuss the algorithms that are used to solve LP problems in any depth, but it is safe to assume that problems with tens of thousands of variables and constraints can be solved with ease. So if you can produce a model of your real-world situation, without too many wild assumptions, in the form of an LP then you know you can get a solution. So we next need to see what a Linear Programming problem consists of. To do so, we first introduce the notion of a linear expression. A linear expression is a sum of the following form

A1 x1 + A x + A x +...+AN xN · 2 · 2 3 · 3 ·

What is modeling? Why use models? 9 Applications of optimization with Xpress-MP Optimization models

• Optimization models, as all mathematical models, are never an exact representation of reality but a good approximation of it

• We need to make working assumptions, for example: – Lathe hours are never more than 160 – Available wood is exactly 200 kg – Prices are constant – We sell all chess sets

• There are usually many different models for the same real problem

Optimization modeling is an art

• Numerical complexity (and even the ability to find an optimal solution) depends on the optimization model we have formulated

• Good optimization models must be – Descriptive enough to capture the most significant aspects of the decision problem (variables, costs, constraints)

– Simple enough to be able to solve the resulting optimization problem

“Make everything as simple as possible, but not simpler.” — Albert Einstein

• An optimization project provides rational (quantitative) decisions based on the available information

• Even if the solution is not really applied, it provides a good suggestion to the decision maker (you should only make large chess sets)

• Making the model and analyzing the solution allows a better understanding of the problem at hand (small chess sets are not profitable)

• We can analyze the robustness of the solution with respect to variations in the data (big changes of solution for small changes of prices?)

• AMPL (A Modeling Language for Mathematical Programming) most used modeling language, supports several solvers

• GAMS (General Algebraic Modeling System) is one of the first modeling languages

• GNU MathProg a subset of AMPL associated with the free package GLPK (GNU Linear Programming Kit)

• YALMIP MATLAB-based modeling language

• CVX (CVXPY) Modeling language for convex problems in MATLAB ( )

• CASADI + IPOPT Nonlinear modeling + automatic differentiation, nonlinear programming solver (MATLAB, , C++)

• Optimization Toolbox’ modeling language (part of MATLAB since R2017b)

• PYOMO -based modeling language

• GEKKO -based mixed-integer nonlinear modeling language

• PYTHON-MIP -based modeling language for mixed-integer linear programming

• PuLP A linear programming modeler for

• JuMP A modeling language for linear, quadratic, and nonlinear constrained optimization problems embedded in

• Model and solve the problem using YALMIP (Löfberg, 2004)

xs = sdpvar(1,1); xl = sdpvar(1,1);

Constraints = [3*xs+2*xl <= 4*40, 1*xs+3*xl <= 200, ... xs >= 0, xl >= 0] Profit = 5*xs+20*xl;

optimize(Constraints,-Profit)

value(xs),value(xl),value(Profit)

• Model and solve the problem using CVX (Grant, Boyd, 2013)

cvx_clear cvx_begin variable xs(1) variable xl(1)

Profit = 5*xs+20*xl;

maximize Profit

subject to 3*xs+2*xl <= 4*40; % maximum lathe-hours 1*xs+3*xl <= 200; % available kg of boxwood xs>=0; xl>=0; cvx_end

xs,xl,Profit

• Model and solve the problem using CASADI + IPOPT (Andersson, Gillis, Horn, Rawlings, Diehl, 2018) (Wächter, Biegler, 2006)

import casadi.* xs=SX.sym('xs'); xl=SX.sym('xl');

Profit = 5*xs+20*xl; Constraints = [3*xs+2*xl-4*40; 1*xs+3*xl-200];

prob=struct('x',[xs;xl],'f',-Profit,'g',Constraints); solver = nlpsol('solver','ipopt', prob); res = solver('lbx',[0;0],'ubg',[0;0]);

Profit = -res.f; xs = res.x(1); xl = res.x(2);

• Model and solve the problem using Optimization Toolbox (The Mathworks, Inc.)

xs=optimvar('xs','LowerBound',0); xl=optimvar('xl','LowerBound',0);

Profit = 5*xs+20*xl; C1 = 3*xs+2*xl-4*40<=0; C2= 1*xs+3*xl-200<=0;

prob=optimproblem('Objective',Profit,'ObjectiveSense','max'); prob.Constraints.C1=C1; prob.Constraints.C2=C2;

[sol,Profit] = solve(prob);

xs=sol.xs; xl=sol.xl;

• Model and solve the problem in Python using PYTHON-MIP5:

from mip import *

m = Model(sense=MAXIMIZE, solver_name=CBC) xs = m.add_var(lb=0) xl = m.add_var(lb=0) m += 3*xs+2*xl <= 4*40 m += 1*xs+3*xl <= 200 m.objective = 5*xs+20*xl m.optimize()

print(xs.x, xl.x)

5https://python-mip.readthedocs.io/

• In this case the optimization model is very simple and we can directly code the LP problem in plain MATLAB or Python:

A=[1 3;3 2]; import scipy as sc b=[200;160]; import numpy as np c=[5 20]; A=np.array([[1,3],[3,2]]) [xopt,fopt]=linprog(... b=np.array([[200],[160]]) -c,A,b,[],[],[0;0]) c=np.array([5,20]) sol=sc.optimize.linprog( -c,A,b,bounds=[0,None])

• The Hybrid Toolbox for MATLAB contains interfaces to various solvers for LP,

QP, MILP, MIQP (http://cse.lab.imtlucca.it/~bemporad/hybrid/toolbox) (Bemporad, 2003-today)

• However, when there are many variables and constraints forming the problem matrices manually can be very time-consuming and error-prone

• We can even model and solve the optimization problem in Excel:

optimization variables cost function B6:C6 =SUMPRODUCT(B6:C6;B2:C2)

Reference:

C. Guéret, C. Prins, M. Sevaux, “Applications of optimization with Xpress-MP,” Translated and revised by S.Heipcke, 1999 Optimization modeling: linear constraints

• Constraints define the set where to look for an optimal solution

• They define relations between decision variables

• When formulating an optimization model we must disaggregate the restrictions appearing in the decision problem into subsets of constraints that we know how to model

• There are many types of constraints we know how to model ...

• Box constraints are the simplest constraints: they define upper and lower bounds on the decision variables

x2 ℓi ≤ xi ≤ ui u2

admissible ℓi ∈ R ∪ {−∞}, ui ∈ R ∪ {+∞} set

ℓ2

x1 0 ℓ1 u1

• Example: ``We cannot sell more than 100 units of Product A'' • Pay attention: some solvers assume nonnegative variables by default!

• When ℓi = ui the constraint becomes xi = ℓi and variable xi becomes redundant. Still it may be worthwhile keeping in the model

©2021 A. Bemporad - Numerical Optimization 53/102 x1 2. Flow constraints total flow x2 x • Flow constraints arise when an item can be divided in different3 streams, or vice versa many streams come together x n 1 x1 X total flow total flow Fmin ≤ xi ≤ Fmax x2 x2 i=1 x3 x3

• Example: ``I can get water from 3 suppliers, S1, S2 and S3. I x1 want to have at least 1000total liters flow available.'' x1 + x2 + x3 ≥ 1000 x2

• Example: ``I have 50 trucks available to rentx3 to 3 customers C1, C2 and C3'' x1 + x2 + x3 ≤ 50

• Losses can be included as well: ``2% water I get from suppliers gets

lost.'' 0.98x1 + 0.98x2 + 0.98x3 ≥ 1000

• Resource constraints take into account that a given resource is limited Xn Rjixi ≤ Rmax,j i=1

• The technological coefficients Rji denote the amount of resource j used per unit of activity i

• Example: ``Small chess sets require 1 kg ``Small chess sets require 3 boxwood, the large ones 3 kg, lathe hours, the large ones 2 h, total available is 200 kg.'' total time is 4×40 h.''

x1 + 3x2 ≤ 200 3x1 + 2x2 ≤ 160

2 3 200 R = [ 3 2 ] ,Rmax = [ 160 ]

• Balance constraints model the fact that “what goes out must in total equal what comes in” L x1in XN XM x1out out in xi = xi + L x2in i=1 i=1 in x3 x2out

• Example: ``I have 100 tons steel and can buy more from

suppliers 1,2,3 to serve customers A,B.'' xA + xB = 100 + x1 + x2 + x3

• Balance can occur between time periods in a multi-time period model

• Example: ``The cash I'll have tomorrow is what I have now plus

what I receive minus what I spend today.'' xt+1 = xt + ut − yt

©2021 A. Bemporad - Numerical Optimization 56/102 5. Quality constraints • Quality constraints are requirements on the average percentage of a certain quality when blending several components P N XN XN α x Pi=1 i i T α x T p x N pmin i i min i i=1 xi i=1 i=1

• Example: ``The average risk of an investment in assets A,B,C, which have risks 25%, 5%, and 12% respectively, must be smaller than 10%'' 0.25xA+0.05xB +0.12xC ≤ 0.1 xA+xB +xC

• The nonlinear quality constraint is converted to a linear one under the

assumption that xi ≥ 0 (if xi = 0 ∀i the constraint becomes redundant)

Objectives and constraints can be often simplified by mathematical transformations and/or adding extra variables

• It is often useful to add extra accounting variables XN y = xi accounting constraint i=1

P N • Of course we can replace y with i=1 xi everywhere in the model (condensed form), but this would make it less readable

• Moreover, keeping y in the model (non-condensed form) may preserve some structural properties that the solver could exploit

• Example: ``The profit at any given year is the difference between

revenues and expenditures'' pt = rt − et

• Blending constraints occur when we want to blend a set of ingredients xi in given percentages αi in the final product x P i N = αi j=1 xj

• Similar to quality constraints, blending constraints can be converted to linear equality constraints XN xi = αixj j=1

• So far we have seen are hard constraints, i.e., that cannot be violated. • Soft constraints are a relaxation, in which the constraint can be violated, usually paying a penalty

XN XN ≤ aij xi bj aij xi ≤ bj + ϵj i=1 i=1

• We call the new variable ϵj panic variable: it should be normally zero but can assume a positive value in case there is no way to fulfill the constraint set

• Example: ``Only 200 kg boxwood are available to make chess sets, but we can buy extra for 6 $/kg''

− maxxs,xℓ,ϵ≥0 5xs + 20xℓ 6ϵ s.t. xs + 3xℓ ≤ 200 + ϵ 3xs + 2xℓ ≤ 160

(Guerét et al., Applications of Optimization with XpressMP, 1999)

The company Steel has received an order for 500 tonnes of steel to be used in shipbuilding. This steel must have the certain characteristics (``grades''). The company has seven different raw materials in stock that may be used forthe production of this steel. The objective is to determine the composition of the steel that minimizes the production cost.

• The company Steel has received an order for 500 tonnes of steel to be used in shipbuilding. • This steel must have the certain characteristics (``grades''). • The company has seven different raw materials in stock that may be used for the production of this steel. • The objective is to determine the composition of the steel that minimizes the production cost.

• Available data of characteristics of steel ordered

chemical element minimum grade maximum grade Carbon (C) 2 3 Copper (Cu) 0.4 0.6 Manganese (Mn) 1.2 1.65

• Available data of grades, available amounts, and prices

raw material C % Cu % Mn % availability [t] cost [e/t] Iron alloy 1 2.5 0 1.3 400 200 Iron alloy 2 3 0 0.8 300 250 Iron alloy 3 0 0.3 0 600 150 Copper alloy 1 0 90 0 500 220 Copper alloy 2 0 96 4 200 240 Aluminum alloy 1 0 0.4 1.2 300 200 Aluminum alloy 2 0 0.6 0 250 165

• “The company has seven different raw materials in stock''

– We need 7 optimization variables x1, . . . , x7

– Their upper and lower bounds are

0 ≤ xj ≤ Rj ,Rj = tons available of raw material j

• “The company Steel has received an order for 500 tonnes of steel'' ∑ 7 – flow constraint y = j=1 xj (produced metal)

– lower bound y ≥ 500 (quantity that needs to be produced)

• “This steel must have the certain characteristics''

min max index i chemical element minimum grade gi maximum grade gi 1 Carbon (C) 2 3 2 Copper (Cu) 0.4 0.6 3 Manganese (Mn) 1.2 1.65

e index j raw material C% Pj1 Cu % Pj2 Mn % Pj3 availability [t] Rj cost [ /t] cj 1 Iron alloy 1 2.5 0 1.3 400 200 2 Iron alloy 2 3 0 0.8 300 250 3 Iron alloy 3 0 0.3 0 600 150 4 Copper alloy 1 0 90 0 500 220 5 Copper alloy 2 0 96 4 200 240 6 Aluminum alloy 1 0 0.4 1.2 300 200 7 Aluminum alloy 2 0 0.6 0 250 165

Quality constraints on grades for each chemical element i = 1, 2, 3: P ( P 7 7 max Pjixj Pjixj ≤ g y gmin ≤ Pj=1 ≤ gmax Pj=1 i i 7 i 7 ≥ min i=1 xj j=1 Pjixj gi y

where Pji = percentage of metal i contained in raw material j

• “The objective is to determine the composition of theP steel that minimizes the production cost'' 7 The cost function to minimize is j=1 cjxj • The complete optimization model is finally P 7 minx,y j=1 cjxj ≤ ≤ s.t. 0 xPj Rj 7 y = j=1 xj ≥ Py 500 7 ≤ max Pj=1 Pjixj gi y 7 ≥ min j=1 Pjixj gi y

• The problem is a linear programming problem. The solution is

∗ ∗ ∗ ∗ ∗ ∗ ∗ x1 = 400, x3 = 39.7763, x5 = 2.7613, x6 = 57.4624, x2 = x4 = x7 = 0 [t]

with optimal production cost f(x∗) = 98122 e

• Model and solve the problem using YALMIP (Löfberg, 2004)

x=sdpvar(7,1); y=sum(x);

Constraints = [x>=0, x<=R, y>=500, ... P'*x<=B(:,2)*y, P'*x>=B(:,1)*y];

cost = sum(c.*x);

optimize(Constraints,cost)

value(x),value(cost)

• Model and solve the problem using CVX (Grant, Boyd, 2013)

cvx_clear cvx_begin variable x(7)

cost = sum(c.*x);

minimize cost

subject to y=sum(x); y>=500; x>=0; x<=R; P'*x<=B(:,2)*y; P'*x>=B(:,1)*y;

cvx_end

x,cost

• Linear programs only allow minimizing a linear combination of the optimization variables

• However, by introducing new variables, we can minimize any convex piecewise affine (PWA) function

Result Every convex piecewise affine function n a0 x + b4 ℓ : R → R can be represented as the `(x) 4 a x + b max of affine functions, and vice versa 10 1

(Schechter, 1987) a30 x + b3

Example: { ′ ′ } ℓ(x) = max a1x + b1, . . . , a4x + b4 x a20 x + b2

• Minimization of a convex PWA function ℓ(x):

ϵ minϵ,x ϵ ′  ϵ ≥ a x + b1  1 ϵ ≥ a′ x + b s.t. 2 2  ϵ ≥ a′ x + b x  3 3 ≥ ′ ϵ a4x + b4 ≥ { ′ ′ ′ ′ } • By construction ϵ max a1x + b1, a2x + b2, a3x + b3, a4x + b4 • By contradiction it is easy to show that at the optimum we have that

{ ′ ′ ′ ′ } ϵ = max a1x + b1, a2x + b2, a3x + b3, a4x + b4

• Convex PWA constraints ℓ(x) ≤ 0 can be handled similarly by imposing ′ ≤ ∀ aix + bi 0, i = 1, 2, 3, 4

• minmax objective: we want to minimize the maximum among M given linear ′ objectives fi(x) = aix + bi

min max {fi(x)} s.t. linear constraints x i=1,...,M

′ • Example: asymmetric cost minx max{a x + b, 0}

• Example: minimize the ∞-norm

min ∥Ax − b∥∞ x

m×n m where ∥v∥∞ , maxi=1,...,n |vi| and A ∈ R , b ∈ R . This corresponds to

min max{A1x − b1, −A1x + b1,...,Amx − bm, −Amx + bm} x

©2021 A. Bemporad - Numerical Optimization 71/102 2. Minimize the sum of max objectives • We want to minimize the sum of maxima among given linear objectives ′ fij (x) = aij x + bij XN min max {fij (x)} s.t. linear constraints x i=1,...,Mj j=1 • The equivalent reformulation is P N minϵ,x j=1 ϵj ≥ ′ s.t. ϵj aij x + bij , i = 1,...,Mj, j = 1,...,N (other linear constraints) • Example: minimize the 1-norm

min ∥Ax − b∥1 x P ∥ ∥ , | | ∈ Rm×n ∈ Rm where v 1 i=1,...,n vi and A , b , that corresponds to Xm min max{Aix − bi, −Aix + bi} x i=1 ©2021 A. Bemporad - Numerical Optimization 72/102 3. Linear-fractional program

• We want to minimize the ratio of linear objectives c′x+d minx e′x+f s.t. Ax ≤ b Gx = h over the domain e′x + f > 0 1 • We introduce the new variable z = and replace x with the new e′x + f i variables yi = zxi, i = 1, . . . , n, where 1 = z(e′x + f) = e′y + fz, z ≥ 0 • Since z ≥ 0 then zAx ≤ zb, and the original problem is translated into the LP ′ minz,y c y + dz s.t. Ay − bz ≤ 0 Gy = hz e′y + fz = 1 z ≥ 0 ∗ 1 ∗ ∗ from which we recover x = z∗ y in case z > 0.

• The Chebychev center of a polyhedron P = {x : Ax ≤ b} x* is the center x∗ of the largest ball B(x∗, r∗) = {x : x = x∗ + u, r* ∗ ∥u∥2 ≤ r } contained in P

• The radius r∗ is called the Chebychev radius of P

• A ball B(x, r) is included in P if and only if

sup Ai(x + u) = Aix + r∥Ai∥2 ≤ bi, ∀i = 1, . . . , m, ∥u∥2≤r

m×n m where A ∈ R , b ∈ R , and Ai is the ith row of A.

• Therefore, we can compute the Chebychev center/radius by solving the LP

maxx,r r s.t.Aix + r∥Ai∥2 ≤ bi, i = 1, . . . , m

References:

S. Boyd, L. Vandenberghe, “Convex Optimization,” 2004

S. Boyd, “Convex Optimization,” lecture notes, http://ee364a.stanford.edu, http://ee364b.stanford.edu Convex sets

n • Convex set: A set S ⊆ R is convex if for all x1, x2 ∈ S

λx1 + (1 − λ)x2 ∈ S, ∀λ ∈ [0, 1]

• The convex hull of N points x¯1,..., x¯N is the set of all their convex combinations

P x1 x2 n N S = {x ∈ R : ∃λ ∈ R : x = λix¯i, P x7 ≥ N } λi 0, λi = 1 xN i=1 x3 x6

x4 • A convex cone of N points x¯1,..., x¯N is the set

X x1 n N S = {x ∈ R : ∃λ ∈ R : x = λix¯i, λi ≥ 0} x

′ a • hyperplane {x : a x = b}, a ≠ 0 a’x ≥ b a’x=b { ′ ≤ } ̸ • halfspace x : a x b , a = 0 a’x ≤ b

• polyhedron P = {x : Ax ≤ b, Ex = f}

• (Euclidean) ball B(x0, r) = {x : ∥x − x0∥2 ≤ r} ={x0 + ry : ∥y∥2 ≤ 1}

′ • ellipsoid E = {x :(x − x0) P (x − x0) ≤ 1} ′ with P = P ≻ 0, or equivalently E = {x0 + Ay : ∥y∥2 ≤ 1}, x0 A square and det A ≠ 0

• Any set S = {x ∈ Rn : g(x) ≤ 0} with g : Rn → Rm convex

• The intersection of (any number of) convex sets is convex

• The image of a convex set under an affine function f(x) = Ax + b (A ∈ Rm×n, b ∈ Rm) is convex

S ⊆ Rn convex ⇒ f(S) = {y : y = f(x), x ∈ S} convex

for example: scaling (A diagonal, b = 0), translation (A = 0, b ≠ 0), ′ projection (A = [I 0], b = 0, i.e., f(S) = {y = [ x1 ... xi ] : x ∈ S})

• Recall: f : S → R is a convex function if S is convex and

f(λx1 + (1 − λ)x2) ≤ λf(x1) + (1 − λ)f(x2) Jensen’s inequality ∀x1, x2 ∈ S, λ ∈ [0, 1]

• Sublevel sets Cα of convex functions are convex sets (but not vice versa)

Cα = {x ∈ S : f(x) ≤ α}

• Therefore linear equality constraints Ax = b and inequality constraints g(x) ≤ 0, with g a convex (vector) function, define a convex set

• Examples of convex functions – affine f(x) = a′x + b, for any a ∈ Rn, b ∈ R – exponential f(x) = eax, x ∈ R, for any a ∈ R – power f(x) = xα, x ∈ R, for any α > 1 or α ≤ 0. Example: x2, 1/x for x > 0 – powers of absolute value f(x) = |x|p, x ∈ R, for p ≥ 1 – negative entropy f(x) = x log x, x ∈ R – any norm f(x) = ∥x∥

– maximum f(x) = max(x1, . . . , xn)

• Examples of concave functions – affine f(x) = a′x + b, for any a ∈ Rn, b ∈ R – logarithm f(x) = log x, x ∈ R √ – power f(x) = xα, x ∈ R, for any 0 ≤ α ≤ 1. Example: x, x ≥ 0

– minimum f(x) = min(x1, . . . , xn)

• Recall the first-order condition of convexity: f : Rn → R with convex domain dom f and differentiable is convex if and only if

f(y)

f(y) ≥ f(x)+∇f(x)′(y−x), ∀x, y ∈ dom f f(x) f(x)+∇f(x)’(y-x)

• Second-order condition: Let f : Rn → R with convex domain dom f be twice 2 2 2 ∂ f(x) differentiable and ∇ f(x) its Hessian matrix, [∇ f(x)]ij = . Then f is ∂xi∂xj convex if and only if

∇2f(x) ≽ 0, ∀x ∈ dom f

If ∇2f(x) ≻ 0 for all x ∈ dom f then f is strictly convex.

1. Check directly whether the definition is satisfied (Jensen’s inequality)

2. Check if the Hessian matrix is positive semidefinite (only for twice differentiable functions)

3. Show that f is obtained by combining known convex functions via operations that preserve convexity

• nonnegative scaling: f convex, α ≥ 0 ⇒ αf convex

• sum: f, g convex ⇒ f + g convex

• affine composition: f convex ⇒ f(Ax + b) convex

• pointwise maximum: f1, . . . , fm convex ⇒ maxi fi(x) convex • composition: h convex increasing, f convex ⇒ h(f(x)) convex

General composition rule: h(f1(x), . . . , fk(x)) is convex when h is convex and

h is increasing w.r.t. its ith argument, and fi convex, or h is decreasing w.r.t. its ith argument, and fi concave, or fi is affine for each i = 1, . . . , k

See also dcp.stanford.edu (Diamond 2014)

• The optimization problem

min f(x) or, more min f(x) s.t. g(x) ≤ 0 generally, s.t. x ∈ S Ax = b S convex set n m g : R → R , gi convex with f : X → R convex is a convex optimization problem, where X = {x ∈ Rn : g(x) ≤ 0, Ax = b} or, more generally, X = S.

• Convex programs can be solved to global optimality and many efficient algorithms exist for this (we will see many later) • Although convexity may sound like a restriction, it occurs very frequently in practice (sometimes after some transformations or approximations)

• The objective function has the form – minimize a scalar convex expression, or – maximize a scalar concave expression

• Each of the constraints (if any) has the form – convex expression ≤ concave expression, or – concave expression ≥ convex expression, or – affine expression = affine expression

This framework is used in the CVX, CVXPY, and Convex.jl packages.

• least squares (LS) problem ∥ − ∥2 ∗ ′ −1 ′ min Ax b 2 x = (|A A{z) A} b pseudoinverse of A Adrien-Marie Legendre (1752–1833) • nonnegative least squares (NNLS) (Lawson, Hanson, 1974) ∥ − ∥2 min Ax b 2 s.t. x ≥ 0

• bounded-variable least squares (BVLS) (Stark,Parker, 1995) J. Carl Friedrich Gauss ∥ − ∥2 (1777–1855) min Ax b 2 s.t. ℓ ≤ x ≤ u

• constrained least squares ∥ − ∥2 min Ax b 2 s.t. Ax ≤ b, Ex = f

• The least squares cost is a special case of quadratic cost Ax b  x* 1 2 1 ′ ′ ′ ′ 1 + = constant ∥Ax − b∥ = x A Ax − b Ax + b b 2x0Qx c0x 2 2 2 • A generalization of constrained least squares is quadratic programming (QP) 1 min x′Qx + c′x 2 ′ ≽ s.t. Ax ≤ b Q = Q 0 Ex = f

• If Q = L′L ≻ 0 we can complete the squares by setting y = Lx + (L−1)′c and convert the QP into a LS problem: 1 1 1 x′Qx + c′x = ∥Lx − (−L−1)′c∥2− c′Q−1c 2 2 2 2

• We want to solve the LP with random cost c

min c′x E[c] =c, ¯ Var[c] = E[(c − c¯)(c − c¯)′] = Σ s.t. Ax ≤ b, Ex = f

• c′x is a random variable with expectation E[c′x] =cx ¯ and variance Var[c′x] = x′Σx

• We want to trade off the expectation of c′x with its variance (=risk) with a risk aversion coefficient γ ≥ 0

• This is equivalent to a QP:

min E[c′x] + γ Var[c′x] minc ¯′x + γx′Σx s.t. Ax ≤ b, Ex = f s.t. Ax ≤ b, Ex = f

• The following ℓ1-penalized linear regression problem is called LASSO (least absolute shrinkage and selection operator):

1∥ − ∥2 ∥ ∥ ∈ Rm×n ∈ Rm min Ax b 2 + λ x 1 A , b x 2 • The tuning parameter λ ≥ 0 determines the tradeoff between fitting Ax ≈ b (λ small) and making x sparse (λ large) • By splitting x in the difference of its positive and negative parts, x = y − z, y, z ≥ 0 we get the positive semidefinite QP with 2n variables

1∥ − − ∥2 ′ min A(y z) b 2 + λ1 (y + z) y,z≥0 2

′ ∗ ∗ where 1 = [1 ... 1]. At optimality at least one of yi , zi will be zero ∥ ∥2 ∥ ∥2 • A small Tikhonov regularization σ( y 2 + z 2) makes the QP strictly convex

• Solve LASSO problem 56

54 1∥ − ∥2 ∥ ∥ min Ax b 2 + λ x 1 52 x 2

50 ∈ R3000×1000 ∈ R3000 A , b 48

46 • A, B = random matrices 10-4 10-3 10-2 10-1 100 101 102

• A sparse with 3000 nonzero entries 1000

800

• Problem solved by QP for different λ’s 600

400 • CPU time ranges from 8.5 ms to 1.17 s 200 using osQP (http://osqp.org) 0 (Stellato, Banjac, Goulart, Bemporad, Boyd, 2020) 10-4 10-3 10-2 10-1 100 101 102

• If we add quadratic constraints in a QP we get the quadratically constrained quadratic program (QCQP)

1 ′ ′ min 2 x Qx + c x 1 ′ ′ ≤ s.t. 2 x Pix + dix + hi 0, i = 1, . . . , m Ax = b

• QCQP is a convex problem if Q, Pi ≽ 0, i = 1, . . . , m

• If P1,...,Pm ≻ 0, the feasible region X of the QCQP is the intersection of m ellipsoids and p hyperplanes (b ∈ Rp)

• Polyhedral constraints (halfspaces) are a special case when Pi = 0

• A generalization of LP, QP, and QCQP is second-order cone programming (SOCP) min c′x ∥ ∥ ≤ ′ s.t. Fix + gi 2 dix + hi, i = 1, . . . , m Ax = b

n1×n p×n with Fi ∈ R , A ∈ R

• If Fi = 0 the SOC constraint becomes a linear inequality constraint

• If di = 0 (hi ≥ 0) the SOC constraint becomes a quadratic constraint

• The quadratic constraint x′F ′F x + d′x + h ≤ 0 is equivalent to the SOC constraint h i 1 (1+d′x+h) 1 ′ 2 ≤ (1 − d x − h) F x 2 2

• We want to solve the LP with uncertain constraint coefficients ai min c′x ′ ≤ s.t. aix bi, i = 1, . . . , m

• Assume ai can be anything in the ellipsoid Ei = {a¯i + Piy, ∥y∥2 ≤ 1}, n×n n Pi ∈ R , where a¯i ∈ R is the center of Ei min c′x ′ ≤ ∀ ∈ E s.t. aix bi, ai i, i = 1, . . . , m • The constraint is equivalent to { ′ } ≤ , where supai∈Ei aix bi { ′ } { ′ } ′ ∥ ′ ∥ sup aix = sup (¯ai + Piy) x =a ¯ix + Pi x 2 ai∈Ei ∥y∥2≤1 • The original robust LP is therefore equivalent to the SOCP min c′x ′ ∥ ′ ∥ ≤ s.t. a¯ix + Pi x 2 bi, i = 1, . . . , m

∈ 1 • For given ηi [ 2 , 1] we want to solve the LP with chance constraints min c′x ′ ≤ ≥ s.t. prob(aix bi) ηi, i = 1, . . . , m ′ − ′ − 2 ′ • Let α = aix bi, α¯ =a ¯ix bi, σ¯ = x Σix. The cumulative distributionR β 2 function (CDF) of α ∼ N (¯α, σ¯) is F (α) = Φ( α−α¯ ), Φ(β) = √1 e−t /2dt σ¯ 2π −∞ − − ′ ′ − ≤ α¯ bi a¯ix ≥ prob(aix bi 0) = F (0) = Φ = Φ ηi σ¯ ∥Lix∥2 • The original LP with random constraints is equivalent to the SOCP

′ min c x 2 ′ − 1 ∥ ∥ ≤ 0 s.t. a¯ix + Φ (ηi) Lix 2 bi, i = 1, . . . , m -2 0 0.2 0.4 0.6 0.8 1 −1 ≥ ≥ 1 where the inverse CDF Φ (ηi) 0 since ηi 2 (Boyd, Vandenberghe, 2004)

• Goal: find the largest box B contained inside a polyhedron P = {x ∈ Rn : Ax ≤ b} x*+y* • Let y ∈ Rn = vector of dimensions of B and x ∈ Rn = vertex of B with lowest coordinates x* • Problem to solve: Q n maxx,y i=1 yi nonlinear, nonconvex, s.t.A(x + diag(v)y) ≤ b, ∀v ∈ {0, 1}n many constraints! y ≥ 0

• Reformulate as maximize log(volume), remove redundant constraints: Xn convex problem minx,y − log(yi) i=1 + + { } s.t. Ax + A y ≤ b, y ≥ 0 Aij = max Aij , 0

©2021 A. Bemporad - Numerical Optimization 94/102 Semidefinite program (SDP) • A semidefinite program (SDP) is an optimization problem in which we have constraints on positive semidefiniteness of matrices ′ minx c x s.t. x1F1 + x2F2 + ... + xnFn + G ≼ 0 Ax = b

where F1,F2,...,Fn,G are (wlog) symmetric m × m matrices • The constraint is called linear matrix inequality (LMI) 6

• Multiple LMIs can be combined in a single LMI using block-diagonal matrices

1 1 1 h i h i h i x1F + ... + xnF + G ≼ 0 F 1 0 F 1 0 1 1 n 1 x +... n x + G 0 ≼ 0 2 2 2 0 F 2 1 0 F 2 n 0 G2 x1F1 + ... + xnFn + G ≼ 0 1 n

Many interesting problems can be formulated (or approximated) as SDPs

6 ′ The LMI constraint means z (x1F1 + x2F2 + ... + xnFn + G)z ≤ 0, ∀z ≥ 0

SDP generalizes LP, QP, QCQP, SOCP:

• an LP can be recast as an SDP

min c′x min c′x s.t. Ax ≤ b s.t. diag(Ax − b) ≼ 0

• an SOCP can be recast as an SDP

′ min c′x min hc x i (d′ x+h )IF x+g ∥ ∥ ≤ ′ i i i i ≽ s.t. Fix + gi 2 dix + hi s.t. ′ ′ 0 (Fix+gi) dix+hi i = 1, . . . , m i = 1, . . . , m

• Good SDP packages exist (SeDuMi, SDPT3, Mathworks LMI Toolbox, ...)

Rn → R R { ∈ R } • A monomial function f : ++ ++, where ++ = x : x > 0 , has the form a1 a2 an ∈ R f(x) = cx1 x2 . . . xn , c > 0, ai Rn → R • A posynomial function f : ++ ++ is the sum of monomials

XK a1k a2k ank ∈ R f(x) = ckx1 x2 . . . xn , ck > 0, aik k=1

• A geometric program (GP) is the following optimization problem min f(x)

s.t. gi(x) ≤ 1, i = 1, . . . , m hi(x) = 1, i = 1, . . . , p

with f, gi posynomials, hi monomials.

• Introduce the change of variables yi = log xi. The optimizer is the same if we minimize log f instead of f and take the log of both sides of the constraints

a1 an • The logarithm of a monomial fM (x) = cx1 . . . xn becomes affine in y

a1 an aiy1 anyn ′ log fM (x) = log(cx1 . . . xn ) = log(ce . . . e ) = a y + b, b = log c P K a1k ank • The logarithm of a posynomial fP (x) = k=1 ckx1 . . . xn becomes ! K X ′ a y+bk log fP (x) = log e k , bk = log ck k=1

y • One can prove that F (y) = log fP (e ) is convex and so it is the program P K a′ y+b min log e k k Pk=1 K ′ ciky+dik ≤ s.t. log k=1 e 0, i = 1, . . . , m Ey + f = 0

• Maximize the volume of a box-shaped structure with height h, width w, depth d

• Constraints:

– total wall area 2(hw + hd) ≤ Awall

– floor area wd ≤ Aﬂr – upper and lower bounds on aspect ratios α ≤ h/w ≤ β, γ ≤ w/d ≤ δ

• The problem can be cast as the following GP min h−1w−1d−1 s.t. 2 hw + 2 hd ≤ 1 Awall Awall 1 wd ≤ 1 Aflr −1 ≤ 1 −1 ≤ αh w 1, β hw 1 −1 ≤ 1 −1 ≤ γwd 1, δ w d 1

• We solve the problem in MATLAB: alpha=0.5; beta=2; gamma=0.5; delta=2; Awall=1000; Afloor=500;

CVX YALMIP cvx_begin gp quiet sdpvar h w d variables h w d % obj. function = box volume C = [alpha <= h/w <= beta, maximize(h*w*d) gamma <= d/w <= delta, h>=0, subject to w>=0]; 2*(h*w + h*d) <= Awall; C = [C, 2*(h*w+h*d) <= Awall, w*d <= Afloor; w*d <= Afloor]; alpha <= h/w <= beta; gamma <= d/w <= delta; optimize(C,-(h*w*d)) cvx_end opt_volume = cvx_optval; yalmip.github.io/tutorial/geometricprogramming

• Result: max volume = 5590.17, h∗ = 11.1803, w∗ = 22.3599, d∗ = 22.3614

• We solve the problem in PYTHON:

CVXPY import cvxpy as cp constraints = [ 2*(h*w + h*d) <= Awall, alpha = 0.5 w*d <= Afloor, beta = 2.0 alpha <= h/w, h/w <= beta, gamma = 0.5 gamma <= d/w, d/w <= delta] delta = 2.0 Awall = 1000.0 problem = cp.Problem(cp.Maximize Afloor = 500.0 (obj), constraints) problem.solve(gp=True) h = cp.Variable(pos=True) w = cp.Variable(pos=True) print("h: ", h.value) d = cp.Variable(pos=True) print("w: ", w.value) print("d: ", d.value) obj = h * w * d print("volume: ", problem.value)

• Substituting the objective f with a monotonically increasing function of f can simplify the problem √ ≥ – Example:√ min x with x 0, is a nonconvex problem, but we can minimize ( x)2 = x instead

∏ – Example: max f(x) = n x is a nonconvex problem, but the function ∑ i=1 i n log(f(x)) = i=1 log(xi) is concave

• Sometimes a nonconvex problem can be transformed into a convex problem by making a nonlinear transformation of the optimization variables (as in GP)