Convex Optimization Problems

Prof. Daniel P. Palomar The Hong Kong University of Science and Technology (HKUST)

MAFS6010R - Portfolio Optimization with R MSc in Financial Mathematics Fall 2019-20, HKUST, Hong Kong Outline

1 Optimization Problems

2 Convex Optimization

3 Quasi-Convex Optimization

4 Classes of Convex Problems: LP, QP, SOCP, SDP

5 Multicriterion Optimization (Pareto Optimality)

6 Solvers Outline

1 Optimization Problems

2 Convex Optimization

3 Quasi-Convex Optimization

4 Classes of Convex Problems: LP, QP, SOCP, SDP

5 Multicriterion Optimization (Pareto Optimality)

6 Solvers Optimization Problem in Standard Form

General optimization problem in standard form:

minimize f0 (x) x subject to fi (x) ≤ 0 i = 1,..., m hi (x) = 0 i = 1,..., p

where

x = (x1,..., xn) is the optimization variable n f0 : R −→ R is the objective function n fi : R −→ R, i = 1,..., m are inequality constraint functions n hi : R −→ R, i = 1,..., p are equality constraint functions.

? Goal: find an optimal solution x that minimizes f0 while satisfying all the constraints.

D. Palomar Convex Problems 4 / 48 Optimization Problem in Standard Form

Feasibility:

a point x ∈ dom f0 is feasible if it satisfies all the constraints and infeasible otherwise a problem is feasible if it has at least one feasible point and infeasible otherwise. Optimal value:

? p = inf {f0 (x) | fi (x) ≤ 0, i = 1,..., m, hi (x) = 0, i = 1,..., p}

p? = ∞ if problem infeasible (no x satisfies the constraints) p? = −∞ if problem unbounded below.

Optimal solution: x? such that f (x?) = p? (and x? feasible).

D. Palomar Convex Problems 5 / 48 Global and Local Optimality

? A feasible x is optimal if f0 (x) = p ; Xopt is the set of optimal points. A feasible x is locally optimal if is optimal within a ball Examples: ? f0 (x) = 1/x, dom f0 = R++: p = 0, no optimal point 3 ? f0 (x) = x − 3x: p = −∞, local optimum at x = 1.

D. Palomar Convex Problems 6 / 48 Implicit Constraints

The standard form optimization problem has an explicit constraint:

m p \ \ x ∈ D = dom fi ∩ dom fi i=0 i=1 D is the domain of the problem

The constraints fi (x) ≤ 0, hi (x) = 0 are the explicit constraints A problem is unconstrained if it has no explicit constraints Example: minimize log b − aT x
x is an unconstrained problem with implicit constraint b > aT x.

D. Palomar Convex Problems 7 / 48 Feasibility Problem

Sometimes, we don’t really want to minimize any objective, just to find a feasible point:

find x x subject to fi (x) ≤ 0 i = 1,..., m hi (x) = 0 i = 1,..., p

This feasibility problem can be considered as a special case of a general problem:

minimize 0 x subject to fi (x) ≤ 0 i = 1,..., m hi (x) = 0 i = 1,..., p

where p? = 0 if constraints are feasible and p? = ∞ otherwise.

D. Palomar Convex Problems 8 / 48 Outline

1 Optimization Problems

2 Convex Optimization

3 Quasi-Convex Optimization

4 Classes of Convex Problems: LP, QP, SOCP, SDP

5 Multicriterion Optimization (Pareto Optimality)

6 Solvers Convex Optimization Problem

Convex optimization problem in standard form:

minimize f0 (x) x subject to fi (x) ≤ 0 i = 1,..., m Ax = b

where f0, f1,..., fm are convex and equality constraints are affine. Local and global optima: any locally optimal point of a convex problem is globally optimal. Most problems are not convex when formulated. Reformulating a problem in convex form is an art, there is no systematic way.

D. Palomar Convex Problems 10 / 48 Example

The following problem is nonconvex (why not?):

minimize x2 + x2 x 1 2 2
subject to x1/ 1 + x2 ≤ 0 2 (x1 + x2) = 0 The objective is convex. The equality constraint function is not affine; however, we can rewrite it as x1 = −x2 which is then a linear equality constraint. The inequality constraint function is not convex; however, we can rewrite it as x1 ≤ 0 which again is linear. We can rewrite it as minimize x2 + x2 x 1 2 subject to x1 ≤ 0 x1 = −x2

D. Palomar Convex Problems 11 / 48 Equivalent Reformulations

Eliminating/introducing equality constraints:

minimize f0 (x) x subject to fi (x) ≤ 0 i = 1,..., m Ax = b

is equivalent to

minimize f0 (Fz + x0) z subject to fi (Fz + x0) ≤ 0 i = 1,..., m

where F and x0 are such that Ax = b ⇐⇒ x = Fz + x0 for some z.

D. Palomar Convex Problems 12 / 48 Equivalent Reformulations

Introducing slack variables for linear inequalities:

minimize f0 (x) x T subject to ai x ≤ bi i = 1,..., m is equivalent to

minimize f0 (x) x,s T subject to ai x + si = bi i = 1,..., m si ≥ 0

D. Palomar Convex Problems 13 / 48 Equivalent Reformulations

Epigraph form: a standard form convex problem is equivalent to minimize t x,t subject to f0 (x) − t ≤ 0 fi (x) ≤ 0 i = 1,..., m Ax = b Minimizing over some variables:

minimize f0 (x, y) x,y subject to fi (x) ≤ 0 i = 1,..., m is equivalent to

minimize f˜0 (x) x subject to fi (x) ≤ 0 i = 1,..., m

where f˜0 (x) = infy f0 (x, y).

D. Palomar Convex Problems 14 / 48 Outline

1 Optimization Problems

2 Convex Optimization

3 Quasi-Convex Optimization

4 Classes of Convex Problems: LP, QP, SOCP, SDP

5 Multicriterion Optimization (Pareto Optimality)

6 Solvers Quasiconvex Optimization

Quasi-convex optimization problem in standard form:

minimize f0 (x) x subject to fi (x) ≤ 0 i = 1,..., m Ax = b n where f0 : R −→ R is quasiconvex and f1,..., fm are convex. Observe that it can have locally optimal points that are not (globally) optimal:

D. Palomar Convex Problems 16 / 48 Quasiconvex Optimization

Convex representation of sublevel sets of a quasiconvex function f0: there exists a family of convex functions φt (x) for fixed t such that

f0 (x) ≤ t ⇐⇒ φt (x) ≤ 0.

Example: p (x) f (x) = 0 q (x)

with p convex, q concave, and p (x) ≥ 0, q (x) > 0 on dom f0. We can choose: φt (x) = p (x) − tq (x)

for t ≥ 0, φt (x) is convex in x p (x) /q (x) ≤ t if and only if φt (x) ≤ 0.

D. Palomar Convex Problems 17 / 48 Quasiconvex Optimization

Solving a quasiconvex problem via convex feasibility problems: the idea is to solve the epigraph form of the problem with a sandwich technique in t: for fixed t the epigraph form of the original problem reduces to a feasibility convex problem

φt (x) ≤ 0, fi (x) ≤ 0 ∀i, Ax ≤ b

if t is too small, the feasibility problem will be infeasible if t is too large, the feasibility problem will be feasible start with upper and lower bounds on t (termed u and l, resp.) and use a sandwitch technique (bisection method): at each iteration use t = (l + u) /2 and update the bounds according to the feasibility/infeasibility of the problem.

D. Palomar Convex Problems 18 / 48 Outline

1 Optimization Problems

2 Convex Optimization

3 Quasi-Convex Optimization

4 Classes of Convex Problems: LP, QP, SOCP, SDP

5 Multicriterion Optimization (Pareto Optimality)

6 Solvers Linear Programming (LP)

LP: minimize cT x + d x subject to Gx ≤ h Ax = b

Convex problem: affine objective and constraint functions. Feasible set is a polyhedron:

D. Palomar Convex Problems 20 / 48 `1- and `∞- Norm Problems as LPs

`∞-norm minimization:

minimize kxk x ∞ subject to Gx ≤ h Ax = b is equivalent to the LP

minimize t t,x subject to −t1 ≤ x ≤ t1 Gx ≤ h Ax = b.

D. Palomar Convex Problems 21 / 48 `1- and `∞- Norm Problems as LPs

`1-norm minimization:

minimize kxk x 1 subject to Gx ≤ h Ax = b is equivalent to the LP P minimize ti t,x i subject to −t ≤ x ≤ t Gx ≤ h Ax = b.

D. Palomar Convex Problems 22 / 48 Linear-Fractional Programming

Linear-fractional programming:

minimize cT x + d
/ eT x + f
x subject to Gx ≤ h Ax = b

 T with dom f0 = x | e x + f > 0 . It is a quasiconvex optimization problem (solved by bisection). Interestingly, the following LP is equivalent:

minimize cT y + dz y,z subject to Gy ≤ hz Ay = bz eT y + fz = 1 z ≥ 0

D. Palomar Convex Problems 23 / 48 Quadratic Programming (QP)

Quadratic programming: minimize (1/2) xT Px + qT x + r x subject to Gx ≤ h Ax = b

Convex problem (assuming P ∈ Sn  0): convex quadratic objective and affine constraint functions. Minimization of a convex quadratic function over a polyhedron:

D. Palomar Convex Problems 24 / 48 Quadratically Constrained QP (QCQP)

Quadratically constrained QP:

T T minimize (1/2) x P0x + q x + r0 x 0 T T subject to (1/2) x Pi x + qi x + ri ≤ 0 i = 1,..., m Ax = b

n Convex problem (assuming Pi ∈ S  0): convex quadratic objective and constraint functions.

D. Palomar Convex Problems 25 / 48 Second-Order Cone Programming (SOCP)

Second-order cone programming:

minimize f T x x T subject to kAi x + bi k ≤ ci x + di i = 1,..., m Fx = g

Convex problem: linear objective and second-order cone constraints

For Ai row vector, it reduces to an LP.

For ci = 0, it reduces to a QCQP. More general than QCQP and LP.

D. Palomar Convex Problems 26 / 48 Semidefinite Programming (SDP)

SDP: minimize cT x x subject to x1F1 + x2F2 + ··· + xnFn  G Ax = b

Inequality constraint is called linear matrix inequality (LMI). Convex problem: linear objective and linear matrix inequality (LMI) constraints. Observe that multiple LMI constraints can always be written as a single one.

D. Palomar Convex Problems 27 / 48 Semidefinite Programming (SDP)

LP and equivalent SDP: minimize cT x x minimize cT x subject to Ax ≤ b x subject to diag (Ax − b)  0 SOCP and equivalent SDP:

minimize f T x x T subject to kAi x + bi k ≤ ci x + di , i = 1,..., m

minimize f T x x  T
 ci x + di IAi x + bi subject to T T  0, i = 1,..., m (Ai x + bi ) ci x + di

D. Palomar Convex Problems 28 / 48 Semidefinite Programming (SDP)

Eigenvalue minimization:

minimize λmax (A (x)) x

where A (x) = A0 + x1A1 + ··· + xnAn, is equivalent to SDP

minimize t x subject to A (x)  tI

It follows from

λmax (A (x)) ≤ t ⇐⇒ A (x)  tI

D. Palomar Convex Problems 29 / 48 Outline

1 Optimization Problems

2 Convex Optimization

3 Quasi-Convex Optimization

4 Classes of Convex Problems: LP, QP, SOCP, SDP

5 Multicriterion Optimization (Pareto Optimality)

6 Solvers Scalarization for Multicriterion Problems

Consider a multicriterion optimization with q different objectives:

f0 (x) = (F1 (x) ,..., Fq (x)) .

To find Pareto optimal points, minimize the positive weighted sum:

T λ f0 (x) = λ1F1 (x) + ··· + λqFq (x) .

Example: regularized least-squares:

minimize kAx − bk2 + γ kxk2 . x 2 2

D. Palomar Convex Problems 31 / 48 References on Convex Problems

Chapter 4 of Stephen Boyd and Lieven Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge University Press, 2004.

https://web.stanford.edu/~boyd/cvxbook/

D. Palomar Convex Problems 32 / 48 Outline

1 Optimization Problems

2 Convex Optimization

3 Quasi-Convex Optimization

4 Classes of Convex Problems: LP, QP, SOCP, SDP

5 Multicriterion Optimization (Pareto Optimality)

6 Solvers Solvers

A solver is an engine for solving a particular type of mathematical problem, such as a convex program. Every programming language (e.g., Matlab, Octave, R, Python, C, C++) has a long list of available solvers to choose from. Solvers typically handle only a certain class of problems, such as LPs, QPs, SOCPs, SDPs, or GPs. They also require that problems be expressed in a standard form. Most problems do not immediately present themselves in a standard form, so they must be transformed into standard form.

D. Palomar Convex Problems 34 / 48 Solver Example: Matlab’s linprog

A program for solving LPs:

x = linprog( c, A, b, A_eq, B_eq, l, u )

Problems must be expressed in the following standard form:

minimize cT x x subject to Ax ≤ b Aeqx = beq l ≤ x ≤ u

D. Palomar Convex Problems 35 / 48 Conversion to Standard Form: Common Tricks

Representing free variables as the difference of nonnegative variables:

x free =⇒ x+ − x−, x+ ≥ 0, x− ≥ 0

Elimininating inequality constraints using slack variables:

aT x ≤ b =⇒ aT x + s = b, s ≥ 0

Splitting equality constraints into inequalities:

aT x = b =⇒ aT x ≤ b, aT x ≥ b

D. Palomar Convex Problems 36 / 48 Solver Example: SeDuMi

A program for solving LPs, SOCPs, SDPs, and related problems:

x = sedumi( A, b, c, K )

Solves problems of the form:

minimize cT x x subject to Ax = b x ∈ K , K1 × K2 × · · · × KL

n where each set Ki ⊆ R i , i = 1, 2,..., L is chosen from a very short list of cones. The Matlab variable K gives the number, types, and dimensions of the cones Ki .

D. Palomar Convex Problems 37 / 48 Solver Example: SeDuMi

Cones supported by SeDuMi: free variables: Rni ni a nonnegative orthant: R+ (for linear inequalities) a real or complex second-order cone:

n n Q , {(x, y) ∈ R × R | kxk2 ≤ y} n n Qc , {(x, y) ∈ C × R | kxk2 ≤ y} a real or complex semidefinite cone:

n n n×n T o S+ , X ∈ R | X = X , X  0 n n n×n H o H+ , X ∈ C | X = X , X  0

The cones must be arranged in this order, i.e., the free variables first, then the nonnegative orthants, then the second-order cones, then the semidefinite cones. D. Palomar Convex Problems 38 / 48 Example: Norm Approximation

Consider the norm approximation problem:

minimize kAx − bk x

An optimal value x? minimizes the residuals

T rk = ak x − bk , k = 1, 2,..., m

according to the measure defined by the norm k·k Obviously, the value of x? depends significantly upon the choice of that norm...... and so does the process of conversion to standard form

D. Palomar Convex Problems 39 / 48 Example: Euclidean or `2-Norm

Norm approximation problem:

 1/2 minimize kAx − bk = Pm aT x − b
2 x 2 k=1 k k

No need to use any solver here: this is a least squares (LS) problem, with an analytic solution:

x? = (AT A)−1AT b

In Matlab or Octave, a single command computes the solution: >‌> x = A \ b Similarly, in R: >‌> x = solve(A, b)

D. Palomar Convex Problems 40 / 48 Example: Chebyshev or `∞-Norm

Norm approximation problem:

T minimize kAx − bk∞ = minimize max ak x − bk x x 1≤k≤m

This can be expressed as a linear program:

minimize t x,t subject to −t1 ≤ Ax − b ≤ t1

or, equivalently,

 x  minimize  0T 1  x,t t  A −1   x   b  subject to ≤ −A −1 t −b

D. Palomar Convex Problems 41 / 48 Example: Chebyshev or `∞-Norm

Recall the final formulation:  x  minimize  0T 1  x,t t  A −1   x   b  subject to ≤ −A −1 t −b

Matlab’s linprog call: >‌> xt = linprog( [zeros(n,1); 1], ... [A,-ones(m,1); -A,-ones(m,1)], ... [b; -b] ) >‌> x = xt(1:n)

D. Palomar Convex Problems 42 / 48 Example: Manhattan or `1-Norm

Norm approximation problem:

Pm T minimize kAx − bk1 = minimize k=1 ak x − bk x x

This can be expressed as a linear program:

minimize 1T t x,t subject to −t ≤ Ax − b ≤ t

or, equivalently,

 x  minimize  0T 1T  x,t t  A −I   x   b  subject to ≤ −A −I t −b

D. Palomar Convex Problems 43 / 48 Example: Manhattan or `1-Norm

Recall the final formulation:  x  minimize  0T 1T  x,t t  A −I   x   b  subject to ≤ −A −I t −b

Matlab’s linprog call: >‌> xt = linprog( [zeros(n,1); ones(n,1)], ... [A,-eye(m,1); -A,-eye(m,1)], ... [b; -b] ) >‌> x = xt(1:n)

D. Palomar Convex Problems 44 / 48 Example: Constrained Euclidean or `2-Norm Constrained norm approximation problem: minimize kAx − bk x 2 subject to Cx = d l ≤ x ≤ u

This is not a least squares problem, but it is QP and an SOCP. This can be expressed as minimize t x,y,t,sl ,su subject to Ax − b = y Cx = d x − sl = l x + su = u sl , su ≥ 0 kyk2 ≤ t

D. Palomar Convex Problems 45 / 48 Example: Constrained Euclidean or `2-Norm

Equivalently: minimize  0T 0T 0T 0T 1  x¯ x,y,t,sl ,su  A −I   b   C   d  subject to   x¯ ≤    I −I   l  II u n n n m x¯ ∈ R × R+ × R+ × Q SeDuMi call: >‌> AA = [ A, zeros(m,n), zeros(m,n), -eye(m), 0; C, zeros(p,n), zeros(p,n) zeros(p,n), 0; eye(n), -eye(n), zeros(n,n), zeros(n,n), 0; eye(n), zeros(n,n), eye(n), zeros(n,n), 0 ] >‌> bb = [ b; d; l; u ] >‌> cc = [ zeros(3*n+m,1); 1 ] >‌> K.f = n; K.l = 2*n; K.q = m + 1; >‌> xsyz = sedumi( AA, bb, cc, K ) >‌> x = xsyz(1:n) D. Palomar Convex Problems 46 / 48 Modeling Frameworks: cvx

A modeling framework simplifies the use of a numerical technology by shielding the user from the underlying mathematical details. Examples: SDPSOL, YALMIP, cvx, etc. cvx is designed to support convex optimization or, more specifically, disciplined convex programming (available in Matlab, Python, R, Julia, Octave, etc.) People don’t simply write optimization problems and hope that they are convex; instead they draw from a "mental library"of functions and sets with known convexity properties and combine them in ways that convex analysis guarantees will produce convex results. Disciplined convex programming formalizes this methodology. Links: cvx: http://cvxr.com cvx user guide: http://web.cvxr.com/cvx/doc/CVX.pdf cvx for R (cvxr): https://github.com/cvxgrp/CVXR

D. Palomar Convex Problems 47 / 48 Thanks

For more information visit:

https://www.danielppalomar.com