Convex Optimization Problems Prof. Daniel P. Palomar The Hong Kong University of Science and Technology (HKUST) MAFS6010R - Portfolio Optimization with R MSc in Financial Mathematics Fall 2019-20, HKUST, Hong Kong Outline 1 Optimization Problems 2 Convex Optimization 3 Quasi-Convex Optimization 4 Classes of Convex Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers Outline 1 Optimization Problems 2 Convex Optimization 3 Quasi-Convex Optimization 4 Classes of Convex Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers Optimization Problem in Standard Form General optimization problem in standard form: minimize f0 (x) x subject to fi (x) ≤ 0 i = 1;:::; m hi (x) = 0 i = 1;:::; p where x = (x1;:::; xn) is the optimization variable n f0 : R −! R is the objective function n fi : R −! R; i = 1;:::; m are inequality constraint functions n hi : R −! R; i = 1;:::; p are equality constraint functions. ? Goal: find an optimal solution x that minimizes f0 while satisfying all the constraints. D. Palomar Convex Problems 4 / 48 Optimization Problem in Standard Form Feasibility: a point x 2 dom f0 is feasible if it satisfies all the constraints and infeasible otherwise a problem is feasible if it has at least one feasible point and infeasible otherwise. Optimal value: ? p = inf ff0 (x) j fi (x) ≤ 0; i = 1;:::; m; hi (x) = 0; i = 1;:::; pg p? = 1 if problem infeasible (no x satisfies the constraints) p? = −∞ if problem unbounded below. Optimal solution: x? such that f (x?) = p? (and x? feasible). D. Palomar Convex Problems 5 / 48 Global and Local Optimality ? A feasible x is optimal if f0 (x) = p ; Xopt is the set of optimal points. A feasible x is locally optimal if is optimal within a ball Examples: ? f0 (x) = 1=x, dom f0 = R++: p = 0, no optimal point 3 ? f0 (x) = x − 3x: p = −∞, local optimum at x = 1. D. Palomar Convex Problems 6 / 48 Implicit Constraints The standard form optimization problem has an explicit constraint: m p \ \ x 2 D = dom fi \ dom fi i=0 i=1 D is the domain of the problem The constraints fi (x) ≤ 0; hi (x) = 0 are the explicit constraints A problem is unconstrained if it has no explicit constraints Example: minimize log b − aT x x is an unconstrained problem with implicit constraint b > aT x. D. Palomar Convex Problems 7 / 48 Feasibility Problem Sometimes, we don’t really want to minimize any objective, just to find a feasible point: find x x subject to fi (x) ≤ 0 i = 1;:::; m hi (x) = 0 i = 1;:::; p This feasibility problem can be considered as a special case of a general problem: minimize 0 x subject to fi (x) ≤ 0 i = 1;:::; m hi (x) = 0 i = 1;:::; p where p? = 0 if constraints are feasible and p? = 1 otherwise. D. Palomar Convex Problems 8 / 48 Outline 1 Optimization Problems 2 Convex Optimization 3 Quasi-Convex Optimization 4 Classes of Convex Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers Convex Optimization Problem Convex optimization problem in standard form: minimize f0 (x) x subject to fi (x) ≤ 0 i = 1;:::; m Ax = b where f0; f1;:::; fm are convex and equality constraints are affine. Local and global optima: any locally optimal point of a convex problem is globally optimal. Most problems are not convex when formulated. Reformulating a problem in convex form is an art, there is no systematic way. D. Palomar Convex Problems 10 / 48 Example The following problem is nonconvex (why not?): minimize x2 + x2 x 1 2 2 subject to x1= 1 + x2 ≤ 0 2 (x1 + x2) = 0 The objective is convex. The equality constraint function is not affine; however, we can rewrite it as x1 = −x2 which is then a linear equality constraint. The inequality constraint function is not convex; however, we can rewrite it as x1 ≤ 0 which again is linear. We can rewrite it as minimize x2 + x2 x 1 2 subject to x1 ≤ 0 x1 = −x2 D. Palomar Convex Problems 11 / 48 Equivalent Reformulations Eliminating/introducing equality constraints: minimize f0 (x) x subject to fi (x) ≤ 0 i = 1;:::; m Ax = b is equivalent to minimize f0 (Fz + x0) z subject to fi (Fz + x0) ≤ 0 i = 1;:::; m where F and x0 are such that Ax = b () x = Fz + x0 for some z. D. Palomar Convex Problems 12 / 48 Equivalent Reformulations Introducing slack variables for linear inequalities: minimize f0 (x) x T subject to ai x ≤ bi i = 1;:::; m is equivalent to minimize f0 (x) x;s T subject to ai x + si = bi i = 1;:::; m si ≥ 0 D. Palomar Convex Problems 13 / 48 Equivalent Reformulations Epigraph form: a standard form convex problem is equivalent to minimize t x;t subject to f0 (x) − t ≤ 0 fi (x) ≤ 0 i = 1;:::; m Ax = b Minimizing over some variables: minimize f0 (x; y) x;y subject to fi (x) ≤ 0 i = 1;:::; m is equivalent to minimize f~0 (x) x subject to fi (x) ≤ 0 i = 1;:::; m where f~0 (x) = infy f0 (x; y). D. Palomar Convex Problems 14 / 48 Outline 1 Optimization Problems 2 Convex Optimization 3 Quasi-Convex Optimization 4 Classes of Convex Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers Quasiconvex Optimization Quasi-convex optimization problem in standard form: minimize f0 (x) x subject to fi (x) ≤ 0 i = 1;:::; m Ax = b n where f0 : R −! R is quasiconvex and f1;:::; fm are convex. Observe that it can have locally optimal points that are not (globally) optimal: D. Palomar Convex Problems 16 / 48 Quasiconvex Optimization Convex representation of sublevel sets of a quasiconvex function f0: there exists a family of convex functions φt (x) for fixed t such that f0 (x) ≤ t () φt (x) ≤ 0: Example: p (x) f (x) = 0 q (x) with p convex, q concave, and p (x) ≥ 0, q (x) > 0 on dom f0. We can choose: φt (x) = p (x) − tq (x) for t ≥ 0, φt (x) is convex in x p (x) =q (x) ≤ t if and only if φt (x) ≤ 0. D. Palomar Convex Problems 17 / 48 Quasiconvex Optimization Solving a quasiconvex problem via convex feasibility problems: the idea is to solve the epigraph form of the problem with a sandwich technique in t: for fixed t the epigraph form of the original problem reduces to a feasibility convex problem φt (x) ≤ 0; fi (x) ≤ 0 8i; Ax ≤ b if t is too small, the feasibility problem will be infeasible if t is too large, the feasibility problem will be feasible start with upper and lower bounds on t (termed u and l, resp.) and use a sandwitch technique (bisection method): at each iteration use t = (l + u) =2 and update the bounds according to the feasibility/infeasibility of the problem. D. Palomar Convex Problems 18 / 48 Outline 1 Optimization Problems 2 Convex Optimization 3 Quasi-Convex Optimization 4 Classes of Convex Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers Linear Programming (LP) LP: minimize cT x + d x subject to Gx ≤ h Ax = b Convex problem: affine objective and constraint functions. Feasible set is a polyhedron: D. Palomar Convex Problems 20 / 48 `1- and `1- Norm Problems as LPs `1-norm minimization: minimize kxk x 1 subject to Gx ≤ h Ax = b is equivalent to the LP minimize t t;x subject to −t1 ≤ x ≤ t1 Gx ≤ h Ax = b: D. Palomar Convex Problems 21 / 48 `1- and `1- Norm Problems as LPs `1-norm minimization: minimize kxk x 1 subject to Gx ≤ h Ax = b is equivalent to the LP P minimize ti t;x i subject to −t ≤ x ≤ t Gx ≤ h Ax = b: D. Palomar Convex Problems 22 / 48 Linear-Fractional Programming Linear-fractional programming: minimize cT x + d = eT x + f x subject to Gx ≤ h Ax = b T with dom f0 = x j e x + f > 0 . It is a quasiconvex optimization problem (solved by bisection). Interestingly, the following LP is equivalent: minimize cT y + dz y;z subject to Gy ≤ hz Ay = bz eT y + fz = 1 z ≥ 0 D. Palomar Convex Problems 23 / 48 Quadratic Programming (QP) Quadratic programming: minimize (1=2) xT Px + qT x + r x subject to Gx ≤ h Ax = b Convex problem (assuming P 2 Sn 0): convex quadratic objective and affine constraint functions. Minimization of a convex quadratic function over a polyhedron: D. Palomar Convex Problems 24 / 48 Quadratically Constrained QP (QCQP) Quadratically constrained QP: T T minimize (1=2) x P0x + q x + r0 x 0 T T subject to (1=2) x Pi x + qi x + ri ≤ 0 i = 1;:::; m Ax = b n Convex problem (assuming Pi 2 S 0): convex quadratic objective and constraint functions. D. Palomar Convex Problems 25 / 48 Second-Order Cone Programming (SOCP) Second-order cone programming: minimize f T x x T subject to kAi x + bi k ≤ ci x + di i = 1;:::; m Fx = g Convex problem: linear objective and second-order cone constraints For Ai row vector, it reduces to an LP. For ci = 0, it reduces to a QCQP. More general than QCQP and LP. D. Palomar Convex Problems 26 / 48 Semidefinite Programming (SDP) SDP: minimize cT x x subject to x1F1 + x2F2 + ··· + xnFn G Ax = b Inequality constraint is called linear matrix inequality (LMI). Convex problem: linear objective and linear matrix inequality (LMI) constraints. Observe that multiple LMI constraints can always be written as a single one. D. Palomar Convex Problems 27 / 48 Semidefinite Programming (SDP) LP and equivalent SDP: minimize cT x x minimize cT x subject to Ax ≤ b x subject to diag (Ax − b) 0 SOCP and equivalent SDP: minimize f T x x T subject to kAi x + bi k ≤ ci x + di ; i = 1;:::; m minimize f T x x T ci x + di IAi x + bi subject to T T 0; i = 1;:::; m (Ai x + bi ) ci x + di D.