CS/ECE/ISyE 524 Introduction to Optimization Spring 2017{18 13. Cones and semidefinite constraints Geometry of cones Second order cone programs Example: robust linear program Semidefinite constraints Laurent Lessard (www.laurentlessard.com) What is a cone? A set of points C 2 Rn is called a cone if it satisfies: I αx 2 C whenever x 2 C and α > 0. I x + y 2 C whenever x 2 C and y 2 C. Similar to a subspace, but α > 0 instead of α 2 R. (this is a critical difference!) Simple examples: jxj ≤ y and y ≥ 0 1.5 1.5 1.0 1.0 0.5 0.5 -1.5 -1.0 -0.5 0.5 1.0 1.5 -1.5 -1.0 -0.5 0.5 1.0 1.5 13-2 What is a cone? A slice of a cone is its intersection with a subspace. We are interested in convex cones (all slices are convex). Can be polyhedral, ellipsoidal, or something else... 13-3 What is a cone? Polyhedral cone recipe: 1. Begin with your favorite polyhedron Ax ≤ b where x 2 Rn 2. fAx ≤ bt; t ≥ 0g is a polyhedral cone in (x; t) 2 Rn+1 3. The slice t = 1 is the original polyhedron. 2.5 2.0 1.5 1.0 0.5 0.5 1.0 1.5 2.0 2.5 13-4 What is a cone? Ellipsoidal cone recipe: 1. Ellipsoid x TPx + qTx + r ≤ 0 where P 0 and x 2 Rn 2. Complete the square () kAx + bk ≤ c 3. fkAx + btk ≤ ctg is an ellipsoidal cone in (x; t) 2 Rn+1 4. The slice t = 1 is the original ellipsoid. 2.5 2.0 1.5 1.0 0.5 0.5 1.0 1.5 2.0 2.5 13-5 Second-order cone constraint A second-order cone constraint is the set of points x 2 Rn: kAx + bk ≤ cTx + d Every SOC constraint is a slice (set t = 1) of the cone kAx + btk ≤ cTx + dt. It's not always a cone itself! Special cases: If A = 0, we have a linear inequality (hyperplane) If c = 0, it's a slice of an ellipsoidal cone Every SOC constraint describes a convex set. 13-6 Second-order cone constraint A second-order cone constraint is the set of points x 2 Rn: kAx + bk ≤ cTx + d If you square both sides... ( kAx + bk2 ≤ (cTx + d)2 kAx + bk ≤ cTx + d () cTx + d ≥ 0 The quadratic inequality is: x T(ATA − ccT)x + 2(bTA − dcT)x + (bTb − d 2) ≤ 0 This may be nonconvex! 13-7 Second-order cone constraint A second-order cone constraint is the set of points x 2 Rn: kAx + bk ≤ cTx + d Example: 1.5 1.0 0 If A = 1 0 and c = and b = d = 0: 0.5 1 -1.5 -1.0 -0.5 0.5 1.0 1.5 -0.5 jxj ≤ y 1.5 1.0 Squaring both sides leads to: 0.5 2 2 -1.5 -1.0 -0.5 0.5 1.0 1.5 x − y ≤ 0 and y ≥ 0 -0.5 13-8 Special case: rotated second-order cone A rotated second-order cone is the set x 2 Rn, y; z 2 R: x Tx ≤ yz; y ≥ 0; z ≥ 0 With n = 1, this looks like: 13-9 Special case: rotated second-order cone A rotated second-order cone is the set x 2 Rn, y; z 2 R: x Tx ≤ yz; y ≥ 0; z ≥ 0 Can put into standard form: 4x Tx ≤ 4yz 4x Tx + y 2 + z 2 ≤ 4yz + y 2 + z 2 4x Tx + (y − z)2 ≤ (y + z)2 p4x Tx + (y − z)2 ≤ y + z 2x ≤ y + z y − z 13-10 SOCPs A second-order cone program (SOCP) has the form: minimize cTx x T subject to: kAi x + bi k ≤ ci x + di for i = 1;:::; m Every LP is an SOCP (just make each Ai = 0) Every convex QP and QCQP is an SOCP I convert quadratic cost to epigraph form (add a variable) I convert quadratic constraints to SOCP (complete square) 13-11 Implementation details A second-order cone program (SOCP) has the form: minimize cTx x T subject to: kAi x + bi k ≤ ci x + di for i = 1;:::; m In JuMP, you can specify SOCP using: @constraint(m, norm(A*x+b) <= dot(c,x)+d) works with ECOS, SCS, Mosek, Gurobi, Ipopt. Can also specify rotated cones directly in Mosek, Ipopt. 13-12 Example: robust LP Consider a linear program with each linear constraint separately written out: maximize cTx x T subject to: ai x ≤ bi for i = 1;:::; m Suppose there is uncertainty in some of the ai vectors. Say for example that ai =a ¯i + ρu wherea ¯i is a nominal value and u is the uncertainty. box constraint: kuk1 ≤ 1 ball constraints: kuk2 ≤ 1 13-13 Robust LP with box constraint T Substituting ai =a ¯i + ρu into ai x ≤ bi , obtain: T T a¯i x + ρu x ≤ bi for all uncertain u box constraint: If this must hold for all u with kuk1 ≤ 1, then it holds for the worst-case u. Therefore: n n n T X X X u x = ui xi ≤ jui jjxi j ≤ jxi j = kxk1 i=1 i=1 i=1 Then we have T a¯i x + ρkxk1 ≤ bi 13-14 Robust LP with box constraint With a box constraint ai =a ¯i + ρu with kuk1 ≤ 1 maximize cTx x T subject to: ai x ≤ bi for i = 1;:::; m Is equivalent to the optimization problem maximize cTx x T subject to:a ¯i x + ρkxk1 ≤ bi for i = 1;:::; m 13-15 Robust LP with box constraint With a box constraint ai =a ¯i + ρu with kuk1 ≤ 1 maximize cTx x T subject to: ai x ≤ bi for i = 1;:::; m ... which is equivalent to the linear program: maximize cTx x;t n T X subject to:a ¯i x + ρ tj ≤ bi for i = 1;:::; m j=1 − tj ≤ xj ≤ tj for j = 1;:::; n 13-16 Robust LP with box constraint 1.5 1.5 1.0 1.0 0.5 0.5 -1.5 -1.0 -0.5 0.5 1.0 1.5 2.0 -1.5 -1.0 -0.5 0.5 1.0 1.5 2.0 -0.5 -0.5 -1.0 -1.0 -1.5 -1.5 T T ai x ≤ bi ai x + 0:2kxk1 ≤ bi New region is smaller, still a polyhedron More robust to uncertain constraints 13-17 Robust LP with ball constraint T Substituting ai =a ¯i + ρu into ai x ≤ bi , obtain: T T a¯i x + ρu x ≤ bi for all uncertain u ball constraint: If this must hold for all u with kuk2 ≤ 1, then it holds for the worst-case u. Using Cauchy-Schwarz inequality: T u x ≤ kuk2kxk2 ≤ kxk2 Then we have T a¯i x + ρkxk2 ≤ bi (a second-order cone constraint!) 13-18 Robust LP with ball constraint With a ball constraint ai =a ¯i + ρu with kuk2 ≤ 1 maximize cTx x T subject to: ai x ≤ bi for i = 1;:::; m Is equivalent to the optimization problem maximize cTx x T subject to:a ¯i x + ρkxk2 ≤ bi for i = 1;:::; m which is an SOCP 13-19 Robust LP with ball constraint 1.5 1.5 1.0 1.0 0.5 0.5 -1.5 -1.0 -0.5 0.5 1.0 1.5 2.0 -1.5 -1.0 -0.5 0.5 1.0 1.5 2.0 -0.5 -0.5 -1.0 -1.0 -1.5 -1.5 T T ai x ≤ bi ai x + 0:2kxk2 ≤ bi New region is smaller, no longer a polyhedron More robust to uncertain constraints 13-20 Matrix variables Sometimes, the decision variable is a matrix X . Can always just think of X 2 Rm×n as x 2 Rmn. Linear functions: mn X T ck xk = c x k=1 m n X X T Cij Xij = trace(C X ) i=1 j=1 Linear program: maximize trace(C TX ) X T subject to: trace(Ai X ) ≤ bi for i = 1;:::; k 13-21 Matrix variables If a decision variable is a symmetric matrix X = X T 2 Rn×n, we can represent it as a vector x 2 Rn(n+1)=2. 2 3 x1 2 3 6x27 x1 x2 x3 6 7 6x37 4x2 x4 x55 () 6 7 6x47 x3 x5 x6 6 7 4x55 x6 The constraint X 0 is called a semidefinite constraint. What does it look like geometrically? 13-22 The PSD cone The set of matrices X 0 are a convex cone in Rn(n+1)=2 x y Example: The set 0 of points in 3 satisfy: y z R xz ≥ y 2; x ≥ 0; z ≥ 0 This is a rotated second-order cone! Equivalent to: 2y ≤ x + z x − z 13-23 More complicated example "1 x y# The set of (x; y; z) satisfying x 1 z 0 is the solution of: y z 1 3×3 X 2 R ; X 0; X11 = 1; X22 = 1; X33 = 1 13-24 Spectrahedra Two common set representations: T I variables x1;:::; xk , constants Qi = Qi , and constraint: Q0 + x1Q1 + ::: xk Qk 0 (linear matrix inequality) I variable X 0 and the constraints: T trace(Ai X ) ≤ bi (linear constraint form) These sets are called spectrahedra.