Solvers for Convex Optimization Problems Prof. Daniel P. Palomar

Home , Convex optimization

Solvers for Convex Optimization Problems

Prof. Daniel P. Palomar The Hong Kong University of Science and Technology (HKUST)

ELEC5470/IEDA6100A - Convex Optimization Fall 2020-21, HKUST, Hong Kong Outline

1 What’s a Solver?

2 Examples of Solvers Matlab’s linprog SeDuMi

3 Example: Norm Approximation

4 Modeling Frameworks Outline

1 What’s a Solver?

2 Examples of Solvers Matlab’s linprog SeDuMi

3 Example: Norm Approximation

4 Modeling Frameworks Solvers

A solver is an engine for solving a particular type of mathematical problem, such as a convex program. Every programming language (e.g., Matlab, Octave, R, Python, C, C++) has a long list of available solvers to choose from. Solvers typically handle only a certain class of problems, such as LPs, QPs, SOCPs, SDPs, or GPs. They also require that problems be expressed in a standard form. Most problems do not immediately present themselves in a standard form, so they must be transformed into standard form.

D. Palomar Convex Problems 4 / 23 Outline

1 What’s a Solver?

2 Examples of Solvers Matlab’s linprog SeDuMi

3 Example: Norm Approximation

4 Modeling Frameworks Outline

1 What’s a Solver?

2 Examples of Solvers Matlab’s linprog SeDuMi

3 Example: Norm Approximation

4 Modeling Frameworks Solver Example: Matlab’s linprog

A program for solving LPs:

x = linprog( c, A, b, A_eq, B_eq, l, u )

Problems must be expressed in the following standard form:

minimize cT x x subject to Ax ≤ b Aeqx = beq l ≤ x ≤ u

As standard forms go, this one is quite ﬂexible: in fact, the ﬁrst step linprog often does is to convert this problem into a more restricted standard form!

D. Palomar Convex Problems 7 / 23 Conversion to Standard Form: Common Tricks

Representing free variables as the diﬀerence of nonnegative variables:

x free =⇒ x+ − x−, x+ ≥ 0, x− ≥ 0

Elimininating inequality constraints using slack variables:

aT x ≤ b =⇒ aT x + s = b, s ≥ 0

Splitting equality constraints into inequalities:

aT x = b =⇒ aT x ≤ b, aT x ≥ b

D. Palomar Convex Problems 8 / 23 Outline

1 What’s a Solver?

2 Examples of Solvers Matlab’s linprog SeDuMi

3 Example: Norm Approximation

4 Modeling Frameworks Solver Example: SeDuMi

A program for solving LPs, SOCPs, SDPs, and related problems:

x = sedumi( A, b, c, K )

Solves problems of the form:

minimize cT x x subject to Ax = b x ∈ K , K1 × K2 × · · · × KL

n where each set Ki ⊆ R i , i = 1, 2,..., L is chosen from a very short list of cones.

The Matlab variable K gives the number, types, and dimensions of the cones Ki .

D. Palomar Convex Problems 10 / 23 Solver Example: SeDuMi

Cones supported by SeDuMi: free variables: Rni ni a nonnegative orthant: R+ (for linear inequalities) a real or complex second-order cone:

n n Q , {(x, y) ∈ R × R | kxk2 ≤ y} n n Qc , {(x, y) ∈ C × R | kxk2 ≤ y} a real or complex semideﬁnite cone:

n n n×n T o S+ , X ∈ R | X = X , X 0 n n n×n H o H+ , X ∈ C | X = X , X 0

The cones must be arranged in this order, i.e., the free variables ﬁrst, then the nonnegative orthants, then the second-order cones, then the semideﬁnite cones. D. Palomar Convex Problems 11 / 23 Outline

1 What’s a Solver?

2 Examples of Solvers Matlab’s linprog SeDuMi

3 Example: Norm Approximation

4 Modeling Frameworks Example: Norm Approximation

Consider the norm approximation problem:

minimize kAx − bk x

An optimal value x? minimizes the residuals

T rk = ak x − bk , k = 1, 2,..., m

according to the measure deﬁned by the norm k·k Obviously, the value of x? depends signiﬁcantly upon the choice of that norm...... and so does the process of conversion to standard form

D. Palomar Convex Problems 13 / 23 Example: Euclidean or `2-Norm

Norm approximation problem:

1/2 minimize kAx − bk = Pm aT x − b 2 x 2 k=1 k k

No need to use any solver here: this is a least squares (LS) problem, with an analytic solution: x? = (AT A)−1AT b In Matlab or Octave, a single command computes the solution: >‌> x = A \ b Similarly, in R: >‌> x = solve(A, b)

D. Palomar Convex Problems 14 / 23 Example: Chebyshev or `∞-Norm

Norm approximation problem:

T minimize kAx − bk∞ = minimize max ak x − bk x x 1≤k≤m

This can be expressed as a linear program:

minimize t x,t subject to −t1 ≤ Ax − b ≤ t1

or, equivalently, x minimize 0T 1 x,t t A −1 x b subject to ≤ −A −1 t −b

D. Palomar Convex Problems 15 / 23 Example: Chebyshev or `∞-Norm

Recall the ﬁnal formulation: x minimize 0T 1 x,t t A −1 x b subject to ≤ −A −1 t −b

Matlab’s linprog call: >‌> xt = linprog( [zeros(n,1); 1], ... [A,-ones(m,1); -A,-ones(m,1)], ... [b; -b] ) >‌> x = xt(1:n)

D. Palomar Convex Problems 16 / 23 Example: Manhattan or `1-Norm

Norm approximation problem:

Pm T minimize kAx − bk1 = minimize k=1 ak x − bk x x

This can be expressed as a linear program:

minimize 1T t x,t subject to −t ≤ Ax − b ≤ t

or, equivalently, x minimize 0T 1T x,t t A −I x b subject to ≤ −A −I t −b

D. Palomar Convex Problems 17 / 23 Example: Manhattan or `1-Norm

Recall the ﬁnal formulation: x minimize 0T 1T x,t t A −I x b subject to ≤ −A −I t −b

Matlab’s linprog call: >‌> xt = linprog( [zeros(n,1); ones(n,1)], ... [A,-eye(m,1); -A,-eye(m,1)], ... [b; -b] ) >‌> x = xt(1:n)

D. Palomar Convex Problems 18 / 23 Example: Constrained Euclidean or `2-Norm Constrained norm approximation problem: minimize kAx − bk x 2 subject to Cx = d l ≤ x ≤ u

This is not a least squares problem, but it is QP and an SOCP. This can be expressed as minimize t x,y,t,sl ,su subject to Ax − b = y Cx = d x − sl = l x + su = u sl , su ≥ 0 kyk2 ≤ t

D. Palomar Convex Problems 19 / 23 Example: Constrained Euclidean or `2-Norm T T T T Equivalently: minimize 0 0 0 0 1 ¯x x,y,t,sl ,su  A −I   b   C   d  subject to   ¯x ≤    I −I   l  II u n n n m ¯x ∈ R × R+ × R+ × Q SeDuMi call: >‌> AA = [ A, zeros(m,n), zeros(m,n), -eye(m), 0; C, zeros(p,n), zeros(p,n) zeros(p,n), 0; eye(n), -eye(n), zeros(n,n), zeros(n,n), 0; eye(n), zeros(n,n), eye(n), zeros(n,n), 0 ] >‌> bb = [ b; d; l; u ] >‌> cc = [ zeros(3*n+m,1); 1 ] >‌> K.f = n; K.l = 2*n; K.q = m + 1; >‌> xsyz = sedumi( AA, bb, cc, K ) >‌> x = xsyz(1:n) D. Palomar Convex Problems 20 / 23 Outline

1 What’s a Solver?

2 Examples of Solvers Matlab’s linprog SeDuMi

3 Example: Norm Approximation

4 Modeling Frameworks Modeling Frameworks: cvx

A modeling framework simpliﬁes the use of a numerical technology by shielding the user from the underlying mathematical details. Examples: SDPSOL, YALMIP, cvx, etc. cvx is designed to support convex optimization or, more speciﬁcally, disciplined convex programming (available in Matlab, Python, R, Julia, Octave, etc.) People don’t simply write optimization problems and hope that they are convex; instead they draw from a "mental library"of functions and sets with known convexity properties and combine them in ways that convex analysis guarantees will produce convex results. Disciplined convex programming formalizes this methodology. Links: cvx: http://cvxr.com cvx user guide: http://web.cvxr.com/cvx/doc/CVX.pdf cvx for R (cvxr): https://github.com/cvxgrp/CVXR

D. Palomar Convex Problems 22 / 23 Thanks

For more information visit:

https://www.danielppalomar.com