2 - 1 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 2. Convexity and Duality
Convex sets and functions Convex optimization problems Standard problems: LP and SDP Feasibility problems Algorithms Certi cates and separating hyperplanes Duality and geometry Examples: LP and and SDP Theorems of alternatives 2 - 2 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Basic Nomenclature
A set S Rn is called a ne if x, y S implies x + (1 )y S for all R; i.e., the line through x,∈ y is contained in S ∈ ∈ convex if x, y S implies x + (1 )y S for all [0, 1]; i.e., the line segment∈ between x and y is contained∈ in S. ∈ a convex cone if x, y S implies x + y S for all , 0; i.e., the pie slice between x∈and y is contained in∈S.
A function f : Rn R is called → a ne if f( x + (1 )y) = f(x) + (1 )f(y) for all R and ∈ x, y Rn; i.e., f is equals a linear function plus a constant f = Ax+b ∈ convex if f x + (1 )y f(x) + (1 )f(y) for all [0, 1] ∈ and x, y Rn ∈ 2 - 3 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Properties of Convex Functions f + f is convex if f and f are 1 2 1 2 f(x) = max f (x), f (x) is convex if f and f are { 1 2 } 1 2 g(x) = sup f(x, y) is convex if f(x, y) is convex in x for each y y convex functions are continuous on the interior of their domain f(Ax + b) is convex if f is Af(x) + b is convex if f is g(x) = infy f(x, y) is convex if f(x, y) is jointly convex the sublevel set n x R f(x) { ∈ | } is convex if f is convex; (the converse is not true) 2 - 4 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Convex Optimization Problems
minimize f0(x) subject to f (x) 0 for all i = 1, . . . , m i hi(x) = 0 for all i = 1, . . . , p
This problem is called a convex program if the objective function f is convex 0 the inequality constraints f are convex i the equality constraints h are a ne i 2 - 5 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Linear Programming (LP) In a linear program, the objective and constraint functions are a ne.
minimize cT x subject to Ax = b Cx d Example
minimize x1 + x2 subject to 3x + x 3 1 2 x 1 2 x 4 1 x + 5x 20 1 2 x + 4x 20 1 2 2 - 6 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Linear Programming Every linear program may be written in the standard primal form
minimize cT x subject to Ax = b x 0 n Here x R , and x 0 means xi 0 for all i ∈
The nonnegative orthant x Rn x 0 is a convex cone. ∈ | This convex cone de nes the partial ordering on Rn Geometrically, the feasible set is the intersection of an a ne set with a convex cone. 2 - 7 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Semide nite Programming
minimize trace CX subject to trace AiX = bi for all i = 1, . . . , m X 0
The variable X is in the set of n n symmetric matrices n n n T S = A R A = A ∈ | X 0 means X is positive semide nite As for LP, the feasible set is the intersection of an a ne set with a convex cone, in this case the positive semide nite cone n X S X 0 ∈ | Hence the feasible set is convex. 2 - 8 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 SDPs with Explicit Variables We can also explicitly parametrize the a ne set to give
minimize cT x subject to F + x F + x F + + xnFn 0 0 1 1 2 2 where F0, F1, . . . , Fn are symmetric matrices.
The inequality constraint is called a linear matrix inequality; e.g., x 3 x + x 1 1 1 2 x1 + x2 x2 4 0 0 1 0 x 1 which is equivalent to 3 0 1 1 1 0 0 1 0 0 4 0 + x1 1 0 0 + x2 1 1 0 0 1 0 0 0 0 1 0 0 0 2 - 9 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 The Feasible Set is Semialgebraic
The (basic closed) semialgebraic set de ned by polynomials f1, . . . , fm is
n x R fi(x) 0 for all i = 1, . . . , m ∈ | n o
The feasible set of an SDP is a semialgebraic set.
Because a matrix A 0 if and only if det(Ak) > 0 for k = 1, . . . , n where Ak is the submatrix of A consisting of the rst k rows and columns. 2 - 10 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 The Feasible Set
For example 3 x (x + x ) 1 1 1 2 0 (x1 + x2) 4 x2 0 1 0 x 1 is equivalent to the polynomial inequalities
0 < 3 x 1 0 < (3 x )(4 x ) (x + x )2 1 2 1 2 0 < x ((3 x )(4 x ) (x + x )2) (4 x ) 1 1 2 1 2 2 2 - 11 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Feasible Sets of SDP If S is the feasible set of an SDP, then S is de ned by polynomials. m m S = x R A0 + Aixi 0 ∈ | n Xi=1 o In fact, S is the closure of the connected component containing 0 of m C = x R f(x) > 0 ∈ | where f = det A + m A x and A 0 0 i=1 i i 0 P Question: for what polynomials f is C the feasible set of an SDP?
What about (x, y) x4 + y4 1 ? It is convex and semialgebraic | 2 - 12 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 A Necessary Condition A simple necessary condition Example: x4 + y4 < 1 cannot consider the line x = zt; then be represented in the form m A0 + A1x + A2y 0 det A0 + t Aizi = 0 i=1 X must vanish at exactly n real points 1.5
1
any line through 0 must intersect 0.5 x f(x) = 0 exactly n times | 0
−0.5 Helton and Vinnikov show this condi- tion is also su cient (subject to ad- −1 ditional technical assumptions) −1.5 −1.5 −1 −0.5 0 0.5 1 1.5 2 - 13 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Projections 4 4 But x + y < 1 is the projection of 1.5 the feasible set of the SDP 1 1 w z w 1 0 0 0.5 z 0 1 0 w x −0.5 0 x 1 −1 z y 0 −1.5 y 1 −1.5 −1 −0.5 0 0.5 1 1.5 Polyehedra are closed under projection. Spectrahedra and basic semi- algebraic sets are not. Easy to optimize over the projection. How to do this systematically? What if the set is not convex? 2 - 14 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Convex Optimization Problems For a convex optimization problem, the feasible set n S = x R fi(x) 0 and hj(x) = 0 for all i, j ∈ | is convex. So w e can write the problem as
minimize f0(x) subject to x S ∈
This approach emphasizes the geometry of the problem. For a convex optimization problem, any local minimum is also a global minimum. 2 - 15 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Feasibility Problems We are also interested in feasibility problems as follows. Does there exist x Rn which satis es ∈ f (x) 0 for all i = 1, . . . , m i hi(x) = 0 for all i = 1, . . . , p
If there does not exist such an x, the problem is described as infeasible. 2 - 16 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Feasibility Problems We can always convert an optimization problem into a feasibility problem; does there exist x Rn such that ∈ f (x) t 0 f (x) 0 i hi(x) = 0 Bisection search over the parameter t nds the optimal. y n è (f1(x); f0(x)) j x 2 R é
? ? (f1(x ); f0(x ))
z 2 - 17 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Feasibility Problems Conversely, we can convert feasibility problems into optimization problems. e.g. the feasibility problem of nding x such that
f (x) 0 for all i = 1, . . . , m i can be solved as
minimize y subject to f (x) y for all i = 1, . . . , m i where there are n + 1 variables x Rn and y R ∈ ∈
This technique may be used to nd an initial feasible point for optimization algorithms 2 - 18 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Algorithms For convex optimization problems, there are several good algorithms interior-point algorithms work well in theory and practice for certain classes of problems, (e.g. LP and SDP) there is a worst-case time-complexity bound conversely, some convex optimization problems are known to be NP- hard problems are speci ed either in standard form, for LPs and SDPs, or via an oracle 2 - 19 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Certi cates Consider the feasibility problem
n Does there exist x R which satis es ∈ f (x) 0 for all i = 1, . . . , m i hi(x) = 0 for all i = 1, . . . , p
There is a fundamental asymmetry between establishing that There exists at least one feasible x The problem is infeasible
To show existence, one needs a feasible point x Rn. ∈ To show emptiness, one needs a a certi cate of infeasibility; a mathematical proof that the problem is infeasible. 2 - 20 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Certi cates and Separating Hyperplanes The simplest form of certi cate is a separating hyperplane. The idea is that a hyperplane L Rn breaks Rn into two half-spaces, n T n T H1 = x R b x a and H2 = x R b x > a ∈ | ∈ | n o n o If two bounded, closed and convex sets are disjoint, there is a hyperplane that separates them.
So to prove infeasibility of
f (x) 0 for i = 1, 2 i we need to computationally show that n n x R f1(x) 0 H1 and x R f2(x) 0 H2 { ∈ | } { ∈ | } Even though such a hyperplane exists, the computation may not be easy 2 - 21 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Duality We’d like to solve
minimize f0(x) subject to f (x) 0 for all i = 1, . . . , m i hi(x) = 0 for all i = 1, . . . , p de ne the Lagrangian for x Rn, Rm and Rp by ∈ ∈ ∈ m p L(x, , ) = f0(x) + ifi(x) + ihi(x) Xi=1 Xi=1 and the Lagrange dual function
g( , ) = inf L(x, , ) x Rn ∈ We allow g( , ) = when there is no nite in mum ∞ 2 - 22 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Duality The dual problem is
maximize g( , ) subject to 0 We call , dual feasible if 0 and g( , ) is nite.
The dual function g is always concave, even if the primal problem is not convex 2 - 23 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Weak Duality For any primal feasible x and dual feasible , we have
g( , ) f (x) 0 because m p g( , ) f (x) + f (x) + h (x) 0 i i i i Xi=1 Xi=1 f (x) 0
A feasible , provides a certi cate that the primal optimal is greater than g( , ) many interior-point methods simultaneously optimize the primal and the dual problem; when f0(x) g( , ) ε we know that x is ε suboptimal 2 - 24 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Strong Duality p? is the optimal value of the primal problem, d? is the optimal value of the dual problem
Weak duality means p? d?
If p? = d? we say strong duality holds. Equivalently, we say the duality gap p? d? is zero. Constraint quali cations give su cient conditions for strong duality.
An example is Slater’s condition; strong duality holds if the primal problem is convex and strictly feasible. 2 - 25 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Geometric Interpretations The Lagrange dual function is
g( ) = inf L(x, ) x Rn ∈ i.e., the minimum intersection for a given slope
y è (f1(x); f0(x)) j x 2 R é
(õ; 1) (f1(x); f0(x)) T Hõ = è (z; y) j õ z + y = L(x; õ) é L(x; õ) z 2 - 26 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Example: Linear Programming
minimize cT x subject to Ax = b x 0 The Lagrange dual function is
g( , ) = inf cT x + T (b Ax) T x x Rn ∈ bT if c AT = 0 = otherwise ( ∞ So the dual problem is
maximize bT subject to AT c 2 - 27 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Example: Semide nite Programming
minimize trace CX subject to trace AiX = bi for all i = 1, . . . , m X 0 The Lagrange dual is m g(Z, ) = inf trace CX trace ZX + i(bi trace AiX) X Xi=1 bT if C Z m A = 0 = i=1 i i otherwise ( ∞ P So the dual problem is to maximize bT subject to m C Z A = 0 and Z 0 i i Xi=1 2 - 28 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Semide nite Programming Duality The primal problem is
minimize trace CX subject to trace AiX = bi for all i = 1, . . . , m X 0
The dual problem is
maximize bT m subject to A C i i Xi=1 2 - 29 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 The Fourfold Way There are several ways of formulating an SDP for its numerical solution.
Because subspaces can be described Using generators or a basis; Equivalently, the subspace is the range of a linear map x x = B for some { | } x1 1 0 1 x = 1 + 1 = 2 1 2 1 2 x3 2 2 2 1 + 2 2 Through the de ning equations; i.e, as the kernel x Ax = 0 { | } 3 (x1, x2, x3) R 4x1 2x2 x3 = 0 { ∈ | }
Depending on which description we use, and whether we write a primal or dual formulation, we have four possibilities (two primal-dual pairs). 2 - 30 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Example: Two Primal-Dual Pairs 1 0 maximize 2x + 2y minimize trace W 0 1 1 + x y subject to 0 1 0 y 1 x subject to trace W = 2 0 1 0 1 trace W = 2 1 0 W 0 Another, more e cient fomulation which solves the same problem:
1 1 maximize trace Z minimize 2t 1 1 t 1 1 1 0 subject to 0 subject to trace Z = 2 1 t + 1 0 1 Z 0 2 - 31 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Duality Duality has many interpretations; via economics, game-theory, geom- etry. e.g., one may interpret Lagrange multipliers as a price for violating constraints, which may correspond to resource limits or capacity con- straints. Often physical problems associate speci c meaning to certain Lagrange multipliers, e.g. pressure, momentum, force can all be viewed as La- grange multipliers 2 - 32 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Example: Mechanics Spring under compression k m Mass at horizontal position x, equilib- rium at x = 2
k minimize (x 2)2 2 subject to x 1 k The Lagrangian is L(x, ) = (x 2)2 + (x 1) 2 ∂ If is dual optimal and x is primal optimal, then L(x, ) = 0, i.e., ∂x k(x 2) + = 0 so we can interpret as a force 2 - 33 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Feasibility of Inequalities The primal feasibility problem is n does there exist x R such that ∈ f (x) 0 for all i = 1, . . . , m i
The dual function g : Rm R is → m g( ) = sup ifi(x) x Rn ∈ Xi=1 The dual feasibility problem is m does there exist R such that ∈ g( ) < 0 0 2 - 34 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Theorem of Alternatives If the dual problem is feasible, then the primal problem is infeasible.
Proof Suppose the primal problem is feasible, and let x˜ be a feasible point. Then m g( ) = sup ifi(x) x Rn ∈ i=1 m X m ifi(x˜) for all R ∈ Xi=1 and so g( ) 0 for all 0. 2 - 35 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Geometric Interpretation z2 Rm õ S = nz 2 j z õ 0 o
z1
Rm T Hõ = nz 2 j õ z = g(õ) o
T if g( ) < 0 and 0 then the hyperplane H separates S from T , where f1(x) . n T = . x R | ∈ fm(x) 2 - 36 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Certi cates A dual feasible point gives a certi cate of infeasibility of the primal.
If the Lagrange dual function g is easy to compute, and we can show g( ) < 0, then this is a proof that the primal is infeasible.
One way to do this is to have an explicit expression for g( ) = sup L(x, ) x For many problems, we do not know how to do this
Alternatively, given , we may be able to show directly that n L(x, ) < ε for all x R ∈ 2 - 37 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Completion of Squares
If A Rn n and D Rm m are symmetric matrices and B Rn m ∈ ∈ ∈
x T A B x = y BT D y x + A 1By T A x + A 1By + yT (D BT A 1B)y
this gives a test for global positivity: A B 0 A 0 and D BT A 1B 0 BT D ⇐⇒ It is a sum of squares decomposition Applying this recursively, we can certify nonnegativity 2 - 38 Convexity and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.03 Quadratic Optimization The Schur complement gives a general formula for quadratic optimization; if A 0, then T x A B x T T 1 min = y (D B A B)y x y BT D y and the minimizing x is 1 xopt = A By