3 - 1 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 3. Quadratically Constrained Quadratic Programming Quadratic programming MAXCUT Boolean optimization Primal and dual SDP relaxations Randomization Interpretations Examples LQR with binary inputs Rounding schemes 3 - 2 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Quadratic Programming A quadratically constrained quadratic program (QCQP) has the form

minimize f0(x) subject to f (x) 0 for all i = 1, . . . , m i n where the functions fi : R R have the form → T T fi(x) = x Pix + qi x + ri

A very general problem If all the f are convex then the QCQP may be solved by SDP; i but specialized software for second-order cone programming is more ecient 3 - 3 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Example: LQR with Binary Inputs Consider the discrete-time LQR problem

2 x(t + 1) = Ax(t) + Bu(t) minimize y(t) yr(t) subject to k k ( y(t) = Cx(t) where yr is the reference output trajectory, and the input u(t) is constrained by u(t) = 1 for all t = 0, . . . , N.

| | T = 20 Bang−Bang−Cost = 16.927000 1.5

1

0.5

0 Input −0.5

−1

−1.5 0 2 4 6 8 10 12 14 16 18 20 An open-loop LQR-type problem, but Time

3 with a bang-bang input. x1 2 x2 x3 1 x1’ x2’ x3’ 0 State −1

−2

−3 0 2 4 6 8 10 12 14 16 18 20 Time 3 - 4 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 LQR with Binary Inputs 2 The objective y(t) yr(t) is a quadratic function of the input u: k k y(0) 0 0 0 . . . 0 u(0) y(1) CB 0 0 . . . 0 u(1)       y(2) = CAB CB 0 . . . 0 u(2)  .   ......   .        y(t) CAtB CAt 1B . . . CB 0 u(t)             So the problem can be written as:

u T Q r u minimize 1 rT s 1 subject to u +1, 1 for all i i ∈ { } where Q, r, s are functions of the problem data. This is a quadratic boolean optimization problem. 3 - 5 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 MAXCUT given an undirected graph, with no self-loops

vertex set V = 1, . . . , n { } edge set E i, j i, j V, i = j { } | ∈ 6 n o For a subset S V , the capacity of S is the number of edges connecting a node in S to a node not in S the MAXCUT problem

nd S V with maximum capacity the example above shows a cut with capacity 15; this is the maximum 3 - 6 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Example a graph with 12 nodes, 24 edges; the maximum capacity cmax = 20 3 - 7 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Problem Formulation the graph is dened by its adjacency matrix

1 if i, j E Qij = { } ∈ (0 otherwise and specify a cut S by a vector x Rn ∈ 1 if i S xi = ∈ 1 otherwise ( then 1 x x = 2 if i, j is a cut, so the capacity of x is i j { } n n 1 c(x) = (1 x x )Q 4 i j ij Xi=1 Xj=1 3 - 8 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Optimization Formulation so we’d like to solve

minimize xT Qx subject to x 1, 1 for all i = 1, . . . , n i ∈ { }

call the optimal value p?, then the maximum cut is n n 1 1 ? cmax = Q p 4 ij 4 Xi=1 Xj=1 3 - 9 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Boolean Optimization A classic combinatorial problem:

minimize xT Qx subject to x 1, 1 i ∈ { }

Many other examples; knapsack, LQR with binary inputs, etc. Can model the constraints with quadratic equations: x2 1 = 0 x 1, 1 i ⇐⇒ i ∈ { } An exponential number of points. Cannot check them all! The problem is NP-complete (even if Q 0). Despite the hardness of the problem, there are some very good approaches. . . 3 - 10 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 SDP Relaxations We can nd a lower bound via the dual; the primal is

minimize xT Qx subject to x2 1 = 0 i

Let = diag(1, . . . , n), then the Lagrangian is n L(x, ) = xT Qx (x2 1) = xT (Q )x + trace i i Xi=1

The dual is therefore the SDP

maximize trace subject to Q 0 3 - 11 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 SDP Relaxations From this SDP we obtain a primal-dual pair of SDP relaxations

minimize trace QX maximize trace subject to X 0 subject to Q Xii = 1 diagonal

We derived them from Lagrangian and SDP duality But, these SDP relaxations arise in many other ways Well-known in combinatorial optimization, graph theory, etc. Several interpretations 3 - 12 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 SDP Relaxations: Dual Side Gives a simple underestimator of the objective function.

maximize trace subject to Q diagonal

Directly provides a lower bound on the objective: for any feasible x: n xT Qx xT x = x2 = trace ii i Xi=1 The rst inequality follows from Q The second equation from being diagonal The third, from x +1, 1 i ∈ { } 3 - 13 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 SDP Relaxations: Primal Side The original problem is:

minimize xT Qx 2 subject to xi = 1 Let X := xxT . Then

xT Qx = trace QxxT = trace QX

Therefore, X 0, has rank one, and X = x2 = 1. ii i

Conversely, any matrix X with

X 0, X = 1, rank X = 1 ii necessarily has the form X = xxT for some 1 vector x. 3 - 14 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Primal Side Therefore, the original problem can be exactly rewritten as:

minimize trace QX subject to X 0 Xii = 1 rank(X) = 1

Interpretation: lift to a higher dimensional space, from Rn to Sn. Dropping the (nonconvex) rank constraint, we obtain the relaxation.

If the solution X has rank 1, then we have solved the original problem.

Otherwise, rounding schemes to project solutions. In some cases, approxi- mation guarantees (e.g. Goemans-Williamson for MAX CUT). 3 - 15 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Feasible Points and Certicates

minimize trace QX maximize trace subject to X 0 subject to Q Xii = 1 diagonal

Dual relaxations give certied bounds.

Primal relaxations give information about possible feasible points.

Both are solved simultaneously by primal-dual SDP solvers 3 - 16 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Example

minimize 2x1x2 + 4x1x3 + 6x2x3 2 subject to xi = 1

0 1 2 The associated matrix is Q = 1 0 3 . The SDP solutions are: 2 3 0   1 1 1 1 0 0 X = 1 1 1 , = 0 2 0  1 1 1  0 0 5     We have X 0, X = 1, Q 0, and ii trace QX = trace = 8 Since X is rank 1, from X = xxT we recover the optimal x = 1 1 1 T , 3 - 17 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01

p1 1 Spectrahedron 0.5 0 -0.5 We can visualize this (in 3 3): -1 1 1 p1 p2 0.5 X = p 1 p 0  1 3 p2 p3 1 p3 0   in (p1, p2, p3) space. -0.5

-1 1 0.5 0 -0.5 p 2 -1 When optimizing the linear objective function

trace QX = 2p1 + 4p2 + 6p3, the optimal solution is at the vertex (1, 1, 1). 3 - 18 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Primalization

After solving the SDP for X Sn, we’d like to map back to x 1, 1 n ∈ ∈ { } There may not exist an x 1, 1 n such that X = xxT ∈ { }

We can interpret this algebraically: rank X = 1 6 geometrically: X is not a lifted point

We need a procedure for nding a good x given X; called rounding, primalization, or projection. This is hard in general, but for MAXCUT good methods are known 3 - 19 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Randomization Suppose we solve the primal relaxation

minimize trace QX subject to X 0 Xii = 1 for all i = 1, . . . , n and the optimal X is not rank 1. Goemans and Williamson developed the following randomized algorithm for nding a feasible point

T r n Factorize X as X = V V , where V = v1 . . . vn R ∈ Then X = vT v , and since X = 1 this factorization gives n vectors ij i j ii on the unit sphere in Rr Instead of assigning either 1 or 1 to each vertex, we have assigned a point on the unit sphere in Rr to each vertex 3 - 20 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Randomized Slicing

Pick a random vector q Rr, and choose cut ∈ S = i vT q 0 | i Then the probability that i, j is a cut edge is { } angle between vi and vj 1 = arccos vT v i j 1 = arccos X ij

So the expected cut capacity is n n 1 1 c = Q arccos X sdp-expected 2 ij ij Xi=1 Xj=1 3 - 21 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Randomization (MAXCUT only)

arcos(t) 3 SDP gives an upper bound on the cut capacity α(1−t)π/2

2.5 n n 1 c = (1 X )Q 2 sdp-upper-bound 4 ij ij i=1 j=1 X X 1.5

With = 0.878, we have 1

0.5 (1 t) arccos(t) for all t [ 1, 1] 2 ∈ 0 −1 −0.5 0 0.5 1 So we have n n 1 c Q arccos X sdp-upper-bound 2 ij ij Xi=1 Xj=1 1 = c sdp-expected 3 - 22 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Randomization So far, we have c 1 c sdp-upper-bound sdp-expected Also clearly c cmax sdp-expected And cmax c sdp-upper-bound

After solving the SDP, we slice randomly to generate a random family of feasible points.

We can sandwich the expected value of this family as follows. ( = 0.878)

c c cmax c sdp-upper-bound sdp-expected sdp-upper-bound 3 - 23 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Coin-Flipping Approach 1 Suppose we just randomly assigned vertices to S with probability 2; then n n 1 c = Q coinip-expected 4 ij Xi=1 Xj=1 A trivial upper bound on the maximum cut is just the total number of edges n n 1 c = Q trivial-upper-bound 2 ij Xi=1 Xj=1 1 and so ccoinip-expected = 2ctrivial-upper-bound 3 - 24 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Coin-Flipping Approach We have c = 1c coinip-expected 2 trivial-upper-bound c cmax coinip-expected cmax c trivial-upper-bound

Again, we have a sandwich result

1 c = c cmax c 2 trivial-upper-bound coinip-expected trivial-upper-bound 3 - 25 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Example histogram of SDP capacities 4 x 10 64 vertices, 126 edges 2 1.8 1.6

SDP upper bound 116 1.4

1.2

1

0.8

0.6

0.4

0.2

0 0 20 40 60 80 100 120 histogram of coin-ip capacities

1600

1400

1200

1000

800

600

400

200

0 0 20 40 60 80 100 120 3 - 26 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 A General Scheme

BooleanMinimization Lagrangian Primal Duality Relaxation

X SDP ¤ Relaxed Duality Dual-Bound

The relaxed X suggests candidate points. The diagonal matrix certies a lower bound.

Ubiquitous scheme in optimization (convex hulls, fractional colorings, etc. . . ) We will learn systematic ways of constructing these relaxations, and more. . . 3 - 27 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 LQR with Binary Inputs u T Q r u minimize 1 rT s 1 subject to u +1, 1 for all i i ∈ { } for some matrices (Q, r, s) function of the problem data (A, B, C, N).

An SDP dual bound: maximize trace() + Q r subject to 0, diagonal rT s

Let q, q be the optimal value of both problems. Then, q q : u T Q r u u T 0 u = trace + 1 rT s 1 1 0 1 3 - 28 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 LQR with Binary Inputs maximize trace() + Q r subject to 0, diagonal rT s Since (, ) = (0, 0) is always feasible, q 0. Furthermore, the bound is never worse than the LQR solution obtained by dropping the 1 constraint, since = 0, = s rT Q 1r is a feasible point. N LQR cost SDP bound Optimal 10 14.005 15.803 15.803 Example: 15 15.216 16.698 16.705 20 15.364 16.905 16.927 3 - 29 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 The S-procedure A sucient condition for the infeasibility of quadratic inequalities: n T x R x Aix 0 { ∈ | } Again, a primal-dual pair of SDP relaxations:

X 0 A I trace X = 1 i i i i 0 trace A X 0 P i

The basis of many important results in control theory. 3 - 30 Quadratically Constrained Quadratic Programming P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 Structured Singular Value A central paradigm in robust control. y x is a measure of robustness: how big can a structured perturbation be, M without losing stability.

Do the loop equations admit nontrivial solutions?

y = Mx, y2 x2 0 i i Applying the standard SDP relaxation:

d (y2 x2) = xT (MT DM D)x < 0, D = diag(d ), d 0 i i i i i Xi We obtain the standard upper bound:

MT DM D 0, D diagonal, D 0