Lagrangian duality Mauro Passacantando Department of Computer Science, University of Pisa [email protected] Optimization Methods Master of Science in Embedded Computing Systems { University of Pisa http://pages.di.unipi.it/passacantando/om/OM.html M. Passacantando Optimization Methods 1 / 13 { Lagrangian relaxation Consider the general optimization problem 8 < min f (x) g(x) ≤ 0 (P) : h(x) = 0 where x 2 D and the optimal value is v(P). Lagrangian function L : D × Rm × Rp ! R is m p X X L(x; λ, µ) := f (x) + λi gi (x) + µj hj (x) i=1 j=1 M. Passacantando Optimization Methods 2 / 13 { Lagrangian relaxation and dual function Definition Given λ ≥ 0 and µ 2 Rp, the problem min L(x; λ, µ) x 2 D is called Lagrangian relaxation of (P) and '(λ, µ) = inf L(x; λ, µ) is the x2D Lagrangian dual function. Dual function ' I is concave because inf of linear functions w.r.t (λ, µ) I can be −∞ I can be not differentiable M. Passacantando Optimization Methods 3 / 13 { Lagrangian relaxation and dual function Theorem Given λ ≥ 0 and µ 2 Rp, we have '(λ, µ) ≤ v(P): Proof. If x 2 Ω, i.e. g(x) ≤ 0; h(x) = 0, then m X L(x; λ, µ) = f (x) + λi gi (x) ≤ f (x); i=1 hence '(λ, µ) = min L(x; λ, µ) ≤ min L(x; λ, µ) ≤ min f (x) = v(P) x2D x2Ω x2Ω M. Passacantando Optimization Methods 4 / 13 { Lagrangian dual problem Definition The problem max '(λ, µ) (D) λ ≥ 0 is called Lagrangian dual problem of (P). Dual problem consists in finding the best lower bound of v(P). Dual problem is a convex problem, even if (P) is not convex. M. Passacantando Optimization Methods 5 / 13 { Lagrangian dual problem Example 1 - Linear Programming. Primal problem: min cTx (P) Ax ≥ b Lagrangian function: L(x; λ) = cTx + λT(b − Ax) = λTb + (cT − λTA)x Dual function: ( −∞ if cT − λTA 6= 0 '(λ) = min L(x; λ) = n T T T x2R λ b if c − λ A = 0 Dual problem: 8 max λTb max '(λ) < −! λTA = cT (D) λ ≥ 0 : λ ≥ 0 is a linear programming problem. Exercise. What is the dual of (D)? M. Passacantando Optimization Methods 6 / 13 { Lagrangian dual problem Example 2 - Least norm solution of linear equations. Primal problem: min x Tx (P) Ax = b Lagrangian function: L(x; µ) = x Tx + µT(Ax − b). n Dual function: '(µ) = minx2R L(x; µ). L(x; µ) is quadratic and strongly convex w.r.t x, thus 1 r L = 2x + ATµ = 0 () x = − ATµ, x 2 1 T T T hence '(µ) = − 4 µ AA µ − b µ. Dual problem: max − 1 µTAATµ − bTµ 4 (D) µ 2 Rp is an unconstrained convex quadratic programming problem. M. Passacantando Optimization Methods 7 / 13 { Lagrangian dual problem Exercise. Find the dual problem of a generic quadratic programming problem min 1 x TQx + cTx 2 (P) Ax ≤ b where Q is a symmetric positive definite matrix. M. Passacantando Optimization Methods 8 / 13 { Weak duality Theorem (weak duality) For any optimization problem we have v(D) ≤ v(P). Strong duality, i.e., v(D) = v(P), does not hold in general. Example 1. 8 2 < min −x x − 1 ≤ 0 v(P) = −1 : −x ≤ 0 2 L(x; λ) = −x + λ1(x − 1) − λ2x, 2 '(λ) = min L(x; λ) = −∞ 8 λ 2 R ; x2R hence v(D) = −∞. M. Passacantando Optimization Methods 9 / 13 { Weak duality Example 2. min 2x 4 + x 3 − 20x 2 + x x 2 − 2x − 3 ≤ 0 Primal problem: min 2x4+x3−20x2+x s.t. x2−2x−3 <= 0 Dual problem −35 10 −40 0 −45 −10 −50 −20 −55 −60 −30 −65 −40 −70 −1 −0.5 0 0.5 1 1.5 2 2.5 3 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Primal optimal solution x ∗ ' 2:0427, v(P) ' −38:0648. Dual optimal solution λ∗ ' 2:68, v(D) ' −46:0838. M. Passacantando Optimization Methods 10 / 13 { Strong duality Theorem (strong duality) If the problem 8 < min f (x) g(x) ≤ 0 : h(x) = 0 is convex, there exists an optimal solution x ∗, and ACQ holds at x ∗, then KKT multipliers (λ∗; µ∗) associated to x ∗ are an optimal solution of the dual problem and v(D) = v(P). Proof. L(x; λ, µ) is convex with respect to x, thus v(D) ≥ '(λ∗; µ∗) = min L(x; λ∗; µ∗) = L(x ∗; λ∗; µ∗) = f (x ∗) = v(P) ≥ v(D) x M. Passacantando Optimization Methods 11 / 13 { Strong duality Strong duality can hold also for some nonconvex problems. Example. 2 2 min −x1 − x2 2 2 v(P) = −1 x1 + x2 − 1 ≤ 0 2 2 2 2 2 2 L(x; λ) = −x1 − x2 + λ(x1 + x2 − 1) = (λ − 1)x1 + (λ − 1)x2 − λ. ( −∞ if λ < 1 '(λ) = −λ if λ ≥ 1 hence λ∗ = 1 is the dual optimum and v(D) = −1. M. Passacantando Optimization Methods 12 / 13 { Exercises Exercise 1. Consider the problem 8 n P 2 > min xi < i=1 n P > xi ≥ 1 : i=1 I Discuss existence and uniqueness of optimal solutions I Find the optimal solution and the optimal value I Write the dual problem I Solve the dual problem and check whether strong duality holds Exercise 2. Given a; b 2 R with a < b, consider the problem min x 2 a ≤ x ≤ b I Find the optimal solution and the optimal value for any a; b I Solve the dual problem and check whether strong duality holds M. Passacantando Optimization Methods 13 / 13 {.