‣ LP Duality ‣ Strong Duality Theorem ‣ Bonus Proof of LP Duality

LINEAR PROGRAMMING II LINEAR PROGRAMMING II

‣ LP duality ‣ LP duality ‣ strong duality theorem ‣ Strong duality theorem ‣ bonus proof of LP duality ‣ Bonus proof of LP duality ‣ applications ‣ Applications

Lecture slides by Kevin Wayne

Last updated on 7/25/17 11:09 AM

LP duality LP duality

Primal problem. Primal problem. (P) max 13A + 23B (P) max 13A + 23B s. t. 5A + 15B ≤ 480 s. t. 5A + 15B ≤ 480 4A + 4B ≤ 160 4A + 4B ≤ 160 35A + 20B ≤ 1190 35A + 20B ≤ 1190 A , B ≥ 0 A , B ≥ 0

€ € Goal. Find a lower bound on optimal value. Goal. Find an upper bound on optimal value.

Easy. Any feasible solution provides one. Ex 1. Multiply 2nd inequality by 6: 24 A + 24 B ≤ 960.

Ex 1. (A, B) = (34, 0) ⇒ z* ≥ 442 ⇒ z* = 13 A + 23 B ≤ 24 A + 24 B ≤ 960. Ex 2. (A, B) = (0, 32) ⇒ z* ≥ 736 objective function Ex 3. (A, B) = (7.5, 29.5) ⇒ z* ≥ 776 Ex 4. (A, B) = (12, 28) ⇒ z* ≥ 800

3 4 LP duality LP duality

€ € Goal. Find an upper bound on optimal value. Goal. Find an upper bound on optimal value.

Ex 2. Add 2 times 1st inequality to 2nd inequality: ≤ Ex 2. Add 1 times 1st inequality to 2 times 2nd inequality: ≤

⇒ z* = 13 A + 23 B ≤ 14 A + 34 B ≤ 1120. ⇒ z* = 13 A + 23 B ≤ 13 A + 23 B ≤ 800.

Recall lower bound. (A, B) = (34, 0) ⇒ z* ≥ 442 Combine upper and lower bounds: z* = 800.

5 6

LP duality LP duality: economic interpretation

Primal problem. Brewer: find optimal mix of beer and ale to maximize profits. (P) max 13A + 23B s. t. 5A + 15B ≤ 480 4A + 4B ≤ 160 (P) max 13A + 23B 35A + 20B ≤ 1190 s. t. 5A + 15B ≤ 480 A , B ≥ 0 4A + 4B ≤ 160 35A + 20B ≤ 1190 A , B ≥ 0 Idea. Add nonnegative combination€ (C, H, M) of the constraints s.t.

13A + 23B ≤ (5C + 4H + 35M ) A + (15C + 4H + 20M ) B € ≤ 480C +160H +1190M Entrepreneur: buy individual resources from brewer at min cost. ・C, H, M = unit price for corn, hops, malt. Dual problem. Find best such upper bound. Brewer won’t agree to sell resources if 5C + 4H + 35M < 13. € ・

(D) min 480C + 160H + 1190M (D) min 480C + 160H + 1190M s. t. 5C + 4H + 35M ≥ 13 s. t. 5C + 4H + 35M ≥ 13 15C + 4H + 20M ≥ 23 15C + 4H + 20M ≥ 23 C , H , M ≥ 0 C , H , M ≥ 0

7 8

€ € LP duals Double dual

Canonical form. Canonical form.

(P) max cT x (D) min yT b (P) max cT x (D) min yT b s. t. Ax ≤ b s. t. AT y ≥ c s. t. Ax ≤ b s. t. AT y ≥ c x ≥ 0 y ≥ 0 x ≥ 0 y ≥ 0

€ € € €

Property. The dual of the dual is the primal. Pf. Rewrite (D) as a maximization problem in canonical form; take dual.

(D') max − yT b (DD) min − cT z s. t. −AT y ≤ −c s. t. −(AT )T z ≥ −b y ≥ 0 z ≥ 0

9 € € 10

Taking duals

LP dual recipe. LINEAR PROGRAMMING II

‣ LP duality Primal (P) maximize minimize Dual (D) ‣ strong duality theorem a x = bi yi unrestricted

constraints a x ≤ b yi ≥ 0 variables ‣ bonus proof of LP duality a x ≥ b i yi ≤ 0 ‣ applications aTy ≥ c j xj ≥ 0 T a y ≤ cj variables x ≤ 0 constraints j aTy = c unrestricted j

Pf. Rewrite LP in standard form and take dual.

11 LP strong duality LP weak duality

Theorem. [Gale–Kuhn–Tucker 1951, Dantzig–von Neumann 1947] Theorem. For A ∈ ℜm×n, b ∈ ℜm, c ∈ ℜn, if (P) and (D) are nonempty, For A ∈ ℜm×n, b ∈ ℜm, c ∈ ℜn, if (P) and (D) are nonempty, then max = min. then max ≤ min.

(P) max cT x (D) min yT b (P) max cT x (D) min yT b s. t. Ax ≤ b s. t. AT y ≥ c s. t. Ax ≤ b s. t. AT y ≥ c x ≥ 0 y ≥ 0 x ≥ 0 y ≥ 0

Generalizes:€ € € € ・Dilworth’s theorem. Pf. Suppose x ∈ ℜm is feasible for (P) and y ∈ ℜn is feasible for (D). ・König–Egervary theorem. ・y ≥ 0, A x ≤ b ⇒ yT A x ≤ yT b ・Max-flow min-cut theorem. ・x ≥ 0, AT y ≥ c ⇒ yT A x ≥ cT x ・von Neumann’s minimax theorem. ・Combine: cT x ≤ yT A x ≤ yT b. • ・…

Pf. [ahead]

13 14

Projection lemma Projection lemma

Weierstrass’ theorem. Let X be a compact set, and let f(x) be a continuous Weierstrass’ theorem. Let X be a compact set, and let f(x) be a continuous function on X. Then min { f(x) : x ∈ X } exists. function on X. Then min { f(x) : x ∈ X } exists.

Projection lemma. Let X ⊂ ℜm be a nonempty closed convex set, and Projection lemma. Let X ⊂ ℜm be a nonempty closed convex set, and let y ∉ X. Then there exists x* ∈ X with minimum distance from y. let y ∉ X. Then there exists x* ∈ X with minimum distance from y. Moreover, for all x ∈ X we have (y – x*)T (x – x*) ≤ 0. Moreover, for all x ∈ X we have (y – x*)T (x – x*) ≤ 0.

obtuse angle Pf. || y – x || 2 ・Define f(x) = || y – x ||. x Want to apply Weierstrass, but X not x y ・ y necessarily bounded. x* ・X ≠ φ ⇒ there exists xʹ ∈ X. x* ・Define Xʹ = { x ∈ X : || y – x || ≤ || y – xʹ || } xʹ so that Xʹ is closed, bounded, and Xʹ min { f(x) : x ∈ X } = min { f(x) : x ∈ Xʹ }. X ・By Weierstrass, min exists. X

15 16 Projection lemma Separating hyperplane theorem

Weierstrass’ theorem. Let X be a compact set, and let f(x) be a continuous Theorem. Let X ⊂ ℜm be a nonempty closed convex set, and let y ∉ X. function on X. Then min { f(x) : x ∈ X } exists. Then there exists a hyperplane H = { x ∈ ℜm : aTx = α } where a ∈ ℜm, α ∈ ℜ that separates y from X. Projection lemma. Let X ⊂ ℜm be a nonempty closed convex set, and let y ∉ X. Then there exists x* ∈ X with minimum distance from y. aT x ≥ α for all x ∈ X aTy < α Moreover, for all x ∈ X we have (y – x*)T (x – x*) ≤ 0. Pf. Pf. ・Let x* be closest point in X to y. x* min distance ⇒ || y – x* || 2 ≤ || y – x || 2 for all x ∈ X. By projection lemma, ・・ y ・By convexity: if x ∈ X, then x* + ε (x – x*) ∈ X for all 0 < ε < 1. (y – x*)T (x – x*) ≤ 0 for all x ∈ X 2 2 * T ・|| y – x*|| ≤ || y – x* – ε(x – x*) || ・Choose a = x – y ≠ 0 and α = a x*. x* = || y – x*|| 2 + ε2 ||(x – x*)|| 2 – 2 ε (y – x*)T (x - x*) ・If x ∈ X, then aT(x – x*) ≥ 0; ・Thus, (y – x*)T (x - x*) ≤ ½ ε ||(x – x*)|| 2. thus ⇒ aTx ≥ aTx* = α. Letting +, we obtain the desired result. • Also, T T || ||2 • x ・ ε → 0 ・ a y = a (x* – a) = α – a < α

X H = { x ∈ ℜm : aTx = α} 17 18

Farkas’ lemma Another theorem of the alternative

Theorem. For A ∈ ℜm×n , b ∈ ℜm exactly one of the following Corollary. For A ∈ ℜm×n , b ∈ ℜm exactly one of the following two systems two systems holds: holds:

(II) ∃ y ∈ ℜm (I) ∃ x ∈ ℜn (II) ∃ y ∈ ℜm (I) ∃ x ∈ ℜn s. t. AT y ≥ 0 s. t. Ax = b s. t. AT y ≥ 0 s. t. Ax ≤ b yT b < 0 x ≥ 0 yT b < 0 x ≥ 0 y ≥ 0

Pf.€ [not both] Suppose x satisfies€ (I) and y satisfies (II). € € Then 0 > yT b = yTAx ≥ 0, a contradiction. Pf. Apply Farkas’ lemma to: Pf. [at least one] Suppose (I) infeasible. We will show (II) feasible. (I' ) x n , s m (II') ∃ y ∈ ℜm Consider S = { A x : x ≥ 0 } so that S closed, convex, b ∉ S. ∃ ∈ ℜ ∈ ℜ ・ T ・Let y ∈ ℜm, α ∈ ℜ be a hyperplane that separates b from S: s. t. A x + I s = b s. t. A y ≥ 0 x, s 0 I y ≥ 0 y Tb < α, yTs ≥ α for all s ∈ S. ≥ yT b < 0 ・0 ∈ S ⇒ α ≤ 0 ⇒ yTb < 0 T for all T since can be arbitrarily large. • ・y Ax ≥ α x ≥ 0 ⇒ y A ≥ 0 x €

19 € 20 LP strong duality LP strong duality

Theorem. [strong duality] For A ∈ ℜm×n, b ∈ ℜm, c ∈ ℜn, if (P) and (D) are m nonempty then max = min. (II) ∃ y ∈ ℜ , z ∈ ℜ s. t. AT y − cz ≥ 0 T T T y b −α z < 0 (P) max c x (D) min y b T y, z ≥ 0 s. t. Ax ≤ b s. t. A y ≥ c x ≥ 0 y ≥ 0

Let y, z be a solution€ to (II). Pf. [max min] Weak LP duality. € ≤ € Pf. [min ≤ max] Suppose max < α. We show min < α. Case 1. [z = 0] ・Then, { y ∈ ℜm : AT y ≥ 0, yTb < 0, y ≥ 0 } is feasible. (I) ∃ x ∈ ℜn (II) ∃ y ∈ ℜm , z ∈ ℜ ・Farkas Corollary ⇒ { x ∈ ℜn : Ax ≤ b, x ≥ 0 } is infeasible. s. t. Ax ≤ b s. t. AT y − c z ≥ 0 ・Contradiction since by assumption (P) is nonempty. −cT x ≤ −α yT b − α z < 0 y, z 0 x ≥ 0 ≥ Case 2. [z > 0] Scale , so that satisfies (II) and ・ y z y z = 1. By definition of , (I) infeasible (II) feasible by Farkas’ corollary. Resulting y feasible to (D) and yTb < . • € ・ α € ⇒ ・ α

21 22

Strong duality theorem

Theorem. For A ∈ ℜm×n, b ∈ ℜm, c ∈ ℜn, if (P) and (D) are nonempty, LINEAR PROGRAMMING II then max = min.

‣ LP duality

‣ strong duality theorem (P) max cT x (D) min yT b s. t. Ax = b ‣ bonus proof of LP duality s. t. AT y ≥ c x ≥ 0 ‣ applications

€ €

24 Review: simplex tableaux Simplex tableaux: dual solution

T -1 T -1 subtract cB AB times constraints subtract cB AB times constraints

T T (cT cT A−1 A ) x Z cT A−1 b T T (cT cT A−1 A ) x Z cT A−1 b cB xB + cN xN = Z N − B B N N = − B B cB xB + cN xN = Z N − B B N N = − B B I x A−1 A x A−1 b I x A−1 A x A−1 b AB xB + AN xN = b B + B N N = B AB xB + AN xN = b B + B N N = B x , x 0 x , x 0 xB , xN ≥ 0 B N ≥ xB , xN ≥ 0 B N ≥

initial tableaux tableaux corresponding to basis B initial tableaux tableaux corresponding to basis B -1 -1 multiply by AB multiply by AB € € € € -1 -1 Primal solution. xB = AB b ≥ 0, xN = 0 Primal solution. xB = AB b ≥ 0, xN = 0 T T -1 T T -1 Optimal basis. cN – cB AB AN ≤ 0 Optimal basis. cN – cB AB AN ≤ 0 T T -1 Dual solution. y = c B AB

T T T y A = y AB y AN T T −1 [ ] y b = cB AB b T -1 T -1 = cB AB AB cB AB AN = cT x + cT x [ ] B B B B T T -1 T = c c A A = c x [ B B B N ] cT cT ≥ [ B N ] min ≤ max = cT dual feasible 25 26 €

€ Simplex algorithm: LP duality

T -1 subtract cB AB times constraints LINEAR PROGRAMMING II T T (cT cT A−1 A ) x Z cT A−1 b cB xB + cN xN = Z N − B B N N = − B B −1 −1 A x A x b I xB + AB AN xN = AB b B B + N N = ‣ LP duality x , x 0 xB , xN ≥ 0 B N ≥ ‣ strong duality theorem initial tableaux tableaux corresponding to basis B -1 multiply by AB ‣ alternate proof of LP duality € ‣ applications € -1 Primal solution. xB = AB b ≥ 0, xN = 0 T T -1 Optimal basis. cN – cB AB AN ≤ 0 T T -1 Dual solution. y = c B AB

Simplex algorithm yields constructive proof of LP duality.

27 LP duality: economic interpretation LP duality: sensitivity analysis

Brewer: find optimal mix of beer and ale to maximize profits. Q. How much should brewer be willing to pay (marginal price) for additional supplies of scarce resources? (P) max 13A + 23B A* = 12 s. t. 5A + 15B ≤ 480 B* = 28 A. corn $1, hops $2, malt $0. 4A + 4B ≤ 160 OPT = 800 35A + 20B ≤ 1190 A , B ≥ 0

Q. Suppose a new product “light beer” is proposed. It requires 2 corn, 5 Entrepreneur: buy€ individual resources from brewer at min cost. hops, 24 malt. How much profit must be obtained from light beer to justify diverting resources from production of beer and ale? (D) min 480C + 160H + 1190M C* = 1 H* = 2 s. t. 5C + 4H + 35M ≥ 13 A. At least 2 ($1) + 5 ($2) + 24 ($0) = $12 / barrel. 15C + 4H + 20M ≥ 23 M* = 0 OPT = 800 C , H , M ≥ 0

€ LP duality. Market clears. 29 30

LP is in NP ∩ co-NP

LP. For A ∈ ℜm×n, b ∈ ℜm, c ∈ ℜn, α ∈ ℜ, does there exist x ∈ ℜn such that: Ax = b, x ≥ 0, cT x ≥ α ?

Theorem. LP is in NP ∩ co-NP. Pf. ・Already showed LP is in NP. ・If LP is infeasible, then apply Farkas’ lemma to get certificate of infeasibility:

(II) ∃ y ∈ ℜm , z ∈ ℜ s. t. AT y ≥ 0 yT b − α z < 0 z ≥ 0 or equivalently, yT b – α z = –1

€