LINEAR PROGRAMMING II LINEAR PROGRAMMING II ‣ LP duality ‣ LP duality ‣ strong duality theorem ‣ Strong duality theorem ‣ bonus proof of LP duality ‣ Bonus proof of LP duality ‣ applications ‣ Applications Lecture slides by Kevin Wayne Last updated on 7/25/17 11:09 AM LP duality LP duality Primal problem. Primal problem. (P) max 13A + 23B (P) max 13A + 23B s. t. 5A + 15B ≤ 480 s. t. 5A + 15B ≤ 480 4A + 4B ≤ 160 4A + 4B ≤ 160 35A + 20B ≤ 1190 35A + 20B ≤ 1190 A , B ≥ 0 A , B ≥ 0 € € Goal. Find a lower bound on optimal value. Goal. Find an upper bound on optimal value. Easy. Any feasible solution provides one. Ex 1. Multiply 2nd inequality by 6: 24 A + 24 B ≤ 960. Ex 1. (A, B) = (34, 0) ⇒ z* ≥ 442 ⇒ z* = 13 A + 23 B ≤ 24 A + 24 B ≤ 960. Ex 2. (A, B) = (0, 32) ⇒ z* ≥ 736 objective function Ex 3. (A, B) = (7.5, 29.5) ⇒ z* ≥ 776 Ex 4. (A, B) = (12, 28) ⇒ z* ≥ 800 3 4 LP duality LP duality Primal problem. Primal problem. (P) max 13A + 23B (P) max 13A + 23B s. t. 5A + 15B ≤ 480 s. t. 5A + 15B ≤ 480 4A + 4B ≤ 160 4A + 4B ≤ 160 35A + 20B ≤ 1190 35A + 20B ≤ 1190 A , B ≥ 0 A , B ≥ 0 € € Goal. Find an upper bound on optimal value. Goal. Find an upper bound on optimal value. Ex 2. Add 2 times 1st inequality to 2nd inequality: ≤ Ex 2. Add 1 times 1st inequality to 2 times 2nd inequality: ≤ ⇒ z* = 13 A + 23 B ≤ 14 A + 34 B ≤ 1120. ⇒ z* = 13 A + 23 B ≤ 13 A + 23 B ≤ 800. Recall lower bound. (A, B) = (34, 0) ⇒ z* ≥ 442 Combine upper and lower bounds: z* = 800. 5 6 LP duality LP duality: economic interpretation Primal problem. Brewer: find optimal mix of beer and ale to maximize profits. (P) max 13A + 23B s. t. 5A + 15B ≤ 480 4A + 4B ≤ 160 (P) max 13A + 23B 35A + 20B ≤ 1190 s. t. 5A + 15B ≤ 480 A , B ≥ 0 4A + 4B ≤ 160 35A + 20B ≤ 1190 A , B ≥ 0 Idea. Add nonnegative combination€ (C, H, M) of the constraints s.t. 13A + 23B ≤ (5C + 4H + 35M ) A + (15C + 4H + 20M ) B € ≤ 480C +160H +1190M Entrepreneur: buy individual resources from brewer at min cost. ・C, H, M = unit price for corn, hops, malt. Dual problem. Find best such upper bound. Brewer won’t agree to sell resources if 5C + 4H + 35M < 13. € ・ (D) min 480C + 160H + 1190M (D) min 480C + 160H + 1190M s. t. 5C + 4H + 35M ≥ 13 s. t. 5C + 4H + 35M ≥ 13 15C + 4H + 20M ≥ 23 15C + 4H + 20M ≥ 23 C , H , M ≥ 0 C , H , M ≥ 0 7 8 € € LP duals Double dual Canonical form. Canonical form. (P) max cT x (D) min yT b (P) max cT x (D) min yT b s. t. Ax ≤ b s. t. AT y ≥ c s. t. Ax ≤ b s. t. AT y ≥ c x ≥ 0 y ≥ 0 x ≥ 0 y ≥ 0 € € € € Property. The dual of the dual is the primal. Pf. Rewrite (D) as a maximization problem in canonical form; take dual. (D') max − yT b (DD) min − cT z s. t. −AT y ≤ −c s. t. −(AT )T z ≥ −b y ≥ 0 z ≥ 0 9 € € 10 Taking duals LP dual recipe. LINEAR PROGRAMMING II ‣ LP duality Primal (P) maximize minimize Dual (D) ‣ strong duality theorem a x = bi yi unrestricted constraints a x ≤ b yi ≥ 0 variables ‣ bonus proof of LP duality a x ≥ b i yi ≤ 0 ‣ applications aTy ≥ c j xj ≥ 0 T a y ≤ cj variables x ≤ 0 constraints j aTy = c unrestricted j Pf. Rewrite LP in standard form and take dual. 11 LP strong duality LP weak duality Theorem. [Gale–Kuhn–Tucker 1951, Dantzig–von Neumann 1947] Theorem. For A ∈ ℜm×n, b ∈ ℜm, c ∈ ℜn, if (P) and (D) are nonempty, For A ∈ ℜm×n, b ∈ ℜm, c ∈ ℜn, if (P) and (D) are nonempty, then max = min. then max ≤ min. (P) max cT x (D) min yT b (P) max cT x (D) min yT b s. t. Ax ≤ b s. t. AT y ≥ c s. t. Ax ≤ b s. t. AT y ≥ c x ≥ 0 y ≥ 0 x ≥ 0 y ≥ 0 Generalizes:€ € € € ・Dilworth’s theorem. Pf. Suppose x ∈ ℜm is feasible for (P) and y ∈ ℜn is feasible for (D). ・König–Egervary theorem. ・y ≥ 0, A x ≤ b ⇒ yT A x ≤ yT b ・Max-flow min-cut theorem. ・x ≥ 0, AT y ≥ c ⇒ yT A x ≥ cT x ・von Neumann’s minimax theorem. ・Combine: cT x ≤ yT A x ≤ yT b. • ・… Pf. [ahead] 13 14 Projection lemma Projection lemma Weierstrass’ theorem. Let X be a compact set, and let f(x) be a continuous Weierstrass’ theorem. Let X be a compact set, and let f(x) be a continuous function on X. Then min { f(x) : x ∈ X } exists. function on X. Then min { f(x) : x ∈ X } exists. Projection lemma. Let X ⊂ ℜm be a nonempty closed convex set, and Projection lemma. Let X ⊂ ℜm be a nonempty closed convex set, and let y ∉ X. Then there exists x* ∈ X with minimum distance from y. let y ∉ X. Then there exists x* ∈ X with minimum distance from y. Moreover, for all x ∈ X we have (y – x*)T (x – x*) ≤ 0. Moreover, for all x ∈ X we have (y – x*)T (x – x*) ≤ 0. obtuse angle Pf. || y – x || 2 ・Define f(x) = || y – x ||. x Want to apply Weierstrass, but X not x y ・ y necessarily bounded. x* ・X ≠ φ ⇒ there exists xʹ ∈ X. x* ・Define Xʹ = { x ∈ X : || y – x || ≤ || y – xʹ || } xʹ so that Xʹ is closed, bounded, and Xʹ min { f(x) : x ∈ X } = min { f(x) : x ∈ Xʹ }. X ・By Weierstrass, min exists. X 15 16 Projection lemma Separating hyperplane theorem Weierstrass’ theorem. Let X be a compact set, and let f(x) be a continuous Theorem. Let X ⊂ ℜm be a nonempty closed convex set, and let y ∉ X. function on X. Then min { f(x) : x ∈ X } exists. Then there exists a hyperplane H = { x ∈ ℜm : aTx = α } where a ∈ ℜm, α ∈ ℜ that separates y from X. Projection lemma. Let X ⊂ ℜm be a nonempty closed convex set, and * aT x ≥ α for all x ∈ X let y ∉ X. Then there exists x ∈ X with minimum distance from y. aTy < α Moreover, for all x ∈ X we have (y – x*)T (x – x*) ≤ 0. Pf. Pf. ・Let x* be closest point in X to y. x* min distance ⇒ || y – x* || 2 ≤ || y – x || 2 for all x ∈ X. By projection lemma, ・ ・ y ・By convexity: if x ∈ X, then x* + ε (x – x*) ∈ X for all 0 < ε < 1. (y – x*)T (x – x*) ≤ 0 for all x ∈ X 2 2 * T ・|| y – x*|| ≤ || y – x* – ε(x – x*) || ・Choose a = x – y ≠ 0 and α = a x*. x* = || y – x*|| 2 + ε2 ||(x – x*)|| 2 – 2 ε (y – x*)T (x - x*) ・If x ∈ X, then aT(x – x*) ≥ 0; ・Thus, (y – x*)T (x - x*) ≤ ½ ε ||(x – x*)|| 2. thus ⇒ aTx ≥ aTx* = α. Letting +, we obtain the desired result. • Also, T T || ||2 • x ・ ε → 0 ・ a y = a (x* – a) = α – a < α X m T H = { x ∈ ℜ : a x = α} 17 18 Farkas’ lemma Another theorem of the alternative Theorem. For A ∈ ℜm×n , b ∈ ℜm exactly one of the following Corollary. For A ∈ ℜm×n , b ∈ ℜm exactly one of the following two systems two systems holds: holds: (II) ∃ y ∈ ℜm (I) ∃ x ∈ ℜn (II) ∃ y ∈ ℜm (I) ∃ x ∈ ℜn s. t. AT y ≥ 0 s. t. Ax = b s. t. AT y ≥ 0 s. t. Ax ≤ b yT b < 0 x ≥ 0 yT b < 0 x ≥ 0 y ≥ 0 Pf.€ [not both] Suppose x satisfies€ (I) and y satisfies (II). € € Then 0 > yT b = yTAx ≥ 0, a contradiction. Pf. Apply Farkas’ lemma to: Pf. [at least one] Suppose (I) infeasible. We will show (II) feasible. (I' ) x n , s m (II') ∃ y ∈ ℜm Consider S = { A x : x ≥ 0 } so that S closed, convex, b ∉ S. ∃ ∈ ℜ ∈ ℜ ・ T ・Let y ∈ ℜm, α ∈ ℜ be a hyperplane that separates b from S: s. t. A x + I s = b s. t. A y ≥ 0 x, s 0 I y ≥ 0 y Tb < α, yTs ≥ α for all s ∈ S. ≥ yT b < 0 ・0 ∈ S ⇒ α ≤ 0 ⇒ yTb < 0 T for all T since can be arbitrarily large. • ・y Ax ≥ α x ≥ 0 ⇒ y A ≥ 0 x € 19 € 20 LP strong duality LP strong duality Theorem. [strong duality] For A ∈ ℜm×n, b ∈ ℜm, c ∈ ℜn, if (P) and (D) are m nonempty then max = min. (II) ∃ y ∈ ℜ , z ∈ ℜ s. t. AT y − cz ≥ 0 T T T y b −α z < 0 (P) max c x (D) min y b T y, z ≥ 0 s. t. Ax ≤ b s. t. A y ≥ c x ≥ 0 y ≥ 0 Let y, z be a solution€ to (II). Pf. [max min] Weak LP duality. € ≤ € Pf. [min ≤ max] Suppose max < α. We show min < α. Case 1. [z = 0] ・Then, { y ∈ ℜm : AT y ≥ 0, yTb < 0, y ≥ 0 } is feasible. (I) ∃ x ∈ ℜn (II) ∃ y ∈ ℜm , z ∈ ℜ ・Farkas Corollary ⇒ { x ∈ ℜn : Ax ≤ b, x ≥ 0 } is infeasible. s. t. Ax ≤ b s. t. AT y − c z ≥ 0 ・Contradiction since by assumption (P) is nonempty. −cT x ≤ −α yT b − α z < 0 y, z 0 x ≥ 0 ≥ Case 2. [z > 0] Scale , so that satisfies (II) and ・ y z y z = 1.