ORIE 6300 Mathematical Programming I September 6, 2016 Lecture 5 Lecturer: Damek Davis Scribe: Rundong Wu 1 Review A while back, we defined polyhedrons and polytopes as follows. n Definition 1 A Polyhedron is P = fx 2 R : Ax ≤ bg Definition 2 A Polytope is given by Q = conv(v1; v2; :::; vk), where the vi are the vertices of the polytope, for k finite. Also recall the equivalence of extreme points, vertices and basic feasible solutions, and recall the definition of a bounded polyhedron. Definition 3 A polyhedron P is bounded iff 9M > 0 such that jjxjj ≤ M; 8x 2 P . We showed bounded polyhedra were polytopes by taking the extreme points and seeing that they were the vertices for P as a polytope. Recall also the Separating Hyperplane Theorem from a previous lecture. n Theorem 1 (Separating Hyperplane) Let C ⊆ R be a closed, nonempty and convex set. Let n y 2 R nC and let 1 x∗ = P (y) := argmin kx − yk2: C x2C 2 ∗ Then there exists a number b 2 R, such that with a = y − x , we have (8x 2 C) aT x ≤ aT x∗ < b < aT y: 2 The polar of a set Now we want to prove that polytopes are bounded polyhedra. To do this, we need to introduce one more concept. n Definition 4 Let S ⊆ R . We call the set ◦ n T S = fz 2 R : z x ≤ 1; 8x 2 Sg; the polar set of S. 5-1 Figure 1: The polar of the `1 ball is the `1 norm ball. Example 1 (Polars of the `1 and `1 balls.) Consider S = f(x1; x2): −1 ≤ x1 ≤ 1; −1 ≤ x2 ≤ 1g, the region is shown in the left of Figure 1. ◦ By definition, x 2 S if, and only if, jx1j + jx2j = sup(z1;z2)2Sfx1z1 + x2z2g ≤ 1. Thus, ◦ n S = fx 2 R : jx1j + jx2j ≤ 1g, which is shown on the right hand side of Figure 1. ◦◦ ◦ ◦ Now let's consider S . By definition, x 2 (S ) if, and only if, maxfjz1j; jz2jg = supjx1j+jx2|≤1fx1z1+ ◦ ◦ x2z2g ≤ 1. Thus, (S ) = f(x1; x2): −1 ≤ x1 ≤ 1; −1 ≤ x2 ≤ 1g = S. n ◦◦ ◦ ◦ Lemma 2 If C is a closed convex subset of R with 0 2 C, then C := (C ) = C. Proof: • (⊇) Suppose that x 2 C. Then x 2 C◦◦ if, and only if, zT x ≤ 1 for all z 2 C◦. This is clearly true because z 2 C◦ implies that zT x ≤ 1. • (⊆) We will show that if x2 = C, then x2 = C◦◦. First note that C is closed and convex with at least z = 0 2 C. If x2 = C, then by the Separating Hyperplane Theorem, there exists n T T 0 6= a 2 R and b 2 R with a x > b > a z for all z 2 C. Since 0 2 C, we have b > 0. Let a~ = a=b 6= 0. Thereforea ~T x > 1 > a~T z, for all z 2 C. This impliesa ~ 2 C◦: Buta ~T x > 1, so x2 = C◦◦: ◦◦ Therefore C = C: 3 Polytopes are Bounded Polyhedra Now we can prove our result, at least sort of. We'll assume that 0 is in the interior of the polytope. We claim that this can be done without loss of generality; this is because we can translate the polytope to have 0 2 P , apply the following proof and then translate back if needed. n Theorem 3 If Q ⊆ R is a polytope with 0 in the interior of Q, then Q is a (bounded) polyhedron. Proof: Our proof strategy is as follows. (1) We will first show that the polar of a polytope is a polyhedron. (2) We then show that that since the polytope has 0 in its interior, then the polar of the polytope is bounded. So then P = Q◦ is a bounded polyhedron. (3) We know from a previous lecture that any bounded polyhedron is a polytope, so P = Q◦ is a polytope. (4) But then applying the proof that the polar of a polytope is a polyhedron, we get that P ◦ = Q◦◦ = Q (by the lemma above) is a polyhedron. It is easy to prove that a polytope is bounded. Thus, we must prove (1) and (2). 5-2 (1) We first prove that the polar of Q is a polyhedron. Let P = Q◦: Then we know that ◦ ◦◦ n P = Q = Q: Since Q is a polytope, Q = convfv1; : : : ; vkg for some k finite vectors v1; : : : ; vk 2 R . We claim that n T P = fz 2 R : vi z ≤ 1; i = 1; : : : ; kg: ◦ n T T T One the one hand, P = Q = fz 2 R : x z ≤ 1; 8x 2 Qg, so vi z = z vi ≤ 1 for i = 1; 2; : : : ; k, n T T which implies that P ⊆ fz 2 R : vi z ≤ 1; i = 1; : : : ; kg. On the other hand, if z vi ≤ 1 for Pk P i = 1; : : : ; k, then for any x 2 Q; with x = i=1 λivi for some λi ≥ 0; i λi = 1, we have k k k T T X X T X z x = z λivi = λi(z vi) ≤ λi = 1; is=1 i=1 i=1 which proves the claim. Thus P is a polyhedron. n (2) Because 0 2 int(Q), there exists some > 0 such that all x 2 R with kxk < lie in Q. If z 2 P , z 6= 0, then, because kxk < , we have z x = 2 Q: 2 jjzjj Because P = Q◦, zT z 2 xT z ≤ 1 ) ≤ 1 ) jjzjj ≤ : 2jjzjj Hence P is a bounded polyhedron. 4 Normal Cone Modern optimization theory crucially relies on a concept called the normal cone. n Definition 5 Let S ⊂ R be a closed, convex set. The normal cone of S is the set-valued mapping n n NS : R ! 2R , given by fg 2 nj (8z 2 S) gT (z − x) ≤ 0g if x 2 S N (x) = R S ; if x2 = S Figure 2: Normal cones of several convex sets. 5-3 Example 2 We compute several normal cones; see Figure 2. 1. Let S = fzg. n if x = z N (x) = R S ; otherwise 2. Let S = [0; 1]. 8 if x = 0 > R≤0 <> if x = 1 N (x) = R≥0 S f0g if x 2 (0; 1) > : ; otherwise n 3. Let S = fx j kxk ≤ 1; x 2 R g: 8 < R≥0x if kxk = 1 NS(x) = f0g if kxk < 1 : ; otherwise 4. The normal cone of a triangle, computed at some but not all points, is depicted in Figure 2. Normal cones satisfy several useful properties. n Proposition 4 Let S ⊆ R be a nonempty, closed, convex set. Then the following hold: 1. If x 2 S, then NS(x) is a convex cone, i:e: (8λ1 ≥ 0); (8λ2 ≥ 0) ; (8g1 2 NS(x)) ; (8g2 2 NS(x)) λ1g1 + λ2g2 2 NS(x): n 2. Let y 2 R nS, then PS(y) = x () y − x 2 NS(x). 3. If x 2 int(S), then NS(x) = f0g. 4. If x 2 S and x2 = int(S), then NS(x) ) f0g. Proof: 1. We leave the proof of part 1 as an exercise. 2. ()): By separating hyperplane theorem, with a = y − PS(y) = y − x, 9 b 2 R, s:t: T T T (8z 2 S) a z ≤ a PS(y) ≤ a y T ) a (z − PS(y)) ≤ 0 ) a 2 NS(PS(y)) ((): If y − x 2 NS(x), then 8z 2 S (y − x)T (z − x) ≤ 0 () (y − x)T (y − x) + (y − x)T (z − y) ≤ 0 ) jjy − xjj2 ≤ (y − x)T (y − z) ≤ jjy − xjj jjy − zjj (Cauchy − Schwarz inequality) ) jjy − xjj ≤ jjy − zjj; 8z 2 S ) x = PS(y) 5-4 3. Suppose that g 2 NS(x). Because x 2 int(S), there exists > 0, such that x + g 2 S. Therefore, we have gT ((x + g) − x) ≤ 0 ) gT g ≤ 0 ) g = 0: Hence, NS(x) = f0g. k n k 4. Since x2 = int(S), there exists a sequence y 2 R nS, s:t: y ! x as k ! 1. We leave it as k k k an exercise to prove that if x = PS(y ), then x ! x, k ! 1. k yk−xk k k Let g = jjyk−xkjj . Then, by part 5, we have g 2 NS(x ). Without loss of generality, we can k n assume that g ! g 2 R , with kgk = 1. k k We claim that g 2 NS(x). To prove the claim, note that since g 2 NS(x ), we have (gk)T (z − xk) ≤ 0, and gT (z − x) = (g − gk)T (z − x) + (gk)T (z − x) = (g − gk)T (z − x) + (gk)T (xk − x) + (gk)T (z − xk) ≤ (g − gk)T (z − x) + (gk)T (xk − x) ! 0 as k ! 1 T So for all z 2 S, we have g (z − x) ≤ 0, which means g 2 NS(x). Obviously 0 2 NS(x), so f0g ( NS(x). Proposition 5 shows that normal cones detect the boundary and interior of convex sets. n Corollary 5 Let S ⊆ R be a nonempty, closed, convex set. Then • NS(x) = f0g if, and only if, x 2 int(S). • NS(x) ) f0g if, and only if, x 2 Snint(S). 5-5.