10-725: Optimization Fall 2013 Lecture 5: September 10 Lecturer: Geoff Gordon/Ryan Tibshirani Scribes: Jun Ma Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications. They may be distributed outside this class only with the permission of the Instructor. This lecture's notes illustrate some uses of various LATEX macros. Take a look at this and imitate. 5.1 Convex Sets 5.1.1 Closed and open sets Let C Rn. ⊆ Definition 5.1 The affine hull of C is the smallest affine set that contains C. Pk Pk aff(C) = θixi xi C; θi = 1; θi R . f i=1 j 2 i=1 2 g An example of an affine set is the solution set of a system of linear equations, C = x Ax = b ;A m×n n f j g 2 R ;B R . If x1; x2 C, θ R, then A(θx1 + (1 θ)x2) = θAx1 + (1 θ)Ax2 = θb + (1 θ)b = b. 2 2 2 − − − Let x be some point in Rn, and B(x; ) be a ball of radius centered at x. Then Definition 5.2 x is on the boundary of C, @C, if for all > 0, B(x; ) C = and B(x; ) Cc = . \ 6 ; \ 6 ; Definition 5.3 x is in the interior of C, int C, if > 0 : B(x; ) C. 9 ⊂ Definition 5.4 x is in the relative interior of C, rel int C, if > 0 : B(x; ) aff(C) C. 9 \ ⊆ Definition 5.5 The closure of C, cl C = C @C. [ Definition 5.6 The relative boundary of C, rel @C = cl C rel int C. n Definition 5.7 C is closed if @C C, open if @C C = , and compact iff it is closed and bounded in Rn. ⊂ \ ; 5-1 5-2 Lecture 5: September 10 5.1.2 Convex sets and examples Figure 5.1: Example of a convex set (left) and a non-convex set (right). Definition 5.8 A set C Rn is convex iff the line segment between any two points in C is completely contained in C. ⊆ C is convex x1; x2 C; θ [0; 1]; θx1 + (1 θ)x2 C. () 8 2 8 2 − 2 This definition of convexity holds for any number of points in C, even infinite countable sums: C is convex P1 P1 xi C; θi 0; i = 1; 2;:::; ; i=1 θi = 1, the convex combination i=1 θixi C, if the series converges.() 8 2 ≥ 1 2 In general, C is convex iff for any random variable X over C, P (X C) = 1, its expectation is also in C: R 2 E(X) = xP (x)dx C, if the integral exists. C 2 Definition 5.9 C is strictly convex x1 = x2 C; θ (0; 1); θx1 + (1 θ)x2 int C. () 8 6 2 8 2 − 2 Intuitively, C is a strictly convex set iff a line segment between two points on the boundary, x1 and x2, intersects the boundary only at x1 and x2. 1 p=∞ p=2 0.5 p=1 p=0.5 0 −0.5 −1 −1 −0.5 0 0.5 1 Figure 5.1: Lp norm balls are convex sets for p 1, and non-convex for p < 1. ≥ Lecture 5: September 10 5-3 Some examples of convex sets: n The empty set , the singleton set x0 , and the complete space R . • ; f g n n n Lines x R x = θx1 + (1 θx2); x1; x2 R ; θ R , and line segments x R x = θx1 + (1 • f 2 j n − 2 2 g f 2 j − θx2); x1; x2 R ; θ [0; 1] 2 2 g Hyperplanes x Rn aT x = b; a Rn; b R , and halfspaces x Rn aT x b; a Rn; b R • f 2 j 2 2 g f 2 j ≤ 2 2 g n Euclidean balls B(x0; r) = x R x0 x 2 r • f 2 jk − k ≤ g 1 n Pn p p Lp balls, p 1. x R x0 x p r , where x p = ( i=1 xi ) . Lp balls for p (0; 1) are not • convex. ≥ f 2 jk − k ≤ g k k j j 2 n T Polyhedron: the solution set of a finite number of linear equalities and inequalities. P = x R aj x • T f 2 j ≤ bj; j = 1; : : : ; m; c x = di; i = 1; : : : ; p , or P = x Ax b; Cx = d i g f j ≤ g Polytope (bounded polyhedron), intersection of halfspaces and hyperplanes. • 5.1.3 Convex and conic hulls Pk Pk Definition 5.10 The convex hull of a set C, conv[C] = θixi xi C; θi 0 i = 1; : : : ; k; θi = f i=1 j 2 ≥ 8 i=1 1; k Z+ . 2 g Properties of convex hull: conv[C] is the smallest convex set that contains C. • conv[C] is convex. • C conv[C] • ⊆ C; C0 convex sets, C C0 = conv[C] C0 • 8 ⊂ ) ⊆ Definition 5.11 C is a cone x C; θ 0 = θx C. A convex cone is a cone which is also a convex set. () 2 ≥ ) 2 Definition 5.12 The conic hull of a set C, cone[C] = x x = θ1x1 + ::: + θkxk; θi 0; xi C f j ≥ 2 g n n×n An example of a convex cone is the set of symmetric PSD matrices, + = A R A 0 . If A; B n ; θ 0, then xT (θA + (1 θ)B)x = θxT Ax + (1 θ)xT Bx 0: S f 2 j g 2 S+ ≥ − − ≥ 5-4 Lecture 5: September 10 5.1.4 Convex set representations Figure 5.1: Representation of a convex set as the convex hull of a set of points (left), and as the intersection of a possibly infinite number of halfspaces (right). 5.1.4.1 Convex hull representation Let C Rn be a closed convex set. Then C can be written as conv(X), the convex hull of possibly infinitely many points⊆ ( X = ). Also, any closed convex set is the convex hull of itself. j j 1 5.1.4.2 Dual representation with halfspaces n T Let C R be a closed convex set. Then C can be written as i x ai x + bi 0 , the intersection of possibly⊆ infinitely many closed halfspaces. Also, every closed convex[ f setj is the intersection≤ g of all halfspaces that contain it. 5.1.5 Covexity preserving operations Let C Rn be a convex set. Then, the following operations preserve convexity: 2 Translation C + b • Scaling αC • Intersection If D is a convex set, then C D is convex. In general, if Sα is a convex set α A, then • \ 8 2 α2 Sα is convex. \ A Affine function Let A Rm×n, and b Rn. Then AC + b = Ax + b x C Rm is convex. • 2 2 f j 2 g ⊆ n m Set Sum Let C1 R and C2 R be convex sets. Then C1 + C2 = x1 + x2 x1 C1; x2 C2 is • convex. ⊆ ⊆ f j 2 2 g n+m Direct Sum C1 C2 = (x1; x2) R ; x1 C1; x2 C2 • × f 2 2 2 g n Perspective projection(pinhole camera) If C R R++ is a convex set, then P (C) is also convex, • where ⊂ × n P (x) = P (x1; x2; : : : ; xn; t) = (x1=t; x2=t; : : : ; xn=t) R 2 Ax+b m×n m n T Linear fractional function Let f(x) = T ;A R ; b R ; c R ; d R; dom f = x c x + • c x+d 2 2 2 2 f j d > 0 . Then f(C) Rm is convex. g 2 An operation which does not preserve convexity is set union. Lecture 5: September 10 5-5 5.1.6 Separating hyperplane theorem Figure 5.1: The hyperplane x aT x = b separates the disjoint convex sets C and D f j g Theorem 5.13 For convex sets C; D Rn;C D = ; a Rn; b R, such that x C; aT x b; y D; aT y b. ⊆ \ ; 9 2 2 8 2 ≤ 8 2 ≥ Let C1 and C2 be two convex sets. Then T T Definition 5.14 C1 and C2 are strongly separated if a [C1 + B(0; )] > b and a [C2 + B(0; )] < b. T Definition 5.15 C1 and C2 are properly separated if it is not the case that both C1 x : a = b and T ⊆ f g C2 x : a x = b ⊆ f g T T Definition 5.16 C1 and C2 are strictly separated if a x > b x C1 and a x < b x C2. 8 2 8 2 n Theorem 5.17 If C1;C2 are non-empty convex sets in R , with cl C1 cl C2 = , and either C1 or C2 \ ; bounded, there exists a hyperplane separating C1 and C2 strongly. 1 Consider x = (x1; x2) R++ R. An example where strong separation fails is when C1 = x x2 , 2 × f j ≥ x1 g C2 = x x2 0 . Here, x2 = 0 separates C1 and C2, but does not strongly separate them, since neither of the setsf j are≤ bounded.g n Theorem 5.18 If C1;C2 are non-empty convex sets in R , there exists a hyperplane separating C1 and C2 strongly infx 2C ;x 2C x1 x2 > 0 dist(C1;C2) > 0 0 cl (C1 C2). () 1 1 2 2 fj − jg () () 2 − 5-6 Lecture 5: September 10 5.1.7 Supporting hyperplane theorem T T Figure 5.1: The hyperplane x a x = a x0 supports C at x0/ f j g Theorem 5.19 For any convex set C and any boundary point x0, there exists a supporting hyperplane for C T T T T at x0.