10-725: Convex Optimization Fall 2013 Lecture 4: Convexity Lecturer: Barnab´asP´oczos Scribes: Jessica Chemali, David Fouhey, Yuxiong Wang Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications. They may be distributed outside this class only with the permission of the Instructor. The goals of this lecture are to: 1. give definitions that are important to convexity as well as examples of convex sets and basic properties; 2. define convex functions and their properties, as well as some examples. 4.1 Basic Definitions We begin by formalizing a few mathematical objects that we will use throughout the lecture: n n Definition 4.1 A line passing through x1 and x2 2 R forms the set fx 2 R jx = θx1 + (1 − θ)x2; θ 2 Rg. Definition 4.2 The line segment passing through x1 and x2 is defined similarly, but with θ confined to [0; 1], n or fx 2 R jx = θx1 + (1 − θ)x2; θ 2 [0; 1]g: Figure 4.1: Line and line segment. n Definition 4.3 A Euclidean ball with radius r centered at x0, B(xo; r), is: fx 2 R jjjx − x0jj2 ≤ rg Pn p 1=p Definition 4.4 An Lp ball is the equivalent, but for the Lp distance, or jjxjjp = ( i=1 jxij ) fx 2 n R jjjx − x0jjp ≤ rg Definition 4.5 A half-space HS(a; b) is a set of points on one side of a hyperplane: fx 2 RnjaT x ≤ bg . 4-1 4-2 Lecture 4: Convexity Figure 4.2: Halfspace. Definition 4.6 A hyperplane HS(a; b) is defined similarly, but with equality, or fx 2 RnjaT x = bg . Figure 4.3: Hyperplane. Definition 4.7 A cone is any set such that if x 2 C and θ ≥ 0 then θx 2 C. We can immediately use these to give a definition of an affine set. Specifically, a set C is affine if for any points in it, the line through them is also contained in C. For example, the affine hull of a circle in R2 is R2 2 and the affine hull of a line in R is R. Written more formally, for x1; x2 2 C and θ 2 C, θx1 + (1 − θ)x2 2 C. This definition also leads to the definition of the affine hull of a set, which is the smallest affine set containing C. The affine hull can be written out explicitly as: ( k ) X Aff[C] = θ1x1 + ··· + θkxkjxi 2 C; θi = 1 i=1 Similarly, a conic hull is Cone[C] = fxjx = θ1x1 + ::: + θkxk; θi ≥ 0; xi 2 Cg Lecture 4: Convexity 4-3 Figure 4.4: Cone. Figure 4.5: Conic Hull. Before we proceed with the remaining definitions, we also need to give a few topological definitions. Given a set C, Definition 4.8 We say that x is on the boundary of C, or @C, if for some small enough > 0, the epsilon- radius ball centered at x covers both inside and outside the set, or B(x; ) \ C 6= ; and B(x; ) \ Cc 6= ;. Definition 4.9 We say that x is in the interior of C if 9 > 0 s.t. B(x; ) ⊂ C. Definition 4.10 We say that the relative interior of C, rel int C is the same as the interior, but with the ball restricted to the affine hull of C. Specifically, x is in rel int C if B(x; ) \ Aff[C] ⊂ C. This definition is more natural in some ways. Consider a line segment in a 2D-space. No points in the space are in the interior the line segment since any epsilon ball contains points off the line; however, if we restrict our consideration to the affine hull of the line segment (the equivalent line), some points are. We can then use set operations to take these and generate more definitions and properties of C: 1. The closure of C, cl C, is C [ @C. 2. The relative boundary of C, rel@C, is cl Cnrel int C. 3. C is closed if @C ⊂ C 4. C is open if @C \ C = ; 4-4 Lecture 4: Convexity 5. C is compact if it is closed and bounded in Rn. 4.2 Convex sets 4.2.1 Definitions Definition 4.11 A set C ⊂ Rn is convex if for any two points in C, the line segment joining them is contained in C. Formally, it is convex if and only if for all x1; x2 2 C and θ 2 [0; 1], θx1 + (1 − θ)x2 2 C. Figure 4.6: Convex sets. Definition 4.12 A convex set is strictly convex if for any two points in the set in general position, the line segment less the endpoints is contained in int C. Formally, if for all x1; x2 2 C with x1 6= x2 and θ 2 (0; 1) θx1 + (1 − θ)x2 2 int C. Definition 4.13 The convex hull of a set C is the set of all convex combinations of the points in C k X + conv[C] = fθ1x1 + ··· + θkxkjxi 2 C; θi ≥ 0 8i = 1; : : : ; k; θi = 1; k 2 Z g i=1 Pk Here, we are constraining θi in both ways that we did before: θi ≥ 0 as in the conic hull, and i=1 θi = 1 as in the affine hull. Just as in the affine hull case, the conv[C] is the smallest convex set that also contains C. Note that: conv[C] is convex and C ⊂ conv[C]. Figure 4.7: Convex hull. Lecture 4: Convexity 4-5 4.2.2 Examples Example 4.14 Many of the objects defined earlier are convex sets: lines, line segments, hyperplanes and d half spaces. Additionally, the empty set ; and singleton sets fxg are convex, as are complete spaces R . Lp Balls are convex for p ≥ 1, but are not for 0 < p < 1. Example 4.15 A more complex example of a convex set is a polyhedron. This, at least in this course, is a solution set to a finite set number of linear equalities and inequalities (i.e., fxjAx ≤ b; Cx = dg. If the solution set is bounded, we call it a polytope. As we will see later, this is easy to show as convex, as it is a an intersection of halfspaces (from Ax ≤ b) and hyperplanes (from Cx = d), each of which are convex. Example 4.16 One can also construct convex sets via the convex combination of points. The definition of the convex hull gives it for a finite set of points. One can also define it for an infinite sum: given P1 P1 x1; x2;::: 2 C and θ1; θ2;::: ≥ 0 such that θi = 1, then θixi 2 C so long as the series converges. i=1 R i=1 Similarly, one can use a density function p, and C p(x)xdx 2 C if the integral exists. Example 4.17 We can extend this to more complex domains, such as matrices. For instance, the set of n n × n positive semi-definite (PSD) matrices form a convex cone which we denote S+. Recall that a matrix A is PSD if and only if A = AT and xT Ax ≥ 0 for any x 2 Rn; equivalently a symmetric matrix is PSD if all its eigenvalues are positive. This may also be written as A ≥ 0, which suggests a partial ordering defined by the PSD property, namely, M ≥ N if and only if M − N is PSD. We can prove convexity by linearity: given PSD matrices A and B, xT [θA + (1 − θ)B]x] = θxT Ax + (1 − θ)xT Bx ≥ 0 as it is a convex sum of non-negative terms (since A and B are PSD). 4.2.3 Representations We can represent a convex set in two equivalent ways. Definition 4.18 The primal representation represents a convex set C using its convex hull: a convex com- bination of its points. The number of points might be finite (like in the case of a polyhedron) or infinite (like the case of a circle). Definition 4.19 The dual representation represents a closed set C as the intersection of all the closed halfspaces containing it. 4.2.4 Convexity Preserving Operations One easy way to show that a set is convex is to construct it from convex sets via convexity preserving operations. Here are a few. Given convex sets C; D ⊂ Rn, b 2 Rn, and A 2 Rm×n, α 2 R, the following sets are convex: 1. Translation: C + b = fx + b : x 2 Cg 2. Scaling: αC = fαx : x 2 Cg 3. Intersection: C \ D; this can also be extended to an infinite number of sets. 4-6 Lecture 4: Convexity Figure 4.8: Dual representation as Intersection of hyperplanes. 4. Affine: AC + b = fAx + bjx 2 Cg ⊂ Rm 5. Set sum: C + D = fx + yjx 2 C; y 2 Dg. 6. Direct sum: C × D = f(x; y) 2 Rn+m; x 2 C; y 2 Dg n A more involved operation is perspective projection: suppose C ⊂ R × R++ is convex, then P (C) is also convex, where P is perspective projection, or: n P (x) = P (x1; : : : ; xn; t) = (x1=t; x2=t; : : : ; xn=t) 2 R This can be extended further to linear-fractional functions. Theorem: image of linear-fractional functions. Let A 2 Rm×n, c 2 Rn, and b 2 Rm, d 2 R.