Lecture Notes on Convex Analysis and Iterative Algorithms Ilker_ Bayram [email protected] About These Notes These are the lectures notes of a graduate course I offered in the Dept. of Elec- tronics and Telecommunications Engineering at Istanbul Technical University. My goal was to get students acquainted with methods of convex analysis, to make them more comfortable in following arguments that appear in recent signal processing literature, and understand/analyze the proximal point al- gorithm, along with its many variants. In the first half of the course, convex analysis is introduced at a level suitable for graduate students in electrical engi- neering (i.e., some familiarity with the notion of a convex set, convex functions from other courses). Then several other algorithms are derived based on the proximal point algorithm, such as the Douglas-Rachford algorithm, ADMM, and some applications to saddle point problems. There are no references in this version. I hope to add some in the future. Ilker_ Bayram December, 2018 Contents 1 Convex Sets2 1.1 Basic Definitions..........................2 1.2 Operations Preserving Convexity of Sets.............3 1.3 Projections onto Closed Convex Sets...............7 1.4 Separation and Normal Cones................... 10 1.5 Tangent and Normal Cones.................... 12 2 Convex Functions 15 2.1 Operations That Preserve the Convexity of Functions...... 17 2.2 First Order Differentiation..................... 18 2.3 Second Order Differentiation.................... 20 3 Conjugate Functions 22 4 Duality 27 4.1 A General Discussion of Duality.................. 27 4.2 Lagrangian Duality......................... 31 4.3 Karush-Kuhn-Tucker (KKT) Conditions............. 34 5 Subdifferentials 35 5.1 Motivation, Definition, Properties of Subdifferentials...... 35 5.2 Connection with the KKT Conditions............... 40 5.3 Monotonicity of the Subdifferential................ 40 6 Applications to Algorithms 45 6.1 The Proximal Point Algorithm................... 45 6.2 Firmly-Nonexpansive Operators.................. 48 6.3 The Dual PPA and the Augmented Lagrangian......... 52 6.4 The Douglas-Rachford Algorithm................. 53 6.5 Alternating Direction Method of Multipliers........... 57 6.6 A Generalized Proximal Point Algorithm............. 60 1 1 Convex Sets This first chapter introduces convex sets and discusses some of their properties. Having a solid understanding of convex sets is very useful for convex analysis of functions because we can and will regard a convex function as a special representation of a convex set, namely its epigraph. 1.1 Basic Definitions Definition 1. A set C 2 Rn is said to be convex if x 2 C, x0 2 C implies that αx + (1 − α)x 2 C for any α 2 [0; 1]. Consider the sets below. Each pair of (x; x0) we select in the set on the left defines a line segment which lies inside the set. However, this is not the case for the set on the right. Even though we are able to find line segments with endpoints inside the set (as in (x; x0)), this is not true in general, as exemplified by (y; y0). x0 y y0 x x x0 For the examples below, decide if the set is convex or not and prove whatever you think is true. n Example 1 (Hyperplane). For given s 2 R , r 2 R, consider the set Hs;r = fx 2 Rn : hs; xi = rg. Notice that this is a subspace for r = 0. Example 2 (Affine Subspace). This is a set V 2 Rn such that if x 2 V and x0 2 V , then αx + (1 − α)x0 2 V for all α 2 R. n − Example 3 (Half Space). For given s 2 R , r 2 R, consider the set Hs;r = fx 2 Rn : hs; xi ≤ rg. Example 4 (Cone). A cone K 2 Rn is a set such that if x 2 K, then αx 2 K for all α > 0. Note that a cone may be convex or non-convex. See below for an example of a convex (left) and a non-convex (right) cone. 2 Exercise 1. Let K be a cone. Show that K is convex if and only if x; y 2 K implies x + y 2 K. 1.2 Operations Preserving Convexity of Sets Proposition 1 (Intersection of Convex Sets). Let C1, C2,... Ck be convex sets. Show that C = \iCi is convex. Proof. Exercise! Question 1. What happens if the intersection is empty? Is the empty set convex? This simple result is useful for characterizing linear systems of equations or inequalities. Example 5. For a matrix A, the solution set of Ax = b is an intersection of hyperplanes. Therefore it is convex. For the example above, we can in fact say more, thanks to the following vari- ation of Prop.1. Exercise 2. Show that the intersection of a finite collection of affine subspaces is an affine subspace. Let us continue with systems of linear inequalities. Example 6. For a matrix A, the solution set of Ax ≤ b is an intersection of half spaces. Therefore it is convex. Proposition 2 (Cartesian Products of Convex Sets). Suppose C1,. , Ck are n convex sets in R . Then the Cartesian product C1 × · · · × Ck is a convex set in Rn×···×n. Proof. Exercise! Given an operator F and a set C, we can apply F to elements of C to obtain the image of C under F . We will denote that set as FC. If F is linear then it preserves convexity. Proposition 3 (Linear Transformations of Convex Sets). Let M be a matrix. If C is convex, then MC is also convex. Consider now an operator that just adds a vector dto its operand. This is a translation operator. Geometrically, it should be obvious that translation preserves convexity. It is a good exercise to translate this mental picture to an algebraic expression and show the following. 3 Proposition 4 (Translation). If C is convex, then the set C + d = fx : x = v + d for some v 2 Cgis also convex. Given C1, C2, consider the set of points of the form v = v1 + v2, where vi 2 Ci. This set is denoted by C1 + C2and is called the Minkowski sum of C1and C2. We have the following result concerning Minkowski sums. Proposition 5 (Minkowski Sums of Convex Sets). If C1 and C2 are convex then C1 + C2is also convex. Proof. Observe that C1 + C2 = II (C1 × C2): Thus it follows by Prop.2 and Prop.3 that C1 + C2 is convex. Example 7. The Minkowski sum of a rectangle and a disk in R2 in shown below. 3 2 + = 1 1 2 Definition 2 (Convex Combination). Consider a finite collection of points x1, ..., xk. x is said to be a convex combination of xi's if x satisfies x = α1 x1 + ··· + αk xk for some αi such that αi ≥ 0 for all i; k X αk = 1: i=1 Definition 3 (Convex Hull). The set of all convex combinations of a set C is called the convex hull of C and is denoted as Co(C). Below are two examples, showing the convex hull of the sets C = fx1; x2g, and 0 C = fy1; y2; y3g. 4 y3 y2 x2 x1 y1 Notice that in the definition of the convex hull, the set C does not have to be convex (in fact, C is not convex in the examples above). Also, regardless of the dimension of C, when constructing Co(C), we can consider convex combinations of any number of finite elements chosen from C. In fact, if we denote the set of all convex combinations of kelements from C as Ck, then we can show that Ck ⊂ Ck+m for m ≥ 0. An interesting result, which we will not use is in this course, is the following. Exercise 3 (Caratheodory's Theorem). Show that, if C 2 Rn, then Co(C) = Cn+1. The following proposition justifies the term `convex' in the definition of the convex hull. Proposition 6. The convex hull of a set C is convex. Proof. Exercise! The convex hull of C is the smallest convex set that contains C. More precisely, we have the following. Proposition 7. If D = Co(C), and if E is a convex set with C ⊂ E, then D ⊂ E. Proof. The idea is to show that for any integer k, Econtains all convex com- binations involving k elements from C. For this, we will present an argument based on induction. We start with k = 2. Suppose x1; x2 2 C. This implies x1; x2 2 E. Since E is convex, we have αx1 + (1 − α)x2 2 E, for all α 2 [0; 1]. Since x1, x2 were arbitrary elements of C, it follows that Econtains all convex combinations of any two elements from C. Suppose now that Econtains all convex combinations of any k − 1 elements Pk−1 from C. That is, if x1,... xk−1 are in C, then for i=1 αi = 1, and αi ≥ 0, we Pk−1 th have i=1 αi xi 2 E. Suppose we pick a k element, say xk, from C. Also, let α1,... αkbe on the unit simplex, with αk 6= 0 (if αk = 0, we have nothing 5 to prove). Observe that k k−1 X X αi xi = αk xk + αi xi i=1 i=1 k−1 X αi = α x + (1 − α ) x : k k k 1 − α i i=1 k Notice that k−1 X αi = 1; 1 − α i=1 k α i ≥ 0; for all i: 1 − αk Therefore, k−1 X αi y = x 1 − α i i=1 k is an element of E since it is a convex combination of k − 1 elements of C.