Convex Analysis1 Patrick Cheridito Princeton University, Spring 2013 1Many thanks to Andreas Hamel for providing his lecture notes. Large parts of chapters 2-4 are based on them. 2 Contents Contents 2 1 Convex Analysis in Rd 5 1.1 Subspaces, affine sets, convex sets, cones and half-spaces . 5 1.2 Separation results in finite dimensions . 8 1.3 Linear, affine and convex functions . 12 1.4 Derivatives, directional derivatives and sub-gradients . 16 1.5 Convex conjugates . 19 1.6 Inf-convolution . 23 2 General Vector Spaces 27 2.1 Definitions . 27 2.2 Zorn's lemma and extension results . 30 2.3 Algebraic interior and separation results . 33 2.4 Directional derivatives and sub-gradients . 35 3 Topological Vector Spaces 37 3.1 Topological spaces . 37 3.2 Continuous linear functionals and extension results . 42 3.3 Separation with continuous linear functionals . 44 3.4 Continuity of convex functions . 47 3.5 Derivatives and sub-gradients . 48 3.6 Dual pairs . 50 3.7 Convex conjugates . 53 3.8 Inf-convolution . 55 4 Convex Optimization 59 4.1 Perturbation and the dual problem . 59 4.2 Lagrangians and saddle points . 65 4.3 Karush{Kuhn{Tucker-type conditions . 67 3 4 CONTENTS Chapter 1 d Convex Analysis in R The following notation is used: • d 2 N := f1; 2;:::g d • ei is the i-th unit vector in R Pd d • hx; yi := i=1 xiyi for x; y 2 R p • jjxjj := hx; xi for x 2 Rd d • B"(x) := y 2 R : jjx − yjj ≤ " • R+ := fx 2 R : x ≥ 0g, R++ := fx 2 R : x > 0g • x _ y := max fx; yg and x ^ y := min fx; yg for x; y 2 R 1.1 Subspaces, affine sets, convex sets, cones and half-spaces Definition 1.1.1 Let C be a subset of Rd. C is a subspace of Rd if λx + y 2 C for all x; y 2 C and λ 2 R: C is an affine set if λx + (1 − λ)y 2 C for all x; y 2 C and λ 2 R: C is a convex set if λx + (1 − λ)y 2 C for all x; y 2 C and λ 2 [0; 1]: C is a cone if λx 2 C for all x 2 C and λ 2 R++: 5 6 CHAPTER 1. CONVEX ANALYSIS IN RD Exercise 1.1.2 Let C; D be non-empty subsets of Rd. 1. Show that if C; D are subspaces, then so is C − D := fx − y : x 2 C; y 2 Dg ; and the same is true for affine sets, convex sets and cones. 2. Show that if C is affine, then C + v is affine for every v 2 Rk. 3. Show that if C is affine and contains 0, it is a subspace. 4. Show that if C is affine and v 2 C, then C − v = C − C is a subspace. 5. Show that the intersection of arbitrarily many subspaces is a subspace, and that the same is true for affine subsets, convex subsets and cones. 6. Show that there exists a smallest subspace containing C, and that the same is true for affine sets, convex sets and cones. Definition 1.1.3 If C is a non-empty subset of Rd, we denote by lin C, aff C, conv C, cone C the smallest subspace, affine set, convex set, cone containing C, respectively. Exercise 1.1.4 Let C be a non-empty subset of Rd. Show that ( n ) X lin C = λixi : n 2 N; λi 2 R; xi 2 C i=1 ( n n ) X X aff C = λixi : n 2 N; λi 2 R; xi 2 C; λi = 1 i=1 i=1 ( n n ) X X conv C = λixi : n 2 N; λi 2 R+; xi 2 C; λi = 1 i=1 i=1 cone C = fλx : λ 2 R++; x 2 Cg Definition 1.1.5 The dimension of an affine subset M of Rd is the dimension of the subspace M − M. The dimension of an arbitrary subset C is the dimension of aff C. Definition 1.1.6 Let C be a non-empty subset of Rd. The dual cone of C is the set ∗ d C := z 2 R : hx; zi ≥ 0 for all x 2 C : Exercise 1.1.7 Show that the dual cone C∗ of a non-empty subset C ⊆ Rd is a closed convex cone and C is contained in C∗∗. Definition 1.1.8 The recession cone 0+C of a subset C of Rd consists of all y 2 R satisfying x + λy 2 C for all x 2 C and λ 2 R++: Every y 2 0+C n f0g is called a direction of recession for C. 1.1. SUBSPACES, AFFINE SETS, CONVEX SETS, CONES AND HALF-SPACES7 Definition 1.1.9 Let C be a subset of Rd. The closure cl C of C is the smallest closed subset of Rd containing C. The interior int C consists of all x 2 C such that B"(x) ⊆ C for some " 2 R++. The relative interior ri C is the set of all x 2 C such that B"(x) \ aff C ⊆ C for some " 2 R++. The boundary of C is the set bd C := cl C n int C. The relative boundary is rbd C := cl C n ri C Exercise 1.1.10 1. Show that an affine subset of Rd is closed. 2. Show that the closure of a cone is a cone. 3. Show that the closure of a convex set is convex. Lemma 1.1.11 Let C be a non-empty convex subset of Rd and λ 2 (0; 1]. If int C 6= ;, then λint C + (1 − λ)cl C ⊆ int C: (1.1.1) If ri C 6= ;, then λri C + (1 − λ)cl C ⊆ ri C (1.1.2) In particular, int C and ri C are convex. Proof. Let x 2 int C, y 2 cl C and λ 2 (0; 1]. There exists " > 0 such that B2"(x) ⊆ C and z 2 C such that (1 − λ)jjy − zjj ≤ λε. Choose v 2 Bλε(0). Then v 1 − λ w = + (y − z) 2 B (0); λ λ 2" and therefore, λx + (1 − λ)y + v = λ(x + w) + (1 − λ)z 2 C: d This shows (1.1.1). (1.1.2) follows by working in aff C instead of R . Lemma 1.1.12 Let C be a convex subset of Rd. Then int C 6= ; if and only if aff C = Rd. Proof. If x 2 int C, then 0 2 int C − x, and it follows that d aff (C) − x = aff (C − x) = lin (C − x) = R : On the other hand, if aff C = Rd, choose x 2 C. Then d lin (C − x) = aff (C − x) = aff (C) − x = R : So there there exist d vectors x1; : : : ; xd in C such that vi := xi − x are linearly independent. Since C is convex, one has 1 1 (x + x + ··· + x ) + λv 2 C for jλj ≤ and i = 1; : : : ; d; d + 1 1 d i d + 1 8 CHAPTER 1. CONVEX ANALYSIS IN RD and therefore, 1 (x + x + ··· + x ) + V ⊆ C; d + 1 1 d nPd Pd 1 o Pd where V := i=1 λivi : i=1 jλij ≤ d+1 . So since n(x) := i=1 jλij for x = Pd d i=1 λivi defines a norm and all norms on R are equivalent, there exists an " > 0 such that 1 (x + x + ··· + x ) + B (0) ⊆ C: d + 1 1 d " Corollary 1.1.13 Let C be a non-empty convex subset of Rd. Then ri C is dense in C. In particular, ri C is non-empty. Proof. If C consists of only one point x0, then ri C = C = fx0g. If C contains at least two different points, one can, by shifting, assume that one of them is 0. Then lin C = aff C is at least one-dimensional. So by restricting to lin C, one can assume that lin C = Rd. It follows from Lemma 1.1.12 that ri C 6= ;. Now the corollary follows from Lemma 1.1.11. Definition 1.1.14 A half-space in Rd is a set of the form d d x 2 R : hx; zi ≥ c for some z 2 R n f0g and c 2 R: d d We say a subset C of R is supported at x0 2 C by z 2 R n f0g if hx0; zi = infx2C hx; zi. d d Note that if a subset C of R is supported at x0 2 C by some z 2 R n f0g, then x0 is in the boundary of C and C is contained in the half-space d x 2 R : hx; zi ≥ hx0; zi : 1.2 Separation results in finite dimensions d Lemma 1.2.1 Let C be a non-empty closed subset of R . Then there exists x0 2 C such that jjx0jj = inf jjxjj: x2C If in addition, C is convex, then x0 is unique. Proof. For fixed y 2 C, the set fx 2 C : jjxjj ≤ jjyjjg is closed and bounded. So the existence of x0 follows because the norm is continuous. If C is convex and x0; x1 are two different norm minimizers, one has x + x jj 0 1 jj < jjx jj = jjx jj; 2 0 1 a contradiction.