Introduction to convex optimization Teemu Pennanen April 3, 2019 2 Contents 1 Convexity 7 1.1 Examples . 7 1.2 Extended real-valued functions . 16 1.3 Convexity . 18 1.4 Convexity in algebraic operations . 23 1.5 Convex sets and functions under scaling . 32 1.6 Separation of convex sets . 36 2 Topological properties 39 2.1 Topological spaces . 39 2.2 Existence of optimal solutions . 42 2.3 Compactness in Euclidean spaces . 46 2.4 Interiors of convex sets . 47 2.5 Continuity of convex functions . 50 3 Duality 55 3.1 Conjugate convex functions . 55 3.2 Parametric optimization and saddle-point problems . 64 3.3 Calculus of subgradients and conjugates . 68 3.4 Parametric optimization and saddle-point problems: Part II . 72 3.5 Parametric dual . 75 3.6 Some closedness criteria . 76 3 4 CONTENTS Preface Optimization problems arise whenever decisions are to be made. Many phenom- ena in natural sciences can also be described in terms of minima or maxima of certain functions. In the words of Leonhard Euler, \...nothing whatsoever takes place in the universe in which some relation of maximum or minimum does not appear". These lecture notes study general aspects of convex optimization problems where one seeks to minimize a convex function over a linear space. In a cer- tain sense, convex optimization problems form the nicest class of optimization problems. Stronger analytical results e.g. on the existence and uniqueness of solutions and on optimality conditions are available as soon as the problem is known to be convex. Convexity is essential also in the duality theory of opti- mization. Under convexity, one can treat nonsmooth and infinite-dimensional problems with the same ease as the smooth and finite-dimensional ones. Con- vexity is indispensable in numerical optimization in higher dimensions. When analyzing a given optimization problem, convexity is the first thing to look for. Many aspects of more specific problem classes such as stochastic optimiza- tion, robust optimization, calculus of variations, optimal control, semidefinite programming and multicriteria optimization can be treated under convex analy- sis. Although convexity rules out some important problems e.g. in combinatorial optimization, it arises quite naturally in many applications. Classical applica- tion fields include operations research, engineering, physics and economics. More recently, convex optimization has found important applications in mathemati- cal finance and financial engineering. Even some combinatorial problems can be analyzed with techniques of convex analysis. Selected applications will be treated in the following sections. These notes study convex optimization in general topological vector spaces. The generality is motivated by various important applications e.g. in physics and financial economics which go beyond finite-dimensional spaces. In stochas- tic optimization and mathematical finance, one often encounters topological vector spaces which are not even locally convex. The material is divided into three chapters according to mathematical structure. After outlining some ap- plications, the first chapter studies convex optimization in general (real) vector spaces. Chapter 2 studies optimization problems in topological vector spaces. The last chapter is devoted to duality theory in locally convex topological vec- tor spaces. The necessary topological and functional analytic concepts will be 5 6 CONTENTS introduced as needed. Most of the material in these notes is collected from [11], [13], [16] and [7]. Chapter 1 Convexity This chapter studies convex optimization problems in general vector spaces. The aim is to summarize some basic facts in convexity that do not require topology or duality. Most results are quite simple but already useful in practice. We start by studying convex functions on the real line and then proceed to study convexity preserving operations that result in convex functions on more general vector spaces. Knowledge of such operations is useful in identification of convexity in applications as well as in building tractable optimization models in decision sciences. We will also study differential and asymptotic scaling properties of convex sets and functions. Such properties are involved in conditions for optimality and existence of solutions, respectively. We will see that convexity makes the scaling behaviour remarkably tractable and allows for useful geometric interpre- tations and calculus rules in practice. We end this chapter by presenting the fundamental separation theorems that yield more familiar separation theorems when applied in topological vector spaces. Before going to the general theory, we outline some applications where con- vexity arises quite naturally. That these problems are in fact convex, will be verified as applications as we proceed with the general theory. 1.1 Examples This section discusses some well-known optimization problems where convexity has an important role. The choice of applications is rather arbitrarily and many important applications e.g. in statistics and inverse problems have been omitted. Mathematical programming Mathematical programming is the discipline that studies numerical solution of optimization problems mostly in finite-dimensional spaces of real or integer vari- ables. Computationally efficient algorithms are available for many classes of 7 8 CHAPTER 1. CONVEXITY convex optimization problems over real variables. Even in combinatorial opti- mization, many computational techniques are based on convex approximations. Optimization routines are usually designed for classes of optimization problems that can be written in a \standard" format. The classical linear programming format is minimize x · c over x 2 Rn + (LP) subject to Ax = b; Rn Rm×n Rm Pn where c 2 , A 2 , b 2 and c · x = i=1 cixi. Linear programming models were studied already by Kantorovich in the 1930's as a general format of planning problems in operations research; see [6]. They gained popularity with George Dantzig's invention of the simplex algorithm [3] which is still widely used. For large problems, modern interior point methods often outperform simplex methods both in theory and practice. An important special case of the LP-model is the linear network flow problem X A minimize ci;jxi;j over x 2 R (i;j)2A X X subject to xi;j − xj;i = si 8i 2 N fjj(i;j)2Ag fjj(j;i)2Ag li;j ≤ xi;j ≤ ui;j 8(i; j) 2 A; where N is the set of nodes and A ⊆ f(i; j) j i; j 2 N g is the set of arcs of a network. A notable feature of such models is that if the parameters si, li;j and ui;j have integer values, then there is an optimal solution x with all xi;j integer; see e.g. Rockafellar [15] or Bertsekas [2]. Important special cases of the general network optimization problem include shortest path, assignment, transportation and max-flow problems. There are also natural infinite-dimensional extensions where the finite network is replaced by a measurable space (Ω; F) and flows are described in terms of measures on the product space (Ω × Ω; F ⊗ F). The most famous instance is the Monge-Kantorovich mass transportation problem; see e.g. [9]. Nonlinear extensions can be found in [?]. Replacing the linear functions in the LP-model by more general ones, one obtains the nonlinear programming model n minimize f0(x) over x 2 R (NLP) subject to fj(x) ≤ 0; j = 1; : : : ; m: This is a common format for computational optimization routines. More flexible and often more natural format is the composite optimization model minimize h(F (x)) over x 2 Rn; (CO) n m+1 where F = (f0; : : : ; fm) is a vector-valued function from R to R and h is a m+1 function on R . In this model, all the functions fj are treated symmetrically 1.1. EXAMPLES 9 much as in multicriteria optimization. Allowing h to take on infinite-values, one obtains (NLP) as a special case of (CO); see Section 1.4 below. Some modern convex optimization solvers assume the problem format minimize c · x over x 2 K (CP) subject to Ax = b; where K ⊂ Rn is a convex cone, A is a matrix and b is a vector of appropriate Rn dimensions. Note that with K = +, (CP) becomes (LP). With appropri- ate choices of K, one can express quite a large class of nonlinear optimization problems in the above format. See e.g. the MOSEK \modeling cookbook" [?]. Stochastic optimization Most decision making problem in practice involve uncertain factors that have an essential effect on the outcome of a decision. Let X be the space where an optimal decision is sought and assume that the decision maker describes her views about the uncertain factors by a probability space (Ω; F;P ). The basic static stochastic optimization problem can be written as minimize Ef(x; ·) over x 2 X; (SP) where f : X × Ω ! R is such that ! 7! f(x; !) is measurable for every x 2 X. Despite its simplicity, the above format is quite universal and has applica- tions from financial risk management and derivative pricing to statistics and machine learning; see below. Maximum entropy principle One of the fundamental principles in probabilistic modeling is that of maximum entropy. In a basic form, it states that if all one knows about the uncertainty j m over possible states Ω of a system is the expectations of a collection fφ gj=1 of random variables, then one should describe the system by a probability measure that solves the problem P dP minimize E ln over p 2 Pµ; dµ (ME) subject to EP φj = φ¯j j = 1; : : : ; m; where Pµ denotes the set of probability measures that are absolutely continuous with respect to a given reference measure µ on Ω. The objective is known as the entropy of P with respect to µ. The rationale is that the maximum entropy measure is the most \uninformative" among the probability measures that are consistent with the information on the expectations of φj.