DOCSLIB.ORG

Course Notes: Convex Analysis and Optimization

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Course notes: Convex Analysis and Optimization Dmitriy Drusvyatskiy May 27, 2019 ii Contents 1 Review of Fundamentals 1 1.1 Inner products and linear maps . .1 1.2 Norms . .2 1.3 Eigenvalue and singular value decompositions of matrices . .4 1.4 Point-set topology and differentiability . .4 1.5 Fundamental theorems of calculus & accuracy in approximation8 2 Smooth minimization 13 2.1 Optimality conditions: Smooth Unconstrained . 13 2.2 Convexity, a first look . 15 2.3 Rates of convergence . 19 2.4 Two basic methods . 20 2.4.1 Majorization view of gradient descent . 21 2.4.2 Newton's method . 24 2.5 Computational complexity for smooth convex minimization . 26 2.6 Conjugate Gradient Method . 29 2.7 Optimal methods for smooth convex minimization . 34 2.7.1 Fast gradient methods . 34 2.7.2 Fast gradient methods through estimate sequences . 40 2.7.3 Optimal quadratic averaging . 46 3 Convex geometry and analysis 55 3.1 Basic convex geometry . 55 3.1.1 Separation theorems . 58 3.1.2 Cones and polarity . 60 3.1.3 Tangents and normals . 61 3.2 Convex functions: basic operations and continuity . 63 3.3 The Fenchel conjugate . 67 3.4 Differential properties . 70 3.5 Directional derivative . 72 3.6 The value function . 74 3.7 Duality and subdifferential calculus . 75 3.7.1 Fenchel-Rockafellar duality . 75 iii iv CONTENTS 3.7.2 Lagrangian Duality . 78 3.8 Moreau-Yosida envelope and the proximal map . 82 3.9 Orthogonally invariant functions . 84 Chapter 1 Review of Fundamentals 1.1 Inner products and linear maps Throughout, we fix an Euclidean space E, meaning that E is a finite- dimensional real vector space endowed with an inner product h·; ·i. Recall that an inner-product on E is an assignment h·; ·i: E × E ! R satisfying the following three properties for all x; y; z 2 E and scalars a; b 2 R: (Symmetry) hx; yi = hy; xi (Bilinearity) hax + by; zi = ahx; zi + bhy; zi (Positive definiteness) hx; xi ≥ 0 and equality hx; xi = 0 holds if and only if x = 0. The most familiar example is the Euclidean space of n-dimensional col- umn vectors Rn, which unless otherwise stated we always equip with the Pn T dot-product hx; yi := i=1 xiyi. One can equivalently write hx; yi = x y. A basic result of linear algebra shows that all Euclidean spaces E can be identified with Rn for some integer n, once an orthonormal basis is cho- sen. Though such a basis-specific interpretation can be useful, it is often distracting, with the indices hiding the underlying geometry. Consequently, it is often best to think coordinate-free. The space of real m × n-matrices Rm×n furnishes another example of an Euclidean space, which we always equip with the trace product hX; Y i := T P tr X Y . Some arithmetic shows the equality hX; Y i = i;j XijYij. Thus the trace product on Rm×n is nothing but the usual dot-product on the ma- trices stretched out into long vectors. This viewpoint, however, is typically not very fruitful, and it is best to think of the trace product as a standalone object. An important Euclidean subspace of Rn×n is the space of real sym- metric n × n-matrices Sn, along with the trace product hX; Y i := tr XY . 1 2 CHAPTER 1. REVIEW OF FUNDAMENTALS For any linear mapping A: E ! Y, there exists a unique linear mapping A∗ : Y ! E, called the adjoint, satisfying hAx; yi = hx; A∗yi for all points x 2 E; y 2 Y: In the most familiar case of E = Rn and Y = Rm, the matrix representing A∗ is simply the transpose of the matrix representing A. Exercise 1.1. Given a collection of real m × n matrices A1;A2;:::;Al, define the linear mapping A: Rm×n ! Rl by setting A(X) := (hA1;Xi; hA2;Xi;:::; hAl;Xi): ∗ Show that the adjoint is the mapping A y = y1A1 + y2A2 + ::: + ylAl. Linear mappings A between E and itself are called linear operators, and are said to be self-adjoint if equality A = A∗ holds. Self-adjoint operators on Rn are precisely those operators that are representable as symmetric matrices. A self-adjoint operator A is positive semi-definite, denoted A 0, whenever hAx; xi ≥ 0 for all x 2 E: Similarly, a self-adjoint operator A is positive definite, denoted A 0, when- ever hAx; xi > 0 for all 0 6= x 2 E: A positive semidefinite linear operator A is positive definite if and only if A is invertible. Consider a self-adjoint operator A. A number λ is an eigenvalue of X if there exists a vector 0 6= v 2 E satisfying Av = λv. Any such vector v is called an eigenvector corresponding to λ. The Rayleigh-Ritz theorem shows that the following relation always holds: hAu; ui λ (A) ≤ ≤ λ (A) for all u 2 E n f0g; min hu; ui max where λmin(A) and λmax(A) are the minimal and maximal eigenvalues of A, respectively. Consequently, an operator A is positive semidefinite if and only λmin(A) ≥ 0 and A is positive definite if and only λmin(A) > 0. 1.2 Norms A norm on a vector space V is a function k·k: V! R for which the following three properties hold for all point x; y 2 V and scalars a 2 R: (Absolute homogeneity) kaxk = jaj · kxk (Triangle inequality) kx + yk ≤ kxk + kyk 1.2. NORMS 3 (Positivity) Equality kxk = 0 holds if and only if x = 0. The inner product in the Euclidean space E always induces a norm kxk := phx; xi. Unless specified otherwise, the symbol kxk for x 2 E will always denote this induced norm. For example, the dot product on Rn p 2 2 induces the usual 2-norm kxk2 = x1 + ::: + xn, while the trace product m×n p T on R induces the Frobenius norm kXkF = tr (X X). n Other important norms are the lp−norms on R : ( p p 1=p (jx1j + ::: + jxnj ) for 1 ≤ p < 1 kxkp = : maxfjx1j;:::; jxnjg for p = 1 The most notable of these are the l1, l2, and l1 norms. For an arbitrary norm k · k on E, the dual norm k · k∗ on E is defined by kvk∗ := maxfhv; xi : kxk ≤ 1g: n For p; q 2 [1; 1], the lp and lq norms on R are dual to each other whenever p−1 + q−1 = 1. For an arbitrary norm k · k on E, the Cauchy-Schwarz inequality holds: jhx; yij ≤ kxk · kyk∗: Exercise 1.2. Given a positive definite linear operator A on E, show that the assignment hv; wiA := hAv; wi is an inner product on E, with the in- p duced norm kvkA = hAv; vi. Show that the dual norm with respect to ∗ p −1 the original inner product is kvkA = kvkA−1 = hA v; vi. All norms on E are \equivalent" in the sense that any two are within a constant factor of each other. More precisely, for any two norms ρ1(·) and ρ2(·), there exist constants α; β ≥ 0 satisfying αρ1(x) ≤ ρ2(x) ≤ βρ1(x) for all x 2 E: Case in point, for any vector x 2 Rn, the relations hold: p kxk2 ≤ kxk1 ≤ nkxk2 p kxk1 ≤ kxk2 ≤ nkxk1 kxk1 ≤ kxk1 ≤ nkxk1: For our purposes, the term \equivalent" is a misnomer: the proportionality constants α; β strongly depend on the (often enormous) dimension of the vector space E. Hence measuring quantities in different norms can yield strikingly different conclusions. Consider a linear map A: E ! Y, and norms k · ka on E and k · kb on Y. We define the induced matrix norm kAka;b := max kAxkb: x: kxka≤1 4 CHAPTER 1. REVIEW OF FUNDAMENTALS The reader should verify the inequality kAxkb ≤ kAka;bkxka: In particular, if k · ka and k · kb are the norms induced by the inner products in E and Y, then the corresponding matrix norm is called the operator norm of A and will be denoted simply by kAk. In the case E = Y and a = b, we simply use the notation kAka for the induced norm. n m Exercise 1.3. Equip R and R with the lp-norms. Then for any matrix A 2 Rm×n, show the equalities kAk1 = max kA•jk1 j=1;:::;n kAk1 = max kAi•k1 i=1;:::;n where A•j and Ai• denote the j'th column and i'th row of A, respectively. 1.3 Eigenvalue and singular value decompositions of matrices The symbol Sn will denote the set of n × n real symmetric matrices, while O(n) will denote the set of n × n real orthogonal matrices { those satisfying XT X = XXT = I. Any symmetric matrix A 2 Sn admits an eigenvalue decomposition, meaning a factorization of the form A = UΛU T with U 2 O(n) and Λ 2 Sn a diagonal matrix. The diagonal elements of Λ are precisely the eigenvalues of A and the columns of U are corresponding eigenvectors. More generally, any matrix A 2 Rm×n admits a singular value decom- position, meaning a factorization of the form A = UDV T , where U 2 O(m) and V 2 O(n) are orthogonal matrices and D 2 Rm×n is a diagonal matrix with nonnegative diagonal entries. The diagonal elements of D are uniquely defined and are called the singular values of A. Supposing without loss of generality m ≤ n, the singular values of A are precisely the square roots of the eigenvalues of AAT .
Recommended publications
  • 1 Overview 2 Elements of Convex Analysis

    1 Overview 2 Elements of Convex Analysis

    AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 2 | Wednesday, January 27th 1 Overview In our previous lecture we discussed several applications of optimization, introduced basic terminol- ogy, and proved Weierstass' theorem (circa 1830) which gives a sufficient condition for existence of an optimal solution to an optimization problem. Today we will discuss basic definitions and prop- erties of convex sets and convex functions. We will then prove the separating hyperplane theorem and see an application of separating hyperplanes in machine learning. We'll conclude by describing the seminal perceptron algorithm (1957) which is designed to find separating hyperplanes. 2 Elements of Convex Analysis We will primarily consider optimization problems over convex sets { sets for which any two points are connected by a line. We illustrate some convex and non-convex sets in Figure 1. Definition. A set S is called a convex set if any two points in S contain their line, i.e. for any x1; x2 2 S we have that λx1 + (1 − λ)x2 2 S for any λ 2 [0; 1]. In the previous lecture we saw linear regression an example of an optimization problem, and men- tioned that the RSS function has a convex shape. We can now define this concept formally. n Definition. For a convex set S ⊆ R , we say that a function f : S ! R is: • convex on S if for any two points x1; x2 2 S and any λ 2 [0; 1] we have that: f (λx1 + (1 − λ)x2) ≤ λf(x1) + 1 − λ f(x2): • strictly convex on S if for any two points x1; x2 2 S and any λ 2 [0; 1] we have that: f (λx1 + (1 − λ)x2) < λf(x1) + (1 − λ) f(x2): We illustrate a convex function in Figure 2.
  • CORE View Metadata, Citation and Similar Papers at Core.Ac.Uk

    CORE View Metadata, Citation and Similar Papers at Core.Ac.Uk

    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Bulgarian Digital Mathematics Library at IMI-BAS Serdica Math. J. 27 (2001), 203-218 FIRST ORDER CHARACTERIZATIONS OF PSEUDOCONVEX FUNCTIONS Vsevolod Ivanov Ivanov Communicated by A. L. Dontchev Abstract. First order characterizations of pseudoconvex functions are investigated in terms of generalized directional derivatives. A connection with the invexity is analysed. Well-known first order characterizations of the solution sets of pseudolinear programs are generalized to the case of pseudoconvex programs. The concepts of pseudoconvexity and invexity do not depend on a single definition of the generalized directional derivative. 1. Introduction. Three characterizations of pseudoconvex functions are considered in this paper. The first is new. It is well-known that each pseudo- convex function is invex. Then the following question arises: what is the type of 2000 Mathematics Subject Classification: 26B25, 90C26, 26E15. Key words: Generalized convexity, nonsmooth function, generalized directional derivative, pseudoconvex function, quasiconvex function, invex function, nonsmooth optimization, solution sets, pseudomonotone generalized directional derivative. 204 Vsevolod Ivanov Ivanov the function η from the definition of invexity, when the invex function is pseudo- convex. This question is considered in Section 3, and a first order necessary and sufficient condition for pseudoconvexity of a function is given there. It is shown that the class of strongly pseudoconvex functions, considered by Weir [25], coin- cides with pseudoconvex ones. The main result of Section 3 is applied to characterize the solution set of a nonlinear programming problem in Section 4. The base results of Jeyakumar and Yang in the paper [13] are generalized there to the case, when the function is pseudoconvex.
  • Applications of Convex Analysis Within Mathematics

    Applications of Convex Analysis Within Mathematics

    Noname manuscript No. (will be inserted by the editor) Applications of Convex Analysis within Mathematics Francisco J. Arag´onArtacho · Jonathan M. Borwein · Victoria Mart´ın-M´arquez · Liangjin Yao July 19, 2013 Abstract In this paper, we study convex analysis and its theoretical applications. We first apply important tools of convex analysis to Optimization and to Analysis. We then show various deep applications of convex analysis and especially infimal convolution in Monotone Operator Theory. Among other things, we recapture the Minty surjectivity theorem in Hilbert space, and present a new proof of the sum theorem in reflexive spaces. More technically, we also discuss autoconjugate representers for maximally monotone operators. Finally, we consider various other applications in mathematical analysis. Keywords Adjoint · Asplund averaging · autoconjugate representer · Banach limit · Chebyshev set · convex functions · Fenchel duality · Fenchel conjugate · Fitzpatrick function · Hahn{Banach extension theorem · infimal convolution · linear relation · Minty surjectivity theorem · maximally monotone operator · monotone operator · Moreau's decomposition · Moreau envelope · Moreau's max formula · Moreau{Rockafellar duality · normal cone operator · renorming · resolvent · Sandwich theorem · subdifferential operator · sum theorem · Yosida approximation Mathematics Subject Classification (2000) Primary 47N10 · 90C25; Secondary 47H05 · 47A06 · 47B65 Francisco J. Arag´onArtacho Centre for Computer Assisted Research Mathematics and its Applications (CARMA),
  • Convex) Level Sets Integration

    Convex) Level Sets Integration

    Journal of Optimization Theory and Applications manuscript No. (will be inserted by the editor) (Convex) Level Sets Integration Jean-Pierre Crouzeix · Andrew Eberhard · Daniel Ralph Dedicated to Vladimir Demjanov Received: date / Accepted: date Abstract The paper addresses the problem of recovering a pseudoconvex function from the normal cones to its level sets that we call the convex level sets integration problem. An important application is the revealed preference problem. Our main result can be described as integrating a maximally cycli- cally pseudoconvex multivalued map that sends vectors or \bundles" of a Eu- clidean space to convex sets in that space. That is, we are seeking a pseudo convex (real) function such that the normal cone at each boundary point of each of its lower level sets contains the set value of the multivalued map at the same point. This raises the question of uniqueness of that function up to rescaling. Even after normalising the function long an orienting direction, we give a counterexample to its uniqueness. We are, however, able to show uniqueness under a condition motivated by the classical theory of ordinary differential equations. Keywords Convexity and Pseudoconvexity · Monotonicity and Pseudomono- tonicity · Maximality · Revealed Preferences. Mathematics Subject Classification (2000) 26B25 · 91B42 · 91B16 Jean-Pierre Crouzeix, Corresponding Author, LIMOS, Campus Scientifique des C´ezeaux,Universit´eBlaise Pascal, 63170 Aubi`ere,France, E-mail: [email protected] Andrew Eberhard School of Mathematical & Geospatial Sciences, RMIT University, Melbourne, VIC., Aus- tralia, E-mail: [email protected] Daniel Ralph University of Cambridge, Judge Business School, UK, E-mail: [email protected] 2 Jean-Pierre Crouzeix et al.
  • Ce Document Est Le Fruit D'un Long Travail Approuvé Par Le Jury De Soutenance Et Mis À Disposition De L'ensemble De La Communauté Universitaire Élargie

    Ce Document Est Le Fruit D'un Long Travail Approuvé Par Le Jury De Soutenance Et Mis À Disposition De L'ensemble De La Communauté Universitaire Élargie

    AVERTISSEMENT Ce document est le fruit d'un long travail approuvé par le jury de soutenance et mis à disposition de l'ensemble de la communauté universitaire élargie. Il est soumis à la propriété intellectuelle de l'auteur. Ceci implique une obligation de citation et de référencement lors de l’utilisation de ce document. D'autre part, toute contrefaçon, plagiat, reproduction illicite encourt une poursuite pénale. Contact : [email protected] LIENS Code de la Propriété Intellectuelle. articles L 122. 4 Code de la Propriété Intellectuelle. articles L 335.2- L 335.10 http://www.cfcopies.com/V2/leg/leg_droi.php http://www.culture.gouv.fr/culture/infos-pratiques/droits/protection.htm THESE` pr´esent´eepar NGUYEN Manh Cuong en vue de l'obtention du grade de DOCTEUR DE L'UNIVERSITE´ DE LORRAINE (arr^et´eminist´eriel du 7 Ao^ut 2006) Sp´ecialit´e: INFORMATIQUE LA PROGRAMMATION DC ET DCA POUR CERTAINES CLASSES DE PROBLEMES` EN APPRENTISSAGE ET FOUILLE DE DONEES.´ Soutenue le 19 mai 2014 devant le jury compos´ede Rapporteur BENNANI Youn`es Professeur, Universit´eParis 13 Rapporteur RAKOTOMAMONJY Alain Professeur, Universit´ede Rouen Examinateur GUERMEUR Yann Directeur de recherche, LORIA-Nancy Examinateur HEIN Matthias Professeur, Universit´eSaarland Examinateur PHAM DINH Tao Professeur ´em´erite, INSA-Rouen Directrice de th`ese LE THI Hoai An Professeur, Universit´ede Lorraine Co-encadrant CONAN-GUEZ Brieuc MCF, Universit´ede Lorraine These` prepar´ ee´ a` l'Universite´ de Lorraine, Metz, France au sein de laboratoire LITA Remerciements Cette th`ese a ´et´epr´epar´ee au sein du Laboratoire d'Informatique Th´eorique et Appliqu´ee (LITA) de l'Universit´ede Lorraine - France, sous la co-direction du Madame le Professeur LE THI Hoai An, Directrice du laboratoire Laboratoire d'Informatique Th´eorique et Ap- pliqu´ee(LITA), Universit´ede Lorraine et Ma^ıtre de conf´erence CONAN-GUEZ Brieuc, Institut Universitaire de Technologie (IUT), Universit´ede Lorraine, Metz.
  • Lipschitz Continuity of Convex Functions

    Lipschitz Continuity of Convex Functions

    Lipschitz Continuity of Convex Functions Bao Tran Nguyen∗ Pham Duy Khanh† November 13, 2019 Abstract We provide some necessary and sufficient conditions for a proper lower semi- continuous convex function, defined on a real Banach space, to be locally or globally Lipschitz continuous. Our criteria rely on the existence of a bounded selection of the subdifferential mapping and the intersections of the subdifferential mapping and the normal cone operator to the domain of the given function. Moreover, we also point out that the Lipschitz continuity of the given function on an open and bounded (not necessarily convex) set can be characterized via the existence of a bounded selection of the subdifferential mapping on the boundary of the given set and as a consequence it is equivalent to the local Lipschitz continuity at every point on the boundary of that set. Our results are applied to extend a Lipschitz and convex function to the whole space and to study the Lipschitz continuity of its Moreau envelope functions. Keywords Convex function, Lipschitz continuity, Calmness, Subdifferential, Normal cone, Moreau envelope function. Mathematics Subject Classification (2010) 26A16, 46N10, 52A41 1 Introduction arXiv:1911.04886v1 [math.FA] 12 Nov 2019 Lipschitz continuous and convex functions play a significant role in convex and nonsmooth analysis. It is well-known that if the domain of a proper lower semicontinuous convex function defined on a real Banach space has a nonempty interior then the function is continuous over the interior of its domain [3, Proposition 2.111] and as a consequence, it is subdifferentiable (its subdifferential is a nonempty set) and locally Lipschitz continuous at every point in the interior of its domain [3, Proposition 2.107].
  • On the Ekeland Variational Principle with Applications and Detours

    On the Ekeland Variational Principle with Applications and Detours

    Lectures on The Ekeland Variational Principle with Applications and Detours By D. G. De Figueiredo Tata Institute of Fundamental Research, Bombay 1989 Author D. G. De Figueiredo Departmento de Mathematica Universidade de Brasilia 70.910 – Brasilia-DF BRAZIL c Tata Institute of Fundamental Research, 1989 ISBN 3-540- 51179-2-Springer-Verlag, Berlin, Heidelberg. New York. Tokyo ISBN 0-387- 51179-2-Springer-Verlag, New York. Heidelberg. Berlin. Tokyo No part of this book may be reproduced in any form by print, microfilm or any other means with- out written permission from the Tata Institute of Fundamental Research, Colaba, Bombay 400 005 Printed by INSDOC Regional Centre, Indian Institute of Science Campus, Bangalore 560012 and published by H. Goetze, Springer-Verlag, Heidelberg, West Germany PRINTED IN INDIA Preface Since its appearance in 1972 the variational principle of Ekeland has found many applications in different fields in Analysis. The best refer- ences for those are by Ekeland himself: his survey article [23] and his book with J.-P. Aubin [2]. Not all material presented here appears in those places. Some are scattered around and there lies my motivation in writing these notes. Since they are intended to students I included a lot of related material. Those are the detours. A chapter on Nemyt- skii mappings may sound strange. However I believe it is useful, since their properties so often used are seldom proved. We always say to the students: go and look in Krasnoselskii or Vainberg! I think some of the proofs presented here are more straightforward. There are two chapters on applications to PDE.
  • NOTE on the CONTINUITY of M-CONVEX and L-CONVEX FUNCTIONS in CONTINUOUS VARIABLES 1. Introduction Two Kinds of Convexity Concept

    NOTE on the CONTINUITY of M-CONVEX and L-CONVEX FUNCTIONS in CONTINUOUS VARIABLES 1. Introduction Two Kinds of Convexity Concept

    Journal of the Operations Research Society of Japan 2008, Vol. 51, No. 4, 265-273 NOTE ON THE CONTINUITY OF M-CONVEX AND L-CONVEX FUNCTIONS IN CONTINUOUS VARIABLES Kazuo Murota Akiyoshi Shioura University of Tokyo Tohoku University (Received March 27, 2008) Abstract M-convex and L-convex functions in continuous variables constitute subclasses of convex func- tions with nice combinatorial properties. In this note we give proofs of the fundamental facts that closed proper M-convex and L-convex functions are continuous on their effective domains. Keywords: Combinatorial optimization, convex function, continuity, submodular func- tion, matroid. 1. Introduction Two kinds of convexity concepts, called M-convexity and L-convexity, play primary roles in the theory of discrete convex analysis [6]. They are originally introduced for functions in integer variables by Murota [4, 5], and then for functions in continuous variables by Murota{ Shioura [8, 10]. M-convex and L-convex functions in continuous variables constitute subclasses of convex functions with additional combinatorial properties such as submodularity and diagonal dom- inance (see, e.g., [6{11]). Fundamental properties of M-convex and L-convex functions are investigated in [9], such as equivalent axioms, subgradients, directional derivatives, etc. Con- jugacy relationship between M-convex and L-convex functions under the Legendre-Fenchel transformation is shown in [10]. Subclasses of M-convex and L-convex functions are investi- gated in [8] (polyhedral M-convex and L-convex functions) and in [11] (quadratic M-convex and L-convex functions). As variants of M-convex and L-convex functions, the concepts of M\-convex and L\-convex functions are also introduced by Murota{Shioura [8, 10], where \M\" and \L\" should be read \M-natural" and \L-natural," respectively.
  • Optimization Over Nonnegative and Convex Polynomials with and Without Semidefinite Programming

    Optimization Over Nonnegative and Convex Polynomials with and Without Semidefinite Programming

    OPTIMIZATION OVER NONNEGATIVE AND CONVEX POLYNOMIALS WITH AND WITHOUT SEMIDEFINITE PROGRAMMING GEORGINA HALL ADISSERTATION PRESENTED TO THE FACULTY OF PRINCETON UNIVERSITY IN CANDIDACY FOR THE DEGREE OF DOCTOR OF PHILOSOPHY RECOMMENDED FOR ACCEPTANCE BY THE DEPARTMENT OF OPERATIONS RESEARCH AND FINANCIAL ENGINEERING ADVISER:PROFESSOR AMIR ALI AHMADI JUNE 2018 c Copyright by Georgina Hall, 2018. All rights reserved. Abstract The problem of optimizing over the cone of nonnegative polynomials is a fundamental problem in computational mathematics, with applications to polynomial optimization, con- trol, machine learning, game theory, and combinatorics, among others. A number of break- through papers in the early 2000s showed that this problem, long thought to be out of reach, could be tackled by using sum of squares programming. This technique however has proved to be expensive for large-scale problems, as it involves solving large semidefinite programs (SDPs). In the first part of this thesis, we present two methods for approximately solving large- scale sum of squares programs that dispense altogether with semidefinite programming and only involve solving a sequence of linear or second order cone programs generated in an adaptive fashion. We then focus on the problem of finding tight lower bounds on polyno- mial optimization problems (POPs), a fundamental task in this area that is most commonly handled through the use of SDP-based sum of squares hierarchies (e.g., due to Lasserre and Parrilo). In contrast to previous approaches, we provide the first theoretical framework for constructing converging hierarchies of lower bounds on POPs whose computation simply requires the ability to multiply certain fixed polynomials together and to check nonnegativ- ity of the coefficients of their product.
  • Sum of Squares and Polynomial Convexity

    Sum of Squares and Polynomial Convexity

    CONFIDENTIAL. Limited circulation. For review only. Sum of Squares and Polynomial Convexity Amir Ali Ahmadi and Pablo A. Parrilo Abstract— The notion of sos-convexity has recently been semidefiniteness of the Hessian matrix is replaced with proposed as a tractable sufficient condition for convexity of the existence of an appropriately defined sum of squares polynomials based on sum of squares decomposition. A multi- decomposition. As we will briefly review in this paper, by variate polynomial p(x) = p(x1; : : : ; xn) is said to be sos-convex if its Hessian H(x) can be factored as H(x) = M T (x) M (x) drawing some appealing connections between real algebra with a possibly nonsquare polynomial matrix M(x). It turns and numerical optimization, the latter problem can be re- out that one can reduce the problem of deciding sos-convexity duced to the feasibility of a semidefinite program. of a polynomial to the feasibility of a semidefinite program, Despite the relative recency of the concept of sos- which can be checked efficiently. Motivated by this computa- convexity, it has already appeared in a number of theo- tional tractability, it has been speculated whether every convex polynomial must necessarily be sos-convex. In this paper, we retical and practical settings. In [6], Helton and Nie use answer this question in the negative by presenting an explicit sos-convexity to give sufficient conditions for semidefinite example of a trivariate homogeneous polynomial of degree eight representability of semialgebraic sets. In [7], Lasserre uses that is convex but not sos-convex. sos-convexity to extend Jensen’s inequality in convex anal- ysis to linear functionals that are not necessarily probability I.
  • Subdifferential Properties of Quasiconvex and Pseudoconvex Functions: Unified Approach1

    Subdifferential Properties of Quasiconvex and Pseudoconvex Functions: Unified Approach1

    JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS: Vol. 97, No. I, pp. 29-45, APRIL 1998 Subdifferential Properties of Quasiconvex and Pseudoconvex Functions: Unified Approach1 D. AUSSEL2 Communicated by M. Avriel Abstract. In this paper, we are mainly concerned with the characteriza- tion of quasiconvex or pseudoconvex nondifferentiable functions and the relationship between those two concepts. In particular, we charac- terize the quasiconvexity and pseudoconvexity of a function by mixed properties combining properties of the function and properties of its subdifferential. We also prove that a lower semicontinuous and radially continuous function is pseudoconvex if it is quasiconvex and satisfies the following optimality condition: 0sdf(x) =f has a global minimum at x. The results are proved using the abstract subdifferential introduced in Ref. 1, a concept which allows one to recover almost all the subdiffer- entials used in nonsmooth analysis. Key Words. Nonsmooth analysis, abstract subdifferential, quasicon- vexity, pseudoconvexity, mixed property. 1. Preliminaries Throughout this paper, we use the concept of abstract subdifferential introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth analysis. The aim of such an approach is to show that a lot of results concerning the subdifferential prop- erties of quasiconvex and pseudoconvex functions can be proved in a unified way, and hence for a large class of subdifferentials. We are interested here in two aspects of convex analysis: the charac- terization of quasiconvex and pseudoconvex lower semicontinuous functions and the relationship between these two concepts. 1The author is indebted to Prof. M. Lassonde for many helpful discussions leading to a signifi- cant improvement in the presentation of the results.
  • Chapter 5 Convex Optimization in Function Space 5.1 Foundations of Convex Analysis

    Chapter 5 Convex Optimization in Function Space 5.1 Foundations of Convex Analysis

    Chapter 5 Convex Optimization in Function Space 5.1 Foundations of Convex Analysis Let V be a vector space over lR and k ¢ k : V ! lR be a norm on V . We recall that (V; k ¢ k) is called a Banach space, if it is complete, i.e., if any Cauchy sequence fvkglN of elements vk 2 V; k 2 lN; converges to an element v 2 V (kvk ¡ vk ! 0 as k ! 1). Examples: Let ­ be a domain in lRd; d 2 lN. Then, the space C(­) of continuous functions on ­ is a Banach space with the norm kukC(­) := sup ju(x)j : x2­ The spaces Lp(­); 1 · p < 1; of (in the Lebesgue sense) p-integrable functions are Banach spaces with the norms Z ³ ´1=p p kukLp(­) := ju(x)j dx : ­ The space L1(­) of essentially bounded functions on ­ is a Banach space with the norm kukL1(­) := ess sup ju(x)j : x2­ The (topologically and algebraically) dual space V ¤ is the space of all bounded linear functionals ¹ : V ! lR. Given ¹ 2 V ¤, for ¹(v) we often write h¹; vi with h¢; ¢i denoting the dual product between V ¤ and V . We note that V ¤ is a Banach space equipped with the norm j h¹; vi j k¹k := sup : v2V nf0g kvk Examples: The dual of C(­) is the space M(­) of Radon measures ¹ with Z h¹; vi := v d¹ ; v 2 C(­) : ­ The dual of L1(­) is the space L1(­). The dual of Lp(­); 1 < p < 1; is the space Lq(­) with q being conjugate to p, i.e., 1=p + 1=q = 1.