DOCSLIB.ORG

Min Common/Max Crossing Duality: a Geometric View of Conjugacy

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Min Common/Max Crossing Duality: a Geometric View of Conjugacy August 2008 LIDS-P-2796 Revised Jan. 2009 Min Common/Max Crossing Duality: A Geometric View of Conjugacy in Convex Optimization1 Dimitri P. Bertsekas2 Abstract We provide a unifying framework for the visualization and analysis of duality, and other issues in convex optimization. It is based on two simple optimization problems that are dual to each other: the min common point problem and the max crossing point problem. Within the insightful geometry of these problems, several of the core issues in convex analysis become apparent and can be analyzed in a unified way. These issues are related to conditions for strong duality in constrained optimization and zero sum games, existence of dual optimal solutions and saddle points, existence of subgradients, and theorems of the alternative. The generality and power of our framework is due to its close connection with the Legendre/Fenchel conjugacy framework. However, the two frameworks offer complementary starting points for analysis and provide alternative views of the geometric foundation of duality: conjugacy emphasizes functional/algebraic descriptions, while min common/max crossing emphasizes set/epigraph descriptions. The min common/max crossing framework is simpler, and seems better suited for visualizing and investigating questions of strong duality and existence of dual optimal solutions. The conjugacy framework, with its emphasis on functional descriptions, is more suitable when mathematical operations on convex functions are involved, and the calculus of conjugate functions can be brought to bear for analysis or computation. 1 Supported by NSF Grant ECCS-0801549. The MC/MC framework was initially developed in joint research with A. Nedic, and A. Ozdaglar. This research is described in the book by Bertsekas, Nedic, and Ozdaglar [BNO03]. The present account is an improved and more comprehensive development. In particular, it contains some more streamlined proofs and some new results, particularly in connection with minimax problems and separable problems. 2 Dimitri Bertsekas is with the Dept. of Electr. Engineering and Comp. Science, M.I.T., Cambridge, Mass., 02139. 1 Contents 1. Introduction . p. 3 2. General Results and Some Special Cases . p. 7 Connection to Conjugate Convex Functions . p. 8 General Optimization Duality . p. 9 Optimization with Inequality Constraints . p. 9 Fenchel Duality Framework . p. 10 Minimax Problems . p. 11 3. Strong Duality Theorems . p. 16 Conditions for Strong Duality . p. 16 Existence of Dual Optimal Solutions . p. 18 Special Cases Involving Convexity and/or Compactness . p. 21 MC/MC and Polyhedral Convexity . p. 23 4. Applications . p. 28 Minimax Theorems . p. 28 Saddle Point Theorems . p. 33 A Nonlinear Version of Farkas' Lemma . p. 39 Convex Programming Duality . p. 43 Theorems of the Alternative . p. 46 Subdifferential Theory . p. 49 5. Nonconvex Problems and Estimates of the Duality Gap . p. 51 Separable Optimization Problems and their Geometry . p. 51 Estimates of Duality Gap in Separable Problems . p. 55 Estimates of Duality Gap in Minimax . p. 58 6. References . p. 59 7. Appendix: Convex Analysis Background . p. 60 Relative Interior and Closure . p. 60 Recession Cones . p. 61 Partial Minimization of Convex Functions . p. 62 Closures of Convex Functions . p. 64 Hyperplane Theorems . p. 67 Conjugate Functions . p. 68 Minimax Theory . p. 69 2 1. Introduction 1. INTRODUCTION Duality in optimization is often viewed as a manifestation of a fundamental dual/conjugate description of a closed convex set as the intersection of all closed halfspaces containing the set. When specialized to the epigraph of a function f : n [ ; ], this description leads to the formalism of the conjugate convex < 7! −∞ 1 function of f, which is defined by h(y) = sup x0y f(x) ; (1.1) x n − 2< and permeates much of convex optimization theory. In this paper, we focus on a framework referred to as min common/max crossing (MC/MC for short). It is related to the conjugacy framework, but does not involve an algebraic definition such as Eq. (1.1), and it is structured to emphasize optimization duality aspects. For this reason it is simpler, and seems better suited for geometric visualization and analysis in many important convex optimization contexts. Our framework aims to capture the most essential optimization-related features of the preceding conjugate description of closed convex sets in two simple geometrical problems, defined by a nonempty subset M of n+1. < (a) Min Common Point Problem: Consider all vectors that are common to M and the (n + 1)st axis. We want to find one whose (n + 1)st component is minimum. (b) Max Crossing Point Problem: Consider nonvertical hyperplanes that contain M in their corresponding \upper" closed halfspace, i.e., the closed halfspace whose recession cone contains the vertical halfline (0; w) w 0 (see Fig. 1.1). We want to find the maximum crossing point of the (n + 1)st axis with j ≥ such a hyperplane. Figure 1.1 suggests that the optimal value of the max crossing problem is no larger than the optimal value of the min common problem, and that under favorable circumstances the two optimal values are equal. Our purpose in this paper is to formalize the analysis of the two problems, to provide conditions that guarantee equality of their optimal values and the existence of their optimal solutions, and to show that they can be used to develop much of the core theory of convex analysis and optimization in a unified way. Mathematically, the min common problem is minimize w subject to (0; w) M: 2 Its optimal value is denoted by w∗, i.e., w∗ = inf w: (0;w) M 2 3 ! Negative Halfspace {x | a!x ≥ b} Negative Halfspace {x | a x ≥ b} ! Negative Halfspace {x | a!x ≥ b} Neg! ative HalfsPpaocseiti{vxe |Haaxlfs≥pabc}e {x | a!x ≤ b} Positive Halfspace {x | a x ≤ b} ! Positive Halfspace {x | a!x ≤ b} Positive Halfspace {x | a x ≤ b} ⊥ aff(C) C C ∩ S⊥ d z x aff(C) C C ∩ S d z x ⊥ ! ⊥ aff(C) CNegCat∩ivSe Hadlfspzacxe {x | a x ≥affb(}C) C C ∩! S d z x ! ! ! ! ! !Hyp! erplane {x | a x = b} = {x | a x = a x} Negative HalHfsyppaecrep{laxn|eaP{oxsi≥|taivbex} H=ablf}sp=ac{ex{|xa|xa=x a≤! xb}} ! ! ! ! ! ! ! NHeygpa! etrivpelaHneal{fsxp|acaex{=x |ba} x=≥{xb}| a x = a x} Negative HalHfsyPppaoecsrietpi{lvaxen|Heaa{lxfs≥|pabcx}e ={xb|}a=x{≤x |b}a x = a x} ! ! !! Negative Hal∗fsPpaocsieti{vxe |Ha∗alxfs≥pabc}e {x | a x ≤ b} ∗ ∗ NPeogsaittiivvee HHaallffssppaaccee {{xx||aaxx≤≥bb}} x x f!αx! +! (1 − α)x" ⊥ x x f!αx + (1 − α)x" ! ∗ PNoseitgiavteivHe∗aHlfasplfaspceac{ex{|xaaff|x(aC≤x) ≥b}Cb}C∗ ∩ S d z∗ x Positive Halfspace {x | a x ≤ b}x x f αx + (1⊥− α)x ! x x f!αx + (1 − α)x" aff(PCo)sitC!iveCHa∩lfSspacde {zx"|xa x ≤ b} ⊥ ⊥ Hyperplane {x | a!x = b} = a{ffx(|Ca)!xC= aC!x∩} S d z x aff(C) !C C ∩! S d z x ! ! ∗ ⊥ ∗ Hyperplane {x | a x ⊥= b} = {x | a x =aaffx(Cx}) xC! C ∩ S d z x x x Negative Halfspacaeff{(xC|)aCx ≥Cb∩} S Nedgaztivxe Halfspace {x | a x ≥ b!} ⊥ ∗! ! ! ! ! ! ∗ Hyperapffl(aCne) {Cx | Ca x∩=S b} d= {zxxx| xa x = a x} PHoyspiteirvpelaHnaelf{sxpa|cae x{x=|ba}x=≤{xb}| a xP=osaitxivx}e Hxalfspac! e {xx|∗a!xx≤ fb}α! x∗ +!(1 − α)x ! x0 − d! Hx1ypxe2r! pxlaxn4e−{xd |xa5 −x =d db} = {x0| −a! xd =x1axx2}x x4 −"d x5 − d d Hyperplane {x | a x = b} = {x | a x∗ = a x} ∗ ! ! ! x Hxypefrp!lαaxne +{x(01|−a−xdα=x)x1"bx}2=x{xx4|−∗adxx=5 a−xd}d ∗ x0 − d ∗x1 x2 x x4 −∗ d x⊥5 − d d x x f!αx + (1 − α)x" axff(Cx) Cf!αCx∩+S (1 −d αz)xx" ∗aff(C) C C∗ ∩ S⊥ d z x ∗ ∗ x x f!αx + (1 ∗− α)x" x x f!αx + (1 − α)x" ∗ ∗x x ! ! ! !∗ !!x x f!αx +! (1 −!α)x" Hyperplane {x | a x = b} = N{xeg|aatixve=Haalxfs}pxacex{x | a x ≥ b} ∗ Negative HalfHspyap!ceerp{lxan| ea {xx≥| ab}x = b} = {x | a x = a x} !∗ x0 − d x!1 x2 x! x0 4 −1 d2x53− d d x x xˆ0 xˆ1 xˆ2 xˆ3 NegatNiveegHataivlfesPpHaoacsleiftsi{Ppvxxeaocs|Hieaxtai{vxlxfes≥p|Haabac}lxefs{≥pxabc|}ea{xx≤| ab}xˆ ≤xˆb}xˆ xˆ ∗ x0 − ∗d x1 x2 x x!4∗!−∗ d x!5xˆ!−0 xˆd1dxˆ2 xˆ3 x !x !xˆ0 xˆ1 xˆ2 xˆ3 NePgaotsiivteiPvNHeoseaHigltafxisvltpfeisvapHecaxeaHcl{efaxsf{lpf!x|saxαpac|axeaxcx{ex≥+x{≤x|(b1}ab|−}xa αx≤N)e≥xbg}"abt}ivexN0He−aglafdstp∗ixva1ecexH2{axlxfs|xpa4acx−∗e≥∗d{xbx}|5 a−xd≥d b} 0 1 2 4 5 x x f!αxxx+ (1 − α)x" x − d x x x x − d x − d d! ! ⊥ ⊥ ! ! Positive HPaolsfsitpivaceeH{axlf|spaaxce≤{xxb0}|−a dxaPx≤ffo1(sbCixt}2i)avffxeC(HPxC4oa)Csl−fistC∩pidvaSe!xcCe5H∩−{adxlSfds|pzdaacxde≤{zxb}|xa x ≤ b} x0 − d x1 x2 x x4 − d x5 − d d Negative Halfspac⊥e {x | a x ≥ b} aff(C)affC(Cx)0C−C∩dSCx⊥1∩xdS2 zx xxd4a−z0 dax1xa52−ad3 d a0 a1 a2 a3 ∗ ! ! !! !! ! ∗ HyperHpxylapPnxeoersp{iltxaivn|eeaH0{⊥xxal=1|fsapb2}axc==e3⊥{{bxx} ||=a{xx≤=| aba}xx=}xˆ0axˆx1}xˆ2 xˆa3⊥0 a1 a2 a⊥3 aff(C) C! C ∩ aS a daz!ax ! x x Hyperplanaeff!{(xC|) aCxaff=C(Cb∩})S=xCˆ!{0xxdˆC|1a∩zxˆ!x2Sxx=ˆ3adx}z xaff(C) CaffC(C∩) SC Cd ∩z Sx d z x Hyperplane {x | a x = b} = {x | a x = a x} 0 1 2 3 x0 − d x1 x2 x x4 − d x5 − d d ∗ ! ∗ ⊥ xˆ xˆ xˆ xˆ Hyperplane {x | a!x =xˆ0Nbx}ˆe1g=axˆ!t{2ivxxˆex|30Ha−!axlfd=s∗pxa1c!xex}2{!xxaffx|xa(4Cx∗−f!)!≥dαCxb!}5C+−∩(d1Sd−! αd)x"z !x ! ! ! HyperplaHneyp{exrp| laanxe={xb}| =a x{x=|ba}x=xH!=y{pxaex|xrpa}lxfaHn!=αye!px{aexrx+p|}la(1nxexˆ−=0{αxˆb)1}|xax="ˆ2x{xˆx=3 |ba} x=={xa|xa}x = a x} xˆ0∗ xˆ1 xˆ∗2 xˆ3 ∗ ∗ Negative HxaPlfsopsxiaxtcivefe{xHxαxa|lafs+!pxαa(≥xc1e−b+{}xα()1|xa−xα≤)xb"} 0 1 2 3 ! ! ! " xˆ xˆ!a0 xˆa1 xaˆ! 2 a3 f(z) Positive H∗ alfspaHcey∗p{exr∗p| laanxe≤{xb∗}| a x = b} f=(z{∗)x | a x =∗a x∗} ∗ x x fxαx x+ a(f10 −αa1xαa)+2x a(31 − α)x x ∗ x fx αxx+f(1αf−x(zα))+x(1 − α)x ! f! (z) " ∗ x"x ⊥ ! a0 !a1 a2 a"3 " z a0 a1 a2 a3 a∗ff(Cx) Cx Cz∩ S d z x xˆ0 xˆ1 xˆ2 xˆ3 ∗ x x ∗⊥ a0 ∗a1 a2 a3 z a0 a1 a2aaff3(xC)x C zC ∩ Sx dx zf!xαxˆ0 +xˆ1(1xˆ2−xˆα3)x" x0 − d x1 x2 x x4 !− d x5 − d d !a0 a1 !a2 a3 ! x0 − dHxy1pexr2pxlxaxnxe∗4
Load more

Recommended publications
  • Lagrangian Duality in Convex Optimization
    Lagrangian Duality in Convex Optimization LI, Xing A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Master of Philosophy in Mathematics The Chinese University of Hong Kong July 2009 ^'^'^xLIBRARr SYSTEf^N^J Thesis/Assessment Committee Professor LEUNG, Chi Wai (Chair) Professor NG, Kung Fu (Thesis Supervisor) Professor LUK Hing Sun (Committee Member) Professor HUANG, Li Ren (External Examiner) Abstract of thesis entitled: Lagrangian Duality in Convex Optimization Submitted by: LI, Xing for the degree of Master of Philosophy in Mathematics at the Chinese University of Hong Kong in July, 2009. Abstract In convex optimization, a strong duality theory which states that the optimal values of the primal problem and the Lagrangian dual problem are equal and the dual problem attains its maximum plays an important role. With easily visualized physical meanings, the strong duality theories have also been wildly used in the study of economical phenomenon and operation research. In this thesis, we reserve the first chapter for an introduction on the basic principles and some preliminary results on convex analysis ； in the second chapter, we will study various regularity conditions that could lead to strong duality, from the classical ones to recent developments; in the third chapter, we will present stable Lagrangian duality results for cone-convex optimization problems under continuous linear perturbations of the objective function . In the last chapter, Lagrange multiplier conditions without constraint qualifications will be discussed. 摘要 拉格朗日對偶理論主要探討原問題與拉格朗日對偶問題的 最優值之間“零對偶間隙”成立的條件以及對偶問題存在 最優解的條件，其在解決凸規劃問題中扮演著重要角色， 並在經濟學運籌學領域有著廣泛的應用。 本文將系統地介紹凸錐規劃中的拉格朗日對偶理論，包括 基本規範條件，閉凸錐規範條件等，亦會涉及無規範條件 的序列拉格朗日乘子。 ACKNOWLEDGMENTS I wish to express my gratitude to my supervisor Professor Kung Fu Ng and also to Professor Li Ren Huang, Professor Chi Wai Leung, and Professor Hing Sun Luk for their guidance and valuable suggestions.
    [Show full text]
  • Duality Gap Estimation Via a Refined Shapley--Folkman Lemma | SIAM
    SIAM J. OPTIM. \bigcircc 2020 Society for Industrial and Applied Mathematics Vol. 30, No. 2, pp. 1094{1118 DUALITY GAP ESTIMATION VIA A REFINED SHAPLEY{FOLKMAN LEMMA \ast YINGJIE BIy AND AO TANG z Abstract. Based on concepts like the kth convex hull and finer characterization of noncon- vexity of a function, we propose a refinement of the Shapley{Folkman lemma and derive anew estimate for the duality gap of nonconvex optimization problems with separable objective functions. We apply our result to the network utility maximization problem in networking and the dynamic spectrum management problem in communication as examples to demonstrate that the new bound can be qualitatively tighter than the existing ones. The idea is also applicable to cases with general nonconvex constraints. Key words. nonconvex optimization, duality gap, convex relaxation, network resource alloca- tion AMS subject classifications. 90C26, 90C46 DOI. 10.1137/18M1174805 1. Introduction. The Shapley{Folkman lemma (Theorem 1.1) was stated and used to establish the existence of approximate equilibria in economy with nonconvex preferences [13]. It roughly says that the sum of a large number of sets is close to convex and thus can be used to generalize results on convex objects to nonconvex ones. n m P Theorem 1.1. Let S1;S2;:::;Sn be subsets of R . For each z 2 conv i=1 Si = Pn conv S , there exist points zi 2 conv S such that z = Pn zi and zi 2 S except i=1 i i i=1 i for at most m values of i. Remark 1.2.
    [Show full text]
  • Subdifferentiability and the Duality
    Subdifferentiability and the Duality Gap Neil E. Gretsky ([email protected]) ∗ Department of Mathematics, University of California, Riverside Joseph M. Ostroy ([email protected]) y Department of Economics, University of California, Los Angeles William R. Zame ([email protected]) z Department of Economics, University of California, Los Angeles Abstract. We point out a connection between sensitivity analysis and the funda- mental theorem of linear programming by characterizing when a linear programming problem has no duality gap. The main result is that the value function is subd- ifferentiable at the primal constraint if and only if there exists an optimal dual solution and there is no duality gap. To illustrate the subtlety of the condition, we extend Kretschmer's gap example to construct (as the value function of a linear programming problem) a convex function which is subdifferentiable at a point but is not continuous there. We also apply the theorem to the continuum version of the assignment model. Keywords: duality gap, value function, subdifferentiability, assignment model AMS codes: 90C48,46N10 1. Introduction The purpose of this note is to point out a connection between sensi- tivity analysis and the fundamental theorem of linear programming. The subject has received considerable attention and the connection we find is remarkably simple. In fact, our observation in the context of convex programming follows as an application of conjugate duality [11, Theorem 16]. Nevertheless, it is useful to give a separate proof since the conclusion is more readily established and its import for linear programming is more clearly seen. The main result (Theorem 1) is that in a linear programming prob- lem there exists an optimal dual solution and there is no duality gap if and only if the value function is subdifferentiable at the primal constraint.
    [Show full text]
  • Arxiv:2011.09194V1 [Math.OC]
    Noname manuscript No. (will be inserted by the editor) Lagrangian duality for nonconvex optimization problems with abstract convex functions Ewa M. Bednarczuk · Monika Syga Received: date / Accepted: date Abstract We investigate Lagrangian duality for nonconvex optimization prob- lems. To this aim we use the Φ-convexity theory and minimax theorem for Φ-convex functions. We provide conditions for zero duality gap and strong duality. Among the classes of functions, to which our duality results can be applied, are prox-bounded functions, DC functions, weakly convex functions and paraconvex functions. Keywords Abstract convexity · Minimax theorem · Lagrangian duality · Nonconvex optimization · Zero duality gap · Weak duality · Strong duality · Prox-regular functions · Paraconvex and weakly convex functions 1 Introduction Lagrangian and conjugate dualities have far reaching consequences for solution methods and theory in convex optimization in finite and infinite dimensional spaces. For recent state-of the-art of the topic of convex conjugate duality we refer the reader to the monograph by Radu Bot¸[5]. There exist numerous attempts to construct pairs of dual problems in non- convex optimization e.g., for DC functions [19], [34], for composite functions [8], DC and composite functions [30], [31] and for prox-bounded functions [15]. In the present paper we investigate Lagrange duality for general optimiza- tion problems within the framework of abstract convexity, namely, within the theory of Φ-convexity. The class Φ-convex functions encompasses convex l.s.c. Ewa M. Bednarczuk Systems Research Institute, Polish Academy of Sciences, Newelska 6, 01–447 Warsaw Warsaw University of Technology, Faculty of Mathematics and Information Science, ul.
    [Show full text]
  • Lagrangian Duality and Perturbational Duality I ∗
    Lagrangian duality and perturbational duality I ∗ Erik J. Balder Our approach to the Karush-Kuhn-Tucker theorem in [OSC] was entirely based on subdifferential calculus (essentially, it was an outgrowth of the two subdifferential calculus rules contained in the Fenchel-Moreau and Dubovitskii-Milyutin theorems, i.e., Theorems 2.9 and 2.17 of [OSC]). On the other hand, Proposition B.4(v) in [OSC] gives an intimate connection between the subdifferential of a function and the Fenchel conjugate of that function. In the present set of lecture notes this connection forms the central analytical tool by which one can study the connections between an optimization problem and its so-called dual optimization problem (such connections are commonly known as duality relations). We shall first study duality for the convex optimization problem that figured in our Karush-Kuhn-Tucker results. In this simple form such duality is known as Lagrangian duality. Next, in section 2 this is followed by a far-reaching extension of duality to abstract optimization problems, which leads to duality-stability relationships. Then, in section 3 we specialize duality to optimization problems with cone-type constraints, which includes Fenchel duality for semidefinite programming problems. 1 Lagrangian duality An interesting and useful interpretation of the KKT theorem can be obtained in terms of the so-called duality principle (or relationships) for convex optimization. Recall our standard convex minimization problem as we had it in [OSC]: (P ) inf ff(x): g1(x) ≤ 0; ··· ; gm(x) ≤ 0; Ax − b = 0g x2S n and recall that we allow the functions f; g1; : : : ; gm on R to have values in (−∞; +1].
    [Show full text]
  • Bounding the Duality Gap for Problems with Separable Objective
    Bounding the Duality Gap for Problems with Separable Objective Madeleine Udell and Stephen Boyd March 8, 2014 Abstract We consider the problem of minimizing a sum of non-convex func- tions over a compact domain, subject to linear inequality and equality constraints. We consider approximate solutions obtained by solving a convexified problem, in which each function in the objective is replaced by its convex envelope. We propose a randomized algorithm to solve the convexified problem which finds an -suboptimal solution to the original problem. With probability 1, is bounded by a term propor- tional to the number of constraints in the problem. The bound does not depend on the number of variables in the problem or the number of terms in the objective. In contrast to previous related work, our proof is constructive, self-contained, and gives a bound that is tight. 1 Problem and results The problem. We consider the optimization problem Pn minimize f(x) = i=1 fi(xi) subject to Ax ≤ b (P) Gx = h; N ni Pn with variable x = (x1; : : : ; xn) 2 R , where xi 2 R , with i=1 ni = N. m1×N There are m1 linear inequality constraints, so A 2 R , and m2 linear equality constraints, so G 2 Rm2×N . The optimal value of P is denoted p?. The objective function terms are lower semi-continuous on their domains: 1 ni fi : Si ! R, where Si ⊂ R is a compact set. We say that a point x is feasible (for P) if Ax ≤ b, Gx = h, and xi 2 Si, i = 1; : : : ; n.
    [Show full text]
  • Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
    Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex Hongyang Zhang Junru Shao Ruslan Salakhutdinov Carnegie Mellon University Carnegie Mellon University Carnegie Mellon University [email protected] [email protected] [email protected] Abstract Several recently proposed architectures of neural networks such as ResNeXt, Inception, Xception, SqueezeNet and Wide ResNet are based on the designing idea of having multiple branches and have demonstrated improved performance in many applications. We show that one cause for such success is due to the fact that the multi-branch architecture is less non-convex in terms of duality gap. The duality gap measures the degree of intrinsic non-convexity of an optimization problem: smaller gap in relative value implies lower degree of intrinsic non-convexity. The challenge is to quantitatively measure the duality gap of highly non-convex problems such as deep neural networks. In this work, we provide strong guarantees of this quantity for two classes of network architectures. For the neural networks with arbitrary activation functions, multi-branch architecture and a variant of hinge loss, we show that the duality gap of both population and empirical risks shrinks to zero as the number of branches increases. This result sheds light on better understanding the power of over-parametrization where increasing the network width tends to make the loss surface less non-convex. For the neural networks with linear activation function and `2 loss, we show that the duality gap of empirical risk is zero. Our two results work for arbitrary depths and adversarial data, while the analytical techniques might be of independent interest to non-convex optimization more broadly.
    [Show full text]
  • A Tutorial on Convex Optimization II: Duality and Interior Point Methods
    A Tutorial on Convex Optimization II: Duality and Interior Point Methods Haitham Hindi Palo Alto Research Center (PARC), Palo Alto, California 94304 email: [email protected] Abstract— In recent years, convex optimization has become a and concepts. For detailed examples and applications, the computational tool of central importance in engineering, thanks reader is refered to [8], [2], [6], [5], [7], [10], [12], [17], to its ability to solve very large, practical engineering problems [9], [25], [16], [31], and the references therein. reliably and efficiently. The goal of this tutorial is to continue the overview of modern convex optimization from where our We now briefly outline the paper. There are two main ACC2004 Tutorial on Convex Optimization left off, to cover sections after this one. Section II is on duality, where we important topics that were omitted there due to lack of space summarize the key ideas the general theory, illustrating and time, and highlight the intimate connections between them. the four main practical applications of duality with simple The topics of duality and interior point algorithms will be our examples. Section III is on interior point algorithms, where focus, along with simple examples. The material in this tutorial is excerpted from the recent book on convex optimization, by the focus is on barrier methods, which can be implemented Boyd and Vandenberghe, who have made available a large easily using only a few key technical components, and yet amount of free course material and freely available software. are highly effective both in theory and in practice. All of the These can be downloaded and used immediately by the reader theory we cover can be readily extended to general conic both for self-study and to solve real problems.
    [Show full text]
  • Log-Concave Duality in Estimation and Control Arxiv:1607.02522V1
    Log-Concave Duality in Estimation and Control Robert Bassett,˚ Michael Casey,: and Roger J-B Wets; Department of Mathematics, Univ. California Davis Abstract In this paper we generalize the estimation-control duality that exists in the linear- quadratic-Gaussian setting. We extend this duality to maximum a posteriori estimation of the system's state, where the measurement and dynamical system noise are inde- pendent log-concave random variables. More generally, we show that a problem which induces a convex penalty on noise terms will have a dual control problem. We provide conditions for strong duality to hold, and then prove relaxed conditions for the piece- wise linear-quadratic case. The results have applications in estimation problems with nonsmooth densities, such as log-concave maximum likelihood densities. We conclude with an example reconstructing optimal estimates from solutions to the dual control problem, which has implications for sharing solution methods between the two types of problems. 1 Introduction We consider the problem of estimating the state of a noisy dynamical system based only on noisy measurements of the system. In this paper, we assume linear dynamics, so that the progression of state variables is Xt`1 “FtpXtq ` Wt`1; t “ 0; :::; T ´ 1 (1) X0 “W0 (2) arXiv:1607.02522v1 [math.OC] 8 Jul 2016 Xt is the state variable{a random vector indexed by a discrete time-step t which ranges from 0 to some final time T . All of the results in this paper still hold in the case that the dimension nx of Xt is time-dependent, but for notational convenience we will assume that Xt P R for t “ 0; :::; T .
    [Show full text]
  • Lecture 11: October 8 11.1 Primal and Dual Problems
    10-725/36-725: Convex Optimization Fall 2015 Lecture 11: October 8 Lecturer: Ryan Tibshirani Scribes: Tian Tong Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications. They may be distributed outside this class only with the permission of the Instructor. 11.1 Primal and dual problems 11.1.1 Lagrangian Consider a general optimization problem (called as primal problem) min f(x) (11.1) x subject to hi(x) ≤ 0; i = 1; ··· ; m `j(x) = 0; j = 1; ··· ; r: We define its Lagrangian as m r X X L(x; u; v) = f(x) + uihi(x) + vj`j(x): i=1 j=1 Lagrange multipliers u 2 Rm; v 2 Rr. Lemma 11.1 At each feasible x, f(x) = supu≥0;v L(x; u; v), and the supremum is taken iff u ≥ 0 satisfying uihi(x) = 0; i = 1; ··· ; m. Pm Proof: At each feasible x, we have hi(x) ≤ 0 and `(x) = 0, thus L(x; u; v) = f(x) + i=1 uihi(x) + Pr j=1 vj`j(x) ≤ f(x). The last inequality becomes equality iff uihi(x) = 0; i = 1; ··· ; m. Proposition 11.2 The optimal value of the primal problem, named as f ?, satisfies: f ? = inf sup L(x; u; v): x u≥0;v ? Proof: First considering feasible x (marked as x 2 C), we have f = infx2C f(x) = infx2C supu≥0;v L(x; u; v). Second considering non-feasible x, since supu≥0;v L(x; u; v) = 1 for any x2 = C, infx=2C supu≥0;v L(x; u; v) = ? 1.
    [Show full text]
  • Stability of the Duality Gap in Linear Optimization
    Set-Valued and Variational Analysis manuscript No. (will be inserted by the editor) Stability of the duality gap in linear optimization M.A. Goberna · A.B. Ridolfi · V.N. Vera de Serio Received: date / Accepted: date Abstract In this paper we consider the duality gap function g that measures the difference between the optimal values of the primal problem and of the dual problem in linear pro- gramming and in linear semi-infinite programming. We analyze its behavior when the data defining these problems may be perturbed, considering seven different scenarios. In partic- ular we find some stability results by proving that, under mild conditions, either the duality gap of the perturbed problems is zero or +¥ around the given data, or g has an infinite jump at it. We also give conditions guaranteeing that those data providing a finite duality gap are limits of sequences of data providing zero duality gap for sufficiently small perturbations, which is a generic result. Keywords Linear programming · Linear semi-infinite programming · Duality gap function · Stability · Primal-dual partition Mathematics Subject Classifications (2010) 90C05, 90C34, 90C31, 90C46 1 Introduction Linear optimization consists in the minimization of a linear objective function subject to linear constraints. Here the duality gap plays an important role both for theoretical and for This research was partially supported by MINECO of Spain and FEDER of EU, Grant MTM2014-59179- C2-01 and SECTyP-UNCuyo Res. 4540/13-R. M.A. Goberna Department of Mathematics, University of Alicante, 03080 Alicante, Spain. E-mail: [email protected] A.B. Ridolfi Universidad Nacional de Cuyo, Facultad de Ciencias Aplicadas a la Industria, Facultad de Ciencias Economicas,´ Mendoza, Argentina.
    [Show full text]
  • Epsilon Solutions and Duality in Vector Optimization
    NOT FOR QUOTATION WITHOUT THE PERMISSION OF THE AUTHOR EPSILON SOLUTIONS AND DUALITY IN VECTOR OPTIMIZATION Istvdn VdLyi May 1987 WP-87-43 Working Papers are interim reports on work of the International Institute for Applied Systems Analysis and have received only limited review. Views or opinions expressed herein do not necessarily represent those of the Institute or of its National Member Organizations. INTERNATIONAL INSTITUTE FOR APPLIED SYSTEMS ANALYSIS 2361 Laxenburg, Austria PREFACE This paper is a continuation of the author's previous investigations in the theory of epsilon-solutions in convex vector optimization and serves as a theoretical background for the research of SDS in the field of multicriteria optimization. With the stress laid on duality theory, the results presented here give some insight into the problems arising when exact solutions have to be substituted by approximate ones. Just like in the scalar case, the available computational techniques fre- quently lead to such a situation in multicriteria optimization. Alexander B. Kurzhanski Chairman Systems and Decision Sciences Area CONTENTS 1. Introduction 2. Epsilon Optimal Elements 3. Perturbation Map and Duality 4. Conical Supports 5. Conclusion 6. References EPSILON SOLUTIONS AND DUALITY IN VECTOR OF'TIMIZATION Istv&n V&lyi The study of epsilon solutions in vector optimization problems was started in 1979 by S. S. Kutateladze [I]. These types of solutions are interesting because of their relation to nondifferentiable optimization and the vector valued extensions of Ekeland's variational principle as considered by P. Loridan [2] and I. Vdlyi [3], but computational aspects are perhaps even more important. In practical situations, namely, we often stop the calculations at values that we consider sufficiently close to the optimal solution, or use algorithms that result in some approximates of the Pareto set.
    [Show full text]