DOCSLIB.ORG

Convex Analysis, Duality and Optimization

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Convex Analysis, Duality and Optimization Convex Analysis, Duality and Optimization Yao-Liang Yu [email protected] Dept. of Computing Science University of Alberta March 7, 2010 Prelude Basic Convex Analysis Convex Optimization Fenchel Conjugate Minimax Theorem Lagrangian Duality References Outline Prelude Basic Convex Analysis Convex Optimization Fenchel Conjugate Minimax Theorem Lagrangian Duality References Notations Used Throughout I C for convex set, S for arbitrary set, K for convex cone, I g(·) is for arbitrary functions, not necessarily convex, I f (·) is for convex functions, for simplicity, we assume f (·) is closed, proper, continuous, and differentiable when needed, I min (max) means inf (sup) when needed, I w.r.t.: with respect to; w.l.o.g.: without loss of generality; u.s.c.: upper semi-continuous; l.s.c.: lower semi-continuous; int: interior point; RHS: right hand side; w.p.1: with probability 1. Historical Note I 60s: Linear Programming, Simplex Method I 70s-80s: (Convex) Nonlinear Programming, Ellipsoid Method, Interior-Point Method I 90s: Convexification almost everywhere I Now: Large-scale optimization, First-order gradient method But... Neither of poly-time solvability and convexity implies the other. NP-Hard convex problems abound. Outline Prelude Basic Convex Analysis Convex Optimization Fenchel Conjugate Minimax Theorem Lagrangian Duality References Convex Sets and Functions Definition (Convex set) A point set C is said to be convex if 8 λ 2 [0; 1]; x; y 2 C, we have λx + (1 − λ)y 2 C. Definition (Convex function) A function f (·) is said to be convex if 1. domf is convex, and, 2. 8 λ 2 [0; 1]; x; y 2 domf , we have f (λx + (1 − λ)y) ≤ λf (x) + (1 − λ)f (y); x Or equivalently, f (·) is convex if its epigraph f t : f (x) ≤ tg is a convex set. I Function h(·) is concave iff −h(·) is convex, I h(·) is called affine (linear) iff it's both convex and concave, I No concave set. Affine set: drop the constraint on λ. More on Convex functions Definition (Strongly Convex Function) f (x) is said to be µ-strongly convex with respect to a norm k · k iff dom f is convex and 8λ 2 [0; 1], λ(1 − λ) f (λx + (1 − λ)y) + µ · kx − yk2 ≤ λf (x) + (1 − λ)f (y): 2 Proposition (Sufficient Conditions for µ-Strong Convexity) 1. Zero Order: definition 2. First Order: 8x; y 2 dom f ; µ f (y) ≥ f (x) + hrf (x); x − yi + kx − yk2: 2 3. Second Order: 8x; y 2 dom f ; hr2f (x)y; yi ≥ µkyk2: Elementary Convex Functions (to name a few) I Negative entropy x log x is convex on x > 0, 1=p h P pi I `p-norm kxkp := i jxi j is convex when p ≥ 1, concave otherwise (except p = 0), P I Log-sum-exp function log i exp(xi ) is convex, same is true for the matrix version log Tr exp(X ) on symmetric matrices, T −1 I Quadratic-over-linear function x Y x is jointly convex in x and Y 0, what if Y 0? I Log-determinant log det X is concave on X 0, what about log det X −1? −1 I Tr X is convex on X 0, I The largest element x[1] = maxi xi is convex; moreover, sum of largest k elements is convex; what about smallest analogies? I The largest eigenvalue of symmetric matrices is convex; moreover, sum of largest k eigenvalues of symmetric matrices is also convex; can we drop the condition symmetric? Compositions Proposition (Affine Transform) AC := fAx : x 2 Cg and A−1C := fx : Ax 2 Cg are convex sets. Similarly, (Af )(x) := min f (y) and (fA)(x) := f (Ax) are convex. Ay=x Proposition (Sufficient but NOT Necessary) f ◦ g is convex if I g(·) is convex and f (·) is non-decreasing, or I g(·) is concave and f (·) is non-increasing. Proof. For simplicity, assume f ◦ g is twice differentiable. Use the second-order sufficient condition. Remark: One needs to check if domf ◦ g is convex! However, this is unnecessary if we consider extended-value functions. Operators Preserving Convexity Proposition (Algebraic) For θ > 0, λC := fθx : x 2 Cg is convex; θf (x) is convex; and f1(x) + f2(x) is convex. Proposition (Intersection v.s. Supremum) I Intersection of arbitrary collection of convex sets is convex; I Similarly, pointwise supremum of arbitrary collection of convex functions is convex. Proposition (Sum v.s. Infimal Convolution) I C1 + C2 := fx1 + x2 : xi 2 Ci g is convex; I Similarly, (f1f2)(x) := infy ff1(y) + f2(x − y)g is convex. Proof. Consider affine transform. What about union v.s. infimum? Needs extra convexification. Convex Hull Definition (Convex Hull) The convex hull of S, denoted convS, is the smallest convex set containing S, i.e. the intersection of all convex sets containing S. Similarly, the convex hull of g(x), denoted convg, is the greatest convex function dominated by g, i.e. the pointwise supremum of all convex functions dominated by g. Theorem (Carath´eodory, 1911) n The convex hull of any set S 2 R is: n+1 n+1 X X fx : x = λi xi ; xi 2 S; λi ≥ 0; λi = 1g: i=1 i=1 We will see how to compute convg later. Cones and Conic Hull Definition (Cone and Positively Homogeneous Function) A set S is called a cone if 8x 2 S; θ ≥ 0; we have θx 2 S. Similarly, a function g(x) is called positively homogeneous if 8θ ≥ 0; g(θx) = θg(x). K is a convex cone if it is a cone and is convex, specifically, if 8x1; x2 2 K; θ1; θ2 ≥ 0; ) θ1x1 + θ2x2 2 K. Similarly, f (x) is positively homogeneous convex if it is positively homogeneous and convex, specifically, if 8x1; x2 2 domf ; θ1; θ2 ≥ 0; ) f (θ1x1 + θ2x2) ≤ θ1f (x1) + θ2f (x2). Remark: Under the above definitions, cones always contain the origin, and positively homogeneous functions equal 0 at the origin. Definition (Conic Hull) The conic hull of S is the smallest convex cone containing S. Similarly, the conic hull of g(x), denoted coneg, is the greatest positively homogeneous convex function dominated by g. Conic Hull cont' Theorem (Carath´eodory, 1911) n The conic hull of any set S 2 R is: n X fx : x = θi xi ; xi 2 S; θi ≥ 0; g: i=1 For convex function f (x), its conic hull is: (conef )(x) = min θ · f (θ−1x): θ≥0 How to compute coneg? Hint: coneg = cone convg, why? Elementary Convex Sets (to name a few) T I Hyperplane a x = α is convex, T I Half space a x ≤ α is convex, I Affine set Ax = b is convex (proof?), I Polyhedra set Ax ≤ b is convex (proof?), Proposition (Level sets) (Sub)level sets of f (x), defined as fx : f (x) ≤ αg are convex. Proof. Consider the intersection of the epigraph of f (x) and the 0T x hyperplane 1 t = α. Remark: A function, with all level sets being convex, is not necessarily convex! We call such functions, with convex domain, quasi-convex. P Convince yourself the `0-norm, defined as kxk0 = i I[xi 6= 0], is not convex. Show that -kxk0 on x ≥ 0 is quasi-convex. Elementary Convex Sets cont' T −1 I Ellipsoid fx :(x − xc ) P (x − xc ) ≤ 1; P 0g or 1=2 fxc + P u : kuk2 ≤ 1g is convex, I Nonnegative orthant x ≥ 0 is a convex cone, I All positive (semi)definite matrices compose a convex cone (positive (semi)definite cone) X 0 (X 0), x I All norm cones f t : kxk ≤ tg are convex, in particular, for the Euclidean norm, the cone is called second order cone or Lorentz cone or ice-cream cone. Remark: This is essentially saying that all norms are convex. `0-norm is not convex? No, but it's not a \norm" either. People call it \norm" unjustly. Outline Prelude Basic Convex Analysis Convex Optimization Fenchel Conjugate Minimax Theorem Lagrangian Duality References Unconstrained Consider the simple problem: min f (x); (1) x where f (·) is defined in the whole space. Theorem (First-order Optimality Condition) A sufficient and necessary condition for x? to be the minimizer of (1) is: 0 2 @f (x?): (2) When f (·) is differentiable, (2) reduces to rf (x?) = 0. Remark: I The minimizer is unique when f (·) is strictly convex, I For general nonconvex functions g(·), the condition in (2) gives only critical (stationary) points, which could be minimizer, maximizer, or nothing (saddle-point). Simply Constrained Consider the constrained problem: min f (x); (3) x2C where f (·) is defined in the whole space. Is rf (x?) = 0 still the optimality condition? If you think so, consider the example: min x: x2[1;2] Theorem (First-order Optimality Condition) A sufficient and necessary condition for x? to be the minimizer of (3) is (assuming differentiability): 8x 2 C; (x − x?)T rf (x?) ≥ 0: (4) Verify this condition is indeed satisfied by the example above. General Convex Program We say a problem is convex if it is of the following form: min f0(x) x2C s:t: fi (x) ≤ 0; i = 1;:::; m Ax = b; where fi (x); i = 0;:::; m are all convex. Remark: I The equality constraint must be affine! But affine functions are free to appear in inequality constraints. I The objective function, being convex, is to be minimized. Sometimes we see maximizinga concave function, no difference (why?). I The inequality constraints are ≤, which lead to a convex feasible region (why?).
Load more

Recommended publications
  • Boosting Algorithms for Maximizing the Soft Margin
    Boosting Algorithms for Maximizing the Soft Margin Manfred K. Warmuth∗ Karen Glocer Gunnar Ratsch¨ Dept. of Engineering Dept. of Engineering Friedrich Miescher Laboratory University of California University of California Max Planck Society Santa Cruz, CA, U.S.A. Santa Cruz, CA, U.S.A. T¨ubingen, Germany Abstract We present a novel boosting algorithm, called SoftBoost, designed for sets of bi- nary labeled examples that are not necessarily separable by convex combinations of base hypotheses. Our algorithm achieves robustness by capping the distribu- tions on the examples. Our update of the distribution is motivated by minimizing a relative entropy subject to the capping constraints and constraints on the edges of the obtained base hypotheses. The capping constraints imply a soft margin in the dual optimization problem. Our algorithm produces a convex combination of hypotheses whose soft margin is within δ of its maximum. We employ relative en- ln N tropy projection methods to prove an O( δ2 ) iteration bound for our algorithm, where N is number of examples. We compare our algorithm with other approaches including LPBoost, Brown- Boost, and SmoothBoost. We show that there exist cases where the numberof iter- ations required by LPBoost grows linearly in N instead of the logarithmic growth for SoftBoost. In simulation studies we show that our algorithm converges about as fast as LPBoost, faster than BrownBoost, and much faster than SmoothBoost. In a benchmark comparison we illustrate the competitiveness of our approach. 1 Introduction Boosting methods have been used with great success in many applications like OCR, text classifi- cation, natural language processing, drug discovery, and computational biology [13].
    [Show full text]
  • On the Dual Formulation of Boosting Algorithms
    On the Dual Formulation of Boosting Algorithms Chunhua Shen, and Hanxi Li Abstract—We study boosting algorithms from a new perspective. We show that the Lagrange dual problems of ℓ1 norm regularized AdaBoost, LogitBoost and soft-margin LPBoost with generalized hinge loss are all entropy maximization problems. By looking at the dual problems of these boosting algorithms, we show that the success of boosting algorithms can be understood in terms of maintaining a better margin distribution by maximizing margins and at the same time controlling the margin variance. We also theoretically prove that, approximately, ℓ1 norm regularized AdaBoost maximizes the average margin, instead of the minimum margin. The duality formulation also enables us to develop column generation based optimization algorithms, which are totally corrective. We show that they exhibit almost identical classification results to that of standard stage-wise additive boosting algorithms but with much faster convergence rates. Therefore fewer weak classifiers are needed to build the ensemble using our proposed optimization technique. Index Terms—AdaBoost, LogitBoost, LPBoost, Lagrange duality, linear programming, entropy maximization. ✦ 1 INTRODUCTION that the hard-margin LPBoost does not perform well in OOSTING has attracted a lot of research interests since most cases although it usually produces larger minimum B the first practical boosting algorithm, AdaBoost, was margins. More often LPBoost has worse generalization introduced by Freund and Schapire [1]. The machine learn- performance. In other words, a higher minimum margin ing community has spent much effort on understanding would not necessarily imply a lower test error. Breiman [11] how the algorithm works [2], [3], [4].
    [Show full text]
  • 1 Overview 2 Elements of Convex Analysis
    AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 2 | Wednesday, January 27th 1 Overview In our previous lecture we discussed several applications of optimization, introduced basic terminol- ogy, and proved Weierstass' theorem (circa 1830) which gives a sufficient condition for existence of an optimal solution to an optimization problem. Today we will discuss basic definitions and prop- erties of convex sets and convex functions. We will then prove the separating hyperplane theorem and see an application of separating hyperplanes in machine learning. We'll conclude by describing the seminal perceptron algorithm (1957) which is designed to find separating hyperplanes. 2 Elements of Convex Analysis We will primarily consider optimization problems over convex sets { sets for which any two points are connected by a line. We illustrate some convex and non-convex sets in Figure 1. Definition. A set S is called a convex set if any two points in S contain their line, i.e. for any x1; x2 2 S we have that λx1 + (1 − λ)x2 2 S for any λ 2 [0; 1]. In the previous lecture we saw linear regression an example of an optimization problem, and men- tioned that the RSS function has a convex shape. We can now define this concept formally. n Definition. For a convex set S ⊆ R , we say that a function f : S ! R is: • convex on S if for any two points x1; x2 2 S and any λ 2 [0; 1] we have that: f (λx1 + (1 − λ)x2) ≤ λf(x1) + 1 − λ f(x2): • strictly convex on S if for any two points x1; x2 2 S and any λ 2 [0; 1] we have that: f (λx1 + (1 − λ)x2) < λf(x1) + (1 − λ) f(x2): We illustrate a convex function in Figure 2.
    [Show full text]
  • CORE View Metadata, Citation and Similar Papers at Core.Ac.Uk
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Bulgarian Digital Mathematics Library at IMI-BAS Serdica Math. J. 27 (2001), 203-218 FIRST ORDER CHARACTERIZATIONS OF PSEUDOCONVEX FUNCTIONS Vsevolod Ivanov Ivanov Communicated by A. L. Dontchev Abstract. First order characterizations of pseudoconvex functions are investigated in terms of generalized directional derivatives. A connection with the invexity is analysed. Well-known first order characterizations of the solution sets of pseudolinear programs are generalized to the case of pseudoconvex programs. The concepts of pseudoconvexity and invexity do not depend on a single definition of the generalized directional derivative. 1. Introduction. Three characterizations of pseudoconvex functions are considered in this paper. The first is new. It is well-known that each pseudo- convex function is invex. Then the following question arises: what is the type of 2000 Mathematics Subject Classification: 26B25, 90C26, 26E15. Key words: Generalized convexity, nonsmooth function, generalized directional derivative, pseudoconvex function, quasiconvex function, invex function, nonsmooth optimization, solution sets, pseudomonotone generalized directional derivative. 204 Vsevolod Ivanov Ivanov the function η from the definition of invexity, when the invex function is pseudo- convex. This question is considered in Section 3, and a first order necessary and sufficient condition for pseudoconvexity of a function is given there. It is shown that the class of strongly pseudoconvex functions, considered by Weir [25], coin- cides with pseudoconvex ones. The main result of Section 3 is applied to characterize the solution set of a nonlinear programming problem in Section 4. The base results of Jeyakumar and Yang in the paper [13] are generalized there to the case, when the function is pseudoconvex.
    [Show full text]
  • Lagrangian Duality in Convex Optimization
    Lagrangian Duality in Convex Optimization LI, Xing A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Master of Philosophy in Mathematics The Chinese University of Hong Kong July 2009 ^'^'^xLIBRARr SYSTEf^N^J Thesis/Assessment Committee Professor LEUNG, Chi Wai (Chair) Professor NG, Kung Fu (Thesis Supervisor) Professor LUK Hing Sun (Committee Member) Professor HUANG, Li Ren (External Examiner) Abstract of thesis entitled: Lagrangian Duality in Convex Optimization Submitted by: LI, Xing for the degree of Master of Philosophy in Mathematics at the Chinese University of Hong Kong in July, 2009. Abstract In convex optimization, a strong duality theory which states that the optimal values of the primal problem and the Lagrangian dual problem are equal and the dual problem attains its maximum plays an important role. With easily visualized physical meanings, the strong duality theories have also been wildly used in the study of economical phenomenon and operation research. In this thesis, we reserve the first chapter for an introduction on the basic principles and some preliminary results on convex analysis ； in the second chapter, we will study various regularity conditions that could lead to strong duality, from the classical ones to recent developments; in the third chapter, we will present stable Lagrangian duality results for cone-convex optimization problems under continuous linear perturbations of the objective function . In the last chapter, Lagrange multiplier conditions without constraint qualifications will be discussed. 摘要 拉格朗日對偶理論主要探討原問題與拉格朗日對偶問題的 最優值之間“零對偶間隙”成立的條件以及對偶問題存在 最優解的條件，其在解決凸規劃問題中扮演著重要角色， 並在經濟學運籌學領域有著廣泛的應用。 本文將系統地介紹凸錐規劃中的拉格朗日對偶理論，包括 基本規範條件，閉凸錐規範條件等，亦會涉及無規範條件 的序列拉格朗日乘子。 ACKNOWLEDGMENTS I wish to express my gratitude to my supervisor Professor Kung Fu Ng and also to Professor Li Ren Huang, Professor Chi Wai Leung, and Professor Hing Sun Luk for their guidance and valuable suggestions.
    [Show full text]
  • Applications of Convex Analysis Within Mathematics
    Noname manuscript No. (will be inserted by the editor) Applications of Convex Analysis within Mathematics Francisco J. Arag´onArtacho · Jonathan M. Borwein · Victoria Mart´ın-M´arquez · Liangjin Yao July 19, 2013 Abstract In this paper, we study convex analysis and its theoretical applications. We first apply important tools of convex analysis to Optimization and to Analysis. We then show various deep applications of convex analysis and especially infimal convolution in Monotone Operator Theory. Among other things, we recapture the Minty surjectivity theorem in Hilbert space, and present a new proof of the sum theorem in reflexive spaces. More technically, we also discuss autoconjugate representers for maximally monotone operators. Finally, we consider various other applications in mathematical analysis. Keywords Adjoint · Asplund averaging · autoconjugate representer · Banach limit · Chebyshev set · convex functions · Fenchel duality · Fenchel conjugate · Fitzpatrick function · Hahn{Banach extension theorem · infimal convolution · linear relation · Minty surjectivity theorem · maximally monotone operator · monotone operator · Moreau's decomposition · Moreau envelope · Moreau's max formula · Moreau{Rockafellar duality · normal cone operator · renorming · resolvent · Sandwich theorem · subdifferential operator · sum theorem · Yosida approximation Mathematics Subject Classification (2000) Primary 47N10 · 90C25; Secondary 47H05 · 47A06 · 47B65 Francisco J. Arag´onArtacho Centre for Computer Assisted Research Mathematics and its Applications (CARMA),
    [Show full text]
  • Convex) Level Sets Integration
    Journal of Optimization Theory and Applications manuscript No. (will be inserted by the editor) (Convex) Level Sets Integration Jean-Pierre Crouzeix · Andrew Eberhard · Daniel Ralph Dedicated to Vladimir Demjanov Received: date / Accepted: date Abstract The paper addresses the problem of recovering a pseudoconvex function from the normal cones to its level sets that we call the convex level sets integration problem. An important application is the revealed preference problem. Our main result can be described as integrating a maximally cycli- cally pseudoconvex multivalued map that sends vectors or \bundles" of a Eu- clidean space to convex sets in that space. That is, we are seeking a pseudo convex (real) function such that the normal cone at each boundary point of each of its lower level sets contains the set value of the multivalued map at the same point. This raises the question of uniqueness of that function up to rescaling. Even after normalising the function long an orienting direction, we give a counterexample to its uniqueness. We are, however, able to show uniqueness under a condition motivated by the classical theory of ordinary differential equations. Keywords Convexity and Pseudoconvexity · Monotonicity and Pseudomono- tonicity · Maximality · Revealed Preferences. Mathematics Subject Classification (2000) 26B25 · 91B42 · 91B16 Jean-Pierre Crouzeix, Corresponding Author, LIMOS, Campus Scientifique des C´ezeaux,Universit´eBlaise Pascal, 63170 Aubi`ere,France, E-mail: [email protected] Andrew Eberhard School of Mathematical & Geospatial Sciences, RMIT University, Melbourne, VIC., Aus- tralia, E-mail: [email protected] Daniel Ralph University of Cambridge, Judge Business School, UK, E-mail: [email protected] 2 Jean-Pierre Crouzeix et al.
    [Show full text]
  • Turbo Decoding As Iterative Constrained Maximum Likelihood Sequence Detection John Maclaren Walsh, Member, IEEE, Phillip A
    1 Turbo Decoding as Iterative Constrained Maximum Likelihood Sequence Detection John MacLaren Walsh, Member, IEEE, Phillip A. Regalia, Fellow, IEEE, and C. Richard Johnson, Jr., Fellow, IEEE Abstract— The turbo decoder was not originally introduced codes have good distance properties, which would be relevant as a solution to an optimization problem, which has impeded for maximum likelihood decoding, researchers have not yet attempts to explain its excellent performance. Here it is shown, succeeded in developing a proper connection between the sub- nonetheless, that the turbo decoder is an iterative method seeking a solution to an intuitively pleasing constrained optimization optimal turbo decoder and maximum likelihood decoding. This problem. In particular, the turbo decoder seeks the maximum is exacerbated by the fact that the turbo decoder, unlike most of likelihood sequence under the false assumption that the input the designs in modern communications systems engineering, to the encoders are chosen independently of each other in the was not originally introduced as a solution to an optimization parallel case, or that the output of the outer encoder is chosen problem. This has made explaining just why the turbo decoder independently of the input to the inner encoder in the serial case. To control the error introduced by the false assumption, the opti- performs as well as it does very difficult. Together with the mizations are performed subject to a constraint on the probability lack of formulation as a solution to an optimization problem, that the independent messages happen to coincide. When the the turbo decoder is an iterative algorithm, which makes the constraining probability equals one, the global maximum of determining its convergence and stability behavior important.
    [Show full text]
  • A Comparative Study of Time Delay Estimation Techniques for Road Vehicle Tracking Patrick Marmaroli, Xavier Falourd, Hervé Lissek
    A comparative study of time delay estimation techniques for road vehicle tracking Patrick Marmaroli, Xavier Falourd, Hervé Lissek To cite this version: Patrick Marmaroli, Xavier Falourd, Hervé Lissek. A comparative study of time delay estimation techniques for road vehicle tracking. Acoustics 2012, Apr 2012, Nantes, France. hal-00810981 HAL Id: hal-00810981 https://hal.archives-ouvertes.fr/hal-00810981 Submitted on 23 Apr 2012 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Proceedings of the Acoustics 2012 Nantes Conference 23-27 April 2012, Nantes, France A comparative study of time delay estimation techniques for road vehicle tracking P. Marmaroli, X. Falourd and H. Lissek Ecole Polytechnique F´ed´erale de Lausanne, EPFL STI IEL LEMA, 1015 Lausanne, Switzerland patrick.marmaroli@epfl.ch 4135 23-27 April 2012, Nantes, France Proceedings of the Acoustics 2012 Nantes Conference This paper addresses road traffic monitoring using passive acoustic sensors. Recently, the feasibility of the joint speed and wheelbase length estimation of a road vehicle using particle filtering has been demonstrated. In essence, the direction of arrival of propagated tyre/road noises are estimated using a time delay estimation (TDE) technique between pairs of microphones placed near the road.
    [Show full text]
  • Applications of LP Duality
    Lecture 6 Duality of LPs and Applications∗ Last lecture we introduced duality of linear programs. We saw how to form duals, and proved both the weak and strong duality theorems. In this lecture we will see a few more theoretical results and then begin discussion of applications of duality. 6.1 More Duality Results 6.1.1 A Quick Review Last time we saw that if the primal (P) is max c>x s:t: Ax ≤ b then the dual (D) is min b>y s:t: A>y = c y ≥ 0: This is just one form of the primal and dual and we saw that the transformation from one to the other is completely mechanical. The duality theorem tells us that if (P) and (D) are a primal-dual pair then we have one of the three possibilities 1. Both (P) and (D) are infeasible. 2. One is infeasible and the other is unbounded. 3. Both are feasible and if x∗ and y∗ are optimal solutions to (P) and (D) respectively, then c>x∗ = b>y∗. *Lecturer: Anupam Gupta. Scribe: Deepak Bal. 1 LECTURE 6. DUALITY OF LPS AND APPLICATIONS 2 6.1.2 A Comment about Complexity Note that the duality theorem (and equivalently, the Farkas Lemma) puts several problems related to LP feasibility and solvability in NP \ co-NP. E.g., Consider the question of whether the equational form LP Ax = b; x ≥ 0 is feasible. If the program is feasible, we may efficiently verify this by checking that a \certificate” point satisfies the equations. By taking this point to be a vertex and appealing to Hwk1 (Problem 4), we see that we may represent this certificate point in size polynomial in the size of the input.
    [Show full text]
  • Duality Theory and Sensitivity Analysis
    4 Duality Theory and Sensitivity Analysis The notion of duality is one of the most important concepts in linear programming. Basically, associated with each linear programming problem (we may call it the primal problem), defined by the constraint matrix A, the right-hand-side vector b, and the cost vector c, there is a corresponding linear programming problem (called the dual problem) which is constructed by the same set of data A, b, and c. A pair of primal and dual problems are closely related. The interesting relationship between the primal and dual reveals important insights into solving linear programming problems. To begin this chapter, we introduce a dual problem for the standard-form linear programming problem. Then we study the fundamental relationship between the primal and dual problems. Both the "strong" and "weak" duality theorems will be presented. An economic interpretation of the dual variables and dual problem further exploits the concepts in duality theory. These concepts are then used to derive two important simplex algorithms, namely the dual simplex algorithm and the primal dual algorithm, for solving linear programming problems. We conclude this chapter with the sensitivity analysis, which is the study of the effects of changes in the parameters (A, b, and c) of a linear programming problem on its optimal solution. In particular, we study different methods of changing the cost vector, changing the right-hand-side vector, adding and removing a variable, and adding and removing a constraint in linear programming. 55 56 Duality Theory and Sensitivity Analysis Chap. 4 4.1 DUAL LINEAR PROGRAM Consider a linear programming problem in its standard form: Minimize cT x (4.1a) subject to Ax = b (4.1b) x:::O (4.1c) where c and x are n-dimensional column vectors, A an m x n matrix, and b an m­ dimensional column vector.
    [Show full text]
  • Optimization Basic Results
    Optimization Basic Results Michel De Lara Cermics, Ecole´ des Ponts ParisTech France Ecole´ des Ponts ParisTech November 22, 2020 Outline of the presentation Magic formulas Convex functions, coercivity Existence and uniqueness of a minimum First-order optimality conditions (the case of equality constraints) Duality gap and saddle-points Elements of Lagrangian duality and Uzawa algorithm More on convexity and duality Outline of the presentation Magic formulas Convex functions, coercivity Existence and uniqueness of a minimum First-order optimality conditions (the case of equality constraints) Duality gap and saddle-points Elements of Lagrangian duality and Uzawa algorithm More on convexity and duality inf h(a; b) = inf inf h(a; b) a2A;b2B a2A b2B inf λf (a) = λ inf f (a) ; 8λ ≥ 0 a2A a2A inf f (a) + g(b) = inf f (a) + inf g(b) a2A;b2B a2A b2B Tower formula For any function h : A × B ! [−∞; +1] we have inf h(a; b) = inf inf h(a; b) a2A;b2B a2A b2B and if B(a) ⊂ B, 8a 2 A, we have inf h(a; b) = inf inf h(a; b) a2A;b2B(a) a2A b2B(a) Independence For any functions f : A !] − 1; +1] ; g : B !] − 1; +1] we have inf f (a) + g(b) = inf f (a) + inf g(b) a2A;b2B a2A b2B and for any finite set S, any functions fs : As !] − 1; +1] and any nonnegative scalars πs ≥ 0, for s 2 S, we have X X inf π f (a )= π inf f (a ) Q s s s s s s fas gs2 2 s2 As as 2As S S s2S s2S Outline of the presentation Magic formulas Convex functions, coercivity Existence and uniqueness of a minimum First-order optimality conditions (the case of equality constraints) Duality gap and saddle-points Elements of Lagrangian duality and Uzawa algorithm More on convexity and duality Convex sets Let N 2 N∗.
    [Show full text]