Conic Linear Programming Yinyu Ye December 2004, revised October 2017 i ii Preface This monograph is developed for MS&E 314, \Conic Linear Programming", which I am teaching at Stanford. Information, lecture slides, supporting mate- rials, and computer programs related to this book may be found at the following address on the World-Wide Web: http://www.stanford.edu/class/msande314 Please report any question, comment and error to the address: [email protected] A little story in the development of semidefinite programming (SDP), a major subclass of conic linear programming. One day in 1990, I visited the Computer Science Department of the University of Minnesota and met a young graduate student, Farid Alizadeh. He, working then on combinatorial optimiza- tion, introduced me “semidefinite optimization" or linear programming over the positive definite matrix cone. We had a very extensive discussion that afternoon and concluded that interior-point linear programming algorithms could be ap- plicable to solving SDPs. I suggested Farid to look at the linear programming (LP) interior-point algorithms and to develop an SDP (primal) potential reduc- tion algorithm. He worked hard for several months, and one afternoon showed up in my office in Iowa City, about 300 miles from Minneapolis. He had every- thing worked out, including potential function, algorithm, complexity bound, and even a \dictionary" list between LP and SDP. But he was stuck on one problem that was on how to keep the symmetry of the scaled directional ma- trix. We went to a bar nearby on Clinton Street in Iowa City (I paid for him since I was a third-year professor then and eager to demonstrate that I could take care of my students). After chatting for a while, I suggested that he should use scaling X−1=2∆X−1=2 to compute symmetric directional matrix ∆, instead of X−1∆ which he was using earlier, where X is the current symmetric positive definite matrix. This way, X + α∆ would remain symmetric with a step-size scalar. He returned to Minneapolis and moved to Berkeley shortly after, and few weeks later sent me an e-mail message telling me that everything had worked out beautifully. At the same time, Nesterov and Nemirovskii developed a more general and powerful theory in extending interior-point algorithms for solving convex pro- grams, where SDP was a special case. Boyd and his group presented a wide range of SDP applications and formulations, many of which were incredibly novel and elegant. Then came the primal-dual algorithms of many authors, the iii iv PREFACE SDP approximation algorithm for Max-Cut, ... { SDP eventually established its full popularity. PREFACE v To Daisun, Fei, Tim, Kaylee and Rylee vi PREFACE Contents Preface iii List of Figures xi 1 Introduction and Preliminaries 1 1.1 Introduction . 1 1.2 Mathematical Preliminaries . 3 1.2.1 Basic notations . 3 1.2.2 Convex sets and cones . 5 1.2.3 Real functions . 10 1.2.4 Inequalities . 12 1.3 Some Basic Decision and Optimization Problems . 13 1.3.1 System of linear equations . 13 1.3.2 Linear least-squares problem . 14 1.3.3 System of linear inequalities . 15 1.3.4 Linear programming (LP) . 16 1.3.5 Quadratic programming (QP) . 19 1.4 Algorithms and Computations . 20 1.4.1 Complexity of problems . 20 1.4.2 Convergence rate . 21 1.5 Basic Computational Procedures . 23 1.5.1 Gaussian elimination method . 23 1.5.2 Choleski decomposition method . 24 1.5.3 The Newton method . 24 1.5.4 Solving ball-constrained linear problem . 25 1.5.5 Solving ball-constrained quadratic problem . 26 1.6 Notes . 26 1.7 Exercises . 27 2 Conic Linear Programming 31 2.1 Conic Linear Programming and its Dual . 31 2.1.1 Dual of conic linear programming . 33 2.2 Farkas' Lemma and Duality Theorem of Conic Linear Programming 35 2.2.1 Alternative theorem for conic systems . 36 vii viii CONTENTS 2.2.2 Duality theorem for conic linear programming . 38 2.2.3 Optimality conditions of conic linear programming . 40 2.3 Exact Low-Rank SDP Solutions . 42 2.3.1 Exact low-rank theorem . 42 2.4 Approximate Low-Rank SDP Solutions . 45 2.4.1 Approximate low-rank theorem . 45 2.4.2 A constructive proof . 47 2.5 Uniqueness of CLP Optimal Solution . 51 2.6 Notes . 53 2.7 Exercises . 54 3 Interior-Point Algorithms 57 3.1 Central Path and Path-Following . 57 3.1.1 Logarithmic barrier function for convex cones . 58 3.1.2 The central path . 63 3.1.3 Path following algorithms . 67 3.2 Potential Reduction Algorithms . 69 3.2.1 Potential functions . 70 3.2.2 Potential reduction algorithms . 72 3.2.3 Analysis of the primal potential-reduction . 74 3.3 Primal-Dual (Symmetric) Algorithm for LP and SDP . 84 3.4 Dual Algorithm for SDP . 87 3.5 Initialization . 95 3.6 Notes . 98 3.7 Exercises . 98 4 SDP for Global Quadratic and Combinatorial Optimization 101 4.1 Approximation . 101 4.2 Ball-Constrained Quadratic Minimization . 102 4.2.1 Homogeneous Case: q = 0 . 102 4.2.2 Non-Homogeneous Case . 103 4.3 Quadratically Constrained Quadratic Problems (QCQP) . 104 4.3.1 Multiple Ellipsoid-Constrained Quadratic Maximization . 105 4.3.2 Binary Quadratic Maximization . 106 4.3.3 Box Constrained Optimization . 107 4.4 Max-Cut Problem . 112 4.4.1 SDP relaxation . 113 4.4.2 Approximation analysis . 113 4.5 Max-Bisection Problem . 115 4.5.1 SDP relaxation . 116 4.5.2 The .651-method of Frieze and Jerrum . 116 4.5.3 A modified rounding and improved analyses . 118 4.5.4 A simple .5-approximation . 121 4.5.5 A .699-approximation . 122 4.6 Notes . 124 4.7 Exercises . 127 CONTENTS ix 5 SDP for Geometry Computation 131 5.1 The basic SDP model . 131 5.2 Wireless sensor network localization . 132 5.2.1 An SDP relaxation model . 133 5.2.2 Probabilistic or error analyses . 135 5.3 General SDP theory on graph realization . 138 5.3.1 Preliminaries . 138 5.3.2 Analysis of the SDP relaxation . 139 5.3.3 Strongly Localizable Problem . 141 5.3.4 A Comparison of Notions . 143 5.3.5 Unique Localizability 6) Strong Localizability . 143 5.3.6 Rigid in R2 6) Unique Localizability . 144 5.3.7 Preliminary computational and simulation results . 145 5.4 Other distance geometry problems . 147 5.4.1 Metric distance embedding . 147 5.4.2 Molecular confirmation . 148 5.4.3 Euclidean ball parking . 148 5.4.4 Data dimensionality reduction . 149 5.5 A special case: The k-radius of P . 150 5.5.1 Deterministic First Rounding . 151 5.5.2 Randomized Second Rounding . 152 5.6 Distributed SDP computing . 155 5.6.1 Preliminary computational and simulation results . 156 5.7 Notes . 156 5.8 Exercises . 159 6 SDP for Robust Optimization 161 6.1 Robust Optimization . 161 6.1.1 Stochastic Method . 162 6.1.2 Sampling Method . 162 6.2 Robust Quadratic Optimization . 162 6.2.1 Ellipsoid uncertainty . 162 6.2.2 S-Lemma . 163 6.2.3 SDP for Robust QP . 164 6.3 General Robust Quadratic Optimization . 164 6.3.1 Ellipsoid uncertainty . 164 6.3.2 SDP formulation . 165 6.4 More Robust Quadratic Optimization . 165 6.4.1 Ellipsoid uncertainty . 165 6.4.2 SDP formulation . 166 6.5 Tool for Robust Optimization . 166 6.5.1 Examples . 167 6.6 Applications . 168 6.6.1 Robust Linear Least Squares . 168 6.6.2 Heat dissipation problem . 168 6.6.3 Portfolio optimization . 169 x CONTENTS 6.7 Exercises . 169 7 SDP for Quantum Computation 171 7.1 Quantum Computation . 171 7.2 Completely Positive Map . 172 7.3 Channel Capacity Problem . 173 7.4 Quantum Interactive Proof System . 174 7.5 Notes . 175 7.6 Exercises . 175 8 Computational Issues 177 8.1 Presolver . 177 8.2 Linear System Solver . 179 8.2.1 Solving normal equation . 180 8.2.2 Solving augmented system . 183 8.2.3 Numerical phase . 184 8.2.4 Iterative method . 188 8.3 High-Order Method . 189 8.3.1 High-order predictor-corrector method . 189 8.3.2 Analysis of a high-order method . ..