DOCSLIB.ORG

Introduction to Optimization

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Introduction to Optimization Computational Optimization Overview Computational Optimization for Economists Sven Leyffer, Jorge More,´ and Todd Munson Mathematics and Computer Division Argonne National Laboratory 1. Introducion to Optimization [Mor´e] {leyffer,more,tmunson}@mcs.anl.gov 2. Continuous Optimization in AMPL [Munson] 3. Optimization Software [Leyffer] 4. Complementarity & Games [Munson] Institute for Computational Economics University of Chicago July 30 – August 9, 2007 Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Outline: Six Topics Introduction Unconstrained optimization Part I • Limited-memory variable metric methods Systems of Nonlinear Equations Introduction, Applications, and Formulations • Sparsity and Newton’s method Automatic Differentiation • Computing sparse Jacobians via graph coloring Constrained Optimization • All that you need to know about KKT conditions Solving optimization problems • Modeling languages: AMPL and GAMS • NEOS Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Topic 1: The Optimization Viewpoint View of Optimization from Applications Modeling Algorithms Software Automatic differentiation tools Application-specific languages 0.12 High-performance architectures 0.1 0.08 0.06 0.04 0.02 0 1 0.8 1 0.6 0.8 0.4 0.6 0.4 0.2 0.2 0 0 Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Classification of Constrained Optimization Problems Classification of Constrained Optimization Software • Formulation min {f(x): xl ≤ x ≤ xu, cl ≤ c(x) ≤ cu} • Interfaces: MATLAB, AMPL, GAMS • Second-order information options: • Differences • Number of variables n • Limited memory • Number of constraints m • Hessian-vector products • Number of linear constraints • Linear solvers • Direct solvers • Number of equality constraints n e • Iterative solvers • Number of degrees of freedom n − ne • Preconditioners 0 • Sparsity of c (x) = (∂icj(x)) • Partially separable problem formulation 2 2 Pm 2 • Sparsity of ∇xL(x, λ) = ∇ f(x) + k=1 ∇ ck(x)λk • Documentation • License Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Life-Cycles Saving Problem Constrained Optimization Software: IPOPT Maximize the utility • T Formulation X t β u(ct) min {f(x): xl ≤ x ≤ xu, c(x) = 0} t=1 where St are the saving, ct is consumption, wt are wages, and • Interfaces: AMPL • Second-order information options: S = (1 + r)S + w − c , 0 ≤ t < T t+1 t t+1 t+1 • Differences • Limited memory with r = 0.2 interest rate, β = 0.9, S = S = 0, and 0 T • Hessian-vector products u(c) = − exp(−c) • Direct solvers: MA27, MA57 • Partially separable problem formulation: None Assume that w = 1 for t < R and w = 0 for t ≥ R. t t • Documentation Question. What are the characteristics of the life-cycle problem? • License Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Life-Cycles Saving Problem: Results Topic 2: Unconstrained Optimization (R, T ) = (30, 50) (R, T ) = (60, 100) Augustin Louis Cauchy (August 21, 1789 – May 23, 1857) Question. Problem formulation to results: How long? Additional information at Mac Tutor www-history.mcs.st-andrews.ac.uk Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Unconstrained Optimization: Background Ginzburg-Landau Model n Given a continuously differentiable f : R 7→ R and Minimize the Gibbs free energy for a homogeneous superconductor n min {f(x): x ∈ R } Z 2 1 4 2 2 2 −|v(x)| + 2 |v(x)| + k[∇ − iA(x)] v(x)k + κ k(∇ × A)(x)k dx generate a sequence of iterates {xk} such that the gradient test D k∇f(xk)k ≤ τ 2 is eventually satisfied v : R → C (order parameter) 2 2 A : R → R (vector potential) n Theorem. If f : R 7→ R is continuously differentiable and bounded below, then there is a sequence {xk} such that Unconstrained problem. Non-convex function. Hessian is singular. lim k∇f(xk)k = 0. k→∞ Unique minimizer, but there is a saddle point. Exercise. Prove this result. Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Unconstrained Optimization Computing the Gradient Hand-coded gradients What can I use if the gradient ∇f(x) is not available? Generally efficient Error prone Geometry-based methods: Pattern search, Nelder-Mead, . The cost is usually less than 5 function evaluations Model-based methods: Quadratic, radial-basis models, . Difference approximations What can I use if the gradient ∇f(x) is available? Conjugate gradient methods f((x + hei) − f(x) ∂if(x) ≈ Limited-memory variable metric methods hi Variable metric methods Choice of hi may be problematic in the presence of noise. Costs n function evaluations 1/2 Accuracy is about the εf where εf is the noise level of f Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Cheap Gradient via Automatic Differentiation Line Search Methods Code generated by automatic differentiation tools A sequence of iterates {xk} is generated via Accurate to full precision xk+1 = xk + αkpk, For the reverse mode the cost is ΩT T {f(x)}. In theory, ΩT ≤ 5. where pk is a descent direction at xk, that is, For the reverse mode the memory is proportional to the number of ∇f(x )T p < 0, intermediate variables. k k and αk is determined by a line search along pk. Exercise Line searches Develop an order n code for computing the gradient of Geometry-based: Armijo, . n Y Model-based: Quadratics, cubic models, . f(x) = xk k=1 Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Powell-Wolfe Conditions on the Line Search Conjugate Gradient Algorithms Given 0 ≤ µ < η ≤ 1, require that Given a starting vector x0 generate iterates via T f(x + αp) ≤ f(x) + µ α∇f(xk) pk sufficent decrease xk+1 = xk + αkpk |∇f(x + αp)T p| ≤ η |∇f(x)T p| curvature condition pk+1 = −∇f(xk) + βkpk where αk is determined by a line search. Three reasonable choices of βk are (gk = ∇f(xk)): 2 FR kgk+1k βk = , Fletcher-Reeves kgkk PR hgk+1, gk+1 − gki βk = 2 , Polak-Rivi`ere kgkk PR+ PR βk = max βk , 0 , PR-plus Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Limited-Memory Variable-Metric Algorithms Recommendations Given a starting vector x0 generate iterates via But what algorithm should I use? xk+1 = xk − αkHk∇f(xk) If the gradient ∇f(x) is not available, then a model-based method is a reasonable choice. Methods based on quadratic interpolation where αk is determined by a line search. are currently the best choice. The matrix Hk is defined in terms of information gathered during the If the gradient ∇f(x) is available, then a limited-memory variable previous m iterations. metric method is likely to produce an approximate minimizer in the least number of gradient evaluations. H is positive definite. k If the Hessian is also available, then a state-of-the-art Storage of Hk requires 2mn locations. implementation of Newton’s method is likely to produce the best Computation of Hk∇f(xk) costs (8m + 1)n flops. results if the problem is large and sparse. Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Topic 3: Newton’s Method Motivation n n Give a continuously differentiable f : R 7→ R , solve f1(x) . f(x) = . = 0 fn(x) Linear models. The mapping defined by 0 Lk(s) = f(xk) + f (xk)s Sir Isaac Newton (January 4, 1643 – March 331, 1727) is a linear model of f near x , and thus it is sensible to choose s such Additional information at Mac Tutor k k that L (s ) = 0 provided x + s is near x . www-history.mcs.st-andrews.ac.uk k k k k k Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Newton’s Method Flow in a Channel Problem Given a starting point x0, Newton’s method generates iterates via Analyze the flow of a fluid during injection into a long vertical channel, assuming that the flow is modeled by the boundary value problem below, f 0(x )s = −f(x ), x = x + s . k k k k+1 k k where u is the potential function and R is the Reynolds number. Computational Issues How do we solve for sk? How do we handle a (nearly) singular f 0(x )? k u0000 = R (u0u00 − uu000) How do we enforce convergence if x is not near a solution? 0 u(0) = 0, u(1) = 1 0 How do we compute/approximate f (xk)? u0(0) = u0(1) = 0 How accurately do we solve for sk? Is the algorithm scale invariant? Is the algorithm mesh-invariant? Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Sparsity Topic 4: Automatic Differentiation Assume that the Jacobian matrix is sparse, and let ρi be the number of non-zeroes in the i-th row of f 0(x). 0 Sparse linear solvers can solve f (x)s = −f(x) in order ρA operations, where ρ = avg{ρ2}. A i Gottfried Wilhelm Leibniz (July 1, 1646 – November 14, 1716) Graph coloring techniques (see Topic 4) can compute or Additional information at Mac Tutor approximate the Jacobian matrix with ρM function evaluations www-history.mcs.st-andrews.ac.uk where ρM = max{ρi} Leyffer, Mor´e, and Munson Computational Optimization Leyffer, Mor´e, and Munson Computational Optimization Computing Gradients and Sparse Jacobians Structurally Orthogonal Columns n m Theorem. Given f : R 7→ R , automatic differentiation tools compute Structurally orthogonal columns do not have a nonzero in the same 0 f (x)v at a cost comparable to f(x) row position.
Load more

Recommended publications
  • Chapter 8 Constrained Optimization 2: Sequential Quadratic Programming, Interior Point and Generalized Reduced Gradient Methods
    Chapter 8: Constrained Optimization 2 CHAPTER 8 CONSTRAINED OPTIMIZATION 2: SEQUENTIAL QUADRATIC PROGRAMMING, INTERIOR POINT AND GENERALIZED REDUCED GRADIENT METHODS 8.1 Introduction In the previous chapter we examined the necessary and sufficient conditions for a constrained optimum. We did not, however, discuss any algorithms for constrained optimization. That is the purpose of this chapter. The three algorithms we will study are three of the most common. Sequential Quadratic Programming (SQP) is a very popular algorithm because of its fast convergence properties. It is available in MATLAB and is widely used. The Interior Point (IP) algorithm has grown in popularity the past 15 years and recently became the default algorithm in MATLAB. It is particularly useful for solving large-scale problems. The Generalized Reduced Gradient method (GRG) has been shown to be effective on highly nonlinear engineering problems and is the algorithm used in Excel. SQP and IP share a common background. Both of these algorithms apply the Newton- Raphson (NR) technique for solving nonlinear equations to the KKT equations for a modified version of the problem. Thus we will begin with a review of the NR method. 8.2 The Newton-Raphson Method for Solving Nonlinear Equations Before we get to the algorithms, there is some background we need to cover first. This includes reviewing the Newton-Raphson (NR) method for solving sets of nonlinear equations. 8.2.1 One equation with One Unknown The NR method is used to find the solution to sets of nonlinear equations. For example, suppose we wish to find the solution to the equation: xe+=2 x We cannot solve for x directly.
    [Show full text]
  • Quadratic Programming GIAN Short Course on Optimization: Applications, Algorithms, and Computation
    Quadratic Programming GIAN Short Course on Optimization: Applications, Algorithms, and Computation Sven Leyffer Argonne National Laboratory September 12-24, 2016 Outline 1 Introduction to Quadratic Programming Applications of QP in Portfolio Selection Applications of QP in Machine Learning 2 Active-Set Method for Quadratic Programming Equality-Constrained QPs General Quadratic Programs 3 Methods for Solving EQPs Generalized Elimination for EQPs Lagrangian Methods for EQPs 2 / 36 Introduction to Quadratic Programming Quadratic Program (QP) minimize 1 xT Gx + g T x x 2 T subject to ai x = bi i 2 E T ai x ≥ bi i 2 I; where n×n G 2 R is a symmetric matrix ... can reformulate QP to have a symmetric Hessian E and I sets of equality/inequality constraints Quadratic Program (QP) Like LPs, can be solved in finite number of steps Important class of problems: Many applications, e.g. quadratic assignment problem Main computational component of SQP: Sequential Quadratic Programming for nonlinear optimization 3 / 36 Introduction to Quadratic Programming Quadratic Program (QP) minimize 1 xT Gx + g T x x 2 T subject to ai x = bi i 2 E T ai x ≥ bi i 2 I; No assumption on eigenvalues of G If G 0 positive semi-definite, then QP is convex ) can find global minimum (if it exists) If G indefinite, then QP may be globally solvable, or not: If AE full rank, then 9ZE null-space basis Convex, if \reduced Hessian" positive semi-definite: T T ZE GZE 0; where ZE AE = 0 then globally solvable ... eliminate some variables using the equations 4 / 36 Introduction to Quadratic Programming Quadratic Program (QP) minimize 1 xT Gx + g T x x 2 T subject to ai x = bi i 2 E T ai x ≥ bi i 2 I; Feasible set may be empty ..
    [Show full text]
  • A Hybrid Global Optimization Method: the One-Dimensional Case Peiliang Xu
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Elsevier - Publisher Connector Journal of Computational and Applied Mathematics 147 (2002) 301–314 www.elsevier.com/locate/cam A hybrid global optimization method: the one-dimensional case Peiliang Xu Disaster Prevention Research Institute, Kyoto University, Uji, Kyoto 611-0011, Japan Received 20 February 2001; received in revised form 4 February 2002 Abstract We propose a hybrid global optimization method for nonlinear inverse problems. The method consists of two components: local optimizers and feasible point ÿnders. Local optimizers have been well developed in the literature and can reliably attain the local optimal solution. The feasible point ÿnder proposed here is equivalent to ÿnding the zero points of a one-dimensional function. It warrants that local optimizers either obtain a better solution in the next iteration or produce a global optimal solution. The algorithm by assembling these two components has been proved to converge globally and is able to ÿnd all the global optimal solutions. The method has been demonstrated to perform excellently with an example having more than 1 750 000 local minima over [ −106; 107].c 2002 Elsevier Science B.V. All rights reserved. Keywords: Interval analysis; Hybrid global optimization 1. Introduction Many problems in science and engineering can ultimately be formulated as an optimization (max- imization or minimization) model. In the Earth Sciences, we have tried to collect data in a best way and then to extract the information on, for example, the Earth’s velocity structures and=or its stress=strain state, from the collected data as much as possible.
    [Show full text]
  • Gauss-Newton SQP
    TEMPO Spring School: Theory and Numerics for Nonlinear Model Predictive Control Exercise 3: Gauss-Newton SQP J. Andersson M. Diehl J. Rawlings M. Zanon University of Freiburg, March 27, 2015 Gauss-Newton sequential quadratic programming (SQP) In the exercises so far, we solved the NLPs with IPOPT. IPOPT is a popular open-source primal- dual interior point code employing so-called filter line-search to ensure global convergence. Other NLP solvers that can be used from CasADi include SNOPT, WORHP and KNITRO. In the following, we will write our own simple NLP solver implementing sequential quadratic programming (SQP). (0) (0) Starting from a given initial guess for the primal and dual variables (x ; λg ), SQP solves the NLP by iteratively computing local convex quadratic approximations of the NLP at the (k) (k) current iterate (x ; λg ) and solving them by using a quadratic programming (QP) solver. For an NLP of the form: minimize f(x) x (1) subject to x ≤ x ≤ x; g ≤ g(x) ≤ g; these quadratic approximations take the form: 1 | 2 (k) (k) (k) (k) | minimize ∆x rxL(x ; λg ; λx ) ∆x + rxf(x ) ∆x ∆x 2 subject to x − x(k) ≤ ∆x ≤ x − x(k); (2) @g g − g(x(k)) ≤ (x(k)) ∆x ≤ g − g(x(k)); @x | | where L(x; λg; λx) = f(x) + λg g(x) + λx x is the so-called Lagrangian function. By solving this (k) (k+1) (k) (k+1) QP, we get the (primal) step ∆x := x − x as well as the Lagrange multipliers λg (k+1) and λx .
    [Show full text]
  • Geometric GSI’19 Science of Information Toulouse, 27Th - 29Th August 2019
    ALEAE GEOMETRIA Geometric GSI’19 Science of Information Toulouse, 27th - 29th August 2019 // Program // GSI’19 Geometric Science of Information On behalf of both the organizing and the scientific committees, it is // Welcome message our great pleasure to welcome all delegates, representatives and participants from around the world to the fourth International SEE from GSI’19 chairmen conference on “Geometric Science of Information” (GSI’19), hosted at ENAC in Toulouse, 27th to 29th August 2019. GSI’19 benefits from scientific sponsor and financial sponsors. The 3-day conference is also organized in the frame of the relations set up between SEE and scientific institutions or academic laboratories: ENAC, Institut Mathématique de Bordeaux, Ecole Polytechnique, Ecole des Mines ParisTech, INRIA, CentraleSupélec, Institut Mathématique de Bordeaux, Sony Computer Science Laboratories. We would like to express all our thanks to the local organizers (ENAC, IMT and CIMI Labex) for hosting this event at the interface between Geometry, Probability and Information Geometry. The GSI conference cycle has been initiated by the Brillouin Seminar Team as soon as 2009. The GSI’19 event has been motivated in the continuity of first initiatives launched in 2013 at Mines PatisTech, consolidated in 2015 at Ecole Polytechnique and opened to new communities in 2017 at Mines ParisTech. We mention that in 2011, we // Frank Nielsen, co-chair Ecole Polytechnique, Palaiseau, France organized an indo-french workshop on “Matrix Information Geometry” Sony Computer Science Laboratories, that yielded an edited book in 2013, and in 2017, collaborate to CIRM Tokyo, Japan seminar in Luminy TGSI’17 “Topoplogical & Geometrical Structures of Information”.
    [Show full text]
  • Implementing Customized Pivot Rules in COIN-OR's CLP with Python
    Cahier du GERAD G-2012-07 Customizing the Solution Process of COIN-OR's Linear Solvers with Python Mehdi Towhidi1;2 and Dominique Orban1;2 ? 1 Department of Mathematics and Industrial Engineering, Ecole´ Polytechnique, Montr´eal,QC, Canada. 2 GERAD, Montr´eal,QC, Canada. [email protected], [email protected] Abstract. Implementations of the Simplex method differ only in very specific aspects such as the pivot rule. Similarly, most relaxation methods for mixed-integer programming differ only in the type of cuts and the exploration of the search tree. Implementing instances of those frame- works would therefore be more efficient if linear and mixed-integer pro- gramming solvers let users customize such aspects easily. We provide a scripting mechanism to easily implement and experiment with pivot rules for the Simplex method by building upon COIN-OR's open-source linear programming package CLP. Our mechanism enables users to implement pivot rules in the Python scripting language without explicitly interact- ing with the underlying C++ layers of CLP. In the same manner, it allows users to customize the solution process while solving mixed-integer linear programs using the CBC and CGL COIN-OR packages. The Cython pro- gramming language ensures communication between Python and COIN- OR libraries and activates user-defined customizations as callbacks. For illustration, we provide an implementation of a well-known pivot rule as well as the positive edge rule|a new rule that is particularly efficient on degenerate problems, and demonstrate how to customize branch-and-cut node selection in the solution of a mixed-integer program.
    [Show full text]
  • A Sequential Quadratic Programming Algorithm with an Additional Equality Constrained Phase
    A Sequential Quadratic Programming Algorithm with an Additional Equality Constrained Phase Jos´eLuis Morales∗ Jorge Nocedal † Yuchen Wu† December 28, 2008 Abstract A sequential quadratic programming (SQP) method is presented that aims to over- come some of the drawbacks of contemporary SQP methods. It avoids the difficulties associated with indefinite quadratic programming subproblems by defining this sub- problem to be always convex. The novel feature of the approach is the addition of an equality constrained phase that promotes fast convergence and improves performance in the presence of ill conditioning. This equality constrained phase uses exact second order information and can be implemented using either a direct solve or an iterative method. The paper studies the global and local convergence properties of the new algorithm and presents a set of numerical experiments to illustrate its practical performance. 1 Introduction Sequential quadratic programming (SQP) methods are very effective techniques for solv- ing small, medium-size and certain classes of large-scale nonlinear programming problems. They are often preferable to interior-point methods when a sequence of related problems must be solved (as in branch and bound methods) and more generally, when a good estimate of the solution is available. Some SQP methods employ convex quadratic programming sub- problems for the step computation (typically using quasi-Newton Hessian approximations) while other variants define the Hessian of the SQP model using second derivative informa- tion, which can lead to nonconvex quadratic subproblems; see [27, 1] for surveys on SQP methods. ∗Departamento de Matem´aticas, Instituto Tecnol´ogico Aut´onomode M´exico, M´exico. This author was supported by Asociaci´onMexicana de Cultura AC and CONACyT-NSF grant J110.388/2006.
    [Show full text]
  • A Parallel Quadratic Programming Algorithm for Model Predictive Control
    MITSUBISHI ELECTRIC RESEARCH LABORATORIES http://www.merl.com A Parallel Quadratic Programming Algorithm for Model Predictive Control Brand, M.; Shilpiekandula, V.; Yao, C.; Bortoff, S.A. TR2011-056 August 2011 Abstract In this paper, an iterative multiplicative algorithm is proposed for the fast solution of quadratic programming (QP) problems that arise in the real-time implementation of Model Predictive Control (MPC). The proposed algorithm–Parallel Quadratic Programming (PQP)–is amenable to fine-grained parallelization. Conditions on the convergence of the PQP algorithm are given and proved. Due to its extreme simplicity, even serial implementations offer considerable speed advantages. To demonstrate, PQP is applied to several simulation examples, including a stand- alone QP problem and two MPC examples. When implemented in MATLAB using single-thread computations, numerical simulations of PQP demonstrate a 5 - 10x speed-up compared to the MATLAB active-set based QP solver quadprog. A parallel implementation would offer a further speed-up, linear in the number of parallel processors. World Congress of the International Federation of Automatic Control (IFAC) This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc.
    [Show full text]
  • 1 Linear Programming
    ORF 523 Lecture 9 Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Any typos should be emailed to a a [email protected]. In this lecture, we see some of the most well-known classes of convex optimization problems and some of their applications. These include: • Linear Programming (LP) • (Convex) Quadratic Programming (QP) • (Convex) Quadratically Constrained Quadratic Programming (QCQP) • Second Order Cone Programming (SOCP) • Semidefinite Programming (SDP) 1 Linear Programming Definition 1. A linear program (LP) is the problem of optimizing a linear function over a polyhedron: min cT x T s.t. ai x ≤ bi; i = 1; : : : ; m; or written more compactly as min cT x s.t. Ax ≤ b; for some A 2 Rm×n; b 2 Rm: We'll be very brief on our discussion of LPs since this is the central topic of ORF 522. It suffices to say that LPs probably still take the top spot in terms of ubiquity of applications. Here are a few examples: • A variety of problems in production planning and scheduling 1 • Exact formulation of several important combinatorial optimization problems (e.g., min-cut, shortest path, bipartite matching) • Relaxations for all 0/1 combinatorial programs • Subroutines of branch-and-bound algorithms for integer programming • Relaxations for cardinality constrained (compressed sensing type) optimization prob- lems • Computing Nash equilibria in zero-sum games • ::: 2 Quadratic Programming Definition 2. A quadratic program (QP) is an optimization problem with a quadratic ob- jective and linear constraints min xT Qx + qT x + c x s.t. Ax ≤ b: Here, we have Q 2 Sn×n, q 2 Rn; c 2 R;A 2 Rm×n; b 2 Rm: The difficulty of this problem changes drastically depending on whether Q is positive semidef- inite (psd) or not.
    [Show full text]
  • Arxiv:1804.07332V1 [Math.OC] 19 Apr 2018
    Juniper: An Open-Source Nonlinear Branch-and-Bound Solver in Julia Ole Kr¨oger,Carleton Coffrin, Hassan Hijazi, Harsha Nagarajan Los Alamos National Laboratory, Los Alamos, New Mexico, USA Abstract. Nonconvex mixed-integer nonlinear programs (MINLPs) rep- resent a challenging class of optimization problems that often arise in engineering and scientific applications. Because of nonconvexities, these programs are typically solved with global optimization algorithms, which have limited scalability. However, nonlinear branch-and-bound has re- cently been shown to be an effective heuristic for quickly finding high- quality solutions to large-scale nonconvex MINLPs, such as those arising in infrastructure network optimization. This work proposes Juniper, a Julia-based open-source solver for nonlinear branch-and-bound. Leverag- ing the high-level Julia programming language makes it easy to modify Juniper's algorithm and explore extensions, such as branching heuris- tics, feasibility pumps, and parallelization. Detailed numerical experi- ments demonstrate that the initial release of Juniper is comparable with other nonlinear branch-and-bound solvers, such as Bonmin, Minotaur, and Knitro, illustrating that Juniper provides a strong foundation for further exploration in utilizing nonlinear branch-and-bound algorithms as heuristics for nonconvex MINLPs. 1 Introduction Many of the optimization problems arising in engineering and scientific disci- plines combine both nonlinear equations and discrete decision variables. Notable examples include the blending/pooling problem [1,2] and the design and opera- tion of power networks [3,4,5] and natural gas networks [6]. All of these problems fall into the class of mixed-integer nonlinear programs (MINLPs), namely, minimize: f(x; y) s.t.
    [Show full text]
  • Practical Implementation of the Active Set Method for Support Vector Machine Training with Semi-Definite Kernels
    PRACTICAL IMPLEMENTATION OF THE ACTIVE SET METHOD FOR SUPPORT VECTOR MACHINE TRAINING WITH SEMI-DEFINITE KERNELS by CHRISTOPHER GARY SENTELLE B.S. of Electrical Engineering University of Nebraska-Lincoln, 1993 M.S. of Electrical Engineering University of Nebraska-Lincoln, 1995 A dissertation submitted in partial fulfilment of the requirements for the degree of Doctor of Philosophy in the Department of Electrical Engineering and Computer Science in the College of Engineering and Computer Science at the University of Central Florida Orlando, Florida Spring Term 2014 Major Professor: Michael Georgiopoulos c 2014 Christopher Sentelle ii ABSTRACT The Support Vector Machine (SVM) is a popular binary classification model due to its superior generalization performance, relative ease-of-use, and applicability of kernel methods. SVM train- ing entails solving an associated quadratic programming (QP) that presents significant challenges in terms of speed and memory constraints for very large datasets; therefore, research on numer- ical optimization techniques tailored to SVM training is vast. Slow training times are especially of concern when one considers that re-training is often necessary at several values of the models regularization parameter, C, as well as associated kernel parameters. The active set method is suitable for solving SVM problem and is in general ideal when the Hessian is dense and the solution is sparse–the case for the `1-loss SVM formulation. There has recently been renewed interest in the active set method as a technique for exploring the entire SVM regular- ization path, which has been shown to solve the SVM solution at all points along the regularization path (all values of C) in not much more time than it takes, on average, to perform training at a sin- gle value of C with traditional methods.
    [Show full text]
  • Treatment of Degeneracy in Linear and Quadratic Programming
    UNIVERSITE´ DE MONTREAL´ TREATMENT OF DEGENERACY IN LINEAR AND QUADRATIC PROGRAMMING MEHDI TOWHIDI DEPARTEMENT´ DE MATHEMATIQUES´ ET DE GENIE´ INDUSTRIEL ECOLE´ POLYTECHNIQUE DE MONTREAL´ THESE` PRESENT´ EE´ EN VUE DE L'OBTENTION DU DIPLOME^ DE PHILOSOPHIÆ DOCTOR (MATHEMATIQUES´ DE L'INGENIEUR)´ AVRIL 2013 c Mehdi Towhidi, 2013. UNIVERSITE´ DE MONTREAL´ ECOLE´ POLYTECHNIQUE DE MONTREAL´ Cette th`eseintitul´ee : TREATMENT OF DEGENERACY IN LINEAR AND QUADRATIC PROGRAMMING pr´esent´ee par : TOWHIDI Mehdi en vue de l'obtention du dipl^ome de : Philosophiæ Doctor a ´et´ed^ument accept´eepar le jury d'examen constitu´ede : M. EL HALLAOUI Issma¨ıl, Ph.D., pr´esident M. ORBAN Dominique, Doct. Sc., membre et directeur de recherche M. SOUMIS Fran¸cois, Ph.D., membre et codirecteur de recherche M. BASTIN Fabian, Doct., membre M. BIRBIL Ilker, Ph.D., membre iii To Maryam To Ahmad, Mehran, Nima, and Afsaneh iv ACKNOWLEDGEMENTS Over the past few years I have had the great opportunity to learn from and work with professor Dominique Orban. I would like to thank him sincerely for his exceptional support and patience. His influence on me goes beyond my academic life. I would like to deeply thank professor Fran¸cois Soumis who accepted me for the PhD in the first place, and has since provided me with his remarkable insight. I am extremely grateful to professor Koorush Ziarati for introducing me to the world of optimization and being the reason why I came to Montreal to pursue my dream. I am eternally indebted to my caring and supportive family ; my encouraging mother, Mehran ; my thoughtful brother, Nima ; my affectionate sister, Afsaneh ; and my inspiration of all time and father, Ahmad.
    [Show full text]