Joseph Young Version: 1.2.1 Release Date: August 11, 2018 Manual licensed under a Creative www.optimojoe.com Commons Attribution-NoDerivatives 4.0 c 2018 by OptimoJoe. Some rights reserved International license Optizelle v1.2.1 OptimoJoe Joseph Young Contents 1 Introduction 3 1.1 Licensing..................................................4 1.2 Support...................................................4 1.3 Brief example................................................5 1.4 History................................................... 12 2 Installation 13 2.1 Downloading................................................ 13 2.2 Installing and Uninstalling......................................... 13 2.3 Dependencies................................................ 14 2.4 Building................................................... 14 2.5 Configuring................................................. 17 2.6 Platform Specific Configuration...................................... 28 3 Basic API 29 3.1 Import Optizelle.............................................. 30 3.2 Import or define the appropriate vector spaces............................. 30 3.3 Define the objective function....................................... 31 3.4 (Optional) Define the constraints..................................... 37 3.5 (Optional) Define the preconditioners.................................. 42 3.6 Create the optimization state....................................... 47 3.7 Set the optimization parameters..................................... 49 3.8 Accumulate the functions......................................... 51 3.9 Call the optimization solver........................................ 54 3.10 Extract the solution............................................ 55 3.11 Compile/run the program......................................... 57 3.11.1 C++................................................. 57 3.11.2 Python/MATLAB/Octave..................................... 57 4 Optimization parameters 58 5 Output 105 1 6 Advanced API 117 6.1 User-defined messaging........................................... 117 6.2 Handling errors............................................... 118 6.3 Customized vector spaces......................................... 120 6.4 Symmetric cone programming....................................... 130 6.5 State manipulation............................................. 142 6.6 Restarts................................................... 146 6.7 Caching Computations........................................... 159 7 Additional examples 173 7.1 Simple equality constrained........................................ 173 7.2 Simple inequality constrained....................................... 181 7.3 Simple constrained............................................. 188 7.4 Rosenbrock advanced API......................................... 197 7.5 Simple constrained advanced API..................................... 212 8 Algorithmic discussion 231 9 Licenses 238 9.1 Optizelle................................................... 238 9.2 JsonCpp................................................... 238 9.3 BLAS/LAPACK.............................................. 239 9.4 CMake.................................................... 240 9.5 WiX..................................................... 243 9.6 GCC..................................................... 244 9.7 TeX Live.................................................. 258 9.8 Python................................................... 260 9.9 NumPy................................................... 265 9.10 MATLAB.................................................. 267 9.11 JSONlab................................................... 267 9.12 Octave.................................................... 268 Index 281 2 1 Introduction Optizelle [op-tuh-zel] is an open source software library designed to solve general purpose nonlinear optimization problems of the form Unconstrained Equality Constrained min f(x) min f(x) x2X x2X st g(x) = 0 Inequality Constrained Constrained min f(x) min f(x) x2X x2X st h(x) 0 st g(x) = 0 h(x) 0 It features • State of the art algorithms { Unconstrained { steepest descent, preconditioned nonlinear-CG (Fletcher-Reeves, Polak-Ribiere, Hestenes- Stiefel), BFGS, Newton-CG, SR1, trust-region Newton, Barzilai-Borwein two-point approximation { Equality constrained { inexact composite-step SQP. { Inequality constrained { primal-dual interior point method for cone constraints (linear, second-order cone, and semidefinite), log-barrier method for cone constraints { Constrained { any combination of the above • Open source { Released under the 2-Clause BSD License { Free and ready to use with both open and closed sourced commercial codes • Multilanguage support { Interfaces to C++, MATLAB/Octave, and Python • Robust computations and repeatability { Can stop, archive, and restart the computation from any optimization iteration { Combined with the multilanguage support, the optimization can be started in one language and mi- grated to another. For example, archived optimization runs that started in Python can be migrated and completed in C++. • User-defined parallelism 3 { Fully compatible with OpenMP, MPI, or GPUs • Extensible linear algebra { Supports user-defined vector algebra and preconditioners { Enables sparse, dense, and matrix-free computations { Ability to define custom inner products and compatibility with preconditioners such as algebraic multi- grid make Optizelle well-suited for PDE constrained optimization • Sophisticated Control of the Optimization Algorithms { Allows the user to insert arbitrary code into the optimization algorithm, which enables custom heuris- tics to be embedded without modifying the source. For example, in signal processing applications, the optimization iterates could be run through a band-pass filter at the end of each optimization iteration. 1.1 Licensing Optizelle is copyrighted by OptimoJoe and licensed under the 2-Clause BSD License: BSD 2-Clause License Copyright 2013-2018 OptimoJoe. Copyright 2012-2013 Sandia Corporation. Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation, the U.S. Government retains certain rights in this software. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. In short, Optizelle is free to use in both open and closed sourced codes. If you do so, we ask that you provide a citation or link to http://www.optimojoe.com. 1.2 Support News, updates, and download information for Optizelle can be found at http://www.optimojoe.com/products/optizelle Our user forum can be found at http://forum.optimojoe.com Finally, if you are interested in paid support and consulting, please contact us at [email protected]. 4 1.3 Brief example In order to see a short example of Optizelle in action, consider the unconstrained minimization of the Rosenbrock function 2 2 2 min (1 − x1) + 100(x2 − x ) : 2 1 x2R In order to optimize this function, we use the following code and parameters, which generates the subsequent output. Language C++ Code // In this example, we setup and minimize the Rosenbrock function. #include <vector> #include <iostream> #include <string> #include <cstdlib> #include"optizelle/optizelle.h" #include"optizelle/vspaces.h" #include"optizelle/json.h" //---Objective0--- // Squares its input template<typename Real> Real sq(Real x){ return x*x; } // Define the Rosenbrock function where // //f(x,y)=(1-x)^2+100(y-x^2)^2 // struct Rosenbrock : public Optizelle::ScalarValuedFunction <double,Optizelle::Rm> { typedef Optizelle::Rm <double> X; // Evaluation of the Rosenbrock function double eval(X::Vector const & x) const{ return sq(1.-x[0])+100.*sq(x[1]-sq(x[0])); } // Gradient void grad( X::Vector const & x, X::Vector & grad ) const{ grad[0]=-400.*x[0]*(x[1]-sq(x[0]))-2.*(1.-x[0]); grad[1]=200.*(x[1]-sq(x[0])); } // Hessian-vector product void hessvec( X::Vector const & x, 5 X::Vector const & dx, X::Vector & H_dx ) const{ H_dx[0]=(1200.*sq(x[0])-400.*x[1]+2)*dx[0]-400.*x[0]*dx[1]; H_dx[1]=-400.*x[0]*dx[0]+200.*dx[1]; } }; //---Objective1--- //---Preconditioner0--- // Definea perfect preconditioner for the Hessian struct RosenHInv : public Optizelle::Operator <double,Optizelle::Rm,Optizelle::Rm> { public: typedef Optizelle::Rm <double> X; typedef X::Vector X_Vector; private: X_Vector& x; public: RosenHInv(X::Vector& x_) : x(x_) {} void eval(X_Vector const & dx,X_Vector