Optimization Toolbox For Use with MATLAB® Thomas Coleman Mary Ann Branch Andrew Grace Computation Visualization Programming User’s Guide Version 2 How to Contact The MathWorks: 508-647-7000 Phone 508-647-7001 Fax The MathWorks, Inc. Mail 24 Prime Park Way Natick, MA 01760-1500 http://www.mathworks.com Web ftp.mathworks.com Anonymous FTP server comp.soft-sys.matlab Newsgroup [email protected] Technical support [email protected] Product enhancement suggestions [email protected] Bug reports [email protected] Documentation error reports [email protected] Subscribing user registration [email protected] Order status, license renewals, passcodes [email protected] Sales, pricing, and general information Optimization Toolbox User’s Guide COPYRIGHT 1990 - 1999 by The MathWorks, Inc. The software described in this document is furnished under a license agreement. The software may be used or copied only under the terms of the license agreement. No part of this manual may be photocopied or repro- duced in any form without prior written consent from The MathWorks, Inc. U.S. GOVERNMENT: If Licensee is acquiring the Programs on behalf of any unit or agency of the U.S. Government, the following shall apply: (a) For units of the Department of Defense: the Government shall have only the rights specified in the license under which the commercial computer software or commercial software documentation was obtained, as set forth in subparagraph (a) of the Rights in Commercial Computer Software or Commercial Software Documentation Clause at DFARS 227.7202-3, therefore the rights set forth herein shall apply; and (b) For any other unit or agency: NOTICE: Notwithstanding any other lease or license agreement that may pertain to, or accompany the delivery of, the computer software and accompanying documentation, the rights of the Government regarding its use, reproduction, and disclo- sure are as set forth in Clause 52.227-19 (c)(2) of the FAR. MATLAB, Simulink, Stateflow, Handle Graphics, and Real-Time Workshop are registered trademarks, and Target Language Compiler is a trademark of The MathWorks, Inc. Other product or brand names are trademarks or registered trademarks of their respective holders. Printing History: November 1990 First printing December 1996 Second printing for MATLAB 5 January 1999 Third printing Revised for Version 2 (Release 11) Contents Before You Begin What Is the Optimization Toolbox? ..................... viii New Features in Version 2 .............................. ix Large-ScaleAlgorithms................................. ix Function Names and Calling Syntax ...................... ix How to Use this Manual ................................ xi Installing the Toolbox ..................................xii Typographical Conventions ........................... xiii Matrix,Vector,andScalarNotation ..................... xiii Tutorial 1 Introduction ......................................... 1-2 Overview ........................................... 1-4 Medium-Scale Examples ............................... 1-6 UnconstrainedExample ............................... 1-6 NonlinearInequalityConstrainedExample ............... 1-8 ConstrainedExamplewithBounds ...................... 1-9 ConstrainedExamplewithGradients................... 1-11 GradientCheck:AnalyticVersusNumeric ............... 1-13 EqualityConstrainedExample ........................ 1-14 Maximization....................................... 1-15 GreaterthanZeroConstraints......................... 1-15 AdditionalArguments:AvoidingGlobalVariables......... 1-15 MultiobjectiveExamples.............................. 1-16 i Large-Scale Examples ................................ 1-29 NonlinearEquationswithJacobian..................... 1-33 Nonlinear Equations with Jacobian Sparsity Pattern . 1-35 Nonlinear Least-Squares with FullJacobianSparsityPattern ....................... 1-37 Nonlinear Minimization with Gradient and Hessian . 1-39 Nonlinear Minimization with Gradient and Hessian Sparsity Pattern ............................................ 1-40 Nonlinear Minimization with Bound Constraints and Banded Preconditioner ...................................... 1-43 Nonlinear Minimization with Equality Constraints . 1-47 Quadratic Minimization with Bound Constraints . 1-49 Linear Least-Squares with Bound Constraints . 1-50 Linear Programming with Equalities and Inequalities . 1-52 Linear Programming with Dense Columns in the Equalities . 1-53 Default Parameter Settings ........................... 1-56 ChangingtheDefaultSettings......................... 1-56 Displaying Iterative Output ........................... 1-58 Output Headings: Medium-Scale Algorithms . 1-58 OutputHeadings:Large-ScaleAlgorithms ............... 1-60 Optimization of Inline Objects Instead of M-Files ....... 1-63 Practicalities ........................................ 1-65 Converting Your Code to Version 2.0 Syntax ............ 1-69 Usingoptimsetandoptimget .......................... 1-70 NewCallingSequences............................... 1-70 ExampleofConvertingfromconstrtofmincon ............ 1-80 ii Contents Introduction to Algorithms 2 Parametric Optimization .............................. 2-2 Unconstrained Optimization ........................... 2-4 Quasi-NewtonMethods................................ 2-5 LineSearch.......................................... 2-8 Quasi-Newton Implementation ........................ 2-11 Hessian Update . .............................. 2-11 LineSearchProcedures............................... 2-11 Least Squares Optimization ........................... 2-17 Gauss-NewtonMethod ............................... 2-18 Levenberg-MarquardtMethod ......................... 2-19 Nonlinear Least Squares Implementation .............. 2-21 Gauss-NewtonImplementation ........................ 2-21 Levenberg-MarquardtImplementation .................. 2-21 Constrained Optimization ............................ 2-23 SequentialQuadraticProgramming(SQP) ............... 2-24 QPSubproblem ..................................... 2-24 SQP Implementation ................................. 2-27 UpdatingtheHessianMatrix.......................... 2-27 QuadraticProgrammingSolution....................... 2-28 LineSearchandMeritFunction........................ 2-31 Multiobjective Optimization .......................... 2-33 Introduction ........................................ 2-33 GoalAttainmentMethod.............................. 2-39 Algorithm Improvements for Goal Attainment Method . 2-40 iii Review .............................................. 2-43 References .......................................... 2-44 Large-Scale Algorithms 3 Large-Scale Optimization .............................. 3-2 Trust Region Methods for Nonlinear Minimization ....... 3-3 Preconditioned Conjugate Gradients ................... 3-6 Linearly Constrained Problems ........................ 3-8 Linear Equality Constraints . .................. 3-8 BoxConstraints...................................... 3-8 Nonlinear Least-Squares .............................. 3-11 Quadratic Programming .............................. 3-12 Linear Least-Squares ................................. 3-13 Large-Scale Linear Programming ..................... 3-14 MainAlgorithm ..................................... 3-14 Preprocessing....................................... 3-17 References .......................................... 3-18 iv Contents Reference 4 Optimization Functions ............................... 4-3 Minimization ........................................ 4-3 EquationSolving ..................................... 4-3 Least-Squares(CurveFitting)........................... 4-4 Utility . ............................... 4-4 DemonstrationsofLarge-ScaleMethods .................. 4-4 DemonstrationsofMedium-ScaleMethods ................ 4-5 Function Arguments .................................. 4-6 v vi Contents Before You Begin Before You Begin What Is the Optimization Toolbox? The Optimization Toolbox is a collection of functions that extend the capability of the MATLAB® numeric computing environment. The toolbox includes routines for many types of optimization including: • Unconstrained nonlinear minimization • Constrained nonlinear minimization, including goal attainment problems, minimax problems, and semi-infinite minimization problems • Quadratic and linear programming • Nonlinear least squares and curve-fitting • Nonlinear system of equation solving • Constrained linear least squares Specialized algorithms for large-scale (sparse) problems are also available (see “Large-Scale Algorithms” in the next section “New Features in Version 2”). All of the toolbox functions are MATLAB M-files, made up of MATLAB statements that implement specialized optimization algorithms. You can view the MATLAB code for these functions using the statement: type function_name You can extend the capabilities of the Optimization Toolbox by writing your own M-files, or by using the toolbox in combination with other toolboxes, or with MATLAB, or Simulink®. viii New Features in Version 2 New Features in Version 2 Version 2 of the Optimization Toolbox offers a number of advances over previous versions. Large-Scale Algorithms The focus of this version is new algorithms for solving large-scale problems, including • Linear programming • Nonlinear least squares with bound constraints • Nonlinear system of equation solving • Unconstrained nonlinear minimization • Nonlinear minimization with bound constraints • Nonlinear minimization with linear equalities • Quadratic problems with