Introduction to Nonsmooth Optimization Adil Bagirov • Napsu Karmitsa Marko M
Total Page:16
File Type:pdf, Size:1020Kb
Introduction to Nonsmooth Optimization Adil Bagirov • Napsu Karmitsa Marko M. Mäkelä Introduction to Nonsmooth Optimization Theory, Practice and Software 123 Adil Bagirov Napsu Karmitsa School of Information Technology and Marko M. Mäkelä Mathematical Sciences, Centre for Department of Mathematics and Statistics Informatics and Applied Optimization University of Turku University of Ballarat Turku Ballarat, VC Finland Australia ISBN 978-3-319-08113-7 ISBN 978-3-319-08114-4 (eBook) DOI 10.1007/978-3-319-08114-4 Library of Congress Control Number: 2014943114 Springer Cham Heidelberg New York Dordrecht London Ó Springer International Publishing Switzerland 2014 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com) Preface Nonsmooth optimization refers to the general problem of minimizing (or maximizing) functions that are typically not differentiable at their minimizers (maximizers). These kinds of functions can be found in many applied fields, for example in image denoising, optimal control, neural network training, data mining, economics, and computational chemistry and physics. Since classical theory of optimization presumes certain differentiability and strong regularity assumptions for the functions to be optimized, it cannot be directly utilized. The aim of this book is to provide an easy-to-read introduction to the theory of nonsmooth optimization and also to present the current state of numerical nonsmooth opti- mization. In addition, the most common cases where nonsmoothness is involved in practical computations are introduced. In preparing this book, all efforts have been made to ensure that it is self-contained. The book is organized into three parts: Part I deals with nonsmooth optimi- zation theory. We first provide an easy-to-read introduction to convex and non- convex analysis with many numerical examples and illustrative figures. Then we discuss nonsmooth optimality conditions from both analytical and geometrical viewpoints. We also generalize the concept of convexity for nonsmooth functions. At the end of the part, we give brief surveys of different generalizations of sub- differentials and approximations to subdifferentials. In Part II, we consider nonsmooth optimization problems. First, we introduce some real-life nonsmooth optimization problems, for instance, the molecular distance geometry problem, protein structural alignment, data mining, hemivari- ational inequalities, the power unit-commitment problem, image restoration, and the nonlinear income tax problem. Then we discuss some formulations which lead to nonsmooth optimization problems even though the original problem is smooth (continuously differentiable). Examples here include exact penalty formulations. We also represent the maximum eigenvalue problem, which is an important component of many engineering design problems and graph theoretical applica- tions. We refer to these problems as semi-academic problems. Finally, a com- prehensive list of test problems—that is, academic problems—used in nonsmooth optimization is given. v vi Preface Part III is a guide to nonsmooth optimization software. First, we give short descriptions and the pseudo-codes of the most commonly used methods for non- smooth optimization. These include different subgradient methods, cutting plane methods, bundle methods, and the gradient sampling method, as well as some hybrid methods and discrete gradient methods. In addition, we introduce some common ways of dealing with constrained nonsmooth optimization problems. We also compare implementations of different nonsmooth optimization methods for solving unconstrained problems. At the end of the part, we provide a table enabling the quick selection of suitable software for different types of nonsmooth optimi- zation problems. The book is ideal for anyone teaching or attending courses in nonsmooth optimization. As a comprehensible introduction to the field, it is also well-suited for self-access learning for practitioners who know the basics of optimization. Furthermore, it can serve as a reference text for anyone—including experts— dealing with nonsmooth optimization. Acknowledgments: First of all, we would like to thank Prof. Herskovits for giving the reason to write a book on nonsmooth analysis and optimization: He once asked why the subject concerned is elusive in all the books and articles dealing with it, and pointed out the lack of an extensive elementary book. In addition, we would like to acknowledge Prof. Kuntsevich and Kappel for providing Shor’s r-algorithm on their web site as well as Prof. Lukšan and Vlcˇek for providing the bundle-Newton algorithm. We are also grateful to the following colleagues and students, all of whom have influenced the content of the book: Annabella Astorino, Ville-Pekka Eronen, Antonio Fuduli, Manlio Gaudioso, Kaisa Joki, Sami Kankaanpää, Refail Kasimbeyli, Yury Nikulin, Gurkan Ozturk, Rami Rakkolainen, Julien Ugon, Dean Webb and Outi Wilppu. The work was financially supported by the University of Turku (Finland), Magnus Ehrnrooth Foundation, Turku University Foundation, Federation Uni- versity Australia, and Australian Research Council. Ballarat, April 2014 Adil Bagirov Turku Napsu Karmitsa Marko M. Mäkelä Contents Part I Nonsmooth Analysis and Optimization 1 Theoretical Background ............................... 3 1.1 Notations and Definitions . 3 1.2 Matrix Calculus . 4 1.3 Hausdorff Metrics. 6 1.4 Functions and Derivatives . 6 2 Convex Analysis ..................................... 11 2.1 Convex Sets. 11 2.1.1 Convex Hulls . 13 2.1.2 Separating and Supporting Hyperplanes . 14 2.1.3 Convex Cones . 22 2.1.4 Contingent and Normal Cones . 27 2.2 Convex Functions. 32 2.2.1 Level Sets and Epigraphs . 37 2.2.2 Subgradients and Directional Derivatives . 38 2.2.3 ε-Subdifferentials . 47 2.3 Links Between Geometry and Analysis . 49 2.3.1 Epigraphs . 49 2.3.2 Level Sets . 51 2.3.3 Distance Function. 53 2.4 Summary. 57 Exercises . 57 3 Nonconvex Analysis .................................. 61 3.1 Generalization of Derivatives . 61 3.1.1 Generalized Directional Derivative . 61 3.1.2 Generalized Subgradients . 64 3.1.3 ε-Subdifferentials . 73 3.1.4 Generalized Jacobians . 76 vii viii Contents 3.2 Subdifferential Calculus . 77 3.2.1 Subdifferential Regularity . 77 3.2.2 Subderivation Rules . 79 3.3 Nonconvex Geometry . 94 3.3.1 Tangent and Normal Cones . 94 3.3.2 Epigraphs and Level Sets . 98 3.3.3 Cones of Feasible Directions . 102 3.4 Other Generalized Subdifferentials . 104 3.4.1 Quasidifferentials . 104 3.4.2 Relationship Between Quasidifferential and Clarke Subdifferential . 109 3.4.3 Codifferentials . 110 3.4.4 Basic and Singular Subdifferentials. 112 3.5 Summary. 112 Exercises . 113 4 Optimality Conditions ................................ 117 4.1 Unconstrained Optimization. 117 4.1.1 Analytical Optimality Conditions . 118 4.1.2 Descent Directions . 120 4.2 Geometrical Constraints . 121 4.2.1 Geometrical Optimality Conditions. 122 4.2.2 Mixed Optimality Conditions . 123 4.3 Analytical Constraints . 126 4.3.1 Geometrical Optimality Conditions. 127 4.3.2 Fritz John Optimality Conditions . 128 4.3.3 Karush-Kuhn-Tucker Optimality Conditions . 130 4.4 Optimality Conditions Using Quasidifferentials . 134 4.5 Summary. 135 Exercises . 136 5 Generalized Convexities ............................... 139 5.1 Generalized Pseudoconvexity . 139 5.2 Generalized Quasiconvexity. 150 5.3 Relaxed Optimality Conditions. 161 5.3.1 Unconstrained Optimization. 162 5.3.2 Geometrical Constraints . 163 5.3.3 Analytical Constraints . 164 5.4 Summary. 166 Exercises . 167 6 Approximations of Subdifferentials ....................... 169 6.1 Continuous Approximations of Subdifferential . 169 6.2 Discrete Gradient and Approximation of Subgradients . 175 Contents ix 6.3 Piecewise Partially Separable