Fundamental Engineering Optimization Methods Second Edition Kamran Iqbal 1 Contents Preface .......................................................................................................................................................... 5 1 Engineering Design Optimization .......................................................................................................... 6 1.1 Introduction .................................................................................................................................. 6 1.2 Optimization Examples in Science and Engineering ..................................................................... 7 1.3 Notation ...................................................................................................................................... 13 2 Mathematical Preliminaries ................................................................................................................ 14 2.1 Set Definitions ............................................................................................................................. 14 2.2 Function Definitions .................................................................................................................... 15 2.3 Gradient Vector and Hessian Matrix ........................................................................................... 15 2.4 Taylor Series Approximation ....................................................................................................... 17 2.5 Properties of Convex Functions .................................................................................................. 18 2.6 Matrix Eigenvalues and Singular Values ..................................................................................... 19 2.7 Quadratic Function Forms .......................................................................................................... 20 2.8 Vector and Matrix Norms ........................................................................................................... 21 2.9 Linear Systems of Equations ....................................................................................................... 21 2.10 Linear Diophantine System of Equations .................................................................................... 23 2.11 Condition Number and Convergence Rates ................................................................................ 24 2.12 Newton’s Method for Solving Nonlinear Equations ................................................................... 25 2.13 Conjugate-Gradient Method for Solving Linear Equations ......................................................... 25 3 Graphical Optimization ....................................................................................................................... 27 3.1 Functional Minimization in One-Dimension ............................................................................... 27 3.2 Graphical Optimization in Two-Dimensions ............................................................................... 28 4 Mathematical Optimization ................................................................................................................ 35 4.1 The Optimization Problem .......................................................................................................... 35 4.2 Optimality criteria for the Unconstrained Problems .................................................................. 36 4.2.1 First Order Necessary Conditions (FONC) ............................................................................... 37 4.2.2 Second Order Conditions (SOC) .............................................................................................. 37 4.3 Optimality Criteria for the Constrained Problems ...................................................................... 39 4.3.3 Equality Constrained Problems ............................................................................................... 39 4.3.4 Inequality Constrained Problems ............................................................................................ 43 4.4 Optimality Criteria for General Optimization Problems ............................................................. 45 2 4.4.1 Optimality Criteria for Convex Optimization Problems .......................................................... 47 4.4.2 Second Order Conditions ........................................................................................................ 48 4.5 A Geometric Viewpoint ............................................................................................................... 50 4.6 Postoptimality Analysis ............................................................................................................... 51 4.7 Duality Theory ............................................................................................................................. 53 4.7.1 Local Duality ............................................................................................................................ 53 4.7.2 Strong and Weak Duality ........................................................................................................ 54 4.7.3 Duality in Convex Optimization Problems .............................................................................. 55 4.7.4 Separable Problems ................................................................................................................ 56 5 Linear Programming Methods ............................................................................................................ 58 5.1 The Standard LP Problem ............................................................................................................ 58 5.2 Solution to the LP Problem ......................................................................................................... 59 5.2.1 The Basic Solution to the LP Problem ..................................................................................... 60 5.3 The Simplex Method ................................................................................................................... 61 5.3.1 The Simplex Algorithm ............................................................................................................ 61 5.3.2 Tableau Implementation of the Simplex Algorithm ................................................................ 63 5.3.1 Obtaining an Initial BFS ........................................................................................................... 65 5.3.2 Final Tableau Properties ......................................................................................................... 70 5.4 Postoptimality Analysis ............................................................................................................... 70 5.5 Duality Theory for the LP Problems ............................................................................................ 74 5.5.1 Fundamental Duality Properties ............................................................................................. 75 5.5.2 The Dual Simplex Method ....................................................................................................... 76 5.5.3 Recovery of the Primal Solution.............................................................................................. 77 5.6 Optimality Conditions for LP Problems ....................................................................................... 80 5.6.1 KKT Conditions for LP Problems .............................................................................................. 81 5.6.2 A Geometric Viewpoint ........................................................................................................... 82 5.7 The Quadratic Programming Problem ........................................................................................ 83 5.7.1 Optimality Conditions for QP Problems .................................................................................. 83 5.7.2 The Dual QP Problem .............................................................................................................. 85 5.8 The Linear Complementary Problem .......................................................................................... 86 5.9 Non-Simplex Methods for Solving LP Problems.......................................................................... 90 6 Discrete Optimization ......................................................................................................................... 93 3 6.1 Discrete Optimization Problems ................................................................................................. 93 6.2 Solution Approaches to Discrete Problems ................................................................................ 94 6.3 Linear Programming Problems with Integral Coefficients .......................................................... 95 6.4 Binary Integer Programming Problems ....................................................................................... 95 6.5 Integer Programming Problems .................................................................................................. 97 6.5.1 The Branch and Bound Method .............................................................................................. 98 6.5.2 The Cutting Plane Method ...................................................................................................