Johannes O. Royset Roger J-B Wets An Optimization Primer April 16, 2021 Springer Nature to our families Contents Preface ................................................... ............. vi Acknowledgements ................................................... .. xi 1 PRELUDE ................................................... ...... 1 1.A The Mathematical Curtain Rise .................................... 1 1.B Data Smoothing ................................................ 5 1.C Optimization under Uncertainty ................................... 7 1.D Convex Analysis ................................................ 10 1.E Estimation and Classification ..................................... 25 1.F Gradient Descent Method ........................................ 28 1.G Newton’s Method ............................................... 34 1.H Acceleration and Regularization ................................... 39 1.I Quasi-Newton Methods .......................................... 45 1.J Coordinate Descent Algorithms ................................... 48 2 CONVEX OPTIMIZATION ......................................... 52 2.A Formulations ................................................... 52 2.B Subderivatives and Subgradients ................................... 60 2.C Subgradient Calculus ............................................ 68 2.D Proximal Gradient Methods ....................................... 75 2.E Linear Constraints ............................................... 81 2.F Karush-Kuhn-Tucker Condition ................................... 86 2.G Interior-Point Method ............................................ 89 2.H Support Vector Machines ......................................... 92 2.I Subgradient Method ............................................. 96 2.J Conic Constraints ............................................... 101 2.K Polyhedral Analysis ............................................. 106 3 OPTIMIZATION UNDER UNCERTAINTY .......................... 116 3.A Product Mix Optimization ........................................ 116 3.B Expectation Functions ........................................... 120 ii Contents iii 3.C Risk Modeling .................................................. 124 3.D Models of Uncertainty ........................................... 133 3.E Risk-Adaptive Design ............................................ 143 3.F Optimality in Stochastic Optimization .............................. 147 3.G Stochastic Gradient Descent ...................................... 153 3.H Simple Recourse Problems ....................................... 160 3.I Control of Water Pollution ........................................ 168 3.J Linear Recourse Problems ........................................ 170 3.K Network Capacity Expansion ..................................... 177 4 MINIMIZATION PROBLEMS ...................................... 179 4.A Formulations ................................................... 179 4.B Network Design and Operation .................................... 189 4.C Epigraphical Approximation Algorithm ............................. 192 4.D Constraint Softening ............................................. 205 4.E Set Analysis ................................................... 207 4.F Robotic Path Planning ........................................... 213 4.G Tangent and Normal Cones I ...................................... 216 4.H Tangent and Normal Cones II ..................................... 229 4.I Subdifferentiability .............................................. 235 4.J Optimality Conditions ........................................... 249 4.K SQP and Interior-Point Methods ................................... 257 5 PERTURBATION AND DUALITY .................................. 261 5.A Rockafellians ................................................... 261 5.B Quantitative Stability ............................................ 272 5.C Lagrangians and Dual Problems ................................... 280 5.D Lagrangian Relaxation ........................................... 291 5.E Saddle Points ................................................... 295 5.F Strong Duality .................................................. 303 5.G Reformulations ................................................. 311 5.H L-Shaped Method ............................................... 316 5.I Monitoring Functions ............................................ 323 5.J Lagrangian Finite-Generation Method .............................. 328 6 WITHOUT CONVEXITY OR SMOOTHNESS ....................... 333 6.A Second-Order Analysis ........................................... 333 6.B Augmented Lagrangians ......................................... 339 6.C Epigraphical Nesting ............................................ 351 6.D Optimality Conditions ........................................... 359 6.E Sup-Projections ................................................. 365 6.F Proximal Composite Method ...................................... 374 6.G Design of Multi-Component Systems ............................... 383 6.H Difference-of-Convex Functions ................................... 389 6.I DC in Regression and Classification ................................ 395 iv Contents 6.J Approximation Errors............................................ 399 7 GENERALIZED EQUATIONS ...................................... 415 7.A Formulations ................................................... 415 7.B Equilibrium in Energy Markets .................................... 423 7.C Traffic Equilibrium .............................................. 429 7.D Reformulation as Minimization Problems ........................... 433 7.E Projection Methods .............................................. 441 7.F Nonsmooth Newton-Raphson Algorithm ............................ 443 7.G Continuity of Set-Valued Mappings ................................ 448 7.H Graphical Approximation Algorithm ............................... 453 7.I Consistent Approximations ....................................... 465 7.J Approximation Errors............................................ 475 8 RISK MODELING AND SAMPLE AVERAGES ....................... 486 8.A Estimation of Optimality Gaps .................................... 486 8.B Risk and Regret ................................................. 491 8.C Risk-Adaptive Data Analytics ..................................... 505 8.D Duality ................................................... ..... 513 8.E Subgradients of Functionals ...................................... 520 8.F Residual Risk and Surrogates ..................................... 528 8.G Sample Average Approximations .................................. 536 8.H Concentration Inequalities ........................................ 549 8.I Diametrical Stochastic Optimization ............................... 556 9 GAMES AND MINSUP PROBLEMS................................. 564 9.A Generalized Nash Games ......................................... 564 9.B Formulation as Minsup Problems .................................. 568 9.C Bifunctions and Solutions ........................................ 572 9.D Lopsided Approximation Algorithm ................................ 580 9.E Lop-Convergence I .............................................. 584 9.F Lop-Convergence II ............................................. 590 9.G Approximation of Games ......................................... 594 9.H Walras Barter Model............................................. 599 10 DECOMPOSITION ................................................ 606 10.AProximal Alternating Gradient Method ............................. 606 10.BLinkage Constraints ............................................. 612 10.CProgressive Decoupling Algorithm ................................. 617 10.DLocal Elicitation ................................................ 631 10.E Decoupling in Stochastic Optimization ............................. 637 10.F Strong Monotonicity ............................................. 647 10.GVariational Convexity and Elicitation ............................... 651 10.HNonlinear Linkage .............................................. 660 References ................................................... .......... 665 Contents v Index ................................................... .............. 669 Preface The concerns with finding minima and maxima were for a long time mainly restricted to mathematical expressions stemming from physical phenomena. The preeminence of being able to do this was underlined by Leonard Euler when he wrote in 1744: “Nothing at all takes place in the universe in which some rule of maximum or minimum does not appear.” The mathematical tools required were supplied by differential calculus as developed by Newton, Leibnitz and many others, not forgetting the Lagrange multiplier strategy for finding the local minima and maxima constrained by smooth equalities. This resulted in a fertile interaction between physics and mathematics, which was instrumental to their shared progress. By the mid-twentieth century, “optimizers” were confronted by a radically new variety of problems coming from organizational and technological issues. Companies wanted to forecast the demand for their products and line up their supply chain accordingly. Airlines had to match crews as well as aircraft-type to specific routes. Engineers needed to plan transportation networks that best serve urban as well as rural areas. Governments aimed to allocate resources efficiently to sectors of the economy. While the decision criteria