[20Pt]Algorithms for Constrained Optimization: [ 5Pt]

Home , 2020s, AMPL, IPOPT

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

Algorithms for Constrained Optimization: The Beneﬁts of General-purpose Software

Michael Saunders MS&E and ICME, Stanford University California, USA

3rd AI+IoT Business Conference Shenzhen, China, April 25, 2019

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 1/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

SOL Systems Optimization Laboratory George Dantzig, Stanford University, 1974 Inventor of the Simplex Method Father of linear programming

Large-scale optimization: Algorithms, software, applications

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 2/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s SOL history

1974 Dantzig and Cottle start SOL 1974–78 John Tomlin, LP/MIP expert 1974–2005 Alan Manne, nonlinear economic models 1975–76 MS, MINOS ﬁrst version 1979–87 Philip Gill, Walter Murray, MS, Margaret Wright (Gang of 4!) 1989– Gerd Infanger, stochastic optimization 1979– Walter Murray, MS, many students 2002– Yinyu Ye, optimization algorithms, especially interior methods This week! UC Berkeley opened George B. Dantzig Auditorium

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 3/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

Optimization problems

Minimize an objective function subject to constraints:  x  min ϕ(x) st ` ≤  Ax  ≤ u c(x)

x variables   A matrix c1(x)  .  c(x) nonlinear functions  .  c (x) `, u bounds m

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 4/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

1970s

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 5/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s 1970s

MS and Bruce Murtagh: MINOS solver (nonlinear objective) George Dantzig: PILOT economic model of US (LP) Alan Manne: ETAMACRO energy model (nonlinear objective)

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 6/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

1980s

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 7/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s 1980s

MINOS solver (sparse nonlinear constraints) NPSOL solver (dense nonlinear constraints)

Optimal Power Flow @ General Electric Aerospace optimization @ NASA, McDonnell-Douglas

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 8/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

1990s

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 9/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s 1990s

Alan Manne: MERGE greenhouse-gas model (sparse nonlinear constraints) Philip Gill: Aerospace trajectory optimization @ McDonnell-Douglas F-4 Minimum time-to-climb DC-Y SSTO Minimum-fuel landing maneuver

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 10/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

Climb to 20,000m Start at sea-level Speed Mach 1

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 11/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s DC-X single-stage-to-orbit prototype

1/3 full-size = 13m tall

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 12/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s DC-Y single-stage-to-orbit full-size

← Taking-oﬀ? Landing?

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 13/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 14/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 15/39 Don’t exceed 3g

New constraint needed:

Optimum starting altitude = 450m(!)

Come back Alan Shephard!

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s DC-Y landing, 2nd OTIS/NPSOL optimization

1st optimization: starting altitude = 900m 2nd optimization: starting altitude = variable

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 16/39 Don’t exceed 3g

Optimum starting altitude = 450m(!)

Come back Alan Shephard!

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s DC-Y landing, 2nd OTIS/NPSOL optimization

1st optimization: starting altitude = 900m 2nd optimization: starting altitude = variable New constraint needed:

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 16/39 Optimum starting altitude = 450m(!)

Come back Alan Shephard!

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s DC-Y landing, 2nd OTIS/NPSOL optimization

1st optimization: starting altitude = 900m 2nd optimization: starting altitude = variable New constraint needed: Don’t exceed 3g

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 16/39 Come back Alan Shephard!

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s DC-Y landing, 2nd OTIS/NPSOL optimization

1st optimization: starting altitude = 900m 2nd optimization: starting altitude = variable New constraint needed: Don’t exceed 3g

Optimum starting altitude = 450m(!)

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 16/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s DC-Y landing, 2nd OTIS/NPSOL optimization

1st optimization: starting altitude = 900m 2nd optimization: starting altitude = variable New constraint needed: Don’t exceed 3g

Optimum starting altitude = 450m(!)

Come back Alan Shephard!

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 16/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

2000s

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 17/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s 2000s

David Saunders, NASA Ames (Calif)

Wˇoshu¯angb¯aot¯ai! 1970: visit Stanford for 1 month (now 49 years) Shape optimization Supersonic airliners Trajectory optimization SHARP (next Space Shuttle)

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 18/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s OAW oblique all-wing airliner

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 19/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s HSCT high speed civil transport

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 20/39 Trajectory optimization with SNOPT Could always abort to Kennedy, Boston, Gander, or Shannon 4000-mile cross-range capability during reentry

Image credit: David Kinney, NASA Ames Research Center

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s CTV crew transfer vehicle

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 21/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s CTV crew transfer vehicle

SHARP design (Slender Hypervelocity Aerothermodynamic Research Probes) Aerothermal performance constraint in (Velocity, Altitude) space, used during trajectory optimization with UHTC materials (Ultra High Temperature Ceramics) to avoid exceeding material limits Trajectory optimization with SNOPT Could always abort to Kennedy, Boston, Gander, or Shannon 4000-mile cross-range capability during reentry

Image credit: David Kinney, NASA Ames Research Center

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 21/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s SNOPT Applications

Trajectory optimization Manoeuver ISS Launch 2 satellites Space Station

Torque minimization Robot at JPL Radiation therapy Daniel Clemente Control problem Paul Keall

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 22/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

2010s

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 23/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s 2010s

David Saunders, NASA Ames (Calif) Orion Apollo-type capsule to ISS and moon MSL (Mars Science Lab) Heat ﬂux during atmospheric entry Stratolaunch Descent trajectory of space vehicle

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 24/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s Crew Exploration Vehicle (Orion)

Tried shape optimization of heat shield and shoulder curvature (but the Apollo folk were pretty close already)

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 25/39 Landing of launched space vehicle

Preliminary computation: Space vehicle will land in Mojave Desert, California OTIS trajectory optimization: Vehicle would land 2500km too soon!

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s Stratolaunch carrier aircraft (ﬁrst ﬂight April 15)

AIAA 2018

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 26/39 Fig. 2 Stratolaunch carrier aircraft.

Downloaded by Stephen Corda on September 20, 2018 | http://arc.aiaa.org DOI: 10.2514/6.2018-5257

Fig. 3 Carrier aircraft dimensions.

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s Stratolaunch carrier aircraft (ﬁrst ﬂight April 15) Landing of launched space vehicle

Preliminary computation: Space vehicle will land in Mojave Desert, California OTIS trajectory optimization: Vehicle would land 2500km too soon!

AIAA 2018

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 26/39 Fig. 2 Stratolaunch carrier aircraft.

Downloaded by Stephen Corda on September 20, 2018 | http://arc.aiaa.org DOI: 10.2514/6.2018-5257

Fig. 3 Carrier aircraft dimensions.

1998 BPDN: Basis Pursuit DeNoising Shaobing Chen, David Donoho, MS (Stanford)

2011 1D LF-NMR (Low-Field NMR analysis) Ofer Levi, MS, Berman, Wiesman, . . . (Israel) 2018 2D LF-NMR

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s Signal analysis using PDCO

Primal-Dual interior method for Convex Objective General-purpose MATLAB software min ϕ(x) st Ax = b, ` ≤ x ≤ u Unique feature: A may be an operator Iterative method LSMR computes each search direction

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 27/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s Signal analysis using PDCO

1998 BPDN: Basis Pursuit DeNoising Shaobing Chen, David Donoho, MS (Stanford)

2011 1D LF-NMR (Low-Field NMR analysis) Ofer Levi, MS, Berman, Wiesman, . . . (Israel) 2018 2D LF-NMR

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 27/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s Page 17 of 18 Journal of the American Oil2D Chemists' LF-NMR Society analysis using PDCO 1 2 3 4 5 6 Analysis of biodiesel, olive oil, . . . 7 8 9 10 11 12 2 2 13 minf ,r λ1 kf k1 + λ2 kf k + krk 14 15 16 such that K1FK2 + R = S 17 18 f ≥ 0 19 20 21 For Peer Review 22 F , R = matrix form of variables f , r 23 24 25 26 27 28 PDCO solution f is very sharp 29 30 31 32 33 34 35 36 37 38 1 Fig. 4 2D T1-T2 H LF-NMR energy relaxation spectrum mapping of linseed PUFA oil. 39 40 Optimization(1) Glycerol Software segment (2) Average of double bonds segment3rd AI+IoT (3) Average Business of aliphatic Conference, Shenzhen, April 25, 2019 28/39 41 middle chain (4) Average of tail segment. 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 DQQ = Double-precision / Quad-precision / Quad-precision solvers cold start warm start warm start NCL = Nonlinearly Constrained augmented Lagrangian need interior methods

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s 2010s General-purpose software leads to Applications PDCO ideal for LF-NMR analysis

Applications lead to Algorithms

Systems Biology multiscale models lead to DQQ procedure Ding Ma, MS (Stanford)

Taxation policy models lead to NCL procedure Ding Ma, MS, Judd, Orban Need general-purpose software to implement new procedures

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 29/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s 2010s General-purpose software leads to Applications PDCO ideal for LF-NMR analysis

Applications lead to Algorithms

Systems Biology multiscale models lead to DQQ procedure Ding Ma, MS (Stanford)

Taxation policy models lead to NCL procedure Ding Ma, MS, Judd, Orban Need general-purpose software to implement new procedures

DQQ = Double-precision / Quad-precision / Quad-precision solvers cold start warm start warm start NCL = Nonlinearly Constrained augmented Lagrangian need interior methods

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 29/39 Jan 2017 Conference in Oman Stop in Paris on the way back . . . !

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s DQQ procedure (Ding Ma, MS, . . . )

Multiscale optimization in systems biology Double-precision MINOS+ Quad-precision MINOS

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 30/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s DQQ procedure (Ding Ma, MS, . . . )

Multiscale optimization in systems biology Double-precision MINOS+ Quad-precision MINOS Jan 2017 Conference in Oman Stop in Paris on the way back . . . !

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 30/39 NCL procedure ≈ 10 easier subproblems, updating yk or increasing ρk :

T 1 2 NCk minimize φ(x) + y r + ρk krk x, r k 2 subject to c(x) + r ≥ 0, ` ≤ x ≤ u

Variables r make the nonlinear constraints independent and feasible Interior solvers IPOPT, KNITRO happy as r → 0

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s NCL procedure (Ding Ma, MS, . . . )

Taxation problem minimize φ(x) x subject to c(x) ≥ 0, ` ≤ x ≤ u

Example: 571000 constraints ci (x) ≥ 0, 1500 variables x ∗ −6 10000 constraints ci (x ) ≤ 10 (essentially active)

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 31/39 Variables r make the nonlinear constraints independent and feasible Interior solvers IPOPT, KNITRO happy as r → 0

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s NCL procedure (Ding Ma, MS, . . . )

Taxation problem minimize φ(x) x subject to c(x) ≥ 0, ` ≤ x ≤ u

Example: 571000 constraints ci (x) ≥ 0, 1500 variables x ∗ −6 10000 constraints ci (x ) ≤ 10 (essentially active)

NCL procedure ≈ 10 easier subproblems, updating yk or increasing ρk :

T 1 2 NCk minimize φ(x) + y r + ρk krk x, r k 2 subject to c(x) + r ≥ 0, ` ≤ x ≤ u

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 31/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s NCL procedure (Ding Ma, MS, . . . )

Taxation problem minimize φ(x) x subject to c(x) ≥ 0, ` ≤ x ≤ u

Example: 571000 constraints ci (x) ≥ 0, 1500 variables x ∗ −6 10000 constraints ci (x ) ≤ 10 (essentially active)

NCL procedure ≈ 10 easier subproblems, updating yk or increasing ρk :

T 1 2 NCk minimize φ(x) + y r + ρk krk x, r k 2 subject to c(x) + r ≥ 0, ` ≤ x ≤ u

Variables r make the nonlinear constraints independent and feasible Interior solvers IPOPT, KNITRO happy as r → 0

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 31/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

Summary

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 32/39 New applications → New algorithms to help existing software 2010s: Ding Ma, . . . Systems biology (multiscale) DQQ: combine DoubleMINOS + QuadMINOS Taxation policy NCL: Sequence of problems, warm-start IPOPT, KNITRO

Existing software → Prototype new applications 2018: Ali Eshragh, . . . (Australia) MINOS Find Hamiltonian cycles in graphs Add one subroutine to MINOS simplex method

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s General-Purpose Software

We couldn’t guess the earlier applications of optimization!

Existing software → New applications 1980–now MINOS Energy/economic models NPSOL trajectory optimization, radiation therapy SNOPT trajectory optimization (bigger), robotics PDCO Basis Pursuit signal analysis, LF-NMR analysis

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 33/39 Existing software → Prototype new applications 2018: Ali Eshragh, . . . (Australia) MINOS Find Hamiltonian cycles in graphs Add one subroutine to MINOS simplex method

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s General-Purpose Software

We couldn’t guess the earlier applications of optimization!

New applications → New algorithms to help existing software 2010s: Ding Ma, . . . Systems biology (multiscale) DQQ: combine DoubleMINOS + QuadMINOS Taxation policy NCL: Sequence of problems, warm-start IPOPT, KNITRO

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 33/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s General-Purpose Software

We couldn’t guess the earlier applications of optimization!

Existing software → Prototype new applications 2018: Ali Eshragh, . . . (Australia) MINOS Find Hamiltonian cycles in graphs Add one subroutine to MINOS simplex method

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 33/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

2020s

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 34/39 Xing Lab @ Stanford AI, physics, engineering, biology in medicine Diagnosis, treatment planning, nanotech imaging for precision medicine 1998 Radiation therapy NPSOL: Gamma-Knife Body moves during radiation 2018 Billy Loo, Sami Tantawi @ SLAC (Stanford) FLASH therapy (X-rays or protons) Only 1 second of radiation

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s 2020s

Now and future: Autonomous vehicles Smooth path, failsafe

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 35/39 1998 Radiation therapy NPSOL: Gamma-Knife Body moves during radiation 2018 Billy Loo, Sami Tantawi @ SLAC (Stanford) FLASH therapy (X-rays or protons) Only 1 second of radiation

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s 2020s

Now and future: Autonomous vehicles Smooth path, failsafe Xing Lab @ Stanford AI, physics, engineering, biology in medicine Diagnosis, treatment planning, nanotech imaging for precision medicine

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 35/39 2018 Billy Loo, Sami Tantawi @ SLAC (Stanford) FLASH therapy (X-rays or protons) Only 1 second of radiation

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s 2020s

Now and future: Autonomous vehicles Smooth path, failsafe Xing Lab @ Stanford AI, physics, engineering, biology in medicine Diagnosis, treatment planning, nanotech imaging for precision medicine 1998 Radiation therapy NPSOL: Gamma-Knife Body moves during radiation

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 35/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s 2020s

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 35/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 36/39 AI+IoT organizers Markla, New Fortune YouTube, Youku Yuja Wang

SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s Special thanks

SOL George Dantzig, Richard Cottle

Algorithm coauthors Philip Gill, Walter Murray Software coauthors Bruce Murtagh, Philip Gill, Elizabeth Wong Basis Pursuit Shaobing Chen, David Donoho LF-NMR Ofer Levi, Shirley Berman, Zeev Wiesman, . . . DQQ Ding Ma, Ronan Fleming, Ines Thiele NCL Ding Ma, Ken Judd, Dominique Orban AMPL, IPOPT, KNITRO developers

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 37/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s Special thanks

SOL George Dantzig, Richard Cottle

AI+IoT organizers Markla, New Fortune YouTube, Youku Yuja Wang

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 37/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s YouTube companion Yuja Wang

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 38/39 SOL Optimization 1970s 1980s 1990s 2000s 2010s Summary 2020s Liˇang gèY¯oukùpéngyˇouLángLˇanghéWángYˇuji¯a

Two equally amazing stars (optimal!)

Optimization Software 3rd AI+IoT Business Conference, Shenzhen, April 25, 2019 39/39