Optimization Algorithms and Software at SOL

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Michael Saunders SOL and ICME, Stanford University

Computational Linear Algebra and Optimization for the Digital Economy

International Centre for Mathematical Sciences Edinburgh, Scotland, Oct 31–Nov 1, 2013

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 1/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Abstract

When a colleague knocks on your door with a mathematical problem, great joy ensues on both sides if you happen to have numerical software that can solve the problem. This is a reason for writing “general-purpose software”. If the software is downloadable, similar happiness ensues when it gets used for unexpected applications. You learn about such cases if your software includes bugs(!) or Google does its usual precise search. The Systems Optimization Laboratory (SOL) was founded by George Dantzig and Richard Cottle at Stanford University in 1974 to encourage algorithm and software development in traditional Operations Research areas. Dantzig’s group built increasingly challenging linear models of the US Economy (the PILOT linear programs), while Alan Manne continued to expand his nonlinear economic models (which began before large-scale optimization software existed). These spurred the development of MINOS at SOL (by the speaker and fellow New Zealander Bruce Murtagh) in parallel with Arne Drud’s development of CONOPT at the World Bank and then in Denmark.

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 2/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

In the 1980s, the SOL “Gang of 4” (Gill, Murray, Saunders, and Wright) developed “dense” solvers LSSOL, QPSOL, and NPSOL. After Philip Gill moved to UC San Diego, NPSOL became increasingly important for trajectory optimization at McDonnell-Douglas (now Boeing). This spurred the development of the large-scale optimizers SQOPT and SNOPT. Many years later, MINOS, CONOPT, and SNOPT remain heavily used solvers within the GAMS and AMPL algebraic modeling systems (and on the NEOS server), and LUSOL remains the reliable “engine” for basis handling in MINOS, SQOPT, SNOPT, and other solvers such as PATH and lp solve. A unique feature of LUSOL is its Threshold Rook Pivoting option for estimating the rank of a rectangular sparse matrix. Other solvers developed at SOL include MINRES, MINRES-QLP, LSQR, LSMR, and PDCO. We review the mechanics of the solvers and some unexpected applications that they’ve been put to work on. We conclude with even more unexpected aspects of optimization reported by radio and TV audiences in New Zealand.

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 3/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

1 SOL

2 Solvers for Ax ≈ b

3 Applications of linear solvers

4 Rank of stoichiometric matrices

5 Solvers for optimization

6 Applications of optimization solvers

7 Aerospace

8 AC

9 Optimization in NZ

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 4/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

SOL

Systems Optimization Laboratory Stanford University

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 5/101 George Dantzig Gene Golub 1914–2005 1932–2007

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ SOL Founded 1974 by George Dantzig and Richard Cottle Dantzig, Alan Manne: economic models (linear & nonlinear) Gill, Murray, Saunders, Wright: Software for optimization

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 6/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ SOL Founded 1974 by George Dantzig and Richard Cottle Dantzig, Alan Manne: economic models (linear & nonlinear) Gill, Murray, Saunders, Wright: Software for optimization

George Dantzig Gene Golub Michael Saunders: Optimization software1914–2005 at SOL CLAODE, ICMS, Edinburgh 1932–2007 Oct 31–Nov 1, 2013 6/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ SOL Founded 1974 by George Dantzig and Richard Cottle Dantzig, Alan Manne: economic models (linear & nonlinear) Gill, Murray, Saunders, Wright: Software for optimization

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 7/101 Philip Gill, Elizabeth Wong (UC San Diego) Optimization software NPSOL, QPOPT, SQOPT, SNOPT, SQIC Ronan Fleming, Ines Thiele (UCSD, Iceland, Luxembourg) Flux balance analysis (FBA), Flux variability analysis (FVA) Rank and nullspace of stoichiometric matrices (Need LUSOL, PDCO, SQOPT) Funding: ONR, AFOSR, ARO, DOE, NSF, AHPCRC, . . . , DOE DE-FG02-09ER25917, NIH U01-GM102098

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ SOL Recent collaborators: Chris Paige (McGill), Sou-Cheng Choi (Chicago) MINRES-QLP David Fong (ICME and Facebook) LSMR Xiangrui Meng (ICME and LinkedIn) LSRN Jason Lee, Yuekai Sun (ICME) PNOPT Ding Ma, Nick Henderson, Santiago Akle (ICME) LUSOL, PDCO

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 8/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ SOL Recent collaborators: Chris Paige (McGill), Sou-Cheng Choi (Chicago) MINRES-QLP David Fong (ICME and Facebook) LSMR Xiangrui Meng (ICME and LinkedIn) LSRN Jason Lee, Yuekai Sun (ICME) PNOPT Ding Ma, Nick Henderson, Santiago Akle (ICME) LUSOL, PDCO Philip Gill, Elizabeth Wong (UC San Diego) Optimization software NPSOL, QPOPT, SQOPT, SNOPT, SQIC Ronan Fleming, Ines Thiele (UCSD, Iceland, Luxembourg) Flux balance analysis (FBA), Flux variability analysis (FVA) Rank and nullspace of stoichiometric matrices (Need LUSOL, PDCO, SQOPT) Funding: ONR, AFOSR, ARO, DOE, NSF, AHPCRC, . . . , DOE DE-FG02-09ER25917, NIH U01-GM102098 Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 8/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Sparse linear equations Ax = b and least squares problems Ax ≈ b

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 9/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Sparse direct methods for Ax = b and Ax ≈ b

A = LDU LUSOL (Stanford)

A = QR SPQR (Tim Davis, UFL)

Many sparse solvers HSL Library (RAL, UK) Iain Duﬀ, John Reid, Jennifer Scott, . . .

LUSOL, SPQR, MA27, MA47, MA57, MA67, MA77, . . . oﬀer rank-revealing capability for sparse matrices when used with suitable tolerances

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 10/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Iterative solvers for Ax ≈ b

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 11/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Iterative methods for Ax ≈ b

A may be a sparse matrix or an operator for computing Av and/or ATw

A maybe Hermitian or complex A may have any rank

Symmetric Ax = b A = MINRES-QLP

min kAx − bk A = or or LSQR, LSMR

Tall skinny min kAx − bk A = (real) LSRN

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 12/101 LSQR min krk k residual rk = b − Axk T T T LSMR min A rk residual for A Ax = A b

T kA rk k For LS problems, LSQR, LSMR stop when ≤ α kAk krk k (this is the Stewart backward error)

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ LSQR, LSMR for min kAx − bk

Golub-Kahan process generates Uk = u1 u2 ... uk Vk = v1 v2 ... vk T using products Avj , A uj kth approximation xk = Vk yk for some yk

Choose yk to minimize something

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 13/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ LSQR, LSMR for min kAx − bk

Golub-Kahan process generates Uk = u1 u2 ... uk Vk = v1 v2 ... vk T using products Avj , A uj kth approximation xk = Vk yk for some yk

Choose yk to minimize something

LSQR min krk k residual rk = b − Axk T T T LSMR min A rk residual for A Ax = A b

T kA rk k For LS problems, LSQR, LSMR stop when ≤ α kAk krk k (this is the Stewart backward error)

T kA rk k log (typical) LSMR can stop sooner 10 krk k

Name:lp pilot ja, Dim:2267x940, nnz:14977, id=88 0 LSQR LSMR

−1

−2

−3 log(E2)

−4

−5

−6 0 100 200 300 400 500 600 700 800 900 1000 iteration count

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 14/101 CG min kx − xk kA energy norm of error MINRES min krk k

krk k They stop when ≤ 1 (backward error argument) α kAk kxk k + β kbk

For posdef A, kxk k % for both methods (Steihaug 1983, Fong 2011)

Hence, backward error & for MINRES (but not for CG)

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ CG, MINRES for posdef Ax = b

Lanczos process generates Vk = v1 v2 ... vk using products Avj kth approximation xk = Vk yk for some yk

Choose yk to minimize something

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 15/101 For posdef A, kxk k % for both methods (Steihaug 1983, Fong 2011)

Hence, backward error & for MINRES (but not for CG)

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ CG, MINRES for posdef Ax = b

Lanczos process generates Vk = v1 v2 ... vk using products Avj kth approximation xk = Vk yk for some yk

Choose yk to minimize something

CG min kx − xk kA energy norm of error MINRES min krk k

krk k They stop when ≤ 1 (backward error argument) α kAk kxk k + β kbk

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 15/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ CG, MINRES for posdef Ax = b

Lanczos process generates Vk = v1 v2 ... vk using products Avj kth approximation xk = Vk yk for some yk

Choose yk to minimize something

CG min kx − xk kA energy norm of error MINRES min krk k

krk k They stop when ≤ 1 (backward error argument) α kAk kxk k + β kbk

For posdef A, kxk k % for both methods (Steihaug 1983, Fong 2011)

Hence, backward error & for MINRES (but not for CG)

Name:Schenk_AFE_af_shell8, Dim:504855x504855, nnz:17579155, id=11 Name:Cannizzo_sts4098, Dim:4098x4098, nnz:72356, id=13 −1 0 CG CG MINRES MINRES −2 −1

−2 −3

−3 −4

−4 −5

−5 −6 log(||r||/||x||) log(||r||/||x||) −6

−7 −7

−8 −8

−9 −9

−10 −10 0 200 400 600 800 1000 1200 1400 0 50 100 150 200 250 300 350 400 450 iteration count iteration count

Name:Simon_raefsky4, Dim:19779x19779, nnz:1316789, id=7 Name:BenElechi_BenElechi1, Dim:245874x245874, nnz:13150496, id=22 2 0 CG CG MINRES MINRES 0 −1

−2 −2

−4 −3

−6 −4

−8 −5 log(||r||/||x||) log(||r||/||x||) −10 −6

−12 −7

−14 −8

−16 −9

−18 −10 0 1 2 3 4 5 6 7 8 9 10 0 0.5 1 1.5 2 2.5 3 3.5 4 4 iteration count x 10 iteration count x 10 Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 16/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Stoichiometric matrices in systems biology

Sparse matrices S Rows: Chemical species Cols: Chemical reactions

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 17/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ S for models 1, 2, 3, 4 (all similar)

iIT341 iAF692 0 0 100 100 200 200 300 300 400

400 500 600 0 200 400 0 200 400 600 nz = 2314 nz = 3067 iSB619 iJN746 0 0 100 200 200 300 400 400 600 500 600 800 0 200 400 600 0 500 1000 nz = 3821 nz = 4407 Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 18/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ S for Models 5, 6, 7, 8 (all similar)

iJR904 iNJ661 0 0

200 200

400 400

600 600

800 0 200 400 600 800 1000 0 500 1000 nz = 4503 nz = 4793 iND750 iAF1260 0 0 200 500 400

600 1000 800 1500 1000 0 500 1000 0 500 1000 1500 2000 nz = 5300 nz = 9231 Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 19/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Model 9 (Recon1)

Recon1 0

500

1000

1500

2000

2500

0 500 1000 1500 2000 2500 3000 3500 nz = 14300

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 20/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Model 10 (ThMa = Thermotoga maritima)

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 21/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Model 11 (GlcAer)

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 22/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ S for models 9, 10, 11

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 23/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Rank of stoichiometric matrices

Conservation analysis for biochemical networks

Need rank(S) and nullspace(S T )

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 24/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ rank(S) by SVD

Singular value decomposition S = UDV T

UTU = IV TV = ID diagonal rank(S) = rank(D)

Ideal for rank-estimation but U, V are dense

model 9 (Recon1) 2800 × 3700 17 secs model 10 (ThMa) 15000 × 18000 11 hours model 11 (GlcAer) 62000 × 77000 ∞

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 25/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Singular values of models 1–8 Dense SVD of S T

iIT341 iAF692 iSB619 iJN746 5 5 5 5

0 0 0 0

−5 −5 −5 −5

−10 −10 −10 −10

−15 −15 −15 −15

−20 −20 −20 −20 0 500 0 500 1000 0 500 1000 0 500 1000 log10(σi ) iJR904 iNJ661 iND750 iAF1260 5 5 5 5

0 0 0 0

−5 −5 −5 −5

−10 −10 −10 −10

−15 −15 −15 −15

−20 −20 −20 −20 0 500 1000 0 500 1000 0 1000 2000 0 1000 2000

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 26/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ rank(S) by QR

Householder QR factorization SP = QR

P = col perm QTQ = IR triangular rank(S) = rank(R)

Nearly as reliable as SVD

Dense QR used by Vallabhajosyula, Chickarmane, Sauro (2005)

Sparse QR (SPQR) now available: Davis (2013)

model 9 (Recon1) 2800 × 3700 0.1 secs model 10 (ThMa) 15000 × 18000 2.5 secs model 11 (GlcAer) 62000 × 77000 0.2 secs(!)

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 27/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ rank(S) by LUSOL with Threshold Rook Pivoting

Sparse LU with TRP P1SP2 = LDU

P1, P2 = perms D diagonal rank(S) ≈ rank(D) L, U well-conditioned

Lii = Uii = 1 |Lij | and |Uij | ≤ factol = 4 (or 2 or 1.2, 1.1, . . . )

LUSOL: Main engine in sparse linear/nonlinear optimizers MINOS, SQOPT, SNOPT

model 9 (Recon1) 2800 × 3700 0.1 secs model 10 (ThMa) 15000 × 18000 4.0 secs model 11 (GlcAer) 62000 × 77000 158 secs

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 28/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ rank(S) by LUSOL with Threshold Partial Pivoting

Sparse LU with TPP P1SP2 = LU

P1, P2 = perms U trapezoidal rank(S) ≈ rank(U) L well-conditioned

Lii = 1 |Lij | ≤ factol = 4 (or 2 or 1.2, 1.1, . . . )

LUSOL: Main engine in sparse linear/nonlinear optimizers MINOS, SQOPT, SNOPT

model 9 (Recon1) 2800 × 3700 0.1 secs model 10 (ThMa) 15000 × 18000 0.2 secs model 11 (GlcAer) 62000 × 77000 0.3 secs

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 29/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ SPQR vs LUSOL with Threshold Rook Pivoting

SPQR: S = QR time (secs) model | m n rank(S) | nnz(S) nnz(Q) nnz(R) | SVD SPQR ------Recon1 | 2766 3742 2674 | 14300 2750 21093 | 17.5 0.1 ThMa | 15024 17582 14983 | 326035 844096 10595016 | 11hrs 2.5 GlcAer | 62212 76664 62182 | 913967 1287 916600 | infty 0.2

LUSOL: S = LDU |Lij |, |Uij | ≤ 2.0 nnz(L) nnz(U) | time ------Recon1 | 4280 16463 | 0.1 ThMa | 30962 346122 | 4.1 GlcAer | 635571 1810491 | 186.2

LUSOL: S = LDU |Lij |, |Uij | ≤ 4.0 nnz(L) nnz(U) | time ------Recon1 | 2701 12896 | 0.1 ThMa | 36350 330485 | 4.0 GlcAer | 427456 1584188 | 157.9 Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 30/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ SPQR vs LUSOL with Threshold Rook Pivoting

SPQR: ST = QR time (secs) model | m n rank(S’)| nnz(S) nnz(Q) nnz(R) | SVD SPQR ------Recon1 | 3742 2766 2674 | 14300 107935 36929 | 17.2 0.1 ThMa | 17582 15024 14983 | 326035 624640 605888 | 11hrs 0.7 GlcAer | 76664 62212 62182 | 913967 3573696 4038988 | infty 2.7

T LUSOL: S = LDU |Lij |, |Uij | ≤ 2.0 nnz(L) nnz(U) | time ------Recon1 | 12832 7421 | 0.3 ThMa | 501198 358601 | 37.8 GlcAer | 1996892 709448 | 586.0

T LUSOL: S = LDU |Lij |, |Uij | ≤ 4.0 nnz(L) nnz(U) | time ------Recon1 | 9811 6093 | 0.2 ThMa | 410290 355475 | 14.8 GlcAer | 1823067 711906 | 791.2 Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 31/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ SPQR vs LUSOL with Threshold Partial Pivoting

LUSOL: S = LU |Lij |, |Uij | ≤ 2.0 nnz(L) nnz(U) | time ------Recon1 | 721 13585 | 0.1 ThMa | 7779 324483 | 0.2 GlcAer | 533 913781 | 0.4

LUSOL: S = LU |Lij |, |Uij | ≤ 4.0 nnz(L) nnz(U) | time ------Recon1 | 764 13577 | 0.1 ThMa | 7782 323929 | 0.2 GlcAer | 533 913781 | 0.4 Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 32/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ SPQR vs LUSOL with Threshold Partial Pivoting

T LUSOL: S = LU |Lij | ≤ 2.0 nnz(L) nnz(U) | time ------Recon1 | 9304 7813 | 0.2 ThMa | 81506 268938 | 2.7 GlcAer | 337433 703619 | 126.7

T Sparse LU with TPP P¯1U P¯2 = L¯U¯

L¯ well-conditioned rank(S) ≈ rank(U) ≈ rank(U¯ )

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Rank of stoichiometric S Perhaps

Sparse LU with TPP P1SP2 = LU

L well-conditioned rank(S) ≈ rank(U)

Then

Sparse LU with TRP P¯1UP¯2 = L¯D¯ U¯

L¯, U¯ well-conditioned rank(S) ≈ rank(U) ≈ rank(D¯ )

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 34/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Rank of stoichiometric S Perhaps

Sparse LU with TPP P1SP2 = LU

L well-conditioned rank(S) ≈ rank(U)

Then

Sparse LU with TRP P¯1UP¯2 = L¯D¯ U¯

L¯, U¯ well-conditioned rank(S) ≈ rank(U) ≈ rank(D¯ )

T Sparse LU with TPP P¯1U P¯2 = L¯U¯

L¯ well-conditioned rank(S) ≈ rank(U) ≈ rank(U¯ )

LSQR

in parallel!

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 35/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ LSQR for tomography Huang, Dennis, Wang, Chen 2013 A scalable parallel LSQR algorithm for solving large-scale linear system for tomographic problems: a case study in seismic tomography Procedia Computer Science 18:581–590 He Huang et al. / Procedia Computer Science 18 ( 2013) 581 – 590 585

(/ ....

261M × 38M 1&#*20*0')*+#,$'&%3

(0 min kAx − bk A = (/% -/% -/ 5 billion nonzeros

(0% -0% -0%

!"#$%&'( )*+#,$'& )*+#,$'- )*+#,$'-

(a) A partitioning of the sparse matrix vector product A x 3D stuctural seismology × for five MPI tasks. The colors correspond to different MPI Modest-sized dataset from Los Angelestasks. Note thatBasin the kernel rows (ANGF) at the top of matrix A are partitioned across columns, while the damping rows are par- !"#$%&'( )*+#,$'& )*+#,$'- titioned across rows. The corresponding partitions of vectors x and y are also included. Note that the piece of vector y that (b) Vector reconstruction and A x ( Ak x and Ad x") in Cray XT5 (Kraken) at Oak Ridge National Lab × i × i i × i correspondsT to the kernel rows is replicated across all tasks. one MPI task. Yellow represents local task. Parallelize y ← Ax + y and x ← A y + x Fig. 3: Illustration matrix vector multiplication: A x 2400 cores: 10 times faster than PETSc × algorithm and does not represent an excessive overhead because only 10% of the memory storage for the entire 19200 cores: 33 times faster thanmatrix PETScA is a result of the damping component. Note that we do not use a superscript T for the transposed damping component but rather Adt to reflect that it is a separate copy of matrix within our algorithm. Ad is stored in CSR Michael Saunders: Optimization software at SOL format and Adt is storedCLAODE, in CSCformat. ICMS, Edinburgh Oct 31–Nov 1, 2013 36/101 The vector x is partitioned across columns and is illustrated in Figures 3a and 4a. Vector y has two parts in each task, a replicated kernel piece and a partitioned damping piece corresponding to the damping matrix. The length of kernel part of yki is the same as the number of rows in kernel Ak. The damping part of ydi is partitioned according to the number of rows of damping Adi in each task.

6. Parallel Computation

We next describe the most computationally expensive and complex section of SPLSQR algorithm, i.e., the calculations of y A x + y and x AT y + x. The calculation of y A x + y represents lines (11) - (17) ← × ← × ← × in Figure 1, while x AT y + x represents lines (18) - (24). Both calculations require the reconstruction of ← × the necessary pieces of vectors x and y by communicating with other MPI tasks. However they also diﬀer in key characteristics due to the particular partitioning of matrices Ad and Ak.

6.1. y A x + y ← × Figure 3b illustrates the matrix vector multiplication y A x from the perspective of the yellow task. We ← × begin with a description of calculating yk that involves the kernel matrix. Each task multiplies its local piece of kernel Aki with local piece of xi and yields its kernel part of vector, yki, as indicated by the top part of vector y in Figure 3b. Note that because of the column-based partitioning of Ak, yki is a partial result. Due to Ak’s random and nearly dense feature, the resulting yki has overlap with all the other tasks. Hence a reduction across all tasks is performed on yki to combine the partial results. The ﬁnal result is illustrated in black in Figure 3b. Because the number of rows in the kernel nk is very small relative to the entire number of rows n, the total cost of the reduction only has a modest impact. Next we describe the calculation of the damping component of ydi. Unlike the calculation of yki, the calculation of ydi requires additional pieces of x that the yellow task does not currently own. Note that in Figure 3b, the width of the Adi matrix is greater than xi. The required additional pieces of x are owned by its neighboring SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

SOL optimization solvers

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 37/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Optimization solvers (dense) 2 x LSSOL (f77): min kXx − bk st ` ≤ ≤ u Ax Avoids forming X TX We still don’t have a good method for sparse X + constraints

x QPOPT (f77): min 1 xTHx st ` ≤ ≤ u 2 Ax H may be indeﬁnite

 x  NPSOL (f77): min φ(x) st ` ≤  Ax  ≤ u c(x) Philip Gill has recently completed a new NPSOL that includes elastic bounds to handle infeasible QP subproblems (like SQOPT and SNOPT)

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 38/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Optimization solvers (sparse)

MINOS (f77): Sparse NLP No elastic bounds, but still widely used. f90 version half started.

SQOPT, SNOPT (f77): Sparse convex QP + general NLP Elastic bounds, threadsafe, good for expensive functions

SNOPT9 (f2003): Gill, S, Wong Includes SQIC QP solver Switches from SQOPT’s reduced-gradient method to KKT-factorization + block-LU updates for problems with many degrees of freedom

Change 1 line and recompile ⇒ everything in quad precision SNOPT9’s simplex implementation gives us solutions with astounding accuracy! Great for systems biologists!

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 39/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

NEOS

Free optimization solvers via Argonne National Lab (now Univ of Madison, Wisconsin)

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 40/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ NEOS Solver Statistics for 1 year 1 Oct 2012 -- 30 Sep 2013 Total Jobs 1264001 Solver Submissions MINOS 642263 BDMLP 3747 OOQP 771 sd 33 KNITRO 256367 Couenne 3588 SYMPHONY 692 cplex 29 Gurobi 70179 FilMINT 3424 PATHNLP 666 BiqMac 10 Ipopt 35181 BLMVM 3410 DSDP 627 LGO 3 SNOPT 33197 NMTR 2735 condor 601 lpopt 2 csdp 20142 feaspump 2607 sedumi 585 BDLMP 1 DICOPT 19146 AlphaECP 2538 PSwarm 584 XpressMP 17761 bpmpd 2317 RELAX4 568 BARON 14737 PATH 2036 Clp 503 Cbc 14613 LANCELOT 1972 FortMP 432 MINTO 13515 MILES 1641 sdplr 415 scip 12986 sdpt3 1436 Glpk 336 MOSEK 10837 LRAMBO 1415 ddsip 335 CONOPT 9871 filterMPEC 1289 nsips 300 LOQO 9598 SDPA 1244 icos 249 Bonmin 7320 TRON 1145 bnbs 244 MINLP 6573 SBB 1085 pensdp 234 filter 5438 NLPEC 1053 penbmi 226 concorde 4879 L-BFGS-B 999 PGAPack 216 LINDOGlobal 4597 qsopt_ex 902 proxy 127 MUSCOD-II 4425 ASA 858 xpress 36 Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 41/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ NEOS Solver Statistics for 1 year 1 Oct 2012 -- 30 Sep 2013 Total Jobs 1264001 Category Submissions Input Submissions nco 982242 AMPL 1053956 milp 100864 GAMS 152810 minco 57532 SPARSE_SDPA 22083 lp 42205 MPS 10136 sdp 24909 TSP 4879 go 17955 C 4513 cp 12591 Fortran 4114 bco 5547 CPLEX 3723 co 4889 MOSEL 2182 miocp 4425 MATLAB_BINARY 1802 kestrel 3981 LP 903 uco 2735 ZIMPL 709 lno 2407 SDPA 627 slp 612 DIMACS 564 ndo 601 SMPS 277 sio 300 MATLAB 199 socp 75 SDPLR 198 mip 8 QPS 164 nlp 3 DNLP 2

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 42/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

DICOPT

MINLP solver Ignacio Grossman, Carnegie-Mellon

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 43/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ MINLP model

MS student Rui-Jie Zhou transferring from MS&E to EE, mentioned MINOS had been useful 2 papers in Industrial Engineering and Chemistry Research Zhou, Ji-Juan Li, Hong-Guang Dong, Ignacio Grossmann Part I (44 pages) Multiscale state-space superstructure for interplant water-allocation and heat-exchange networks design with direct and indirect integration schemes in fixed flow rate (FF) processes Part II (42 pages) Extends to fixed contaminant-load (FC) processes and integration of FF and FC processes Nonlinear objective 1500 linear and nonlinear constraints 1600 variables (some binary) GAMS/DICOPT = MINOS + CPLEX or CONOPT + CPLEX

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 44/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ MINLP model

External Internal Sinks Sinks

Water user

Water user External Sources PO_W

Distribution Network (DN) PO_HEN Internal Sources

Figure 1. State-space superstructure for stand-alone WAHEN

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 45/101

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

MINLP model

External Sources External PO_JUN

Sinks Jun1

PO_PL Junn ANT1 tr PO_TR PO_PL Interplant distribution ANT2 network PO_IHEN

Figure 2. Multi-scale state-space superstructure for interplant WAHEN

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 46/101

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

MINLP model

Plant 1 6ºC 889kW 1911kW H1 377ºC H1 97ºC E1 288.1ºC E2 10KW/K 6.17ºC 2116.5kW H2 317ºC 14.51KW/K H2 97ºC E3 171.1ºC 183.3ºC C3 20KW/K

5.49KW/K 557.5kW 1726kW C4 215.5ºC

594.5kW C1 377ºC C1 137ºC H1 337.4ºC 278.1ºC 15KW/K

5.31ºC C2 227ºC C2 80ºC 13KW/K 210ºC, 0.619kg/s

209ºC, 0.619kg/s 183.9kW 5.46ºC H3 105ºC H3 25ºC E5 86.6ºC C2 10KW/K 616.2kW 458.7kW H4 185ºC H4 35ºC E4 93.3ºC C1 5KW/K HPS 291.3kW

C3 185ºC C3 25ºC H2 110.7ºC 49.5ºC 7.5KW/K 557.5kW 633.5KW/K 2000KW/K Plant 2 5ºC

Figure 3. Optimal interplant HEN configuration in Example 1

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 47/101

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

MINLP model

Source 3 Sink 2 4.3kg/s 40kg/s 20ºC 36kg/s 23.2kg/s [26.5, 35.2, 120] Sink 3 40.8kg/s 50ºC Sink 1 12.0℃ 28kg/s 95ºC 10.3kg/s 35kg/s 50ºC [30, 40, 76.7] 840kW 17.7kg/s [31, 28.4, 60] C1 Sink 4 100℃ H1 86.4℃ E1 35.7℃ 2.4kg/s 9.4kg/s 40kg/s 100ºC 57.7kg/s 22.7kg/s 20.6kg/s [30, 40, 76.7] 3287kW 96.4℃ E1 48.4℃ 28.4℃ 30kg/s 32kg/s 4.5kg/s 14.9kg/s 100ºC 14499kW 25.1kg/s 60ºC Source 1 51.2kg/s 7742kW 27.5kg/s Source 2 71.8kg/s Waste Source 4 49.8℃ C2 25.0℃ 56kg/s 2.4kg/s 59.3kg/s Sink 13.3kg/s 95ºC [25.7, 37.4, 100] 100kg/s 100kg/s 30℃ 158.8kg/s Plant 1 TAC= $3,594,621 [38.6, 53.6, 80] 22.6kg/s 45kg/s 60ºC Sink 6 1.3kg/s 18.6kg/s 45kg/s 45ºC 6579kW Sink 7 2786kW Source 8 [40, 39.9, 74.7] Sink 5 14.7kg/s 38kg/s 3.8kg/s H1 74.4℃ 49.2℃ E1 20.3℃ Source 5 21.4kg/s 32kg/s 35ºC 31.7kg/s 21.7kg/s 20ºC 70.4℃ E1 41.6℃ 40kg/s 19.3kg/s [50, 49.1, 91.8] 17kg/s 7.3kg/s 16.4kg/s F C r o 70ºC e l

2786kW 42kg/s 65ºC s d 65.0℃ 15.8℃ h E2

12.7kg/s 128kg/s u 1.7kg/s 21kg/s w [30, 46.3, 68.4] t i a l i t t Sink 8 e 4kg/s 5858kW y r Source 6 28.3kg/s Waste 12kg/s 55kg/s E2 25.8℃ 30kg/s Sink 43kg/s 10ºC 75ºC Source 7 5858kW [40, 39.9, 74.7] Plant 2 TAC= $3,027,041

Figure 4. Optimal stand-alone WAHEN designs in Case 1 of Example 2

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 48/101

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ MINLP model

35kg/s 50ºC [44, 36, 60] 7kg/s 15194kW Source 1 Sink1 96.6℃ E2 45.7℃ 21kg/s 30kg/s 100ºC 23kg/s 40kg/s 20ºC 28.7℃ E1 48kg/s 22.9kg/s [40, 39.1, 86.5] Source 2 8kg/s 1469kW Sink2 42.6kg/s 56kg/s 95ºC 10.5kg/s 28kg/s 95ºC 35.7℃ E2 86.6℃ Sink3 23.7kg/s [30, 40, 76.7] Source 3 15194kW 17.5 4.7kg/s kg/s 36kg/s 40kg/s 100ºC 50ºC 3kg/s 3kg/s H1 Source 4 11.5kg/s 57.5kg/s [30, 40, 76.7] 32kg/s 60ºC 3233kW Sink4

9.3kg/s 15.8kg/s 62.5kg/s 5.5kg/s 3kg/s 7kg/s 18.2℃

E1 Plant 1 1469kW

Plant 2 227.7kg/s [30, 41.6, 93] Waste Sink

2kg/s Sink5 21.1 12.6 Source 5 1.4kg/s 32kg/s 35ºC kg/s kg/s 40kg/s 17.5kg/s [50, 50, 116.4] 70ºC 45kg/s 45ºC Sink6 8919kW 17.1kg/s 30kg/s 21.1kg/s 22kg/s [50, 42.5, 96.2] 22kg/s 75ºC 14.6kg/s 76.1℃ E3 33.8℃ Source 6 21kg/s 11kg/s Source 7 45kg/s 60ºC 55kg/s 42.4kg/s [32.3, 51, 80] 10ºC 42kg/s 65ºC 36.8kg/s Sink8 Sink7 8919kW [30, 52.5, 80] 13.8kg/s 23.2kg/s 23.6℃ E3 65.6℃ 27.4kg/s 38kg/s 20ºC Source 8 3.1kg/s

Figure 5. Optimal direct integrated WAHEN design in Case 2 of Example 2

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 49/101

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

MINLP model

0.1kg/s [42.9, 40, 60] 30kg/s Source 1 12.1kg/s 35kg/s 50ºC 100ºC Sink 1 0.5kg/s 10488kW 8.6kg/s 8.8kg/s Source 2 28kg/s 95ºC 95.1℃ E1 Sink 3 56kg/s 42.6kg/s [40, 40, 98.4] 18.1kg/s 95ºC 4.3kg/s

41.6kg/s 5.4kg/s 35.9kg/s 43kg/s 37℃ 4.7kg/s E1 8.4kg/s 10488kW 36kg/s 6.7kg/s 85.1℃ 2184kW Sink 4 50ºC 40kg/s 100ºC Source 3 3.8kg/s 85.1℃ H1 101℃ 25.9kg/s 32.5kg/s [30, 31, 80] 15.4kg/s Source 4 32kg/s 60ºC 0.7kg/s 40kg/s 20ºC 24.6kg/s [36.9, 55, 81.5] Plant 1 Sink 2

[30, 44.4, 97.1] Waste 220kg/s Jun 1 [50.1, 98.2, 173.2] Jun 2 [60, 89.4, 132.5] Sink

Source 5 40kg/s 70ºC 3.7kg/s 19.4kg/s 25.5 3.6kg/s 16.7kg/s 7.8kg/s 45.9kg/s kg/s 32kg/s 35ºC Source 6 5737kW 14.4kg/s 12.5kg/s [50, 50, 90] 30kg/s 74.0℃ E2 37.8℃ Sink 5 75ºC 1.5kg/s Sink 6 5.9kg/s 45kg/s 45ºC 9.1kg/s 4.0kg/s [40, 56, 100] 8.0kg/s 17.9kg/s 55kg/s 12.6kg/s 14.7kg/s Source 8 10ºC Source 7 2.5kg/s 38kg/s 20.1kg/s 20.7kg/s 20ºC 7.9kg/s Sink 8 13.7kg/s 42kg/s 65ºC E2 63.7℃ 45kg/s 35.3kg/s 21.6kg/s [30, 44.4, 80] 60ºC Sink 7 5737kW Plant 2 [40, 43, 80]

Michael Saunders: Optimization softwareFigure 6. Optimal at SOL indirect integrated WAHEN design in Case 3 of ExampleCLAODE, 2 ICMS, Edinburgh Oct 31–Nov 1, 2013 50/101

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ MINLP model

Lower total annualized cost can be obtained in all examples by solving the corresponding MINLP model

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 51/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

PDCO

Primal-dual interior method for convex optimization

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 52/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ PDCO (Matlab primal-dual convex optimizer)

Nominally

minimize φ(x) x subject to Ax = b, ` ≤ x ≤ u

φ(x) is convex with known gradient and Hessian T P For example, φ(x) = c x or λ kxk1 or xj log(xj )

A may be a sparse matrix or an operator for Av and ATw

Basis Pursuit (BP) Basis Pursuit denoising (BPDN) Chen, Donoho, & S (2001)

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 53/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ PDCO (Matlab primal-dual convex optimizer)

To ensure unique solutions, PDCO solves regularized problems:

1 2 1 2 minimize φ(x) + kD1xk + krk x, r 2 2 subject to Ax + D2r = b, ` ≤ x ≤ u

where D1, D2 are diagonal and positive-deﬁnite

−3 −4 D1 = γI γ = 10 or 10 −3 −4 D2 = δI for linear programs δ = 10 or 10

D2 = I for least squares

Jacek prefers D1, D2 semi-deﬁnite and dynamic!

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 54/101 Jon Dattorro (2010) 10,000 × 200,000 1.1M nonzeros Solve 200,000 times with diﬀerent b Gurobi: average 2 mins PDCO (Matlab): average 1 min

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ PDCO for LP feasibility

Regularized least squares with bounds:

minimize 1 kγxk2 + 1 krk2 x, r 2 2 subject to Ax + r = b, ` ≤ x ≤ u

−4 −4 γ = 10 declare feasible if krk∞ ≤ 10 say

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 55/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ PDCO for LP feasibility

Regularized least squares with bounds:

minimize 1 kγxk2 + 1 krk2 x, r 2 2 subject to Ax + r = b, ` ≤ x ≤ u

−4 −4 γ = 10 declare feasible if krk∞ ≤ 10 say

Jon Dattorro (2010) 10,000 × 200,000 1.1M nonzeros Solve 200,000 times with diﬀerent b Gurobi: average 2 mins PDCO (Matlab): average 1 min

T FBA minimize d ve vf ,vr ,ve subject to Svf − Svr + Se ve = 0

vf , vr ≥ 0, ` ≤ ve ≤ u

d optimizes a biological objective e.g., maximize replication rate in unicellular organisms

ve = exchange ﬂuxes = sources and sinks of chemicals T 2 T PDCO works with A = S −SSe then LL = AD A (sparse Cholesky with D increasingly ill-conditioned) ∗ ∗ ∗ ∗ Solution is v = vf − vr and ve

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 56/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ PDCO applied to Entropy problem

T T EP minimize vf (log vf +c −e) + vr (log vr +c −e) vf ,vr ∗ subject to Svf − Svr = − Se ve

vf , vr > 0

c = any vector, e = (1, 1,..., 1)T ∗ ve = optimal exchange ﬂuxes from FBA Entropy objective function is strictly convex ∗ ∗ Solution vf , vr is thermodynamically feasible (satisﬁes energy conservation and 2nd law of thermodynamics)

Fleming, Maes, S, Ye, Palsson (2012)

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 57/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ PDCO applied to LR-NMR problems Laplace Inversion of Low-Resolution NMR Relaxometry Data Biotech and Environmental Engineering, Ben Gurion Univ, Israel Determine composition of olive oil, rapeseed oil, biodiesel, . . .

Z ∞ −t/T2 s(t) ≈ e x(T2) dT2 x = probability density 0 Standard method (discretize): 2 A s min x − x≥0 λI 0 2 A = discrete Laplace transform (fast operator) λ = Tikhonov regularization Could apply PDCO to handle x ≥ 0 Distorts solution by broadening peaks Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 58/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ PDCO applied to LR-NMR problems

Laplace inversion via basis pursuit denoising:

BPDN minimize α kxk + 1 kλxk2 + 1 krk2 x, r 1 2 2 2 2 subject to Ax + r = s : y x ≥ 0

A = discrete Laplace transform (fast operator) T DA Dw PDCO solves min ∆y − using LSMR, where posdef diagonal D I t becomes increasingly ill-conditioned α helps resolve close adjacent peaks

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 59/101 LAPLACE INVERSION OF LR-NMR RELAXOMETRY DATA 85 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

α = 0

α > 0

Figure 6 Comparison of WinDXP (a)–(d) and PDCO using the universal regularization values for a1 and a2 (e)–(h) solutions on a real LR-NMR dataset acquired from an oil sample. The results are ordered by descending number of scans (descending SNR). Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 60/101

Concepts in Magnetic Resonance Part A (Bridging Education and Research) DOI 10.1002/cmr.a Problem setup (normal numerical people):

[U,S,V] = svd(A);

Astounding line of code (one coauthor):

[S,V,D] = svd(A); (!)

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ PDCO applied to LR-NMR problems

Berman, Leshem, Etziony, Levi, Parmet, S, Wiesman 2013 Novel 1H low ﬁeld nuclear magnetic resonance applications for the ﬁeld of biodiesel Biotechnology for Biofuels 6:55, 20pp Berman, Levi, Parmet, S, Wiesman 2013 Laplace inversion of low-resolution NMR relaxometry data using sparse representation methods Concepts in Magnetic Resonance Part A 42A:3, 72–88

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 61/101 Astounding line of code (one coauthor):

[S,V,D] = svd(A); (!)

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ PDCO applied to LR-NMR problems

Problem setup (normal numerical people):

[U,S,V] = svd(A);

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 61/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ PDCO applied to LR-NMR problems

Problem setup (normal numerical people):

[U,S,V] = svd(A);

Astounding line of code (one coauthor):

[S,V,D] = svd(A); (!)

SQOPT in quad precision

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 62/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Flux Balance Analysis (FBA) on Thermotoga maritima

min cTv subject to Sv = 0, ` ≤ v ≤ u S rows and cols 18210 × 17535 Nonzero Sij 33602 4 −6 max and min |Sij | 2 × 10 and 3 × 10

SQOPT in double precision (15 digits) Feasibility tol 1e-6 Optimality tol 1e-6

SQOPT in quad precision (32 digits) Feasibility tol 1e-15 Optimality tol 1e-15

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 63/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Flux Balance Analysis (FBA) on Thermotoga maritima

min cTv subject to Sv = 0, ` ≤ v ≤ u S rows and cols 18210 × 17535 Nonzero Sij 33602 4 −6 max and min |Sij | 2 × 10 and 3 × 10

SQOPT in double precision (42 secs) SQOPT EXIT 10 -- the problem appears to be infeasible

Problem name ThMa No. of iterations 18500 Objective value 8.2286249495E-07 No. of infeasibilities 9 Sum of infeas 1.9606461069E-03 No. of degenerate steps 11611 Percentage 62.76 Max x (scaled) 3482 8.2E+00 Max pi (scaled) 18210 9.8E-01 Max x 5134 5.9E+00 Max pi 18210 1.0E+00 Max Prim inf(scaled) 32832 1.3E-03 Max Dual inf(scaled) 16417 1.0E+00 Max Primal infeas 32832 5.6E-06 Max Dual infeas 32669 2.3E+02

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 64/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Flux Balance Analysis (FBA) on Thermotoga maritima

min cTv subject to Sv = 0, ` ≤ v ≤ u S rows and cols 18210 × 17535 Nonzero Sij 33602 4 −6 max and min |Sij | 2 × 10 and 3 × 10

Restart SQOPT in quad precision (36 secs) SQOPT EXIT 0 -- finished successfully

Problem name ThMa No. of iterations 498 Objective value 8.7036461686E-07 No. of infeasibilities 0 Sum of infeas 0.0000000000E+00 No. of degenerate steps 220 Percentage 44.18 Max x (scaled) 3482 8.2E+00 Max pi (scaled) 2907 1.3E+00 Max x 5134 5.9E+00 Max pi 15517 1.1E+00 Max Prim inf(scaled) 16475 5.2E-28 Max Dual inf(scaled) 13244 1.9E-32 Max Primal infeas 16475 5.2E-29 Max Dual infeas 13244 4.8E-33

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 65/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Flux Balance Analysis (FBA) on GlcAer

min cTv subject to Sv = 0, ` ≤ v ≤ u S rows and cols 68300 × 76664 Nonzero Sij 926357 5 −5 max and min |Sij | 8 × 10 and 5 × 10

SQOPT in quad precision cold start, no scaling (30786 secs) SQOPT EXIT 0 -- finished successfully

Problem name GlcAer No. of iterations 84685 Objective value -7.0382454070E+05 No. of degenerate steps 62127 Percentage 73.36 Max x 61436 6.3E+07 Max pi 25539 2.4E+07 Max Primal infeas 72623 3.0E-21 Max Dual infeas 17817 2.7E-21

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 66/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Flux Balance Analysis (FBA) on GlcAer

min cTv subject to Sv = 0, ` ≤ v ≤ u S rows and cols 68300 × 76664 Nonzero Sij 926357 5 −5 max and min |Sij | 8 × 10 and 5 × 10

SQOPT in quad precision cold start, with scaling (4642 secs) SQOPT EXIT 0 -- finished successfully

Problem name GlcAer No. of iterations 37025 Objective value -7.0382454070E+05 No. of degenerate steps 28166 Percentage 76.07 Max x (scaled) 59440 3.7E+00 Max pi (scaled) 40165 8.1E+11 Max x 61436 6.3E+07 Max pi 25539 2.4E+07 Max Prim inf(scaled) 81918 7.0E-16 Max Dual inf(scaled) 59325 1.5E-17 Max Primal infeas 81918 1.3E-07 Max Dual infeas 27953 2.0E-22

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 67/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Flux Balance Analysis (FBA) on GlcAer

min cTv subject to Sv = 0, ` ≤ v ≤ u S rows and cols 68300 × 76664 Nonzero Sij 926357 5 −5 max and min |Sij | 8 × 10 and 5 × 10

SQOPT in quad precision warm start, no scaling (28 secs) SQOPT EXIT 0 -- finished successfully

Problem name GlcAer No. of iterations 1 Objective value -7.0382454070E+05 No. of degenerate steps 0 Percentage 0.00 Max x 61436 6.3E+07 Max pi 25539 2.4E+07 Max Primal infeas 141186 7.1E-21 Max Dual infeas 14993 8.9E-23

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 68/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Aerospace Applications

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 69/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ NASA Aerospace Applications

David Saunders 1970 Visit Stanford for 1 month (now 43 years) 1973 Serra House, RA, MS (thanks to Gene) 1974–present NASA Ames Projects OAW Oblique All-Wing supersonic airliner HSCT Supersonic airliner CTV SHARP shuttle design MSL Heat shield for landing Mars rover Curiosity CEV Heat shield for Apollo-type capsule to ISS/moon

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 70/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ OAW oblique all wing airliner

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 71/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ HSCT high speed civil transport

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 72/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

CTV crew transfer vehicle

SHARP design (Slender Hypervelocity Aerothermodynamic Research Probes) Aerothermal performance constraint in (Velocity, Altitude) space, used during trajectory optimization with UHTC materials (Ultra High Temperature Ceramics) to avoid exceeding material limits Trajectory optimization with SNOPT Could always abort to Kennedy, Boston, Gander, or Shannon 4000-mile cross-range capability during reentry

Image credit: David Kinney, NASA Ames Research Center

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 73/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ CEV crew exploration vehicle

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 74/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ McDonnell-Douglas Aerospace Applications

Philip Gill, Rocky Nelson 1979–1988 SOL QPSOL, LSSOL, NPSOL 1988–2007 UC San Diego QPOPT, SQOPT, SNOPT McDonnell-Douglas Space Systems, LA (now Boeing) Projects F-4 Minimum time-to-climb DC-Y SSTO Minimum-fuel landing maneuver

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 75/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ F-4 minimum time-to-climb

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 76/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ DC-Y single-stage-to-orbit

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 77/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 78/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 79/101 Don’t exceed 3g

Optimum starting altitude = 1400ft(!)

Come back Alan Shephard!

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ DC-Y landing, 2nd OTIS/NPSOL optimization

1st optimization: starting altitude = 2800ft 2nd optimization: starting altitude = variable New constraint needed:

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 80/101 Optimum starting altitude = 1400ft(!)

Come back Alan Shephard!

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ DC-Y landing, 2nd OTIS/NPSOL optimization

1st optimization: starting altitude = 2800ft 2nd optimization: starting altitude = variable New constraint needed: Don’t exceed 3g

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 80/101 Come back Alan Shephard!

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ DC-Y landing, 2nd OTIS/NPSOL optimization

1st optimization: starting altitude = 2800ft 2nd optimization: starting altitude = variable New constraint needed: Don’t exceed 3g

Optimum starting altitude = 1400ft(!)

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 80/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ DC-Y landing, 2nd OTIS/NPSOL optimization

1st optimization: starting altitude = 2800ft 2nd optimization: starting altitude = variable New constraint needed: Don’t exceed 3g

Optimum starting altitude = 1400ft(!)

Come back Alan Shephard!

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 80/101 Antony Jameson, Juan Alonso Aerospace Computing Lab MDO Multidisciplinary Design Optimization ASO Aerodynamic Shape Optimization

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Stanford Aerospace Applications

Ilan Kroo Stanford Aircraft Aerodynamics and Design Group Blended Wing-Body Transonic airliner

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 81/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Stanford Aerospace Applications

Ilan Kroo Stanford Aircraft Aerodynamics and Design Group Blended Wing-Body Transonic airliner Antony Jameson, Juan Alonso Aerospace Computing Lab MDO Multidisciplinary Design Optimization ASO Aerodynamic Shape Optimization

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 82/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Blended wing airliner

Ilan Kroo, Michael Holden, Aero/Astro Dept, Stanford 1999

Compute control for stable flight of 17ft-span flying model Collocation model: minimize wing weight (or move CG as far aft as possible) subject to flutter constraints 9000 nonlinear equations 9000 state variables, 7 design variables MINOS 5.5: 26 major iterations 4000 minor iterations (first 3000 = LP) 1500 constraint + Jacobian evaluations = 60000 function evaluations 3 days on SGI Octane

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 83/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ News Flash, 3 March 2007

Mike Ross Naval Postgraduate School, Monterey DIDO: A package for solving optimal control problems Implemented in Matlab Calls TOMLAB/SNOPT for the optimization

GMT 062:19:26 The International Space Station was successfully maneuvered using DIDO/TOMLAB/SNOPT Found zero propellant solutions (globally optimal) Saved NASA $1M fuel cost

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 84/101 (Score 8–5) Come on whale, where are you? (Score 8–6) Dratted whales, we can’t trust them anymore I haven’t had so much fun since the cat died ... (Score 8–9) In the immortal words, ’bugga’

AC 2013 (San Francisco Bay): NZ Herald blog

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ America’s Cup Yacht Race AC 95: NPSOL was used in different ways by both AC95 finalists For Team Dennis Conner, NPSOL was used with Boeing’s TRANAIR CFD system for optimal hull design of Young America For Team NZ, Andy Philpott (University of Auckland) used NPSOL to maximize the velocity around the course of 11 potential hull designs. One design appeared significantly faster and was chosen to become Black Magic

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 85/101 (Score 8–5) Come on whale, where are you? (Score 8–6) Dratted whales, we can’t trust them anymore I haven’t had so much fun since the cat died ... (Score 8–9) In the immortal words, ’bugga’

AC 2013 (San Francisco Bay): NZ Herald blog

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 85/101 (Score 8–6) Dratted whales, we can’t trust them anymore I haven’t had so much fun since the cat died ... (Score 8–9) In the immortal words, ’bugga’

AC 2013 (San Francisco Bay): NZ Herald blog (Score 8–5) Come on whale, where are you?

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 85/101 I haven’t had so much fun since the cat died ... (Score 8–9) In the immortal words, ’bugga’

AC 2013 (San Francisco Bay): NZ Herald blog (Score 8–5) Come on whale, where are you? (Score 8–6) Dratted whales, we can’t trust them anymore

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 85/101 ... (Score 8–9) In the immortal words, ’bugga’

AC 2013 (San Francisco Bay): NZ Herald blog (Score 8–5) Come on whale, where are you? (Score 8–6) Dratted whales, we can’t trust them anymore I haven’t had so much fun since the cat died

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 85/101 (Score 8–9) In the immortal words, ’bugga’

AC 2013 (San Francisco Bay): NZ Herald blog (Score 8–5) Come on whale, where are you? (Score 8–6) Dratted whales, we can’t trust them anymore I haven’t had so much fun since the cat died ...

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 85/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ America’s Cup Yacht Race AC 95: NPSOL was used in different ways by both AC95 finalists For Team Dennis Conner, NPSOL was used with Boeing’s TRANAIR CFD system for optimal hull design of Young America For Team NZ, Andy Philpott (University of Auckland) used NPSOL to maximize the velocity around the course of 11 potential hull designs. One design appeared significantly faster and was chosen to become Black Magic

AC 2013 (San Francisco Bay): NZ Herald blog (Score 8–5) Come on whale, where are you? (Score 8–6) Dratted whales, we can’t trust them anymore I haven’t had so much fun since the cat died ... (Score 8–9) In the immortal words, ’bugga’

Optimization in NZ

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 86/101 Robust solutions RECOVERY CARE gives you ﬁnancial protection from speciﬁed sudden illness. You get cash if you live . . . and cash if you don’t.

No objective function People have been marrying and bringing up children for centuries now. Nothing has ever come of it. (Evening Post, 1977)

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

In New Zealand, the radio/TV guide is called The Listener. Every week a Life in New Zealand column publishes clippings describing local events. The ﬁrst sender receives a $5 Lotto Lucky Dip. The following clippings illustrate some characteristics of optimization problems in the real(?) world.

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 87/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Robust solutions RECOVERY CARE gives you ﬁnancial protection from speciﬁed sudden illness. You get cash if you live . . . and cash if you don’t.

No objective function People have been marrying and bringing up children for centuries now. Nothing has ever come of it. (Evening Post, 1977)

Multiple objectives “I had the choice of running over my team-mate or going onto the grass, so I ran over my team-mate then ran onto the grass”, Rymer recalled later.

Obvious objective He said the fee was increased from $5 to $20 because some people had complained it was not worth writing a cheque for $5.

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 88/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Equilibrium condition “The pedestrian count was not considered high enough to justify an overbridge”, Helen Ritchie said. “And if there continues to be people knocked down on the crossing, the number of pedestrians will dwindle.”

Constraints ENTERTAINERS, DANCE BAND, etc. Vocalist wanted for New Wave rock band, must be able to sing.

DRIVING INSTRUCTOR Part-time position. No experience necessary.

HOUSE FOR REMOVAL in excellent order, $800. Do not disturb tenant.

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 89/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Exactly one feasible solution MATTHEWS RESTAURANT, open 365 nights. Including Mondays.

Buying your own business might mean working 24 hours a day. But at least when you’re self-employed you can decide which 24.

Peters: Oh, it’s not that I don’t want to be helpful. But in this case the answer is that I don’t want to be helpful. (Listener, 1990)

Sergeant J Johnston said when Hall was stopped by a police patrol the defendant denied being the driver, but after it was pointed out he was the only person in the car he admitted to being the driver.

His companion was in fact a transvestite, X, known variously as X or X.

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 90/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Bound your variables By the way, have you ever seen a bird transported without the use of a cage? If you don’t use a cage it will ﬂy away and maybe the same could happen to your cat. Mark my words, we have seen it happen.

Redundant constraints If you are decorating before the baby is born, keep in mind that you may have a boy or a girl.

EAR PIERCING while you wait

CONCURRENT TERM FOR BIGAMY (NZ Herald, 1990)

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 91/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Infeasible constraints I chose to cook myself to be quite sure what was going into the meals.

We apologize to Wellington listeners who may not be receiving this broadcast.

The model 200 is British all the way from its stylish rooﬂine to its French-made Michelin tyres. (NZ Car Magazine)

BALD, 36 yr old, handsome male seeking social times and fun with bald 22 years and upwards female Napier Courier, 28/2/02

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 92/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

≥ or ≤? BUY NOW! At $29.95 these jeans will not last long!

NOT TOO GOOD TO BE TRUE! We can sell your home for much less than you’d expect! (NZ Property Weekly)

The BA 146’s landing at Hamilton airport was barely audible above airport background noise, which admittedly included a Boeing 737 idling in the foreground.

Yesterday Mr Palmer said “The Australian reports are not correct that I’ve seen, although I can’t say that I’ve seen them”.

It will be a chance for all women of this parish to get rid of anything that is not worth keeping but is too good to throw away. Don’t forget to bring your husbands.

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 93/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

≥ or ≤? The French were often more blatant and more active, particularly prop X and number eight Y, but at least one All Black was seen getting his retaliation in ﬁrst.

WHAT EVERY TEENAGER SHOULD KNOW — PARENTS ONLY

“Love Under 17” Persons under 18 not admitted.

“Keeping young people in the dark would not stop them having sex—in fact it usually had the opposite eﬀect,” she said.

NELSON, approximately 5 minutes from airport. Golf course adjacent. Sleeps seven all in single beds. Ideal for honeymoons. (Air NZ News, 1978)

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 94/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Hard or soft constraints The two have run their farm as equal partners for 10 years, with Jan in charge of grass management, Lindsay looking after fertilizer, and both working in the milk shed. “We used to have our staff meetings in bed. That got more difficult when we employed staff!” (NZ country paper)

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 95/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Elastic constraints The Stationary Engine Drivers Union is planning rolling stoppages.

When this happens there are set procedures to be followed and they are established procedures, provided they are followed.

APATHY RAMPANT? Not in Albany—the closing of the electoral rolls saw fully 103.49 percent of the area’s eligible voters signed up.

Auckland City ratepayers are to be reminded that they can pay their rates after they die. (Auckland Herald, 1990)

He was remanded in custody to appear again on Tuesday if he is still in the country.

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 96/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Convergence “There is a trend to open libraries when people can use them”, he says.

Mayor for 15 years, Sir Dove-Myer wants a ﬁnal three years at the helm “to restore sanity and stability in the aﬀairs of the city”.

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 97/101 , but not when to do it.

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Applications (Yachting) It is not particularly dangerous, as it only causes vomiting, hot and cold ﬂushes, diarrhoea, muscle cramping, paralysis, and sometimes death . . . (Boating

New Zealand, 1990)

(Ecological models) CAR POLLUTION SOARS IN CHRISTCHURCH—BUT CAUSE REMAINS MYSTERY

Nappies wanted for window cleaning. Must be used.

(Optimal control) Almost half the women seeking fertility investigations at the clinic knew what to do to get pregnant

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 98/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Applications (Yachting) It is not particularly dangerous, as it only causes vomiting, hot and cold ﬂushes, diarrhoea, muscle cramping, paralysis, and sometimes death . . . (Boating

New Zealand, 1990)

(Ecological models) CAR POLLUTION SOARS IN CHRISTCHURCH—BUT CAUSE REMAINS MYSTERY

Nappies wanted for window cleaning. Must be used.

(Optimal control) Almost half the women seeking fertility investigations at the clinic knew what to do to get pregnant, but not when to do it.

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 98/101 Integer variables 0 or 1 or 2 . . .

When Taupo police arrested a Bay of Plenty man for driving over the limit, they discovered he was a bigamist. Nelson Mail, 5/04

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Binary variables 0 or 1 Sometimes neither is optimal

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 99/101 When Taupo police arrested a Bay of Plenty man for driving over the limit, they discovered he was a bigamist. Nelson Mail, 5/04

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Binary variables 0 or 1 Sometimes neither is optimal Integer variables 0 or 1 or 2 . . .

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 99/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Binary variables 0 or 1 Sometimes neither is optimal Integer variables 0 or 1 or 2 . . .

When Taupo police arrested a Bay of Plenty man for driving over the limit, they discovered he was a bigamist. Nelson Mail, 5/04

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 99/101 Always room for improvement The dean Jim Plummer said the three main qualities for the job were speed, agility and drinking skills. “Actually, Gene has none of those, but he’s still the best seminar organizer and party host we’ve had”, he said.

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 100/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ

Always room for improvement The owner Craig Andrew said the three main qualities for the job were speed, agility and driving skills. “Actually, Merv has none of those, but he’s still the best delivery boy we’ve had”, he said. Always room for improvement The dean Jim Plummer said the three main qualities for the job were speed, agility and drinking skills. “Actually, Gene has none of those, but he’s still the best seminar organizer and party host we’ve had”, he said.

S CCM

SC E ICM

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 100/101 that we don’t learn from history – G. W. F. Hegel

If you’re not ﬁred with enthusiasm, you’ll be ﬁred with enthusiasm – Vince Lombardi

Special thanks! Jacek Gondzio Rachael Tappenden EPSRC, Maxwell Institute

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Final thoughts We learn from history

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 101/101 If you’re not ﬁred with enthusiasm, you’ll be ﬁred with enthusiasm – Vince Lombardi

Special thanks! Jacek Gondzio Rachael Tappenden EPSRC, Maxwell Institute

SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Final thoughts We learn from history that we don’t learn from history – G. W. F. Hegel

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 101/101 you’ll be ﬁred with enthusiasm – Vince Lombardi

Special thanks! Jacek Gondzio Rachael Tappenden EPSRC, Maxwell Institute

If you’re not ﬁred with enthusiasm,

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 101/101 Special thanks! Jacek Gondzio Rachael Tappenden EPSRC, Maxwell Institute

If you’re not ﬁred with enthusiasm, you’ll be ﬁred with enthusiasm – Vince Lombardi

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 101/101 SOL Solvers (Ax ≈ b) Applications (Ax ≈ b) rank(S) Solvers (optimization) Applications (optimization) Aerospace ACNZ Final thoughts We learn from history that we don’t learn from history – G. W. F. Hegel

If you’re not ﬁred with enthusiasm, you’ll be ﬁred with enthusiasm – Vince Lombardi

Special thanks! Jacek Gondzio Rachael Tappenden EPSRC, Maxwell Institute

Michael Saunders: Optimization software at SOL CLAODE, ICMS, Edinburgh Oct 31–Nov 1, 2013 101/101