Revisiting Design Aspects of a QP Solver for WORHP 7Th International Conference on Astrodynamics Tools and Techniques (ICATT)

Revisiting Design Aspects of a QP Solver for WORHP 7th International Conference on Astrodynamics Tools and Techniques (ICATT)

Marcel Jacobse, Christof Büskens

University of Bremen, Center for Industrial Mathematics

November 9, 2018 Center for Industrial Mathematics Contents

1 Sequential Quadratic Programming Approach of WORHP

2 Solving WORHP’s Quadratic Subproblems

3 Eﬃcient Formulation of WORHP’s Subproblems

4 Numerical Results with the new QP Solver

SQP Approach Solving QPs QP Formulation Results 2 / 19 Center for Industrial Mathematics

We Optimize Really Huge Problems

Zentrum für Technomathematik 2 Optimierung & O C Optimale Steuerung

Solver for large-scale nonlinear optimization Interfaces for C, C++, Fortran, AMPL, ... Free for academics, www.worhp.de Method1: Sequential Quadratic Programming (SQP)

1Recently: New penalty interior point method, developed by Renke Kuhlmann SQP Approach Solving QPs QP Formulation Results 3 / 19 QP: 1 T [k] [k] T min d H d + ∇x f (x ) d d∈Rn 2 [k] [k] T s.t. g ≤ g(x ) + ∇x g(x ) d ≤ g

I Approximate NLP with a QP in each iteration

I Use direction d to get new iterate

x [k+1] := x [k] + αd

Center for Industrial Mathematics Sequential Quadratic Programming NLP: min f (x) x∈Rn s.t. g ≤ g(x) ≤ g

SQP Approach Solving QPs QP Formulation Results 4 / 19 I Use direction d to get new iterate

x [k+1] := x [k] + αd

Center for Industrial Mathematics Sequential Quadratic Programming NLP: QP: 1 T [k] [k] T min f (x) min d H d + ∇x f (x ) d n d∈ n x∈R R 2 [k] [k] T s.t. g ≤ g(x) ≤ g s.t. g ≤ g(x ) + ∇x g(x ) d ≤ g

I Approximate NLP with a QP in each iteration

SQP Approach Solving QPs QP Formulation Results 4 / 19 Center for Industrial Mathematics Sequential Quadratic Programming NLP: QP: 1 T [k] [k] T min f (x) min d H d + ∇x f (x ) d n d∈ n x∈R R 2 [k] [k] T s.t. g ≤ g(x) ≤ g s.t. g ≤ g(x ) + ∇x g(x ) d ≤ g

I Approximate NLP with a QP in each iteration

I Use direction d to get new iterate

x [k+1] := x [k] + αd

I Approximate NLP with a QP in each iteration

I Use direction d to get new iterate

x [k+1] := x [k] + αd

SQP Approach Solving QPs QP Formulation Results 4 / 19 Center for Industrial Mathematics Sequential Quadratic Programming

Line Search SQP Methods

I Need to solve a QP in every iteration

Inequality constrained QPs: Solution not trivial Common Solution Methods

Active-Set Methods Interior Point Methods

SQP Approach Solving QPs QP Formulation Results 5 / 19 Experience from using WORHP in many areas Development of new features Code refactoring

I Interest in reworked QP solver I “WORHP QP” (C++)

Center for Industrial Mathematics Solving the Quadratic Program

QPSOL

WORHP uses Fortran code QPSOL to solve the subproblems Mehrotra’s predictor-corrector variant of a primal-dual interior point method

SQP Approach Solving QPs QP Formulation Results 6 / 19 I Interest in reworked QP solver I “WORHP QP” (C++)

Center for Industrial Mathematics Solving the Quadratic Program

QPSOL

WORHP uses Fortran code QPSOL to solve the subproblems Mehrotra’s predictor-corrector variant of a primal-dual interior point method

Experience from using WORHP in many areas Development of new features Code refactoring

SQP Approach Solving QPs QP Formulation Results 6 / 19 Center for Industrial Mathematics Solving the Quadratic Program

QPSOL

WORHP uses Fortran code QPSOL to solve the subproblems Mehrotra’s predictor-corrector variant of a primal-dual interior point method

Experience from using WORHP in many areas Development of new features Code refactoring

I Interest in reworked QP solver I “WORHP QP” (C++)

SQP Approach Solving QPs QP Formulation Results 6 / 19 n X − µ log(xi ) i=1 − µe

Perturb problem with barrier parameter µ Apply Newton’s method QAT 0  ∆x Qx + c + ATy − z       A 0 0  ∆y = −  Ax − b  Z 0 X ∆z ZXe − µe

Center for Industrial Mathematics Interior Point Method QP in Standard Form Optimality Conditions

1 Qx + c + ATy − z = 0 min x TQx + cTx x∈Rn 2 Ax − b = 0 s.t. Ax = b, x ≥ 0 ZXe = 0

SQP Approach Solving QPs QP Formulation Results 7 / 19 Apply Newton’s method QAT 0  ∆x Qx + c + ATy − z       A 0 0  ∆y = −  Ax − b  Z 0 X ∆z ZXe − µe

Center for Industrial Mathematics Interior Point Method QP in Standard Form Optimality Conditions

n T 1 T T X Qx + c + A y − z = 0 min x Qx + c x − µ log(xi ) x∈ n 2 R i=1 Ax − b = 0 s.t. Ax = b, x ≥ 0 ZXe − µe = 0

Perturb problem with barrier parameter µ

SQP Approach Solving QPs QP Formulation Results 7 / 19 Center for Industrial Mathematics Interior Point Method QP in Standard Form Optimality Conditions

n T 1 T T X Qx + c + A y − z = 0 min x Qx + c x − µ log(xi ) x∈ n 2 R i=1 Ax − b = 0 s.t. Ax = b, x ≥ 0 ZXe − µe = 0

SQP Approach Solving QPs QP Formulation Results 7 / 19 x Tz I Based on µcurr = n choose µ = σµcurr with σ ∈ (0, 1) I Mehrotra’s method provides eﬃcient heuristic choice for σ

Take maximum stepsize α ∈ (0, 1] such that new iterates

x + := x + α∆x > 0 and z+ := z + α∆z > 0

Keep iterates somewhat close to center, xi zi ≈ µ Let µ → 0 progressively throughout iterations

Center for Industrial Mathematics Interior Point Method Apply Newton’s method

QAT 0  ∆x Qx + c + ATy − z       A 0 0  ∆y = −  Ax − b  Z 0 X ∆z ZXe − µe

SQP Approach Solving QPs QP Formulation Results 8 / 19 x Tz I Based on µcurr = n choose µ = σµcurr with σ ∈ (0, 1) I Mehrotra’s method provides eﬃcient heuristic choice for σ

Keep iterates somewhat close to center, xi zi ≈ µ Let µ → 0 progressively throughout iterations

Center for Industrial Mathematics Interior Point Method Apply Newton’s method

QAT 0  ∆x Qx + c + ATy − z       A 0 0  ∆y = −  Ax − b  Z 0 X ∆z ZXe − µe

Take maximum stepsize α ∈ (0, 1] such that new iterates

x + := x + α∆x > 0 and z+ := z + α∆z > 0

SQP Approach Solving QPs QP Formulation Results 8 / 19 x Tz I Based on µcurr = n choose µ = σµcurr with σ ∈ (0, 1) I Mehrotra’s method provides eﬃcient heuristic choice for σ

Center for Industrial Mathematics Interior Point Method Apply Newton’s method

QAT 0  ∆x Qx + c + ATy − z       A 0 0  ∆y = −  Ax − b  Z 0 X ∆z ZXe − µe

Take maximum stepsize α ∈ (0, 1] such that new iterates

x + := x + α∆x > 0 and z+ := z + α∆z > 0

Keep iterates somewhat close to center, xi zi ≈ µ Let µ → 0 progressively throughout iterations

SQP Approach Solving QPs QP Formulation Results 8 / 19 Center for Industrial Mathematics Interior Point Method Apply Newton’s method

QAT 0  ∆x Qx + c + ATy − z       A 0 0  ∆y = −  Ax − b  Z 0 X ∆z ZXe − µe

Take maximum stepsize α ∈ (0, 1] such that new iterates

x + := x + α∆x > 0 and z+ := z + α∆z > 0

Keep iterates somewhat close to center, xi zi ≈ µ Let µ → 0 progressively throughout iterations x Tz I Based on µcurr = n choose µ = σµcurr with σ ∈ (0, 1) I Mehrotra’s method provides eﬃcient heuristic choice for σ

SQP Approach Solving QPs QP Formulation Results 8 / 19 2 Calculate average complementarity product of predicted iterates

(x + α ∆x )T(z + α ∆z ) µ := pred pred pred pred pred n

µ 2 3 Set reduction factor σ := pred µcorr 4 Solve Newton-System for µ = σµcurr I corrector step ∆corr 5 Combine for ﬁnal direction

∆ := ∆pred + ∆corr

Center for Industrial Mathematics Mehrotra Predictor-Corrector

1 Solve Newton-System for µ = 0 I predictor step ∆pred

SQP Approach Solving QPs QP Formulation Results 9 / 19 4 Solve Newton-System for µ = σµcurr I corrector step ∆corr 5 Combine for ﬁnal direction

∆ := ∆pred + ∆corr

Center for Industrial Mathematics Mehrotra Predictor-Corrector

1 Solve Newton-System for µ = 0 I predictor step ∆pred 2 Calculate average complementarity product of predicted iterates

(x + α ∆x )T(z + α ∆z ) µ := pred pred pred pred pred n

µ 2 3 Set reduction factor σ := pred µcorr

SQP Approach Solving QPs QP Formulation Results 9 / 19 Center for Industrial Mathematics Mehrotra Predictor-Corrector

1 Solve Newton-System for µ = 0 I predictor step ∆pred 2 Calculate average complementarity product of predicted iterates

(x + α ∆x )T(z + α ∆z ) µ := pred pred pred pred pred n

µ 2 3 Set reduction factor σ := pred µcorr 4 Solve Newton-System for µ = σµcurr I corrector step ∆corr 5 Combine for ﬁnal direction

∆ := ∆pred + ∆corr

SQP Approach Solving QPs QP Formulation Results 9 / 19  γµ if x˜ z˜ ≤ γµ  i i  1 1 2 Calculate target ti := γ µ if x˜i z˜i ≥ γ µ  x˜i z˜i otherwise 3 Use right hand side − 0, 0, X˜ Ze˜ − t to compute corrector direction ∆corr

Center for Industrial Mathematics Added Extensions for new solver WORHP QP

Gondzio’s Multiple Centrality Correctors

1 For an aspired stepsize α˜ > αpred calculate intermediary iterates

x˜ := x +α ˜∆xpred and z˜ := z +α ˜∆zpred

SQP Approach Solving QPs QP Formulation Results 10 / 19 3 Use right hand side − 0, 0, X˜ Ze˜ − t to compute corrector direction ∆corr

Center for Industrial Mathematics Added Extensions for new solver WORHP QP

Gondzio’s Multiple Centrality Correctors

1 For an aspired stepsize α˜ > αpred calculate intermediary iterates

x˜ := x +α ˜∆xpred and z˜ := z +α ˜∆zpred  γµ if x˜ z˜ ≤ γµ  i i  1 1 2 Calculate target ti := γ µ if x˜i z˜i ≥ γ µ  x˜i z˜i otherwise

SQP Approach Solving QPs QP Formulation Results 10 / 19 Center for Industrial Mathematics Added Extensions for new solver WORHP QP

Gondzio’s Multiple Centrality Correctors

1 For an aspired stepsize α˜ > αpred calculate intermediary iterates

x˜ := x +α ˜∆xpred and z˜ := z +α ˜∆zpred  γµ if x˜ z˜ ≤ γµ  i i  1 1 2 Calculate target ti := γ µ if x˜i z˜i ≥ γ µ  x˜i z˜i otherwise 3 Use right hand side − 0, 0, X˜ Ze˜ − t to compute corrector direction ∆corr

SQP Approach Solving QPs QP Formulation Results 10 / 19 Center for Industrial Mathematics Added Extensions for new solver WORHP QP

Weighted Correctors

Instead of combining ∆ := ∆pred + ∆corr combine

∆ := ∆pred + ω∆corr

with a weight ω ∈ (0, 1) such that allowed stepsize is maximized. Use simple line search to avoid bi-level optimization problem

SQP Approach Solving QPs QP Formulation Results 11 / 19 QPSOL’s Formulation 1 min x TQx + cTx x∈Rn 2 s.t. x ≤ x ≤ x Ax = b Cx ≤ d

I Formulation limitations require WORHP to: [k] T Split ∇x g(x ) into A and C Need to duplicate constraints with g > −∞ and g < ∞ i i Create extra row in A for all ﬁxed variables x i = x i

Center for Industrial Mathematics Quadratic Problem Formulation

WORHP’s Subproblem

1 T [k] [k] T min d H d + ∇x f (x ) d d∈Rn 2 s.t. x ≤ x [k] + d ≤ x [k] [k] T g ≤ g(x ) + ∇x g(x ) d ≤ g

SQP Approach Solving QPs QP Formulation Results 12 / 19 I Formulation limitations require WORHP to: [k] T Split ∇x g(x ) into A and C Need to duplicate constraints with g > −∞ and g < ∞ i i Create extra row in A for all ﬁxed variables x i = x i

Center for Industrial Mathematics Quadratic Problem Formulation

WORHP’s Subproblem QPSOL’s Formulation

1 T 1 min dTH[k]d + ∇ f (x [k]) d min x TQx + cTx x n d∈Rn 2 x∈R 2 s.t. x ≤ x [k] + d ≤ x s.t. x ≤ x ≤ x [k] [k] T Ax = b g ≤ g(x ) + ∇x g(x ) d ≤ g Cx ≤ d

SQP Approach Solving QPs QP Formulation Results 12 / 19 Center for Industrial Mathematics Quadratic Problem Formulation

WORHP’s Subproblem QPSOL’s Formulation

1 T 1 min dTH[k]d + ∇ f (x [k]) d min x TQx + cTx x n d∈Rn 2 x∈R 2 s.t. x ≤ x [k] + d ≤ x s.t. x ≤ x ≤ x [k] [k] T Ax = b g ≤ g(x ) + ∇x g(x ) d ≤ g Cx ≤ d

I Formulation limitations require WORHP to: [k] T Split ∇x g(x ) into A and C Need to duplicate constraints with g > −∞ and g < ∞ i i Create extra row in A for all ﬁxed variables x i = x i

SQP Approach Solving QPs QP Formulation Results 12 / 19 Center for Industrial Mathematics Quadratic Problem Formulation

Reworked Solver WORHP QP

1 min x TQx + cTx d∈Rn 2 s.t. x ≤ x ≤ x b ≤ Ax ≤ b

No reordering into equality and inequality constraints needed Fixed variables are eliminated from the problem Constraints with both sided bounds are handled internally

SQP Approach Solving QPs QP Formulation Results 13 / 19 z+z zd := z − z and za := 2

T Qx + c + A zd = 0 Ax − s = 0

(Za − Zd /2)(s − b) = 0 (Za + Zd /2) b − s = 0

Optimality Conditions

Qx + c − ATz + ATz = 0 Ax − s = 0 Z (s − b) = 0 Z b − s = 0

Center for Industrial Mathematics Handling both sided constraints 1 L(x, z, z) = x TQx + cTx + zT(b − Ax) + zT(Ax − b) 2

I Introduce slack variables −b ≤ s ≤ b

SQP Approach Solving QPs QP Formulation Results 14 / 19 z+z zd := z − z and za := 2

T Qx + c + A zd = 0 Ax − s = 0

(Za − Zd /2)(s − b) = 0 (Za + Zd /2) b − s = 0

Center for Industrial Mathematics Handling both sided constraints 1 L(x, z, z) = x TQx + cTx + zT(b − Ax) + zT(Ax − b) 2

I Introduce slack variables −b ≤ s ≤ b Optimality Conditions

Qx + c − ATz + ATz = 0 Ax − s = 0 Z (s − b) = 0 Z b − s = 0

SQP Approach Solving QPs QP Formulation Results 14 / 19 Center for Industrial Mathematics Handling both sided constraints 1 L(x, z, z) = x TQx + cTx + zT(b − Ax) + zT(Ax − b) 2

I Introduce slack variables −b ≤ s ≤ b Optimality Conditions z+z zd := z − z and za := 2

T T T Qx + c − A z + A z = 0 Qx + c + A zd = 0 Ax − s = 0 Ax − s = 0

Z (s − b) = 0 (Za − Zd /2)(s − b) = 0 Z b − s = 0 (Za + Zd /2) b − s = 0

SQP Approach Solving QPs QP Formulation Results 14 / 19 Center for Industrial Mathematics Handling both sided constraints Newton’s method + simpliﬁcations yield  T  ! QA ∆x −1  −1 −1  = A − S Z + S Z ∆zd   −rQ −1  −1 −1 −1 −1  −rA + S Z + S Z −S r + S r with −1 −1 −1 −1 −1 ∆s = S Z + S Z −S r + S r +∆ zd ∆z = S−1 (−r − Z∆s) −1 ∆z = S −r + Z∆s .

SQP Approach Solving QPs QP Formulation Results 15 / 19 Used QP solver optimal acceptable unsuccessful WORHP QP 1039 61 205 QPSOL 971 96 238

Center for Industrial Mathematics Numerical Results CUTEst Test Set

Collection of continous optimization problems Version from May 8, 2018: In total 1305 problems

I Run WORHP using QPSOL and WORHP QP

SQP Approach Solving QPs QP Formulation Results 16 / 19 Center for Industrial Mathematics Numerical Results CUTEst Test Set

Collection of continous optimization problems Version from May 8, 2018: In total 1305 problems

I Run WORHP using QPSOL and WORHP QP

Used QP solver optimal acceptable unsuccessful WORHP QP 1039 61 205 QPSOL 971 96 238

SQP Approach Solving QPs QP Formulation Results 16 / 19 Center for Industrial Mathematics Performance Proﬁles

Iterations CPU Time 90 80

60 80 40

20 70 0 1 2 3 4 5

Function Evaluations 60 % of problems 80

50 60 WORHP QP QPSOL 40 virtual best/worst 40 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 1 2 3 4 5

log2(τ)

SQP Approach Solving QPs QP Formulation Results 17 / 19 Center for Industrial Mathematics Performance Proﬁles (only equal minima)

Iterations CPU Time 80 80

75 60

40 70 20 65

0 1 2 3 4 5 60 Function Evaluations 55 80 % of problems

50 60 45 WORHP QP QPSOL 40 virtual best/worst 40

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 1 2 3 4 5

log2(τ)

SQP Approach Solving QPs QP Formulation Results 18 / 19 I Code, stability and performance improvements for WORHP I More versatile standalone QP solver as well

Center for Industrial Mathematics Conclusion

Implemented a reworked QP solver for WORHP Additional algorithmic features like Gondzio’s multiple centrality correctors and weighted correctors Improved QP formulation for a cleaner interface between NLP and QP layer Greatly simpliﬁed handling of sensitivity derivatives Numerical results show signiﬁcant improvements using the new solver

SQP Approach Solving QPs QP Formulation Results 19 / 19 Center for Industrial Mathematics Conclusion

I Code, stability and performance improvements for WORHP I More versatile standalone QP solver as well

SQP Approach Solving QPs QP Formulation Results 19 / 19