Xpress Non-Linear Solvers

Michael Perregaard Principal Engineer, Xpress Optimization Group FICO

© 2014 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent. Xpress Solver Development Team

► Michael Perregaard Mixed Integer ►Csaba Meszaros Interior Point Solver ►Zsolt Csizmadia Continuous Optimization ►Pietro Belotti Mixed Integer Non-Linear ►Qi Huangfu Dual Simplex ►Stefan Heinz Mixed Integer ►Timo Berthold Mixed Integer

The leader in predictive analytics for decision management Founded: 1956 Profile NYSE: FICO Annual Global Revenues Approximately: $700 million

Scores and related analytic models Products Analytic Applications for risk management, fraud, marketing, mobility and Services Tools for decision management

Clients and 5,000+ clients in 80 countries Markets Industry focus: Banking, insurance, retail, health care

#1 in services operations analytics (IDC) Recent #6 in worldwide analytics software (IDC) Rankings #7 in Business Intelligence, CPM and Analytic Applications (Gartner) #26 in the FinTech 100 (American Banker)

20+ offices worldwide, HQ in San Jose, California, USA 2,400+ employees Offices Regional Hubs: San Rafael (CA); San Diego (CA); New York; Roseville, MN; London; Birmingham (UK); Istanbul; Madrid; Munich; Sao Paulo; Bangalore; Beijing; and Singapore.

FICO® FICO® FICO® TRIAD® FICO® Debt FICO® Falcon® FICO® Adeptra® Customer Origination Customer Manager™ Fraud Manager Fraud Resolution Dialogue Manager Manager FICO® Recovery FICO® Insurance FICO® Adeptra® Applications Manager Management Fraud Manager Risk Intervention FICO® Analytic System™ FICO® Claims Manager Offer Manager Fraud Manager Targeting Consumer and Behavior Collections Consortium Models Small Scorecards Scores Fraud Models Custom / Time-to-Event Business Risk Transaction Custom Fraud Embedded Analytics Models Analytics Models Analytics Economic Application Fraud Impact Models Models

For Any Decision Process B2B: FICO® Score  FICO® Credit Capacity Index™  FICO® Insurance Risk Scores Scores B2C: myFICO®

Business Rules Management: FICO® Blaze Advisor® Tools Predictive Analytics: FICO® Model Builder  FICO® Model Central Optimization: FICO® Optimization Modeler  FICO® Xpress  FICO® Decision Optimizer

Custom Analytics Professional Operational Best Practices Services Strategy Design and Optimization

CONSUME CONTRIBUTE Researchers Entrepreneurs Businesses ISVs

Systems Integrators/ Governments Consultants Corporate Academics Developers

COLLABORATE

FICO Analytic Marketplace Buy

FICO Application Studio

FICO Analytic Decision Optimization Solution Modeler Modeler Modeler Stack

FICO Decision Management Platform Build

FICO Visual Insights Studio

Manage

1983 Dash Optimization founded by Bob Daniel and Robert Ashford. Xpress-MP LP solver released.

1985 MP-Model – modelling language

1986 Mixed Integer Programming (MIP) solver added

2001 Mosel modelling language

2003 XSLP non-linear solver added

2008 Dash Optimization bought by Fair Isaac (now FICO)

2012 Xpress Insight added (now Xpress Optimization Modeller)

►Adapt data and parameters to create and compare scenarios Applications ►Understand trade-offs and sensitivities Services ►Visualize data and results for analysis Optimization ►Collaborate in a multi-user environment

Modeler BENEFITS ►Works in a rich client and a web browser — on premise and in the cloud ►Fully featured APIs including web

►Flexible, modular, easy-to-learn and use

►Development IDE Modeling ►Distributed modeling and cloud enablement Mosel ►Data connections (file, excel, databases, web services)

FEATURES ►Precompiled for efficiency and IP protection ►Fully featured APIs

►High-performance, scalable and robust LP (Simplex|Barrier), MIP, QP, MIQP, Optimization QCQP, MIQCQP, SOCP, MISOCP, NLP, MINLP, and CP engines Optimizer ►Great out-of-the-box performance ― advanced users have full control over NonLinear solution process Kalis ►Utilizes multi-core/CPU machines, automatic tuning FEATURES ►N-best solutions capabilities and advanced infeasibility handling ►Fully featured APIs

Level of complexity

7.00 FICO 6.00 IBM Gurobi 5.00

Knitro

4.00

Better 3.00

2.00 o o offering

1.00 o offering

n n

0.00 LP/QP MIP Robust Nonlinear

► FICO has the most complete optimization offering and all solvers are very competitive ► Robust / (MI)SOCP solver dominates the competition ► FICO has the leading nonlinear offering with applications in particular in finance, insurance, and power/gas/oil industries

Results as of March 30, geometric means of time to optimality, LP/QP geometric mean computed by FICO, MIP 12 threads, nonlinear numbers directly computed from the logs and computed by FICO

min cx Primal simplex LP s.t. 푎푖푥 ≥ 푏 , 푖 ∈ 푀 Dual simplex 푖 Interior point

Quadratic primal simplex min cx + xQx QP Quadratic dual simplex s.t. 푎푖푥 ≥ 푏 , 푖 ∈ 푀 푖 Interior point

min cx + xQx QCQP 푖 푖 Interior point s.t. 푎 푥 + 푥푄 푥 ≥ 푏푖, 푖 ∈ 푀

min cx + xQx 푖 푖 Interior point SOCP s.t. 푎 푥 + 푥푄 푥 ≥ 푏푖, 푖 ∈ 푀 푘 푘 퐷 푥 + 푓푘 ≤ 푔 푥 + ℎ푘, 푘 ∈ 퐶

min cx MILP 푖 Branch and bound s.t. 푎 푥 ≥ 푏푖, 푖 ∈ 푀 푥푗 ∈ ℤ, 푗 ∈ 퐼

min cx + xQx Branch and bound MIQP 푖 + quadratic dual simplex s.t. 푎 푥 ≥ 푏푖, 푖 ∈ 푀 푥푗 ∈ ℤ, 푗 ∈ 퐼 min cx + xQx MIQCQP 푖 푖 Branch and bound s.t. 푎 푥 + 푥푄 푥 ≥ 푏푖, 푖 ∈ 푀 + outer approximation 푥푗 ∈ ℤ, 푗 ∈ 퐼 min cx + xQx 푖 푖 Branch and bound MISOCP s.t. 푎 푥 + 푥푄 푥 ≥ 푏푖, 푖 ∈ 푀 + outer approximation 푘 푘 퐷 푥 + 푓푘 ≤ 푔 푥 + ℎ푘, 푘 ∈ 퐶 푥푗 ∈ ℤ, 푗 ∈ 퐼

Standard non-linear formulation: min f(x) s.t. 푔 푥 ≥ 0, 푖 ∈ 푀 푖

Xpress non-linear formulation: min cx + xQx + f(x) 푖 푖 s.t. 푎 푥 + 푥푄 푥 + 푔푖(푥) ≥ 푏푖, 푖 ∈ 푀 푘 푘 퐷 푥 + 푓푘 ≤ 푔 푥 + ℎ푘, 푘 ∈ 퐶

► Xpress embeds non-linear formulas in a standard problem. ► Solvers: SLP or Knitro

Create linear approximation

Might cut off feasible regions

Solve LP

Repeat

Non-linear problem: (assume non-linear objective moved into constraints) min 푐푥 푖 s.t. 푎 푥 + 푔푖 푥 ≥ 푏푖, 푖 ∈ 푀 Linearize non-linear functions around a solution 푥푘: 푘 푘 푔푖 푥 ≈ 푔푖 푥 + 훻푔푖(푥 − 푥 ) to create LP min 푐푥 푖 푘 푘 s.t. 푎 푥 + 푔푖 푥 + 훻푔푖(푥 − 푥 ) ≥ 푏푖, 푖 ∈ 푀

Solve LP to get next iterate solution: 푥푘+1

Solution will bounce between 푥1 and 푥2

푥1 푥2

Solution will bounce between 푥1 and 푥2

Use trust region

Smaller movement

푥3 푥4

Non-convex non-linear with feasible region.

Iterate solution 푥푘 in infeasible region.

푘 푥 Can result in infeasible LP

Use penalty variables to make LP feasible.

min 푐푥 푖 s.t. 푎 푥 + 푔푖 푥 ≥ 푏푖, 푖 ∈ 푀

Linearize:

min 푐푥 푖 푘 푘 s.t. 푎 푥 + 푔푖 푥 + 훻푔푖(푥 − 푥 ) ≥ 푏푖, 푖 ∈ 푀 Add trust region: min 푐푥 푖 푘 s.t. 푎 푥 + 푔푖 푥 + 훻푔푖∆푥 ≥ 푏푖, 푖 ∈ 푀 푥 = 푥푘 + ∆푥 퐿 ≤ ∆푥 ≤ 푈 Add penalty variables: min 푐푥 + Py 푖 푘 s.t. 푎 푥 + 푔푖 푥 + 훻푔푖∆푥 + 푦 ≥ 푏푖, 푖 ∈ 푀 푥 = 푥푘 + ∆푥 퐿 ≤ ∆푥 ≤ 푈, 푦 ≥ 0

Non-linear problem: KKT optimality conditions: (under regularity conditions) min 푐푥 푖 ∗ ∗ s.t. 푎푖푥 + 푔 푥 ≥ 푏 , 푖 ∈ 푀 푎 푥 + 푔푖 푥 ≥ 푏푖, 푖 ∈ 푀 푖 푖 푖 ∗ 푐 = ∑푖∈푀휇푖(푎 + 훻푔푖 푥 ) 푖 ∗ ∗ SLP problem: 휇푖 푎 푥 + 푔푖 푥 = 0, 푖 ∈ 푀 min 푐푥 + Py 휇푖 ≥ 0, 푖 ∈ 푀 푖 푘 s.t. 푎 푥 + 푔푖 푥 + 훻푔푖∆푥 + 푦 ≥ 푏푖, 푖 ∈ 푀 푥 = 푥푘 + ∆푥 퐿 ≤ ∆푥 ≤ 푈, 푦 ≥ 0

► Trust bounds and penalty variables bound primal and dual variables. ► SLP solution feasible if penalties are zero. ► SLP solution optimal if dual multipliers for trust bounds are zero. ► Optimal, feasible LP solution satisfies KKT complementarity.

Strong convergence Extended convergence

► Liquids 퐴 and 퐵 with densities 훿 퐴 and 훿 퐵 .

► Create blend 퐶 with density 퐿퐶 ≤ 훿 퐶 ≤ 푈퐶 in amount 푉(퐶). ► Select amounts 푉(퐴) and 푉(퐵) to blend.

푉 퐶 = 푉 퐴 +푉 퐵 훿 퐶 푉 퐶 = 훿 퐴 푉 퐴 +훿 퐵 푉 퐵

► Linear problem if 퐴 and 퐵 given. ► Bi-linear if 퐴 and 퐵 are also blends.

► Refinery problems can contain 1000s of blends. ► Blends can be returned to earlier stages!

훿 퐶 푉 퐶 = 훿 퐴 푉 퐴 +훿 퐵 푉 퐵

► 푉(퐶) = 0 results in 훿 퐶 undefined. ► Causes “infinite” LP coefficients ► XSLP detects such edge cases. ► Non-convex problem. ► Use restarts to find better solutions. ► XSLP provides parallel multi-start feature with different starting points. ► Other solutions iterate between fixing 훿 or 푉. ► XSLP approximates both at the same time.

► Solves 1st order approximations. ► Builds on top of a strong LP solver. ► Highly efficient for bi-linear or problems with a large amount of linear constraints. ► Local solver: Global optimality guaranteed only for convex problems. ► Applications: ► Petro-chemical industry. ► Finance. ► Price optimization. ► … ► Comes with a Mixed Integer Programming solver.

► Knitro licensed from ► 2nd order interior point solver. ► Strong for highly non-linear, medium sized problems. ► Integrated with Xpress non-linear solver ► Usable with any non-linear model. ► Xpress non-linear solver decides whether to use Knitro or XSLP. ► Xpress calculates 1st and 2nd order derivatives. ► Numerical derivatives ► Symbolic differentiation ► Automatic differentiation.

XSLP Knitro 1st order 2nd order Bilinear or highly linear Highly non-linear problems problems

Large sized problems Medium sized problems Local solver Local solver

푚푖푛 푓(푥) LP 푠. 푡. 푔 푥 ≥ 0

Problem recognition QP

QCQP

SOCP

XSLP Knitro Optimizer

Michael Perregaard [email protected]