Xpress Non-Linear Solvers
Total Page:16
File Type:pdf, Size:1020Kb
Xpress Non-Linear Solvers Michael Perregaard Principal Engineer, Xpress Optimization Group FICO © 2014 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent. Xpress Solver Development Team ► Michael Perregaard Mixed Integer ►Csaba Meszaros Interior Point Solver ►Zsolt Csizmadia Continuous Optimization ►Pietro Belotti Mixed Integer Non-Linear ►Qi Huangfu Dual Simplex ►Stefan Heinz Mixed Integer ►Timo Berthold Mixed Integer 2 © 2014 Fair Isaac Corporation. Confidential. FICO Snapshot The leader in predictive analytics for decision management Founded: 1956 Profile NYSE: FICO Annual Global Revenues Approximately: $700 million Scores and related analytic models Products Analytic Applications for risk management, fraud, marketing, mobility and Services Tools for decision management Clients and 5,000+ clients in 80 countries Markets Industry focus: Banking, insurance, retail, health care #1 in services operations analytics (IDC) Recent #6 in worldwide analytics software (IDC) Rankings #7 in Business Intelligence, CPM and Analytic Applications (Gartner) #26 in the FinTech 100 (American Banker) 20+ offices worldwide, HQ in San Jose, California, USA 2,400+ employees Offices Regional Hubs: San Rafael (CA); San Diego (CA); New York; Roseville, MN; London; Birmingham (UK); Istanbul; Madrid; Munich; Sao Paulo; Bangalore; Beijing; and Singapore. 3 © 2014 Fair Isaac Corporation. Confidential. FICO Product Portfolio For Specific Decision Processes Customer Collections and Fraud Marketing Origination Mobile Management Recovery Management FICO® FICO® FICO® TRIAD® FICO® Debt FICO® Falcon® FICO® Adeptra® Customer Origination Customer Manager™ Fraud Manager Fraud Resolution Dialogue Manager Manager FICO® Recovery FICO® Insurance FICO® Adeptra® Applications Manager Management Fraud Manager Risk Intervention FICO® Analytic System™ FICO® Claims Manager Offer Manager Fraud Manager Targeting Consumer and Behavior Collections Consortium Models Small Scorecards Scores Fraud Models Custom / Time-to-Event Business Risk Transaction Custom Fraud Embedded Analytics Models Analytics Models Analytics Economic Application Fraud Impact Models Models For Any Decision Process B2B: FICO® Score FICO® Credit Capacity Index™ FICO® Insurance Risk Scores Scores B2C: myFICO® Business Rules Management: FICO® Blaze Advisor® Tools Predictive Analytics: FICO® Model Builder FICO® Model Central Optimization: FICO® Optimization Modeler FICO® Xpress FICO® Decision Optimizer Custom Analytics Professional Operational Best Practices Services Strategy Design and Optimization 4 © 2014 Fair Isaac Corporation. Confidential. A Network of Intelligence Accelerating the Development of Ideas CONSUME CONTRIBUTE Researchers Entrepreneurs Businesses ISVs Systems Integrators/ Governments Consultants Corporate Academics Developers COLLABORATE 5 © 2014 Fair Isaac Corporation. Confidential. FICO Solutions on FICO Solution Stack FICO Analytic Marketplace Buy FICO Application Studio FICO Analytic Decision Optimization Solution Modeler Modeler Modeler Stack FICO Decision Management Platform Build FICO Visual Insights Studio Manage 6 © 2014 Fair Isaac Corporation. Confidential. History of Xpress 1983 Dash Optimization founded by Bob Daniel and Robert Ashford. Xpress-MP LP solver released. 1985 MP-Model – modelling language 1986 Mixed Integer Programming (MIP) solver added 2001 Mosel modelling language 2003 XSLP non-linear solver added 2008 Dash Optimization bought by Fair Isaac (now FICO) 2012 Xpress Insight added (now Xpress Optimization Modeller) 7 © 2014 Fair Isaac Corporation. Confidential. Xpress Optimization Suite ►Adapt data and parameters to create and compare scenarios Applications ►Understand trade-offs and sensitivities Services ►Visualize data and results for analysis Optimization ►Collaborate in a multi-user environment Modeler BENEFITS ►Works in a rich client and a web browser — on premise and in the cloud ►Fully featured APIs including web ►Flexible, modular, easy-to-learn and use ►Development IDE Modeling ►Distributed modeling and cloud enablement Mosel ►Data connections (file, excel, databases, web services) FEATURES ►Precompiled for efficiency and IP protection ►Fully featured APIs ►High-performance, scalable and robust LP (Simplex|Barrier), MIP, QP, MIQP, Optimization QCQP, MIQCQP, SOCP, MISOCP, NLP, MINLP, and CP engines Optimizer ►Great out-of-the-box performance ― advanced users have full control over NonLinear solution process Kalis ►Utilizes multi-core/CPU machines, automatic tuning FEATURES ►N-best solutions capabilities and advanced infeasibility handling ►Fully featured APIs 8 © 2014 Fair Isaac Corporation. Confidential. FICO Xpress Solver Performance Level of complexity 7.00 FICO 6.00 IBM Gurobi 5.00 Knitro 4.00 Better 3.00 2.00 o offering 1.00 o offering n n 0.00 LP/QP MIP Robust Nonlinear ► FICO has the most complete optimization offering and all solvers are very competitive ► Robust / (MI)SOCP solver dominates the competition ► FICO has the leading nonlinear offering with applications in particular in finance, insurance, and power/gas/oil industries Results as of March 30, geometric means of time to optimality, LP/QP geometric mean computed by FICO, MIP 12 threads, nonlinear numbers directly computed from the logs and computed by FICO 9 © 2014 Fair Isaac Corporation. Confidential. FICO Xpress Users 10 © 2014 Fair Isaac Corporation. Confidential. Basic Problem Types min cx Primal simplex LP s.t. 푎푖푥 ≥ 푏 , 푖 ∈ 푀 Dual simplex 푖 Interior point Quadratic primal simplex min cx + xQx QP Quadratic dual simplex s.t. 푎푖푥 ≥ 푏 , 푖 ∈ 푀 푖 Interior point min cx + xQx QCQP 푖 푖 Interior point s.t. 푎 푥 + 푥푄 푥 ≥ 푏푖, 푖 ∈ 푀 min cx + xQx 푖 푖 Interior point SOCP s.t. 푎 푥 + 푥푄 푥 ≥ 푏푖, 푖 ∈ 푀 푘 푘 퐷 푥 + 푓푘 ≤ 푔 푥 + ℎ푘, 푘 ∈ 퐶 11 © 2014 Fair Isaac Corporation. Confidential. Basic Problem Types min cx MILP 푖 Branch and bound s.t. 푎 푥 ≥ 푏푖, 푖 ∈ 푀 푥푗 ∈ ℤ, 푗 ∈ 퐼 min cx + xQx Branch and bound MIQP 푖 + quadratic dual simplex s.t. 푎 푥 ≥ 푏푖, 푖 ∈ 푀 푥푗 ∈ ℤ, 푗 ∈ 퐼 min cx + xQx MIQCQP 푖 푖 Branch and bound s.t. 푎 푥 + 푥푄 푥 ≥ 푏푖, 푖 ∈ 푀 + outer approximation 푥푗 ∈ ℤ, 푗 ∈ 퐼 min cx + xQx 푖 푖 Branch and bound MISOCP s.t. 푎 푥 + 푥푄 푥 ≥ 푏푖, 푖 ∈ 푀 + outer approximation 푘 푘 퐷 푥 + 푓푘 ≤ 푔 푥 + ℎ푘, 푘 ∈ 퐶 푥푗 ∈ ℤ, 푗 ∈ 퐼 12 © 2014 Fair Isaac Corporation. Confidential. Non-linear Problems Standard non-linear formulation: min f(x) s.t. 푔 푥 ≥ 0, 푖 ∈ 푀 푖 Xpress non-linear formulation: min cx + xQx + f(x) 푖 푖 s.t. 푎 푥 + 푥푄 푥 + 푔푖(푥) ≥ 푏푖, 푖 ∈ 푀 푘 푘 퐷 푥 + 푓푘 ≤ 푔 푥 + ℎ푘, 푘 ∈ 퐶 ► Xpress embeds non-linear formulas in a standard problem. ► Solvers: SLP or Knitro 13 © 2014 Fair Isaac Corporation. Confidential. SLP – Sequential Linear Programming Create linear approximation Might cut off feasible regions Solve LP Repeat 14 © 2014 Fair Isaac Corporation. Confidential. Sequential Linear Programming Non-linear problem: (assume non-linear objective moved into constraints) min 푐푥 푖 s.t. 푎 푥 + 푔푖 푥 ≥ 푏푖, 푖 ∈ 푀 Linearize non-linear functions around a solution 푥푘: 푘 푘 푔푖 푥 ≈ 푔푖 푥 + 훻푔푖(푥 − 푥 ) to create LP min 푐푥 푖 푘 푘 s.t. 푎 푥 + 푔푖 푥 + 훻푔푖(푥 − 푥 ) ≥ 푏푖, 푖 ∈ 푀 Solve LP to get next iterate solution: 푥푘+1 15 © 2014 Fair Isaac Corporation. Confidential. Sequential Linear Programming Trust regions Solution will bounce between 푥1 and 푥2 푥1 푥2 16 © 2014 Fair Isaac Corporation. Confidential. Sequential Linear Programming Trust regions Solution will bounce between 푥1 and 푥2 Use trust region Smaller movement 푥3 푥4 17 © 2014 Fair Isaac Corporation. Confidential. Sequential Linear Programming Infeasibility Non-convex non-linear with feasible region. Iterate solution 푥푘 in infeasible region. 푘 푥 Can result in infeasible LP Use penalty variables to make LP feasible. 18 © 2014 Fair Isaac Corporation. Confidential. Sequential Linear Programming Extended LP min 푐푥 푖 s.t. 푎 푥 + 푔푖 푥 ≥ 푏푖, 푖 ∈ 푀 Linearize: min 푐푥 푖 푘 푘 s.t. 푎 푥 + 푔푖 푥 + 훻푔푖(푥 − 푥 ) ≥ 푏푖, 푖 ∈ 푀 Add trust region: min 푐푥 푖 푘 s.t. 푎 푥 + 푔푖 푥 + 훻푔푖∆푥 ≥ 푏푖, 푖 ∈ 푀 푥 = 푥푘 + ∆푥 퐿 ≤ ∆푥 ≤ 푈 Add penalty variables: min 푐푥 + Py 푖 푘 s.t. 푎 푥 + 푔푖 푥 + 훻푔푖∆푥 + 푦 ≥ 푏푖, 푖 ∈ 푀 푘 푥 = 푥 + ∆푥 퐿 ≤ ∆푥 ≤ 푈, 푦 ≥ 0 19 © 2014 Fair Isaac Corporation. Confidential. Sequential Linear Programming Optimality Non-linear problem: KKT optimality conditions: (under regularity conditions) min 푐푥 푖 ∗ ∗ s.t. 푎푖푥 + 푔 푥 ≥ 푏 , 푖 ∈ 푀 푎 푥 + 푔푖 푥 ≥ 푏푖, 푖 ∈ 푀 푖 푖 푖 ∗ 푐 = ∑푖∈푀휇푖(푎 + 훻푔푖 푥 ) 푖 ∗ ∗ SLP problem: 휇푖 푎 푥 + 푔푖 푥 = 0, 푖 ∈ 푀 min 푐푥 + Py 휇푖 ≥ 0, 푖 ∈ 푀 푖 푘 s.t. 푎 푥 + 푔푖 푥 + 훻푔푖∆푥 + 푦 ≥ 푏푖, 푖 ∈ 푀 푥 = 푥푘 + ∆푥 퐿 ≤ ∆푥 ≤ 푈, 푦 ≥ 0 ► Trust bounds and penalty variables bound primal and dual variables. ► SLP solution feasible if penalties are zero. ► SLP solution optimal if dual multipliers for trust bounds are zero. ► Optimal, feasible LP solution satisfies KKT complementarity. 20 © 2014 Fair Isaac Corporation. Confidential. Sequential Linear Programming Convergence Strong convergence Extended convergence 21 © 2014 Fair Isaac Corporation. Confidential. Sequential Linear Programming Local Optima 22 © 2014 Fair Isaac Corporation. Confidential. Sequential Linear Programming Blending Example ► Liquids 퐴 and 퐵 with densities 훿 퐴 and 훿 퐵 . ► Create blend 퐶 with density 퐿퐶 ≤ 훿 퐶 ≤ 푈퐶 in amount 푉(퐶). ► Select amounts 푉(퐴) and 푉(퐵) to blend. 푉 퐶 = 푉 퐴 +푉 퐵 훿 퐶 푉 퐶 = 훿 퐴 푉 퐴 +훿 퐵 푉 퐵 ► Linear problem if 퐴 and 퐵 given. ► Bi-linear if 퐴 and 퐵 are also blends. ► Refinery problems can contain 1000s of blends. ► Blends can be returned to earlier stages! 23 © 2014 Fair Isaac Corporation. Confidential. Sequential Linear Programming Blending Example 훿 퐶 푉 퐶 = 훿 퐴 푉 퐴 +훿 퐵 푉 퐵 ► 푉(퐶) = 0 results in 훿 퐶 undefined. ► Causes “infinite” LP coefficients ► XSLP detects such edge cases. ► Non-convex problem. ► Use restarts to find better solutions. ► XSLP provides parallel multi-start feature with different starting points. ► Other solutions iterate between fixing 훿 or 푉.