Welcome to the webinar Migrating models from other solvers to use the Gurobi solver Renan Garcia, Ph.D.
Optimization Support Engineer at Gurobi Optimization
} Ph.D. in Industrial and Systems Engineering, Georgia Tech } Expert in optimization modeling and software development } Over a decade of experience implementing decision support systems
© 2016 Gurobi Optimization David Nehme, Ph.D.
Principal Consultant at Abremod
} 20 years of software development experience } 15 years implementing CPLEX and Gurobi models } Ph.D. in Operations Research for The University of Texas } Top StackOverflow answerer for both CPLEX and Gurobi tags
© 2016 Gurobi Optimization Agenda
} Switching is easier than you may think
} Common migration scenarios
} Migrating from OPL
} Migrating from CPLEX Concert API
} Best practices, limitations and considerations
© 2016 Gurobi Optimization Switching is Easier Than You May Think
© 2016 Gurobi Optimization Key Migration Issues
} Building the model ◦ How do I build my optimization model? ◦ Do I build it one constraint at a time, or do I build an entire constraint matrix?
} Setting solver parameters ◦ What solver parameters do I change? ◦ What effects are these changes intended to produce? ◦ Am I looking for an optimal solution, or just a good feasible solution?
} Computing and extracting the solution ◦ How do I extract the solution produced by the solver?
© 2016 Gurobi Optimization Interfacing with Gurobi
} Command-Line Tool
} Full-featured Interactive Shell
} Matrix-oriented APIs ◦ C, MATLAB, R
} Object-oriented APIs ◦ C++, Java, .NET, Python
} Modeling systems ◦ Commercial: AIMMS, AMPL, Frontline Solvers, GAMS, MPL, … ◦ Free: CMPL, JuliaOpt, OSI, PuLP, PYOMO, SolverStudio, YALMIP, …
© 2016 Gurobi Optimization Do-It-Yourself Resources on www.gurobi.com
} Switching guidance: https://www.gurobi.com/resources/switching-to-gurobi/switching-overview
} Documentation: https://www.gurobi.com/documentation/ ◦ Quick Start Guides ◦ Reference Manual APIs Attributes Parameters Tuning … ◦ Example Tour 22 functional examples
} Seminars and videos: https://www.gurobi.com/resources/seminars-and-videos/seminars-videos
© 2016 Gurobi Optimization Common Migration Scenarios
© 2016 Gurobi Optimization Scenario 1: Migrating with Model Files
} Example: performance testing
} Export model file with virtually no changes needed to existing code ◦ Gurobi supports several file formats (MPS, LP, …)
} Export guidance: https://www.gurobi.com/resources/switching-to-gurobi/exporting-mps- files-from-competing-solvers
} Use Gurobi Command-Line Tool to solve the model ◦ Usage: gurobi_cl [parameters] filename ◦ Ex: gurobi_cl TimeLimit=3600 misc07.mps
} "Quick-and-dirty" approach ◦ Limited ability to interact with solver (parameters only) ◦ For more control, try Interactive Shell
© 2016 Gurobi Optimization Gurobi Parameters
} Control algorithmic behavior } MIP ◦ Defaults validated against large ◦ Ex. MIPFocus, ImproveStartTime internal test set of models } MIP cuts } Termination criteria ◦ Ex. Cuts, GomoryPasses ◦ Ex. TimeLimit, SolutionLimit } Tuning and distributed } Tolerances algorithms ◦ Ex. MIPGap, BarConvTol ◦ Ex. TuneJobs, WorkerPool
} Simplex and barrier } General ◦ Ex. Method, Crossover ◦ Ex. Presolve, OutputFlag
© 2016 Gurobi Optimization Scenario 2: Using a Modeling System
} Example: model is written in AMPL
} Migrating is extremely easy for solver-independent systems
◦ Obtain license ◦ Set solver to Gurobi ◦ Convert parameter settings
◦ Ex: in AMPL model file, add option solver gurobi_ampl; option gurobi_options 'mipfocus 1';
} Need to migrate your existing model code for single-solver systems ◦ OPL, Mosel, …
© 2016 Gurobi Optimization Gurobi Python Environment
} High-level optimization modeling constructs embedded in Python programming language
} Combines expressiveness of a modeling language with the power and flexibility of a programming language ◦ Bring "feel" of a modeling language to the Python interface
} Requires minimal programming skills to get started
} Support all solver and programming needs
} Several seminars on this topic: https://www.gurobi.com/resources/seminars-and-videos/seminars-videos
© 2016 Gurobi Optimization Interactive Examples
© 2016 Gurobi Optimization Scenario 3: Porting Matrix-Oriented Code
} Example: C program which calls CPLEX Callable Library, Xpress, …
} Gurobi's C API supports sparse matrix format ◦ Standard format used by many solvers Simple arrays represent matrix coefficients and their index positions ◦ Ex: GRBaddconstrs(), GRBaddvars() ◦ Minimal changes required to existing code
} Gurobi also supports advanced features ◦ Callbacks ◦ Advanced simplex routines (querying tableau rows) ◦ …
} Must consider some Gurobi-specific modeling features when porting existing code ◦ Gurobi environments ◦ Lazy updates ◦ Attributes
© 2016 Gurobi Optimization Gurobi Environments
} Models are built from an environment
} Parameters are set on an environment
} Models get their own copy of the environment ◦ Once a model is created, subsequent parameter changes in parent environment are not reflected in model environment ◦ Use getEnv() functions to get the environment from model
} Setting parameters in C ◦ Ex: set time limit of 3600 seconds for parent environment status = GRBsetdblparam(env, "TimeLimit", 3600); ◦ Ex: set presolve level to 2 for model's environment status = GRBsetintparam(GRBgetenv(model), "Presolve", 2);
© 2016 Gurobi Optimization Lazy Updates
} Gurobi updates models in batch mode ◦ Model creation and updates are efficient
} Must call update() functions to use model elements ◦ Ex: Call after creating a variable before using it in a constraint ◦ May require changes to code for other solvers
} UpdateMode=1 parameter setting allows you to use elements immediately
© 2016 Gurobi Optimization Attributes
} Unified system to access model elements ◦ Attributes work the same across all Gurobi interfaces
} Access via a basic set of get/set functions ◦ Attribute name is specified as a parameter ◦ Replaces many functions used by other solvers
} Getting/Setting attributes in C ◦ Use get/set functions by type (int, double, char, string) ◦ Ex: query number of nonzeros in a model status = GRBgetintattr(model, "NumNZs", &nzs); ◦ Ex: query solution vector double x[NUMVARS]; status = GRBgetdblattrarray(model, "X", 0, NUMVARS, x); ◦ Ex: modify constraint RHS to 1 status = GRBsetdblattrelement(model, "RHS", cidx, 1.0);
© 2016 Gurobi Optimization Scenario 4: Porting Object-Oriented Code
} Example: Java program which uses CPLEX Concert Technology
} Gurobi's OO APIs represent models using objects ◦ Objects for variables and constraints ◦ Methods are used to create model elements ◦ Ex: add simple constraint x + y ≥ 1 in C++ c1 = model.addConstr(x + y >= 1, "c1");
} All Gurobi APIs are just thin layers on top of same native C code
} Must consider same Gurobi-specific modeling features when porting ◦ Subsequent parameter changes in parent environment not reflected in model environment Java Ex: model.getEnv().set(GRB.IntParam.Presolve, 2); ◦ Must call model's update() method to use elements (unless UpdateMode=1) ◦ Use get/set methods on objects to access attributes Python Ex: constr.setAttr('RHS', 1.0)
© 2016 Gurobi Optimization Don't Panic, Your Code Often Looks Similar
} CPLEX Concert C++: IloEnv env; // create empty environment IloModel model(env); // create empty model IloNumVar x(env, 0, 10, ILOINT); // add variables model.add(x); // ... model.add(x + 2*y <= 1); // add constraints // ...
} Gurobi C++: GRBEnv env; // create empty environment env.set(GRB_IntParam_UpdateMode, 1); // enable automatic updates GRBModel model(env); // create empty model GRBVar x = model.addVar(0, 10, 0, GRB_INTEGER); // add variables // ... model.addConstr(x + 2*y <= 1); // add constraints // ...
© 2016 Gurobi Optimization Solver-specific Guidance
© 2016 Gurobi Optimization Agenda
} Cover two disparate examples ◦ OPL ◦ ILOG Concert Have a C#, and Java Adapters Working on C++ ◦ Showing Two approaches Translating OPL Using an Adapter for Concert
} General advice on the migrating
© 2016 Gurobi Optimization Switching To Gurobi
} Hurdles ◦ Large code base ◦ Business Logic embedded in API calls
} Mitigating Factors ◦ The time to migrate is not proportional to the size of the codebase ◦ With mps files, you can see how the Gurobi will perform on your specific models.
© 2016 Gurobi Optimization Before You Migrate
} Current State ◦ Actively adding features? ◦ Maintenance only? ◦ Do you have regression tests?
} MPS is your friend ◦ Evaluate relative performance
} LP is also your friend ◦ Name your variables and constraints ◦ Testing your code
© 2016 Gurobi Optimization Migrating From OPL
} OPL includes a domain specific language
} Locked to a Solver (CPLEX)
} Two most likely strategies ◦ Move to a solver-agnostic Language AMPL AIMMS, GAMS, MPL ◦ Migrate to Python
© 2016 Gurobi Optimization Migrating from OPL to Python
} Why Python
◦ Best of Both worlds Powerful as a General Purpose Language As effective as a Domain Specific Language Concise Readable Learnable See Stackoverflow (http://bit.ly/1QEF0fq)
◦ Huge user base
© 2016 Gurobi Optimization OPL to Python
} Feature Comparison
OPL Python Tuples, Sets Tuples, Sets Read from Excel openpyxl, xlrd Read from SQL Sqlalchemy Slicing, Grouping Pandas UI Jupyter Notebook .dat format JSON OPLScript Python
© 2016 Gurobi Optimization OPL To Python
} Installing Python ◦ Use the Anaconda Distribution Especially great on Windows Easy installation for Gurobi Libraries http://www.gurobi.com/downloads/get-anaconda
© 2016 Gurobi Optimization OPL To Python
} Migrate a complete example
} From a book ◦ Planning and Scheduling in Manufacturing and Services ◦ OPL Code is freely available http://bit.ly/20VpATz ◦ Reads from OPL dat file
} Python Code Available on github
© 2016 Gurobi Optimization Approach
} Translate the “.dat” file to JSON
◦ JSON is a standard Almost all programming languages have readers OPL reads it Native format for MongoDB Many application have JSON exporters
◦ Simple Python script to translate Most dat files without embedded logic Using PyParsing Another library included with Anaconda
© 2016 Gurobi Optimization OPL Data
} Can Translate with Python Script
Demand = #[ "Demand": { 2: #[ "2": { 1: [ 20000 30000 15000 40000 ] "Product 1": [ 20000, 30000, 15000, 40000 ], 2: [ 0 50000 30000 50000 ] "Product 2": [ 0, 50000, 30000, 50000 ] ]# }, 3: #[ "3": { 1: [ 10000 5000 15000 40000 ] "Product 1": [ 10000, 5000, 15000, 40000 ], 2: [ 0 10000 0 5000 ] "Product 2": [ 0, 10000, 0, 5000 ] ]# } ]#;
© 2016 Gurobi Optimization OPL Model
} Data Declarations
Lang. Statement OPL int RequiredLotSize = ...;
Python RequiredLotSize = d['RequiredLotSize']
OPL float Demand[Stages, Products, Periods] = ...; Python Demand = series_from_json(d['Demand'], [Stages, Products, Periods]) OPL range Products = 1..2; Python Products = ['Product 1', 'Product 2']
© 2016 Gurobi Optimization OPL Model
} Variable Declarations
dvar float+ x[Factories, Products, Periods]; dvar int+ y[Factories, Stages, Products, Periods] in 0..maxint; dvar float+ z[Products, Periods]; dvar float+ q2[Products, ZPeriods]; dvar float+ v2[Products, Periods]; dvar float+ v3[Products, ZPeriods]; dvar boolean yb[bnds, Periods];
© 2016 Gurobi Optimization Variables
OPL dvar float+ x[Factories, Products, Periods]; Python x = get_vars('x', Factories, Products, Periods) OPL dvar boolean yb[bnds, Periods]; Python yb = get_vars('yb', bnds.index, Periods, vtype=GRB.BINARY)
© 2016 Gurobi Optimization Objective function
} OPL } minimize } sum (t in Periods, j in Products, } i in Factories) } ProdCost[i,j]*x[i,j,t] } Python } model.setObjective( } grb.quicksum([ProdCost[i, j] * x[i, j, t] } for i in Factories } for j in Products } for t in Periods])
© 2016 Gurobi Optimization Constraints
OPL if (RequiredLotSize > 0) forall (i in Factories, j in Products, t in Periods) x[i,j,t] == RequiredLotSize*xlots[i,j,t]; else forall (i in Factories, j in Products, t in Periods) xlots[i,j,t] == 0; Python if RequiredLotSize > 0: [addConstr(x[i, j, t] == RequiredLotSize * xlots[i, j, t]) for i in Factories for j in Products for t in Periods] else: [addConstr(xlots[i, j, t] == 0) for i in Factories for j in Products for t in Periods]
© 2016 Gurobi Optimization Results From Solver
OPL float TotProdCost = sum (t in Periods, j in Products, i in Factories) ProdCost[i,j]*x[i,j,t]; Python TotProdCost = sum([ProdCost[i, j] * x[i, j, t].x for t in Periods for j in Products for i in Factories])
© 2016 Gurobi Optimization OPL to Python
} Consider moving to Python } If you still like modeling languages, try AMPL
© 2016 Gurobi Optimization Migration Library for CPLEX Concert API
© 2016 Gurobi Optimization Migrating From Concert
} Concert ◦ C++ API developed in late 90s ◦ Java and C# versions followed ◦ Good option for using mainstream languages
© 2016 Gurobi Optimization Migrating from Concert
} Two Strategies ◦ Translate code ◦ Use an Adapter
© 2016 Gurobi Optimization Adapter
} Pattern ◦ Object Adapter Pattern
} Physical Examples ◦ Electric plugs ◦ CO2 Filter on Apollo 13
© 2016 Gurobi Optimization Adapter Approach
} Give applications written against the Concert API, access to the Gurobi solver
} Our Adapter ◦ Not a complete implementation of Concert Most applications use a small subset ◦ Enough to make a model run ◦ Free to use Starting point for your application
© 2016 Gurobi Optimization Adapter Approach
} Working Concert Application
} Unlink CPLEX / Concert libraries
} Add Gurobi Library
} Add adapter library ◦ Modify adapter for your code
© 2016 Gurobi Optimization Example
} Makefile ◦ CPPLIB = … -lilo_grb … -lgurobi_c++ -lgurobi65 } Eclipse
© 2016 Gurobi Optimization Adapter Library
} Available for Java and C#
} Working on C++
} Available on Github ◦ https://github.com/abremod/concert2gurobi4cs
© 2016 Gurobi Optimization Adapter Library
} “Object Adapter”, not a “Class Adapter” } Favor Composition over Inheritance
◦ public class IloCplex extends ilog.concert.Algorithm implements IloMPModeler { ◦ GRBEnv env; ◦ GRBModel model; ◦ private boolean vars_synced = true; ◦ private boolean constrs_synced = true;
© 2016 Gurobi Optimization Adapter Library
} C++ specific
◦ IloXXXX objects are handles “Pimpl” ◦ Boost shared_ptr<> covers handle functionality ◦ class IloRange : public IloExtractable { ◦ private: ◦ class Impl; ◦ boost::shared_ptr
© 2016 Gurobi Optimization Limitations
} Adapter probably won’t work out of the box
} Variables tied to models, not environments ◦ No “not extracted exceptions”
} Limited Support for Callbacks
} No support for Goals
© 2016 Gurobi Optimization So what happens next?
} Request a free evaluation license, if you have not already done so ◦ [email protected] ◦ [email protected]
} Set up a free consultation with a consultant from Abremod [email protected]
} Webinar slides: Will be available in the next day or two
} Abremod tools available from: https://github.com/abremod/ilogrb
} Webinar recording: Will be available next week