Solving Technologies (Version of 22Nd September 2021)

Topic 7: Solving Technologies (Version of 22nd September 2021)

Jean-Noel¨ Monette and Pierre Flener

Optimisation Group Department of Information Technology Uppsala University Sweden

Course 1DL441: Combinatorial Optimisation and Constraint Programming, whose part 1 is Course 1DL451: Modelling for Combinatorial Optimisation Outline

1. The MiniZinc Toolchain The MiniZinc Toolchain 2. Comparison Criteria Comparison Criteria 3. SAT SAT 4. SMT & OMT SMT & OMT IP & MIP 5. IP & MIP CP

LS & CBLS 6.CP

Hybrid Technologies 7. LS & CBLS Case Study 8. Hybrid Technologies Choosing a Technology and Backend 9. Case Study 10. Choosing a Technology and Backend

COCP/M4CO 7 - 2 - Outline

1. The MiniZinc Toolchain The MiniZinc Toolchain 2. Comparison Criteria Comparison Criteria 3. SAT SAT 4. SMT & OMT SMT & OMT IP & MIP 5. IP & MIP CP

LS & CBLS 6.CP

Hybrid Technologies 7. LS & CBLS Case Study 8. Hybrid Technologies Choosing a Technology and Backend 9. Case Study 10. Choosing a Technology and Backend

COCP/M4CO 7 - 3 - MiniZinc: Model Once, Solve Everywhere!

The MiniZinc instance Toolchain data Comparison Criteria

SAT SMT & OMT ﬂat backend model ﬂattening IP & MIP model & solver CP

LS & CBLS

Hybrid Technologies techno&solver-speciﬁc (optimal) Case Study predicate deﬁnitions solution Choosing a Technology and Backend From a single language, one has access transparently to a wide range of solving technologies from which to choose.

COCP/M4CO 7 - 4 - Outline

1. The MiniZinc Toolchain The MiniZinc Toolchain 2. Comparison Criteria Comparison Criteria 3. SAT SAT 4. SMT & OMT SMT & OMT IP & MIP 5. IP & MIP CP

LS & CBLS 6.CP

Hybrid Technologies 7. LS & CBLS Case Study 8. Hybrid Technologies Choosing a Technology and Backend 9. Case Study 10. Choosing a Technology and Backend

COCP/M4CO 7 - 5 - Objectives

The MiniZinc Toolchain An overview of some solving technologies: Comparison Criteria

SAT

SMT & OMT to understand their advantages and limitations;

IP & MIP

CP LS & CBLS to help you choose a technology for a particular model; Hybrid Technologies

Case Study

Choosing a Technology to help you adapt a model to a particular technology. and Backend

COCP/M4CO 7 - 6 - Examples (Solving technologies) With general-purpose solvers, taking model&data as input: Boolean satisﬁability (SAT)

The MiniZinc SAT (resp. optimisation) modulo theories (SMT & OMT) Toolchain (Mixed) integer linear programming (IP & MIP) Comparison Criteria Constraint programming (CP) part 2 of 1DL441 SAT ... SMT & OMT IP & MIP Hybrid technologies (LCG = CP + SAT, . . . ) CP Methodologies, usually without modelling and solvers: LS & CBLS Dynamic programming (DP) Hybrid Technologies Greedy algorithms Case Study

Choosing a Approximation algorithms Technology and Backend Local search (LS) Genetic algorithms (GA) ... COCP/M4CO 7 - 7 - How to Compare Solving Technologies?

Modelling Language: The MiniZinc What types of decision variables are available? Toolchain

Comparison Which constraint predicates are available? Criteria Can there be an objective function? SAT SMT & OMT Guarantees: IP & MIP Are solvers exact, given enough time: will they ﬁnd all CP

LS & CBLS solutions, prove optimality, and prove unsatisﬁability? Hybrid If not, is there an approximation ratio? Technologies Case Study Features: Choosing a Technology Can the modeller guide the solving? If yes, then how? and Backend In which areas has the techno been successfully used? How do solvers work?

COCP/M4CO 7 - 8 - How Do Solvers Work? (Hooker, 2012)

Deﬁnition (Solving = Search + Inference + Relaxation) The MiniZinc Toolchain Search: Explore the space of candidate solutions.

Comparison Criteria Inference: Reduce the space of candidate solutions. SAT Relaxation: Exploit solutions to easier problems. SMT & OMT IP & MIP Deﬁnition (Systematic Search) CP

LS & CBLS Progressively build a solution, and backtrack if necessary.

Hybrid Use inference and relaxation to reduce the search effort. Technologies It is used in most SAT, SMT, OMT, CP, LCG, & MIP solvers. Case Study

Choosing a Technology and Backend Deﬁnition (Local Search) Start from a candidate solution and iteratively modify it a bit. It is the basic idea behind LS and GA solvers.

COCP/M4CO 7 - 9 - Outline

1. The MiniZinc Toolchain The MiniZinc Toolchain 2. Comparison Criteria Comparison Criteria 3. SAT SAT 4. SMT & OMT SMT & OMT IP & MIP 5. IP & MIP CP

LS & CBLS 6.CP

Hybrid Technologies 7. LS & CBLS Case Study 8. Hybrid Technologies Choosing a Technology and Backend 9. Case Study 10. Choosing a Technology and Backend

COCP/M4CO 7 - 10 - Boolean Satisﬁability Solving (SAT)

Modelling Language: Only Boolean variables. The MiniZinc Toolchain A set of clauses, denoting their conjunction (/\): Comparison • A clause is a disjunction (\/) of literals. Criteria • A literal is a Boolean variable or its negation. SAT Only for satisfaction problems: no objective function; SMT & OMT otherwise: iterate over candidate objective values. IP & MIP CP Example (in MiniZinc syntax) LS & CBLS Variables: var bool: w, x, y, z; Hybrid Technologies Clauses: Case Study

Choosing a constraint(not w \/ not y) /\ (not x \/ y) Technology /\ (not w \/ x \/ not z) and Backend /\ (x \/ y \/ z) /\ (w \/ not z);

A solution: w=false, x=true, y=true, z=false

COCP/M4CO 7 - 11 - The SAT Problem

Given a clause set, ﬁnd a valuation, that is Boolean values The MiniZinc Toolchain for all the variables, so that all the clauses are satisﬁed.

Comparison Criteria The decision version of this problem is NP-complete. SAT

SMT & OMT

IP & MIP Any combinatorial problem can be encoded into SAT. CP Careful: “encoded into” 6= “reduced from”. LS & CBLS There are recipes to clausify non-Boolean constraints. Hybrid Technologies

Case Study There has been intensive research since the 1960s.

Choosing a Technology and Backend We focus here on systematic search, namely DPLL [Davis-Putnam-Logemann-Loveland, 1962].

COCP/M4CO 7 - 12 - DPLL

Tree Search, upon starting from the empty valuation: The MiniZinc Toolchain 1 Perform inference (see below). Comparison 2 If some clause is unsatisﬁed, then backtrack. Criteria SAT 3 If all variables have a value, then we have a solution. SMT & OMT 4 Select an unvalued variable b and make two branches: IP & MIP one with b = true, and the other one with b = false. CP

LS & CBLS 5 Recursively explore each of the two branches.

Hybrid Technologies Inference: Case Study Unit propagation: If all the literals in a clause evaluate Choosing a Technology to false, except one whose variable has no value yet, and Backend then that literal is made to evaluate to true so that the clause becomes satisﬁed.

COCP/M4CO 7 - 13 - Strategies and Improvements over DPLL

Search Strategies: The MiniZinc Toolchain On which variable to branch next? Comparison Criteria Which branch to explore next? SAT Which search (depth-ﬁrst, breadth-ﬁrst, . . . ) to use? SMT & OMT

IP & MIP

CP Improvements: LS & CBLS Backjumping Hybrid Technologies Clause learning Case Study Restarts Choosing a Technology A lot of implementation details and Backend ...

COCP/M4CO 7 - 14 - SAT Solving

The MiniZinc Guarantee: exact, given enough time. Toolchain

Comparison Criteria Mainly black-box: limited ways to guide the solving. SAT SMT & OMT It can scale to millions of variables and clauses. IP & MIP

LS & CBLS Encoding a problem can yield a huge SAT model.

Hybrid Technologies Some solvers can extract an unsatisfiable core, that is Case Study a subset of clauses that make the model unsatisfiable. Choosing a Technology and Backend It is mainly used in hardware and software verification.

COCP/M4CO 7 - 15 - SAT @ MiniZinc and Uppsala University

The MiniZinc The MiniZinc toolchain was extended with the PicatSAT Toolchain backend, which uses the SAT solver Plingeling. Comparison Criteria

SAT Several research groups at Uppsala University

SMT & OMT use SAT solvers, such as: IP & MIP • Algorithmic Program Veriﬁcation CP • Embedded Systems

LS & CBLS • Programming Languages

Hybrid • Theory for Concurrent Systems Technologies

Case Study

Choosing a My Algorithms & Datastructures 3 (1DL481) course Technology and Backend discusses SAT solving and has a homework where a SAT model is designed and fed to a SAT solver.

COCP/M4CO 7 - 16 - Outline

1. The MiniZinc Toolchain The MiniZinc Toolchain 2. Comparison Criteria Comparison Criteria 3. SAT SAT 4. SMT & OMT SMT & OMT IP & MIP 5. IP & MIP CP

LS & CBLS 6.CP

Hybrid Technologies 7. LS & CBLS Case Study 8. Hybrid Technologies Choosing a Technology and Backend 9. Case Study 10. Choosing a Technology and Backend

COCP/M4CO 7 - 17 - SAT Modulo Theories (SMT) and OMT

Modelling Language:

The MiniZinc Language of SAT: Boolean variables and clauses. Toolchain

Comparison Several theories extend the language, say bit vectors, Criteria uninterpreted functions, or linear integer arithmetic. SAT

SMT & OMT SMT is only for satisfaction problems. IP & MIP OMT (optimisation modulo theories) extends SMT. CP LS & CBLS Deﬁnition Hybrid Technologies A theory Case Study deﬁnes variable types and constraint predicates, and Choosing a Technology is associated with a sub-solver for any conjunction of and Backend the supported constraint predicates. Different SMT/OMT solvers may have different theories.

COCP/M4CO 7 - 18 - Example (Linear integer arithmetic; in MiniZinc syntax) Variables: var int: x; var int: y; Constraints:

The MiniZinc Toolchain constraint x >= 0; constraint y <= 0; Comparison constraint x = y + 1 \/ x = 2 * y; Criteria constraint x = 2 \/ y = -2 \/ x = y; SAT SMT & OMT Unique solution: x = 0, y = 0 IP & MIP Decomposition: CP • Theory constraints, using reiﬁed constraints: LS & CBLS Hybrid a <-> x >= 0; b <-> y <= 0; Technologies c <-> x = y + 1; d <-> x = 2 * y; Case Study e <-> x = 2; f <-> y = -2; g <-> x = y; Choosing a Technology • and Backend Boolean skeleton: a /\ b /\ (c \/ d) /\ (e \/ f \/ g);

COCP/M4CO 7 - 19 - SMT Solving: DPLL(T )

Basic Idea: The MiniZinc Toolchain Separate the theory constraints and Boolean skeleton:

Comparison each variable in the Boolean skeleton represents Criteria whether a constraint holds or not. SAT SMT & OMT Use DPLL to solve the Boolean skeleton. IP & MIP CP If a constraint must hold as per DPLL, LS & CBLS then submit it to the relevant theory solver. Hybrid Technologies A theory solver operates on a constraint conjunction: Case Study • Choosing a It checks whether the conjunction is satisﬁable. Technology • It tries to infer that other constraints must (respectively and Backend cannot) hold and it sets the corresponding Boolean variables to true (respectively false).

COCP/M4CO 7 - 20 - Strategies and Improvements

Search Strategies: The MiniZinc Toolchain On which variable to branch next? Comparison Criteria Which branch to explore next? SAT Which strategy (depth-ﬁrst, breadth-ﬁrst, . . . ) to use? SMT & OMT IP & MIP Improvements to SAT Solving: CP

LS & CBLS See slide 14.

Hybrid Technologies Improvements to the Theory Solvers: Case Study

Choosing a More efﬁcient inference algorithms: incrementality. Technology and Backend Richer theories. ...

COCP/M4CO 7 - 21 - SMT and OMT Solving

The MiniZinc Toolchain Guarantee: exact, given enough time. Comparison Criteria

SAT SMT & OMT Mainly black-box: limited ways to guide the solving. IP & MIP

CP LS & CBLS It is based on very efﬁcient SAT technology. Hybrid Technologies

Case Study

Choosing a Technology It is mainly used in hardware and software veriﬁcation. and Backend

COCP/M4CO 7 - 22 - S/OMT @ MiniZinc and Uppsala University

The MiniZinc toolchain was extended with: • The MiniZinc fzn2smt: generates SMTlib models that can be fed to Toolchain any SMT solver, such as CVC4, Yices 2, Z3,... Comparison • emzn2fzn + fzn2omt: generates models that can be fed Criteria to any OMT solver, such as OptiMathSAT, Z3,... SAT

SMT & OMT The Embedded Systems research group at Uppsala IP & MIP University designs SMT solvers. CP Several other research groups at Uppsala University LS & CBLS use SMT and OMT solvers, such as: Hybrid • Technologies Algorithmic Program Veriﬁcation • Case Study Programming Languages • Choosing a Theory for Concurrent Systems Technology and Backend My Algorithms & Datastructures 3 (1DL481) course discusses SMT solving and has a homework where an SMT model is designed and fed to an SMT solver.

COCP/M4CO 7 - 23 - Outline

1. The MiniZinc Toolchain The MiniZinc Toolchain 2. Comparison Criteria Comparison Criteria 3. SAT SAT 4. SMT & OMT SMT & OMT IP & MIP 5. IP & MIP CP

LS & CBLS 6.CP

Hybrid Technologies 7. LS & CBLS Case Study 8. Hybrid Technologies Choosing a Technology and Backend 9. Case Study 10. Choosing a Technology and Backend

COCP/M4CO 7 - 24 - Integer (Linear) Programming (IP, ILP)

Modelling Language: Only integer variables. The MiniZinc Toolchain A set of linear equality & inequality constraints (no 6=). Comparison Only for optimisation problems (otherwise: optimise a Criteria

SAT value): linear objective function. SMT & OMT Example (in MiniZinc syntax) IP & MIP

CP Variables: var int: p; var int: q; LS & CBLS Constraints: Hybrid Technologies constraint p >= 0; constraint q >= 0; Case Study constraint p+2 * q <= 5; Choosing a constraint3 p + 2 q <= 9; Technology * * and Backend Objective: maximize3 * p + 4 * q; Unique optimal solution: p = 1, q = 2

COCP/M4CO 7 - 25 - Mathematical Programming

0-1 linear programming: The MiniZinc Toolchain linear (in)equalities over variables over domain {0, 1}.

Comparison Criteria Linear programming (LP): SAT linear (in)equalities over ﬂoating-point variables. SMT & OMT IP & MIP Mixed integer (linear) programming (MIP): CP linear (in)equalities over ﬂoating-point & int. variables. LS & CBLS

Hybrid Technologies Quadratic programming (QP): Case Study quadratic objective function. Choosing a Technology and Backend ...

There has been intensive research since the 1940s.

COCP/M4CO 7 - 26 - IP Solving

Basic Idea = Relaxation: The MiniZinc Toolchain Polytime algorithms (such as the interior-point method Comparison Criteria and the ellipsoid method) and exponential-time but

SAT practical algorithms (such as the simplex method)

SMT & OMT exist for solving LP models very efﬁciently. IP & MIP Use them for IP by occasionally relaxing an IP model CP by dropping its integrality requirement on the variables. LS & CBLS

Hybrid Technologies Implementations: Case Study Branch and bound= relaxation+ search. Choosing a Technology and Backend Cutting-plane algorithms= relaxation+ inference. Branch and cut= relaxation+ search+ inference.

COCP/M4CO 7 - 27 - Branch and Bound

Tree Search, upon initialising the incumbent to ±∞: The MiniZinc Toolchain 1 Relax the IP model into an LP model, and solve it. Comparison Criteria 2 If the LP model is unsatisﬁable, then backtrack. SAT 3 If all the variables have an integer value in the optimal SMT & OMT LP solution, then backtrack upon updating, if need be, IP & MIP the incumbent to the objective value of that IP solution. CP

LS & CBLS 4 If the objective value of the optimal LP solution is no

Hybrid better than the incumbent, then backtrack. Technologies 5 Otherwise, some variable v has a non-integer value ρ. Case Study

Choosing a Make two branches: Technology and Backend one with v ≤ b ρc, and the other one with v ≥ dρe. 6 Recursively explore each of the two branches.

COCP/M4CO 7 - 28 - Strategies and Improvements

Search Strategies:

The MiniZinc On which variable to branch next? Toolchain Which branch to explore next? Comparison Criteria Which search (depth-ﬁrst, breadth-ﬁrst, . . . ) to use? SAT

SMT & OMT Improvements: IP & MIP Cutting planes: Add implied linear constraints that CP improve the objective value of the LP relaxation. LS & CBLS Decomposition: Split into a master problem and a Hybrid Technologies subproblem, such as by the Benders decomposition. Case Study Solving the LP relaxation: Choosing a Technology • Primal-dual methods. and Backend • Efﬁcient algorithms for special cases, such as ﬂows. • Incremental solving. ...

COCP/M4CO 7 - 29 - IP Solving

Guarantee: exact, given enough time. The MiniZinc Toolchain Mainly black-box: limited ways to guide the solving. Comparison Criteria It scales well. SAT SMT & OMT Any combinatorial problem can be encoded into IP. IP & MIP There are recipes to linearise non-linear constraints. CP Advantages: LS & CBLS • Hybrid Provides both a lower bound and an upper bound on Technologies the objective value of optimal solutions, if stopped early. Case Study • Naturally extends to MIP solving. Choosing a • ... Technology and Backend Central method of operations research (OR), used in production planning, vehicle routing, . . .

COCP/M4CO 7 - 30 - MIP @ MiniZinc and Uppsala University

The MiniZinc toolchain comes bundled with a backend The MiniZinc Toolchain that can be hooked to the following MIP solvers:

Comparison • Cbc (open-source, bundled); Criteria • CPLEX Optimizer (commercial: requires a license); SAT • FICO Xpress Solver (commercial: requires a license); SMT & OMT • Gurobi Optimizer (commercial: requires a license). IP & MIP CP The Optimisation research group at Uppsala University LS & CBLS uses MIP solvers for 4G / 5G network planning and Hybrid Technologies optimisation, etc. Case Study Choosing a My Algorithms & Datastructures 3 (1DL481) course Technology and Backend discusses MIP solving and has a homework where a MIP model is designed and fed to a MIP solver.

COCP/M4CO 7 - 31 - Outline

1. The MiniZinc Toolchain The MiniZinc Toolchain 2. Comparison Criteria Comparison Criteria 3. SAT SAT 4. SMT & OMT SMT & OMT IP & MIP 5. IP & MIP CP

LS & CBLS 6.CP

Hybrid Technologies 7. LS & CBLS Case Study 8. Hybrid Technologies Choosing a Technology and Backend 9. Case Study 10. Choosing a Technology and Backend

COCP/M4CO 7 - 32 - Constraint Programming (CP)

The MiniZinc Modelling Language = full MiniZinc: Toolchain Boolean, integer, enum, ﬂoat, and / or set variables. Comparison Criteria Constraints based on a large vocabulary of predicates. SAT For satisfaction problems and optimisation problems. SMT & OMT

IP & MIP CP Many solvers: LS & CBLS There will be no standard for what is to be supported: Hybrid Technologies different CP solvers may have different sets of variable Case Study types and constraint predicates (under distinct names). Choosing a Technology Some solvers support even higher-level variable types, and Backend such as graphs and strings, and associated predicates.

COCP/M4CO 7 - 33 - Domains

Definition The MiniZinc Toolchain The domain of a variable v, denoted here by dom(v), is the Comparison Criteria set of values that v can still take during search: SAT The domains of the variables are reduced by search SMT & OMT and by inference (see the next two slides). IP & MIP A variable is said to be fixed if its domain is a singleton. CP LS & CBLS Unsatisfiability occurs if a variable domain goes empty. Hybrid Technologies

Case Study Note the difference between: Choosing a a domain as a technology-independent declarative Technology and Backend entity at the modelling level; and a domain as a procedural data structure for CP solving.

COCP/M4CO 7 - 34 - CP Solving

Tree Search, upon initialising each domain as in the model: The MiniZinc Toolchain Satisfaction problem: Comparison 1 Perform inference (see the next slide). Criteria

SAT 2 If the domain of some variable is empty, then backtrack. SMT & OMT 3 If all variables are ﬁxed, then we have a solution. IP & MIP 4 Select a non-ﬁxed variable v, partition its domain into CP two parts π and π , and make two branches: LS & CBLS 1 2

Hybrid one with v ∈ π1, and the other one with v ∈ π2. Technologies 5 Recursively explore each of the two branches. Case Study

Choosing a Technology Optimisation problem: when a feasible solution is found at and Backend step 3, ﬁrst add the constraint that the next solution must be better and then backtrack.

COCP/M4CO 7 - 35 - CP Inference

Deﬁnition The MiniZinc A propagator for a predicate γ deletes from the current Toolchain

Comparison domains of the variables of a γ-constraint the values that Criteria cannot be part of a solution to that constraint. SAT SMT & OMT Examples IP & MIP

CP For x

LS & CBLS delete 3..4 from dom(x) and -1..1 from dom(y). Hybrid For alldifferent([x,y,z]): if dom(x) = {1,3} = Technologies

Case Study dom(y) and dom(z) = 1..4, then delete 1 & 3 from

Choosing a dom(z) so that it becomes the non-range {2,4}. Technology and Backend The propagator of a constraint remains active as long as the Cartesian product of the domains of its variables is not known to contain only solutions to the constraint.

COCP/M4CO 7 - 36 - Strategies and Improvements

Search Strategies:

The MiniZinc On which variable to branch next? Toolchain How to partition the domain of the chosen variable? Comparison Criteria Which search (depth-ﬁrst, breadth-ﬁrst, . . . ) to use? SAT Improvements: SMT & OMT

IP & MIP Propagators, including for all the predicates in

CP Topic 3: Constraint Predicates. LS & CBLS Not all impossible domain values need to be deleted: Hybrid there is a compromise between algorithm complexity Technologies and achieved inference. Case Study

Choosing a Partition the chosen domain into at least two parts. Technology and Backend Domain representations. Order in which propagators are executed. ... COCP/M4CO 7 - 37 - CP Solving

Guarantee: exact, given enough time. The MiniZinc Toolchain White-box: one can design one’s own propagators and Comparison search strategies, and choose among predeﬁned ones. Criteria SAT The higher-level modelling languages enable (for SMT & OMT details, see Topic 8: Inference & Search in CP & LCG): IP & MIP • inference at a higher level; and CP • search strategies stated in terms of problem concepts. LS & CBLS They inspired the MiniZinc modelling language. Hybrid Technologies Successful application areas: Case Study • Conﬁguration Choosing a Technology • Personnel rostering and Backend • Scheduling and timetabling • Vehicle routing • ...

COCP/M4CO 7 - 38 - CP @ MiniZinc and Uppsala University

The MiniZinc The MiniZinc toolchain was extended with backends for Toolchain numerous CP solvers, such as Gecode (bundled), Comparison Criteria Choco, JaCoP, Mistral, SICStus Prolog,... SAT

SMT & OMT

IP & MIP The Optimisation research group at Uppsala University CP contributes to the design of CP solvers and uses them, LS & CBLS say for air trafﬁc management, the conﬁguration of Hybrid wireless sensor networks, robot task sequencing, etc. Technologies

Case Study

Choosing a Technology Part 2 of Combinatorial Optimisation and Constraint and Backend Programming (1DL441) course covers CP in depth.

COCP/M4CO 7 - 39 - Outline

1. The MiniZinc Toolchain The MiniZinc Toolchain 2. Comparison Criteria Comparison Criteria 3. SAT SAT 4. SMT & OMT SMT & OMT IP & MIP 5. IP & MIP CP

LS & CBLS 6.CP

Hybrid Technologies 7. LS & CBLS Case Study 8. Hybrid Technologies Choosing a Technology and Backend 9. Case Study 10. Choosing a Technology and Backend

COCP/M4CO 7 - 40 - Local Search (LS)

Each variable is ﬁxed all the time. Search proceeds by moves: each move modiﬁes the The MiniZinc Toolchain values of a few variables in the current assignment, Comparison and is selected upon probing the cost impacts of Criteria

SAT several candidate moves, called the neighbourhood.

SMT & OMT Stop when a good enough assignment was found,

IP & MIP or when an allocated resource was exhausted, CP such as time spent or iterations made. LS & CBLS

Hybrid Technologies

Case Study

Choosing a Technology and Backend

Local moves Initial assignment COCP/M4CO 7 - 41 - Example (Travelling Salesperson) Problem: Given a set of cities with connecting roads, ﬁnd a tour (a Hamiltonian circuit) that visits each city

The MiniZinc exactly once, with the minimum travel distance. Toolchain

Comparison Representation: We see the set of cities as vertices V Criteria and the set of roads as edges E in a (not necessarily SAT complete) undirected graph G = (V , E). SMT & OMT 231 108 IP & MIP Gävle Borlänge CP 146 102 12 LS & CBLS 161 Hybrid 166 135 Uppsala Technologies 77 Örebro Västerås Case Study 72 95 Stockholm 113 Choosing a Technology 197 and Backend Example: s: We now design a local-search heuristic for this problem.

COCP/M4CO 7 - 42 - An edge set s ⊆ E that forms a tour: NP-hard! Complete E by adding inﬁnite-distance edges: any permutation of V yields an initial assignment.

Replace two edges on the tour s by two other edges so that s is still a tour.

X The sum of all distances on the tour: Dist(a, b), (a,b)∈s as no constraint violation must be taken into account.

Select a random best neighbour.

Example (Travelling Salesperson: Choices) We must deﬁne: 1 The initial assignment: The MiniZinc Toolchain

Comparison Criteria

SAT 2 SMT & OMT The neighbourhood of candidate moves:

IP & MIP

CP LS & CBLS 3 The cost of an assignment: Hybrid Technologies

Case Study

Choosing a Technology and Backend 4 The neighbour selector:

COCP/M4CO 7 - 43 - : NP-hard! Complete E by adding inﬁnite-distance edges: any permutation of V yields an initial assignment.

Replace two edges on the tour s by two other edges so that s is still a tour.

X The sum of all distances on the tour: Dist(a, b), (a,b)∈s as no constraint violation must be taken into account.

Select a random best neighbour.

Example (Travelling Salesperson: Choices) We must deﬁne: 1 The initial assignment: The MiniZinc Toolchain An edge set s ⊆ E that forms a tour

Comparison Criteria

SAT 2 SMT & OMT The neighbourhood of candidate moves:

IP & MIP

CP LS & CBLS 3 The cost of an assignment: Hybrid Technologies

Case Study

Choosing a Technology and Backend 4 The neighbour selector:

COCP/M4CO 7 - 43 - Complete E by adding inﬁnite-distance edges: any permutation of V yields an initial assignment.

Replace two edges on the tour s by two other edges so that s is still a tour.

X The sum of all distances on the tour: Dist(a, b), (a,b)∈s as no constraint violation must be taken into account.

Select a random best neighbour.

Example (Travelling Salesperson: Choices) We must deﬁne: 1 The initial assignment: The MiniZinc Toolchain An edge set s ⊆ E that forms a tour: NP-hard!

Comparison Criteria

SAT 2 SMT & OMT The neighbourhood of candidate moves:

IP & MIP

CP LS & CBLS 3 The cost of an assignment: Hybrid Technologies

Case Study

Choosing a Technology and Backend 4 The neighbour selector:

COCP/M4CO 7 - 43 - Replace two edges on the tour s by two other edges so that s is still a tour.

X The sum of all distances on the tour: Dist(a, b), (a,b)∈s as no constraint violation must be taken into account.

Select a random best neighbour.

Example (Travelling Salesperson: Choices) We must deﬁne: 1 The initial assignment: The MiniZinc Toolchain An edge set s ⊆ E that forms a tour: NP-hard!

Comparison Complete E by adding inﬁnite-distance edges: Criteria any permutation of V yields an initial assignment. SAT 2 SMT & OMT The neighbourhood of candidate moves:

IP & MIP

CP LS & CBLS 3 The cost of an assignment: Hybrid Technologies

Case Study

Choosing a Technology and Backend 4 The neighbour selector:

COCP/M4CO 7 - 43 - X The sum of all distances on the tour: Dist(a, b), (a,b)∈s as no constraint violation must be taken into account.

Select a random best neighbour.

Example (Travelling Salesperson: Choices) We must deﬁne: 1 The initial assignment: The MiniZinc Toolchain An edge set s ⊆ E that forms a tour: NP-hard!

Comparison Complete E by adding inﬁnite-distance edges: Criteria any permutation of V yields an initial assignment. SAT 2 SMT & OMT The neighbourhood of candidate moves:

IP & MIP Replace two edges on the tour s by two other edges so

CP that s is still a tour. LS & CBLS 3 The cost of an assignment: Hybrid Technologies

Case Study

Choosing a Technology and Backend 4 The neighbour selector:

COCP/M4CO 7 - 43 - Select a random best neighbour.

Example (Travelling Salesperson: Choices) We must deﬁne: 1 The initial assignment: The MiniZinc Toolchain An edge set s ⊆ E that forms a tour: NP-hard!

Comparison Complete E by adding inﬁnite-distance edges: Criteria any permutation of V yields an initial assignment. SAT 2 SMT & OMT The neighbourhood of candidate moves:

IP & MIP Replace two edges on the tour s by two other edges so

CP that s is still a tour. LS & CBLS 3 The cost of an assignment: Hybrid X Technologies The sum of all distances on the tour: Dist(a, b), Case Study (a,b)∈s Choosing a as no constraint violation must be taken into account. Technology and Backend 4 The neighbour selector:

COCP/M4CO 7 - 43 - Example (Travelling Salesperson: Choices) We must deﬁne: 1 The initial assignment: The MiniZinc Toolchain An edge set s ⊆ E that forms a tour: NP-hard!

Comparison Complete E by adding inﬁnite-distance edges: Criteria any permutation of V yields an initial assignment. SAT 2 SMT & OMT The neighbourhood of candidate moves:

IP & MIP Replace two edges on the tour s by two other edges so

COCP/M4CO 7 - 43 - Example (Travelling Salesperson: Sample Run) Three consecutive improving current assignments: The MiniZinc Toolchain 231 108 231 108 231 108 Comparison Gävle Gävle Gävle Criteria Borlänge Borlänge Borlänge 146 146 146 102 12 102 12 102 12 SAT 161 161 161 166 135 Uppsala 166 135 Uppsala 166 135 Uppsala SMT & OMT 77 77 77 Örebro Västerås Örebro Västerås Örebro Västerås 72 72 72 95 Stockholm 95 Stockholm 95 Stockholm IP & MIP 113 113 113 CP 197 197 197 s: s: s: LS & CBLS f(s) = 709 f(s) = 656 f(s) = 530

Hybrid Technologies

Case Study Choosing a ˚ Technology This section is so far based on material by Magnus Agren. and Backend

COCP/M4CO 7 - 44 - Heuristics drive the search to (good enough) solutions: Which decision variables are modiﬁed in a move? Which new values do they get in the move? Metaheuristics drive the search to global optima: The MiniZinc Toolchain Avoid cycles of moves & escape local optima. Comparison Criteria Explore many parts of the search space. SAT Focus on promising parts of the search space. SMT & OMT IP & MIP Examples (Metaheuristics) CP

LS & CBLS Tabu search (1986):

Hybrid forbid recent moves from being done again. Technologies Simulated annealing (1983): Case Study

Choosing a perform random moves and accept degrading ones Technology and Backend with a probability that decreases over time. Genetic algorithms (1975): use a pool of current assignments and cross them.

COCP/M4CO 7 - 45 - Systematic Search (as in SAT, SMT, OMT, MIP, and CP): + Will ﬁnd an (optimal) solution, if one exists. + Will give a proof of unsatisﬁability, otherwise.

The MiniZinc − May take a long time to complete. Toolchain

Comparison − Sometimes does not scale well to large instances. Criteria − May need a lot of tweaking: search strategies, . . . SAT

SMT & OMT Local Search: (Hoos and Stutzle,¨ 2004) IP & MIP CP + May ﬁnd an (optimal) solution, if one exists. LS & CBLS − Can rarely give a proof of unsatisﬁability, otherwise. Hybrid Technologies − Can rarely guarantee that a found solution is optimal. Case Study + Often scales much better to large instances. Choosing a Technology − May need a lot of tweaking: (meta)heuristics, . . . and Backend Local search trades completeness and quality for speed!

COCP/M4CO 7 - 46 - Constraint-Based Local Search (CBLS)

MiniZinc-style modelling language: The MiniZinc • Boolean, integer, and / or set decision variables. Toolchain • Constraints based on a large vocabulary of predicates. Comparison Criteria • Three sorts of constraints: see the next three slides.

SAT • For satisfaction problems and optimisation problems.

SMT & OMT

IP & MIP Fairly recent: around the year 2000.

CP Guarantee: inexact on most instances (that is: there is LS & CBLS

Hybrid no promise to ﬁnd all solutions, to prove optimality, or Technologies to prove unsatisﬁability), without approximation ratio. Case Study Choosing a White-box: one must design a search algorithm, Technology and Backend which probes the cost impacts for guidance.

More scalable than systematic approaches.

COCP/M4CO 7 - 47 - Deﬁnition Each constraint predicate has a violation function: the violation of a constraint is zero if it is satisﬁed,

The MiniZinc else a positive value proportional to its dissatisfaction. Toolchain

Comparison Criteria Example SAT SMT & OMT For a <= b, let α and β be the current values of a and b: IP & MIP deﬁne the violation to be α − β if α 6≤ β, and 0 otherwise. CP

LS & CBLS

Hybrid Deﬁnition Technologies A constraint with violation is explicit in a CBLS model Case Study

Choosing a and soft: it can be violated during search but ought to be Technology satisﬁed in a solution. and Backend

The constraint violations are queried during search.

COCP/M4CO 7 - 48 - Deﬁnition A one-way constraint is explicit in a CBLS model and hard:

The MiniZinc it is kept satisﬁed during search. Toolchain

Comparison Criteria SAT Example SMT & OMT For p = a b, whenever the value α of a or the value β IP & MIP * b p CP of is modiﬁed by a move, the value of is automatically

LS & CBLS modiﬁed by the solver so as to remain equal to α · β.

Hybrid Technologies Case Study CBLS solvers offer a syntax for one-way constraints, Choosing a Technology such as p <== a * b in OscaR.cbls, and Backend but MiniZinc does not make such a syntactic distinction.

COCP/M4CO 7 - 49 - Definition An implicit constraint is not in a CBLS model but hard: it is kept satisfied during search by choosing a feasible initial assignment and only making satisfaction-preserving moves, The MiniZinc Toolchain by the use of a constraint-specific neighbourhood. Comparison Criteria

SAT Example SMT & OMT For alldifferent(...), the initial assignment has IP & MIP distinct values for all variables, and the neighbourhood only CP has moves that swap the values of two variables, assuming LS & CBLS the number of variables is equal to the number of values. Hybrid Technologies

Case Study When building a CBLS model, a MiniZinc backend must:

Choosing a Technology Aptly assort the otherwise all explicit & soft constraints. and Backend Add a suitable heuristic and meta-heuristic. This is much more involved than just ﬂattening and solving.

COCP/M4CO 7 - 50 - Example (Travelling Salesperson: Model and Solve) Recall the model, from Topic 1: Introduction, with a variable Next[c] for each city c: The MiniZinc Toolchain 3 solve minimize sum(c in Cities)(Dist[c,Next[c]]); Comparison 4 constraint circuit(Next);% ideally made implicit Criteria SAT Three consecutive current assignments, SMT & OMT preserving the satisfaction of the circuit(Next) IP & MIP constraint and improving the objective value: CP

231 108 231 108 231 108 LS & CBLS Gävle Gävle Gävle Borlänge Borlänge Borlänge Hybrid 146 102 146 102 146 102 Technologies 12 12 12 161 161 161 135 Uppsala 135 Uppsala 135 Uppsala Case Study 166 77 166 77 166 77 Örebro Västerås Örebro Västerås Örebro Västerås 72 72 72 Choosing a 95 Stockholm 95 Stockholm 95 Stockholm Technology 113 113 113 and Backend 197 197 197 s: s: s: f(s) = 709 f(s) = 656 f(s) = 530

COCP/M4CO 7 - 51 - (CB)LS @ MiniZinc and Uppsala University

The MiniZinc toolchain was extended with: The MiniZinc Toolchain • our fzn-oscar-cbls backend to the OscaR.cbls solver;

Comparison • the Yuck CBLS backend. Criteria

SAT

SMT & OMT The Optimisation research group at Uppsala University

IP & MIP contributes to the design of CBLS solvers.

LS & CBLS Several courses at Uppsala University discuss (CB)LS: Hybrid • Algorithms & Datastructures 3 (1DL481) discusses LS Technologies and has a homework where an LS program is written. Case Study • Artiﬁcial Intelligence (1DL340) discusses LS. Choosing a • Technology Part 2 of Combinatorial Optimisation and Constraint and Backend Programming (1DL441) covers CBLS in some depth. • Machine Learning (1DT071) discusses LS.

COCP/M4CO 7 - 52 - Outline

1. The MiniZinc Toolchain The MiniZinc Toolchain 2. Comparison Criteria Comparison Criteria 3. SAT SAT 4. SMT & OMT SMT & OMT IP & MIP 5. IP & MIP CP

LS & CBLS 6.CP

Hybrid Technologies 7. LS & CBLS Case Study 8. Hybrid Technologies Choosing a Technology and Backend 9. Case Study 10. Choosing a Technology and Backend

COCP/M4CO 7 - 53 - Crossfertilisation

Each technology has advantages and drawbacks.

The MiniZinc Toolchain Good ideas from one techno can be applied to another. Comparison A hybrid technology combines several technologies. Criteria

SAT This can yield new advantages with fewer drawbacks. SMT & OMT Some hybrid technologies are loosely coupled: IP & MIP separate solvers or sub-solvers cooperate. CP Other hybrid technologies are tightly coupled: LS & CBLS

Hybrid a single solver handles the whole model. Technologies Case Study Example (Loose hybrid technology) Choosing a Technology Logic-based Benders decomposition: divide the problem and Backend into two parts: a master problem, solved by IP, and a subproblem, solved by CP.

COCP/M4CO 7 - 54 - Tight Hybrid Technologies: Examples

Example (Lazy clause generation, LCG) The MiniZinc Toolchain Use CP propagators to generate clauses in a SAT solver. Comparison Criteria

SAT Example (Large-neighbourhood search, LNS, on COP) SMT & OMT IP & MIP Follow an LS procedure, but each move is performed by: CP 1 Undo the values for a subset of the variables. LS & CBLS 2 Hybrid Use CP to ﬁnd an (optimal) solution to the subproblem. Technologies

Case Study Choosing a Example (Constrained integer programming, CIP) Technology and Backend Use CP propagators in an IP solver in order to generate linear inequalities for non-linear constraints.

COCP/M4CO 7 - 55 - Hybrids @ MiniZinc and Uppsala University

The MiniZinc Toolchain The MiniZinc toolchain was extended with:

Comparison • LCG backends: Chuffed (bundled), Google CP-SAT; Criteria • a CIP backend: SCIP; SAT • LNS backends: the solvers of the Gecode SMT & OMT and Google CP-SAT backends can perform LNS IP & MIP (prescribed via MiniZinc annotations). CP

LS & CBLS

Hybrid Technologies The Optimisation research group at Uppsala University Case Study contributes to the design of hybrid solvers. Choosing a Technology and Backend

COCP/M4CO 7 - 56 - Outline

1. The MiniZinc Toolchain The MiniZinc Toolchain 2. Comparison Criteria Comparison Criteria 3. SAT SAT 4. SMT & OMT SMT & OMT IP & MIP 5. IP & MIP CP

LS & CBLS 6.CP

Hybrid Technologies 7. LS & CBLS Case Study 8. Hybrid Technologies Choosing a Technology and Backend 9. Case Study 10. Choosing a Technology and Backend

COCP/M4CO 7 - 57 - Example: Pigeonhole Problem

The MiniZinc Example (Pigeonhole) Toolchain Place n pigeons into n − 1 holes so that all pigeons are Comparison Criteria placed and no two pigeons are placed in the same hole. SAT

SMT & OMT

IP & MIP This problem is trivially unsatisﬁable,

CP but is a popular benchmark for solvers.

LS & CBLS

Hybrid We will use this problem to show: Technologies

Case Study how solvers may use different deﬁnitions of the same

Choosing a constraint predicate; Technology and Backend that it is often important for solving efﬁciency to use pre-deﬁned constraint predicates.

COCP/M4CO 7 - 58 - Pigeonhole: Models

Using alldifferent

The MiniZinc Toolchain 1 int: n;% the number of pigeons

Comparison 2 % Hole[p]= the hole of pigeonp: Criteria 3 array[1..n] of var 1..(n-1): Hole; SAT 4 constraint alldifferent(Hole); SMT & OMT 5 solve satisfy; IP & MIP

CP LS & CBLS Using != Hybrid Technologies 1 int: n; Case Study 2 array[1..n] of var 1..(n-1): Hole; Choosing a Technology 3 constraint forall(i,j in 1..n where i < j) and Backend (Hole[i] != Hole[j]);

4 solve satisfy;

COCP/M4CO 7 - 59 - Constraint Predicate Deﬁnitions

Built-in alldifferent for probably all CP solvers

The MiniZinc Toolchain predicate alldifferent_int

Comparison (array[int] of var int: X); Criteria SAT predicate int_ne(var int: x, var int: y); SMT & OMT

IP & MIP CP Non-built-in alldifferent for SMT solvers LS & CBLS

Hybrid predicate alldifferent_int Technologies (array[int] of var int: X) = Case Study forall(i,j in index_set(X) where i < j) Choosing a Technology (X[i] != X[j]); and Backend predicate int_ne(var int: x, var int: y);

COCP/M4CO 7 - 60 - Boolean-isation for SAT solvers

predicate alldifferent_int (array[int] of var int: X) = let{ The MiniZinc Toolchain array[int,int] of var bool: Y = int2bools(X); array[...,...] of var bool: A; Comparison Criteria } in forall(i in ..., j in ...)

SAT ((A[i-1,j] -> A[i,j])

SMT & OMT /\ (Y[i,j] <-> (not A[i-1,j] /\ A[i,j]))); function array[int,int] of var bool: int2bools IP & MIP (array[int] of var int: X) = [...]; CP LS & CBLS When X has n decision variables over domains of size m, Hybrid Y A · Technologies this ladder encoding yields the two arrays and of n m

Case Study Boolean variables (where Y[i,v]=true iff X[i]=v,

Choosing a and A[i,v]=true iff v in X[1..i]) Technology 2 and Backend as well as O(n ) clauses of 2 or 3 literals. This is more compact and usually more efﬁcient than the direct encoding with O(n3) clauses of 2 literals over only Y.

COCP/M4CO 7 - 61 - Linearisation for MIP solvers: Cbc, CPLEX, Gurobi, ...

predicate alldifferent_int (array[int] of var int: X) = The MiniZinc Toolchain let{array[int,int] of var 0..1: Y = Comparison eq_encode(X) Criteria } in forall(d in index_set_2of2(Y)) SAT (sum(i in index_set_1of2(Y)) SMT & OMT

IP & MIP (Y[i,d]) <= 1);

LS & CBLS predicate int_ne(var int: x, var int: y) = Hybrid let{var 0..1: p} Technologies in x - y + 1 <= ub(x - y + 1) (1 - p) Case Study * /\ y - x + 1 <= ub(y - x + 1) p; Choosing a * Technology and Backend % ... continued on next slide ...

COCP/M4CO 7 - 62 - Linearisation for MIP solvers (end)

% ... continued from previous slide ...

The MiniZinc Toolchain function array[int,int] of var int: Comparison Criteria eq_encode(array[int] of var int: X) = SAT [... equality_encoding(...) ...] SMT & OMT IP & MIP predicate equality_encoding(var int: x, CP array[int] of var 0..1: Y) = LS & CBLS x in index_set(Y) Hybrid Technologies /\ Case Study sum(d in index_set(Y))(Y[d]) = 1 Choosing a /\ Technology and Backend sum(d in index_set(Y))(d * Y[d]) = x;

COCP/M4CO 7 - 63 - Pigeonhole: Experimental Comparison

Time, in seconds, to prove unsatisﬁability: The MiniZinc n backend alldifferent != Toolchain 10 mzn-gecode < 1 < 1 Comparison Criteria 10 mzn-gurobi < 1 58 SAT 11 mzn-gecode < 1 9 SMT & OMT 11 mzn-gurobi < 1 285 IP & MIP 12 mzn-gecode < 1 113 CP 12 mzn-gurobi < 1 3704 LS & CBLS

Hybrid 100 mzn-gecode < 1 time-out Technologies 100 mzn-gurobi < 1 time-out Case Study 300 mzn-gecode < 1 time-out Choosing a Technology 300 mzn-gurobi 24 time-out and Backend 100000 mzn-gecode < 1 time-out 1000000 mzn-gecode 5 time-out

COCP/M4CO 7 - 64 - Outline

1. The MiniZinc Toolchain The MiniZinc Toolchain 2. Comparison Criteria Comparison Criteria 3. SAT SAT 4. SMT & OMT SMT & OMT IP & MIP 5. IP & MIP CP

LS & CBLS 6.CP

Hybrid Technologies 7. LS & CBLS Case Study 8. Hybrid Technologies Choosing a Technology and Backend 9. Case Study 10. Choosing a Technology and Backend

COCP/M4CO 7 - 65 - Some Questions for Guidance

Do you need guarantees that a found solution is The MiniZinc Toolchain optimal, that all solutions are found, and that Comparison unsatisﬁability is provable? Criteria

SAT

SMT & OMT What types of variables are in your model?

IP & MIP CP What constraint predicates are in your model? LS & CBLS

Hybrid Technologies Does your problem look like a well-known problem?

Case Study Choosing a How do backends perform on easy problem instances? Technology and Backend What is your favourite technology or backend?

COCP/M4CO 7 - 66 - Some Caveats

Each problem can be modelled in many different ways.

The MiniZinc Different models of the same problem can be more Toolchain suited to different backends. Comparison Criteria The performance on small instances does not always SAT scale up to larger instances. SMT & OMT Sometimes, a good search strategy (see Topic 8: IP & MIP Inference & Search in CP & LCG) is more important CP than a good model. LS & CBLS Not all backends that use the same technology have Hybrid Technologies comparable performance. Case Study Some pure problems can be solved by specialist tools, Choosing a Technology say Concorde for the travelling salesperson problem: and Backend real-life side constraints often make them inapplicable. Some problems are maybe even solvable in polynomial time and space.

COCP/M4CO 7 - 67 - Take-Home Message: There are many solving technologies and backends. The MiniZinc Toolchain It is useful to highlight the commonalities & differences. Comparison Criteria No solving technology or backend can be universally SAT better than all the others, unless P = NP. SMT & OMT

IP & MIP Try them!

CP To go further: LS & CBLS Hybrid John N. Hooker. Technologies

Case Study Integrated Methods for Optimization.

Choosing a 2nd edition, Springer, 2012. Technology and Backend

COCP/M4CO 7 - 68 -