Masarykova univerzita Fakulta}w¡¢£¤¥¦§¨ informatiky !"#$%&'()+,-./012345 Usability improvements of OptaPlanner Benchmarker Diploma thesis Bc. Matej Čimbora Brno, January 2015 Declaration Hereby I declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Bc. Matej Čimbora Advisor: doc. RNDr. Tomáš Pitner, Ph.D. ii Acknowledgement I would like to thank doc. RNDr. Tomáš Pitner, Ph.D. and RNDr. Filip Nguyen for valuable advisory and comments. I owe special thanks to Geoffrey de Smet and Lukáš Petrovický for guidance and help during development process. iii Abstract This thesis is dedicated to improvements of OptaPlanner’s benchmark- ing functionality. Selected usability issues are presented, together with proposed solutions, while the main attention is paid on result persis- tence and comparison. In addition, a short introduction into constraint programming, together with an overview of the main features of Opta- Planner engine is included. iv Keywords OptaPlanner, benchmarking, constraint programming, optimization, Java v Contents 1 Introduction ...... 3 1.1 Goals ...... 3 1.2 Structure ...... 4 2 Constraint programming ...... 5 3 Configuration and basic usage ...... 8 3.1 Modeling the problem ...... 8 3.2 Solver configuration ...... 10 3.2.1 Solution evaluation ...... 12 3.2.2 Termination ...... 15 3.3 Running the solver ...... 15 4 Supported algorithms ...... 17 4.1 Local search algorithms ...... 17 4.1.1 Hill climbing ...... 17 4.1.2 Simulated annealing ...... 18 4.1.3 Tabu search ...... 18 4.1.4 Late acceptance ...... 19 4.2 Construction heuristics ...... 19 4.2.1 First fit ...... 19 4.2.2 Weakest fit ...... 20 4.2.3 Allocate entity from Queue ...... 20 4.2.4 Cheapest insertion ...... 20 4.3 Other ...... 20 5 Benchmarking ...... 21 5.1 Report document ...... 21 5.2 Configuration ...... 23 6 Result persistence and recovery ...... 25 6.1 Result persistence ...... 26 6.2 Resume functionality ...... 30 6.2.1 Usability issues ...... 33 7 Result aggregation ...... 34 7.1 User interface ...... 36 7.1.1 Benchmark selection area ...... 36 7.1.2 Menu bar ...... 38 7.1.3 Information panel ...... 39 7.1.4 Exception handling ...... 41 1 7.1.5 Real world usage ...... 42 8 Usage demonstration ...... 43 8.1 Problem description ...... 43 8.2 Scenario ...... 43 8.3 Running the benchmarker ...... 44 8.4 Data persistence verification ...... 45 8.5 Result aggregation ...... 45 9 Conclusion ...... 50 9.1 Future work ...... 50 A Source code locations ...... 51 A.1 Result persistence and recovery ...... 51 A.2 Aggregator GUI ...... 51 2 1 Introduction Resource allocation plays a vital role in business environment. Ability to allocate limited amount of resources, such as employees, or computa- tional power in an effective manner to meet company objectives is one of the factors, determining potential of its success. This area relates closely to problematics of constraint satisfaction, originating as a branch of ar- tificial intelligence, dating back to 1960s [18]. Rapid development of information technologies in the past decades enabled to tackle such problems utilizing computational resources, using specialized solvers. Popularity of the field is proved by numerous available products (e.g. OptaPlanner [12], IBM ILOG [7], Geocode [3], SICStus [14], Choco [1], etc.). Furthermore, the area of constraint programming has been in- cluded as a part of Java Specification Request (JSR 331)[9]. This thesis is dedicated to OptaPlanner, a lightweight, embeddable planning en- gine that optimizes planning problems[13]. It provides a way to solve the problems in a sophisticated way, combining constructing heuristic and metaheuristic algorithms with Java-based, or rule engine score cal- culation, while maintaining scalability of the solving process. There are numerous use case OptaPlanner is able to handle (e.g. Vehicle routing, Bin packing, Job shop sheduling, etc.). The product was formerly a part of Drools suite and was known as Drools Planner. As of version 6.0.0.Beta1 it exists as a standalone project, belonging to KIE group [10]. It is released under Apache Software License 2.0. 1.1 Goals Optimization of planning problems can be approached in different ways, depending on the type of algorithm used. OptaPlanner provides a fea- ture, which can ease related decision process, named benchmarker. Its purpose is to qualitatively compare how different solving algorithms perform on selected problems and provide comparison overview, incor- porating charts and various statistics. As the benchmarking process may take a long time to finish, a failure in the process, leading to loss of results is undesired. The first goal is to handle this use case by result persistence, enabling even partial results to be reused after the bench- marker crashes. The second goal relates closely to the aforementioned 3 1. Introduction persistence functionality. With the partial results of benchmarker pro- cess stored, it is possible to provide a tool for their further processing. This motivated creation of Swing-based application, allowing compar- ison of the results, originating even from different benchmarker pro- cesses. 1.2 Structure The first chapter Constraint programming briefly describes related field of constraint satisfaction, providing formal definitions and overview of basic terms. The following chapter Configuration and basic usage pro- vides description of the way OptaPlanner approaches constraint satis- faction problems. It covers solving procedure and the most notable con- figuration options. Chapter Supported algorithms contains an overview of majority of OptaPlanner’s heuristic and metaheuristic algorithms. The next chapter Benchmarking describes motivation of benchmarker usage, including examples of charts, generated as a part of its summary document. Result persistence and recovery and Result aggregation are chapters dedicated to newly introduced features. The chapters deal with implementation details, providing code samples, where necessary. Chapter Usage demonstration shows an example usage of aggregator application, developed to improve benchmarker result comparison. The last chapter, Conclusion briefly summarizes the results of this thesis, proposing further enhancements and development of the current state of implementation. 4 2 Constraint programming “Constraint programming is a powerful paradigm for solv- ing combinatorial search problems that draws on a wide range of techniques from artificial intelligence, computer science, databases, programming languages and operations research.” [18] Constraint programming has emerged as a branch of artificial intel- ligence, and currently has many applications in real world information systems (e.g. planning, scheduling). It is based on the concept of deci- sion variables and constraints, which restrict the values that individual variables can take. Its advantage lies in the ability to represent prob- lems in natural way, with variables directly corresponding to problem entities, leading to relatively simple formulation of the problem and better understanding of the solution [11]. This chapter provides a brief introduction into related problematics, starting with definition of the Constraint satisfaction problem [24]. Definition 1 (Constraint satisfaction problem). Constraint satisfac- tion problem (CSP) is defined as a triple (X,D,C), where • X = {x1, ..., xn} is a finite set of variables, • D = {D1, ..., Dn} is a finite set of values (domain), • C = {c1, ..., cm}, ci ⊆ Di1 ×...×Dik is a finite set of constraints. Solution of a CSP is an assignment of a single value from the domain to each variable such that no constraint is violated [17]. Definition 2 (Solution). (d1, ..., dn) ∈ D1 × ... × Dn is a solution of (X,D,C), if: •∀ ci ∈ C defined on xi1 , ..., xik :(di1 , ..., dik ) ∈ ci. Constraint satisfaction problems may have multiple number of so- lutions — in that case the problem is referred to as satisfiable. On the other hand, a problem with no possible solutions is called unsatisfiable. In cases, where the problem is over-constrained (it is not possible to 5 2. Constraint programming satisfy all constraints at the same time), or some solutions are qualita- tively better than the other, it is often advantageous to introduce a level of distinction between individual solutions. This motivates definition of Constraint Statisfaction Optimization Problems [23]. Definition 3 (Constraint Satisfaction Optimization Problem). Con- straint Satisfaction Optimization Problem is defined as a quadriple (X,D,Ch,Cs), where • X = {x1, ..., xn} is a finite set of variables, • D = {D1, ..., Dn} is a finite set of values (domain), • Ch = {C1, ..., Cn}, Ci ⊆ Di1 × ... × Dik is a finite set of hard constraints, where Ci is defined on Y ⊆ X,Yi = {xi1 , ..., xik }, + • Cs = {F1, ..., Fj}, Fj : Dj1 ×...×Djl → R is a finite set of soft constraints, where Fj defined on Qj ⊆ X,Qj = {xj1 , ..., xjl }. Additionally, an objective function is defined as follows: l ~ P ~ ~ ~ • F (d) = Fj(d[Qj), where d[Qj] is projection of d on [Qj]. j=1 Definition of solution for CSOP is slightly different compared to classical Constraint Satisfaction Problem. The solution of CSOP meets ~0 ~ all constraints defined in Ch and additionally F (d ) = maxd~F (d) or ~0 ~ F (d ) = mind~F (d). The difference between hard and soft constraints is obvious — hard constraints cannot be broken, they directly influence feasibility of a solution (e.g. single resource unit can be used only once at the same time). On the other hand, soft constraints can be broken, when it is necessary (e.g. situations, in which it is not possible to match all soft constraints). It is the requirement of evaluation function to de- termine, to what extent will the output be affected in that moment. The result can be influenced by setting the weight of individual con- straints – it determines importance of the constraint. Constraints with higher weight have more significant impact on the resulting evaluation, therefore they are more likely to be met. Some constraints can also have positive effect on the evaluation — they are called positive constraints (e.g. prefer tasks with lower resource usage). The exact definition of the evaluation function depends on the nature of a problem. 6 2. Constraint programming Taking aforementioned definitions into account, several solution types can be distinguished: • Feasible solution – solution, which does not break any hard con- straints. • Optimal solution – solution with the highest possible score for given optimization problem. • Best solution – solution, with the highest score achieved during pre-defined time period. In general, OptaPlanner focuses on finding the best solution — due to large search space it may be ineffective to search for the optimal solution, as the time required to find such solution may not fit the business needs (e.g. ambulance vehicles routing). Search space is defined as a space of states a search can arrive at during the solving process. It can be represented by search tree, where search space size relates to the number of nodes in the tree [26]. It is proportional to the size of domains of individual variables: Definition 4 (Number of nodes in the search tree). Number of nodes in the search tree is defined as: n P 1 + (|Dx1 | ∗ ... ∗ |Dxi |), where n denotes a number of variables. i=1 CSPs are well-known to be NP-hard in general [19]. The time re- quired to find a solution increases exponentially with linear increase of the input [20], which makes the choice of right solving approach cru- cial. For this purpose OptaPlanner provides a variety of heuristic and metaheuristic algorithms, which will be discussed later. 7 3 Configuration and basic usage Tackling optimization problems using OptaPlanner requires several steps to be performed. This chapter focuses on description of internal solving API and description of configuration process. The core part of the solv- ing process is handled by Solver class. The name indicates it represents a mechanism to handle a planning problem and solve it. Diagram 3.1 shows a top-level view on the solver. Individual terms will be introduced further in this chapter. Figure 3.1: Solver overview The solving procedure can be split into 3 parts: • Modeling the problem – creation of domain model of the problem • Solver configuration – setting up attributes influencing the solv- ing process (e.g. used algorithms) • Running the solver – start solving process on specific optimiza- tion problem and get the best solution when finished 3.1 Modeling the problem Creation of domain model for the corresponding optimization problem is the first step of the solving process. One of the advantages of using 8 3. Configuration and basic usage Figure 3.2: Problem representation OptaPlanner lies in the ability to represent the problem naturally using object oriented approach. The fact that the basic configuration is per- formed using annotations, with no other dependencies and processing logic implementation required, makes the objects easily reusable even for other purposes (e.g. marshalling). Diagram 3.2 generally describes how optimization problems are represented in the context of OptaPlan- ner. Solution class resides at the top of the hierarchy. It references all objects included in problem solving (i.e. variables, domain sets). It rep- resents current state of an optimization problem, which is defined by current initialization of variables. The state changes as the assignment of values from domains to variables alters during the solving process, in order to find a better solution. An important feature of solution is that it can be evaluated by evaluation function and current score (eval- uation) can be determined. The outcome of such function may vary depending on constraint definitions and current state of solution. Opti- mization algorithms OptaPlanner ships with utilize the result in order to steer the solving process in the most effective way. They will itera- tively try to improve the assignment, such that the highest evaluation based on the scoring function is obtained. 9 3. Configuration and basic usage In OptaPlanner world, variables are incorporated in a specialized wrapper object — planning entity. A single planning entity usually includes several variables, which are simply attributes of the wrapper. Their value change during solving, while the candidate value is picked from respective domain, which is usually defined in Solution class as a collection of objects. Domains of variables are usually represented by simple POJO (Plain Old Java Object) objects and are denoted as problem facts. Picture 3.3 [13] illustrates aforementioned configuration procedure. Suppose having a simple resource allocation problem, where the goal is to assign a set of processes to a set of computers. Each computer has limited amount of computational power, memory and network band- width, which can be utilized at one moment. Additionally, each process requires certain amount of resources (CPU, memory, network band- width) in order to be successfully performed. The picture shows sim- plicity of problem modeling using OptaPlanner. At first, Computer and Process classes need to be created, each of them defining computational power and memory required, or provided respectively. Obviously, one computer can be assigned several processes, therefore one-to-many rela- tionship is used. The next step lies in identification of a planning entity (object encapsulating variables). As a rule of thumb, it’s usually “many” side of relationship, therefore Process class can be safely annotated with @PlanningEntity annotation. Additionally, computer attribute of Pro- cess class needs to be annotated with @PlanningVariable to make it identified as a variable. Once finished, solution class is created, named CloudBalance, marked by @PlanningSolution annotation, referencing both Computer and Process objects in collections. The one containing all computers serves as a domain for process variables. The last step re- quires @PlanningEntityCollectionProperty annotation to be added to a collection, which references planning variables (processes). 3.2 Solver configuration The purpose of this section is to describe the way OptaPlanner is able to utilize its internal logic and supported algorithms to find solutions of optimization problems. The core part of OptaPlanner is called solver. Its purpose is to 10 3. Configuration and basic usage Figure 3.3: Sample resource allocation problem orchestrate the solving process and recall the best solution, so that it can be obtained once the solving finishes. The solving process itself is usually performed in multiple phases. During each phase, one of the configured solving algorithms is applied on a problem, modifying it to some degree. When a phase finishes, its result (modified solution) serves as a starting point for the following phase. This principle is generally used to solve problems in two stages — during the first stage, a solution is initialized using construction heuristics, whereas the purpose of the second stage is to optimize the solution, usually using metaheuristic algorithms. In order to be able to proceed further, it is important to explain the way a problem is processed during individual phases. As it was mentioned, when a problem is being solved, it changes its state. The changes are caused by modifications of assignment of planning variables to planning entities (identified by appropriate annotations, as described in the previous section) and they provide a way, how to build up a new solution from scratch or to modify the current solution in such way, that a better solution can be found. In context of OptaPlanner, such assignment change is called move. The set of possible moves is picked 11 3. Configuration and basic usage up from the neighbourhood of the current solution. Two solutions are called neighbours if one can be obtained by well defined changes of the other[22]. The component responsible for generation of possible moves is denoted as move selector. There are multiple move selectors OptaPlanner provides by default, that differ by approach they perform moves: • Change move selector – single planning entity and planning value are picked. Planning variable of the entity is assigned the value. • Pillar change move selector – a planning variable, which value is the same for a set of planning entities is changed to a different value for all (or subset of) planning entities. • Swap move selector – two planning entities are picked and all of their planning variables are swapped. • Pillar swap move selector – similar to the previous type. There are two groups of planning entities defined by the common value of planning variables. The values are changed between these groups. Different move selector types can be mixed together, forming hierar- chy of move selectors. The scope of a move selector can be restricted as well, to have a better control over the set of affected planning entities, or planning values respectively. Additionally, it is possible to influence the order in which values or entities are selected, or filtering to possibly avoid unwanted changes in the state of a problem. 3.2.1 Solution evaluation When having a set of possible moves at one’s disposal, it is important to make a decision about the winning move — step, which will be applied at given algorithm iteration. The decision depends on the evaluation of a solution, an essential part of solving process. The evaluation is performed by score director and offers a way of qualitative comparison among different solutions. It helps to determine the best solution after making a step — when solver comes across a solution, which evaluation is better than the best previous solution, it is stored for further access. 12 3. Configuration and basic usage In OptaPlanner world, the evaluation is kept in dedicated object named score. Based on the nature of a problem, one of the available score types can be selected prior to solving. They can be divided into two groups, either depending on the granularity of score type (number of values it is possible to contain in a single score object), or depending on an internal representation of the values contained in a score object. As for the first group, for some problems simple score is sufficient — the one wrapping single numeric value, as breaking any of constraints result in problem requirements not being met (e.g. n queens). On the other hand, sometimes it is advantageous to define several score levels to offer separate treatment to constraints, which ultimately need to be met (e.g. only one lesson may take place at given classroom at the same time) and constraints, which fulfillment is plausible, but not necessary (e.g. prefer classrooms which capacity matches number of students in the lesson). Internal score representation type can possibly have impact on the precision of the outcome. The value of score expresses evaluation of the current state of the problem based on number of constraints that are broken and their respective weight. Negative value indicates some of the constraints are broken. For example, HardSoftScore of 0/- 356 indicates all hard constraints are matched, but some of the soft constraints still remain broken. Evaluation function can be defined in two ways. The first approach is to make use of Java programmatic configuration, which offers two variants (simple and incremental). The choice depends on the use case, in general incremental score calculation is faster, according to docu- mentation. It provides speed and scalability in exchange for complexity of implementation and maintainability of such function. Another way is to make use of Drools rule engine, providing custom implementation of Rete algorithm (Rete OO), biased towards object oriented systems. It offers an alternative way to Java-based score calculation, by providing its own declarative language, where constraints are specified in form of rules, which have the following structure: rule name when 13 3. Configuration and basic usage Listing 3.1: General rule structure Rule name identifies given rule, its name is required to be unique inside the package that a set of related rules reside in. Conditions (left hand side) define prerequisites, which need to be met in order to execute defined set of actions. The process of rule evaluation is done by inference engine, which performs pattern matching — checking rules, loaded into production memory against a set of problem facts, residing in working memory. OptaPlanner defines DroolsScoreDirector class, which serves as an integration point with Drools rule engine. It is responsible for orchestration of the process of score calculation out of working solution. It builds a special object, named score holder, making it available to rule engine as a global variable. Its purpose is to keep track of the current state of score. There are several score holder types and their choice depends on the type of score being used (e.g. HardSoftScoreHolder). The state of score holder usually changes in “actions” part of the rules as a result of unmatched constraint. Code sample 3.2 demonstrates usage of rule from a sample timetabling problem — number of students for a lecture should not exceed capacity of given room. Each additional students will result in soft part of the score to be decreased by 1. rule"roomCapacity" when $room : Room($capacity : capacity) $lecture : Lecture(room == $room, studentSize > $capacity , $studentSize : studentSize) then scoreHolder.addSoftConstraintMatch(kcontext, ($capacity - $studentSize)); end Listing 3.2: Rule sample Rule language offers a large variety of other options (e.g. imports, functions, queries), however their description is out of scope of this thesis. Further details can be found in Drools Expert reference manual [2]. 14 3. Configuration and basic usage 3.2.2 Termination Solver configuration includes termination element. This setting allows user to have control over termination of the solving process, as the amount of time needed to be invested in order to find the optimal solution can be undesirable, especially for problems with big search space. Additionally, optimization algorithms require stopping point to be defined. It can be defined globally on solver level, or locally on phase level (i.e. for individual algorithms). Termination options, which OptaPlanner offers, can be divided into four groups: • Time-based termination, which can be specified using different time units, depending on the user requirements (e.g. seconds, minutes, hours). One can set the solving process to terminate either when an absolute amount of time elapsed, or if the best score has not improved over a specified period of time. • Step count-based termination, which is based on the principle of score change iterations (steps). Solving can be terminated if a given number of iterations has been reached, or if the best score has not improved in a given number of steps. • Best score-based termination, triggered when a defined best score is achieved. In some use cases, it may be sufficient to stop the process when feasible score has been attained, with no hard con- straints broken. • Composed termination, which can be specified as a combination of the previous termination types, using logical AND or OR. The following example contains sample termination definitions: Listing 3.3: Sample termination settings 3.3 Running the solver When the configuration part is finished, the solving process can be started. The first step is to obtain a reference to Solver object. For 15 3. Configuration and basic usage this purpose, SolverFactory class can be utilized. It provides a set of methods, which enable user to specify the source of configuration file. BuildSolver method needs to be invoked afterwards to initialize the solver. SolverFactory solverFactory = SolverFactory. createFromXmlResource(String file); Solver solver = solverFactory.buildSolver(); Listing 3.4: Creating Solver object Programmatic configuration of the solver can be used as well, once SolverConfig reference is obtained. Again, the solver is constructed by invoking buildSolver method. SolverConfig solverConfig = solverFactory.getSolverConfig (); // configuration Solver solver = solverConfig.buildSolver(); Listing 3.5: Obtaining SolverConfig reference When having Solver reference at one’s disposal, the solving process can be initiated by invoking solve method, where solution object needs to be passed as a parameter. Based on termination settings, it will take certain amount of time to finish. GetBestSolution method provides a way, how to get the best solution, which the solver was able to calculate during given time period. Solution solution = new SolutionClass; // (e.g. resource allocation problem) // initialize domain classes solver.solve(solution); SolutionClass bestSolution = (SolutionClass) solver. getBestSolution(); Listing 3.6: Running the Solver 16 4 Supported algorithms OptaPlanner implements a variety of optimization algorithms, which can be classified in the following way: 4.1 Local search algorithms These algorithms belong to the group of algorithms of improvement type (iterative repair). They start with already initialized solution and try to find find better one by making adjustments in the current so- lution. The neighboring solutions are evaluated and one of them is accepted, based on acceptance-rejection criterion defined by implemen- tation of given algorithm. There are several components OptaPlanner uses to configure the search process: • Move selector – already described in the chapter Configuration and basic usage. Its responsibility is to create a set of candidate moves. • Acceptor – the component defining acceptance-rejection criteria. It defines the type of algorithm to be used for move acceptance from the set of candidate moves (e.g. Tabu search, Simulated annealing). • Forager – it is responsible for selecting the next step from the set of accepted moves. In addition, termination needs to be set for local search algorithms to stop, when given criteria are met. 4.1.1 Hill climbing Hill climbing is one of the most basic local search algorithms. Its logic is simple — it starts with an initialized solution (focal point). In each iteration, it examines all adjacent solution (defined by the neighborhood of the current solution) and evaluates them using evaluation (scoring) function. The solution with the highest score is picked to become a new focal point. If there are multiple solutions with the highest score, one of them is chosen nondeterministically. The problem with the algorithm 17 4. Supported algorithms is that it can get stuck at local optimum (a point, where no adjacent points, which improve evaluation can be found), or plateau (a set of adjacent points having the same evaluation), unless a heuristic is used while picking the next focal points (e.g. a degree of randomness), or algorithm restart is performed. This may possibly lead to inability to find the best solution (global maximum). 4.1.2 Simulated annealing The algorithm was originally developed as a model for describing an- nealing process of metals heated to very high temperature and then gradually cooling down. The logic of such process is utilized and it addresses the problem of getting stuck at local optimum by permit- ting to pick even worse-evaluated solutions with decreasing probability when the move acceptance takes place. A random value is generated from interval (0, 1), which is assigned to candidate moves from the neighborhood during each iteration. Provided there is a solution with better evaluation in the neighborhood compared to the current one, it is selected. On the other hand, if no such solution is found, one of the available solutions with worse evaluation is picked. The selection pro- cess is based on comparison of generated value, assigned to every move, and the result of the Metropolis criterion 5. If the value belongs to in- terval (0, function result), the move is selected, otherwise the algorithm proceeds with the next move. As it was mentioned, in the early stages of the solving process (when the temperature is high), the probability of picking sub-optimal move is higher than in later stages. Definition 5 (Metropolis criterion). Metropolis criterion is defined as U < e−∆E/T , where • U ∈ (0, 1), • ∆E = eval(current solution) − eval(candidate solution), • T is the current temperature. 4.1.3 Tabu search Another algorithm with the ability to avoid getting stuck at local op- timum is Tabu search. It achieves this by maintaining a list of last n 18 4. Supported algorithms procedures/affected objects, which are avoided when picking the next solution to continue with. The type of objects to be checked and the size of the list is configurable. For example, entities or values included in one of the previous steps can be used. In every iteration, affected objects are added to the list, and the ones at the end of the list are removed, provided the capacity of the list is exceeded. The size of the list needs to be defined carefully – when it is too small, the problem can get stuck in local optimum. On the other hand, number of visited solu- tions grow with the increase of the list size [25]. Compared to simulated annealing, the decision process has more of deterministic nature. 4.1.4 Late acceptance The main ida of Late acceptance algorithm lies in delaying the compar- ison between neighbourhood solutions by n steps [16]. This is achieved by keeping a list of evaluations, that were current in the last n steps. In every iteration, the algorithm compares current evaluation to the one, that was actual n steps ago. 4.2 Construction heuristics Algorithms listed in this section are also denoted as algorithms of con- structive type. Their purpose is to build up an initial solution from scratch, by gradually assigning values of planning variables of individ- ual planning entities. They do not have to provide feasible solution, as their result often serve only as a base for further improvement of the solution by iterative repair algorithms. OptaPlanner terminates them automatically. 4.2.1 First fit First fit algorithm provides a straightforward approach to solution ini- tialization – it performs assignment of planning variables to planning entities on the first available ocasion, i.e. when the assignment breaks no constraints. If no partial solution fitting all constraints can be found in the set of candidate solutions, then one with the best evaluation is selected. A variation of First fit algorithm, that takes difficulty of assignment of planning entity into account is available – First fit de- 19 4. Supported algorithms creasing. In that case, planning entity difficulty comparator needs to be implemented. 4.2.2 Weakest fit Planning variables can be compared based on their strength — metrics which designates capability of a variable to satisfy a planning entity. During solution initialization, weakest fit algorithm always picks the weakest variable. Implementation of planning value strength compara- tor is required. Its modification – Weakest fit Decreasing is a combina- tion of two algorithms: Weakest fit and First fit decreasing. 4.2.3 Allocate entity from Queue This algorithm provides a generic form of the aforementioned construc- tion heuristics. It allows more detailed control over the ordering of entities/variables, which are put into queue and gradually initialized, based on the ordering criteria. 4.2.4 Cheapest insertion Cheapest insertion always attempts to find assignment of an entity and variable, which breaks the lowest number constraints. On the one hand, it can produce high-quality initial solution. On the other hand, scalability of such algorithm with increasing problem size can raise concerns about justification of its usage. 4.3 Other OptaPlanner also supports exhaustive search algorithms, which offer a guarantee to find the optimal solution. However, implementations (Brute Force, Branch and Bound) suffer from scalability issues, when compared to local search algorithms. Other approaches like Hyper- heuristics, or Evolutionary algorithms are planned for future releases. 20 5 Benchmarking When choosing optimal solving strategy for individual problems, it is essential to have a feature, which allows for comparison of used ap- proaches, based on different point of views. As different search algo- rithms can be suitable to solving different kind of problems, there is no silver bullet for solver configuration and the right settings usually need to be determined experimentally. For this purpose, OptaPlanner offers benchmarking functionality. The idea is simple. Several optimization problems are defined on the input of the benchmarker, alongside with multiple solver configurations. Solving process is afterwards executed for each combination of a problem and a solver (run). The difference between an ordinary solver execution and benchmarker execution lies in production of additional information, which makes the comparison easier. This can, for example, include information about memory con- sumption or the evolution of score over time. Once the benchmarking process finishes, the results are processed and presented in a summary HTML report. The statistics are recorded in form of CSV files, so that further user handling is possible. Based on the report document, it is possible to find the most appropriate configuration for solving specific problem. In the early stages of benchmarking, it may be appropriate to identify the best-performing optimization algorithm, afterwards the benchmarker can be used for more thoughtful configuration tweaking of the picked candidate, or modifying the optimization problem itself, as performance bottleneck is not always necessarily configuration issue. Optimization problem-related issues may arise as well (e.g. poor imple- mentation of evaluation function, using inappropriate structures and data types in the model, etc.). 5.1 Report document Generated HTML document provides an overview of results in a well- organized form. Displayed charts are automatically generated using JFreeChart library [8]. Knowledge, provided by the benchmarker is in- valuable when choosing one solving approach over the other one, or trying to understand nuances of using different algorithms for different problems. The following division briefly describes the main sections of 21 5. Benchmarking the document, and provides several report examples in addition. • Result summary section contains the most important informa- tion – score charts. Right algorithm selection and related deci- sion process is usually heavily influenced by the resulting (best) score an algorithm is able to achieve in a given amount of time. For example, default benchmarker configurations for individual example problems included in OptaPlanner distribution usually mix construction heuristic and metaheuristic algorithms – by benchmarking them on a specific problem, it is obvious that con- struction heuristic algorithms (e.g. First Fit) get outperformed, as metaheuristic algorithms use them as a starting point of a problem optimization. Best score chart 5.1 provides an overview of the best score achieved by different solving algorithms on dif- ferent problems. The results are grouped by a specific problem, the same algorithms use the same colour. Best score scalabil- ity chart shows the influence of problem size on the best score. This may be useful for ruling out incorrect solving approaches for given problem to guarantee scalability of the calculation, as related issues may not occur on small datasets. • Performance summary area contains performance-related charts, which help user to tune solver configuration or fix problem im- plementation in order to improve aspects like solving duration, or number of operations processed per unit of time. • Problem benchmarks section contains problem statistics — a dif- ferent view on the recorded data, achieved by grouping statistics for individual solvers by a common problem dataset. They pro- vide information such as score evolution over time 5.2, or mem- ory consumption. As they may potentially have adverse effect on the performance of the solving process, they are disabled by default. • Solver benchmarks section contains XML configuration of indi- vidual solvers that were used. Useful for reusing the best settings in a different configuration. • Benchmark information provides details about benchmarking environment, the results were created in. Information about op- 22 5. Benchmarking Figure 5.1: Time spent summary chart erating system, Java version, or number of benchmarks run in parallel are crucial in order to guarantee comparability of the results. 5.2 Configuration Configuration procedure is very similar compared to the one, used for ordinary solver, discussed in chapter Configuration and basic usage. In addition, it allows for multiple solvers and planning problems to be de- fined. Apart from the list of problem and solver benchmarks a sample benchmarker configuration may include an output directory, where the generated results will be written, or warm-up period duration, which precedes each test. During this period, JIT optimizations can be per- formed, so that the results are not adversely affected. It is suggested to run a warmup for a period of at least 30 seconds. 23 5. Benchmarking Figure 5.2: Evolution of the best score over time 24 6 Result persistence and recovery As mentioned before, the main functionality of benchmarker module is to compare performance of defined solvers over one or more plan- ning problems. This obviously leads to number of solvers * number of planning problems unique benchmarking runs, that the main process composes of. One of the problems, which aforementioned logic may potentially experience, is unexpected termination of the whole bench- marking process. Given the fact the results are processed only after each of the benchmarking tasks finish, termination of the process leads to inevitable loss of all previously obtained results, as they are only in- memory stored. The problem can lead to bad user experience mainly in situations, when the bencmarking process takes non-trivial amount of time (hours, days) to finish. This leads to the need to define mechanism, which would allow for storing the results of individual benchmarking runs every time any of them finishe their job. The solution brings two main benefits. • If sub-results are persisted, the state of benchmarking process can be restored, if an error occurs in the previous process. Bench- marker should automatically detect such failure and initialize the process to such state, that the previously obtained sub-results are used without the need to calculate them once again. Per- sistence of the results will not affect the performance of other running benchmarks and the state of persisted data will be con- sistent. • If the results are persisted in a consistent way, it would be pos- sible to form a virtual repository of all benchmarking runs. This could be particularly useful to perform comparison of the results, generated by different benchmarker processes, which is not pos- sible at this moment. Each of the two aforementioned fields will be depicted thoroughly in the following chapters, starting with result persistence and recovery, as it has definite impact on the second task (result comparison using the means of their aggregation). 25 6. Result persistence and recovery 6.1 Result persistence To be able to understand steps that were required to store results of in- dividual benchmarking runs better, it is beneficial to review old bench- marker domain model 6.1 at first. Newly created classes are highlighted. Figure 6.1: Benchmarker domain model The main component of the benchmarker domain model hierarchy is PlannerBenchmark interface, defining bencmark() method, which starts the benchmarking process. It holds references to all sub-benchmarks (runs) that the main process consists of. Internally, the runs are repre- sented by SingleBenchmark class implementing Callable interface. This pattern enables the runs to be invoked in an asynchronous fashion, each of them in a separate thread. The logic of SingleBenchmark’s call method is simple – start solving process of a single planning prob- lem using defined solver configuration. PlannerBenchmark component is responsible for invoking individual SingleBenchmark callables. It de- fines ExecutorService, backed up by Java FixedThreadPool, which sub- mits the jobs and waits until all of them finish. In order to fully uti- lize resources provided by multi-core machines, the executor can be tuned to achieve concurrent processing of several benchmarking runs by passing numberOfThreads parameter. Both ProblemBenchmark and SolverBenchmark classes provide different views of the hierarchy, with 26 6. Result persistence and recovery ProblemBenchmark comprising all SingleBenchmarks for a particular planning problem. SolverBenchmark class works in a similar fashion – it incorporates all SingleBenchmarks for given optimization algorithm. PlannerBenchmark, ProblemBenchmark and SolverBenchmark objects are configured and initialized via helper classes – PlannerBenchmark- Config, ProblemBenchmarkConfig and SolverBenchmarkConfig. Their purpose is to parse XML configuration on the input and build the hierarchy with all parameters set properly. Another vital part of the benchmarker are statistics. Their purpose is to capture additional in- formation during solving process, that help to assess the final results of benchmarking process better. They are represented by SingleStatistic, or ProblemStatistic class. At the time of implementation, the following statistics were available: • BestScore • BestSolutionMutation • CalculateCount • ImprovingStepPercentage • MemoryUse • MoveCountPerStep • StepScore The results are stored in CSV (Comma-separated value) format files. For each combination of SingleBenchmark and defined statistics type, one object is created. Storing task required a rapid change in the object model of Opta- planner. The fact that the main result of the solving process is encapsu- lated in SingleBenchmark instances lead to the major part of attention being paid on this class (attributes of SolverBenchmark and Problem- Benchmark related to the result of solving process are derived from SingleBenchmark — e.g. winning single benchmark, average score). At first, attributes which values change during the course of computa- tion had to be located and afterwards moved to a separate class. Sin- gleBenchmarkResult was created for this purpose. The class itself does 27 6. Result persistence and recovery not contain any computational logic, it is solely dedicated to encap- sulate aforementioned attributes, thus it can be denoted as a POJO. The process of introducing SingleBenchmarkResult class was simple — all affected getters and setters of SingleBenchmark class were just redirected to it. The storing process is performed by the means of XStream library [15], which provides a convenient way of transferring object to XML representation and vice versa. Standard usage does not require any additional configuration, the usage is pretty straightforward. No getters nor default constructors are required. class SingleBenchmarkResult { private String id; private long duration; } Xstream xstream = new Xstream(); SingleBenchmarkResult result = new SingleBenchmarkResult( "sample", 42); String xml = xstream.toXML(result); // XML output Listing 6.1: XStream sample usage Constructing an object from XML file is simple as well. SingleBenchmarkResult result = (SingleBenchmarkResult) xstream.fromXML(xml); Listing 6.2: Unmarshalling object from XML file If needed, it is possible to define aliases for classes and fields either using programmatic configuration, or using any of available annota- tions. Xstream allows user to define custom converters, so that the marshalling/unmarshalling process can be controlled and modified if needed. XstreamResumeIO class was created in order to be able to read and write individual SingleBenchmark instances. One of the requirements was that the persistence of the results 28 6. Result persistence and recovery would not affect the performance of OptaPlanner adversely. There- fore the results are stored only after each run completes the job — as it is done in a separate thread, it does not affect other processes in progress. The following code example taken from DefaultPlannerbench- mar.runSingleBenchmarks() shows the critical piece of code. xStreamResumeIO.write(singleBenchmark. getSingleBenchmarkState(), new File(benchmarkOutputDirectory.getPath(), singleBenchmark.getName() +".xml")); Listing 6.3: SingleBenchmarkState persistence The name of SingleBenchmark is derived from by planning problem it represents an optimization algorithm it uses. public String getName() { return problemBenchmark.getName() +"_"+ solverBenchmark.getName(); } Listing 6.4: Resolving name of a SingleBenchmark In addition to persistence of individual SingleBenchmarkResults, it is also useful to store configuration of the benchmarker, in relation to re- covery functionality. The details will be described later in this chapter. The configuration is written in form of resumeBenchmarkConfig.xml file, a copy of input configuration file. The processing of statistics required a different approach. Compared to SingleBenchmarkResult, the amount of data carried in statistics is much higher, as the data is recorded over a certain period of time. Obviously, if the benchmarking run takes long time to finish, storing the data in form of XML could result in large files, with processing being excessively resource-consuming. For this reason comma-separated value (CSV) format was chosen as a storage format, as it allows to store a large data set in an efficient way, compared to XML [21]. for (StatisticType statisticType : singleBenchmark. getSingleStatisticMap().keySet()) { File statisticFile = new File(benchmarkOutputDirectory. getPath(), singleBenchmark.getSingleBenchmarkStatisticFilename( statisticType)); 29 6. Result persistence and recovery singleBenchmark.getSingleStatisticMap().get( statisticType).writeCsvStatistic(statisticFile); } Listing 6.5: Persistence of statistics In this case, the name of particular statistics is derived from the name of SingleBenchmark and the type it represents. public String getSingleBenchmarkStatisticFilename( StatisticType type) { return getName() +"_" + type +".csv"; } Listing 6.6: Resolving name of a statistic WriteCsvStatistics(File outputFile) method was introduced in each of SingleStatistic implementation in order to be able to persist its con- tent. The following example was taken from BestScoreSingleStatistic. It works in a similar fashion for all other statistic types too. public void writeCsvStatistic(File outputFile) { SingleStatisticCsv csv = new SingleStatisticCsv(); for (BestScoreSingleStatisticPoint point : pointList) { long timeMillisSpend = point.getTimeMillisSpend(); Score score = point.getScore(); if (score != null){ csv.addPoint(timeMillisSpend, score.toString()); } } csv.writeCsvSingleStatisticFile(outputFile); } Listing 6.7: Statistic persistence detail SingleStatisticCsv class was created in order to wrap point-value pairs resulting from statistics. It defines method writeCsvSingleStatis- ticFile(File outputFile), which iterates through all the pairs and writes them in the corresponding CSV file. 6.2 Resume functionality Another idea connected with result persistence was the ability to re- run the benchmark after it had been aborted due to a failure. With the 30 6. Result persistence and recovery storing functionality implemented, the task should be rather straight- forward: make use of generated XML files for SingleBenchmarkResult and CSV files encapsulating statistics in order to initialize benchmarker object hierarchy in such way, that the results that were already ob- tained in the previous run would be used and running corresponding SingleBenchmarks would not be necessary. The mechanism was im- plemented in the following way: Provided each benchmark generates HTML report with the summary of results after being successfully fin- ished, with index.html page as a part of the report and resumeBench- markConfig.xml as the result of data persistence, it is possible to as- sume whether the previous benchmarking process ended up successfully or not, based on the presence of both files in the folder of the latest benchmark report. Resolving of the aforementioned folder is based on timestamp included in its name — the latest one can be determined. Additionally, the user is able to control the resume process by passing forceNoResume flag to benchmarker configuration. If set to true, the re- port directory will not be scanned for unfinished benchmark processes and all SingleBenchmarks will be run from scratch. By default, the recovery process takes place automatically, without user interaction. The following code snippet shows the way individual SingleBench- marks are recovered. if (latestResumeBenchmarkDir != null){ File resumeBenchmarkDirContent = new File( latestResumeBenchmarkDir,"resume"); File singleBenchmarkStateFile = new File( resumeBenchmarkDirContent, singleBenchmark.getName() +".xml"); if (singleBenchmarkStateFile.exists()) { SingleBenchmarkState singleBenchmarkState = problemBenchmark.getPlannerBenchmark(). getxStreamResumeIO().read(singleBenchmarkStateFile ); singleBenchmark.setSingleBenchmarkState( singleBenchmarkState); if (Boolean.TRUE.equals(singleBenchmarkState. getSucceeded())) { for (ProblemStatistic problemStatistic : problemBenchmark.getProblemStatisticList()) { File statisticFile = new File( resumeBenchmarkDirContent, singleBenchmark. 31 6. Result persistence and recovery getSingleBenchmarkStatisticFilename( problemStatistic.getProblemStatisticType())); if (!statisticFile.exists()) { throw new IllegalArgumentException("Statistic file(" + statisticFile.getName() +") for singleBenchmark" +"(" + singleBenchmark.getName() +") has not been found in resume directory."); } SingleStatistic singleStatistic = problemStatistic.readSingleStatistic( statisticFile, singleBenchmark.getSolverBenchmark ().getSolverConfig(). getScoreDirectorFactoryConfig()); singleBenchmark.getSingleStatisticMap().put( problemStatistic.getProblemStatisticType(), singleStatistic); } } singleBenchmark.setRecovered(true); } } Listing 6.8: SingleBenchmark recovery At first, latestResumeBenchmarkDir parameter is checked. If the value is set, it basically means the current benchmark is resuming the previous one, as the conditions described before had been met. There- fore benchmark hierarchy initialization will take place. Another check comes up with singleBenchmarkStateFile.exists() condition. If the file exists, it implies SingleBenchmarkResult was persisted in the latest unsuccessful benchmarking process, and should be recovered. State of SingleBenchmark comprise statistics as well, which are loaded if the run finished with success flag set to true (statistics are not stored for un- successful SingleBenchmarks). Another important thing to note is that recovered flag is set to true for resumed SingleBenchmarks. This flag is used by resuming logic to determine, whether corresponding bench- marking run should be performed or not. Resume check is performed in DefaultPlannerBenchmark.runSingleBenchmark() method, as shown below. if (!singleBenchmark.getRecovered()) { Future 32 6. Result persistence and recovery (singleBenchmark); futureMap.put(singleBenchmark, future); } Listing 6.9: Initiating benchmark execution As mentioned before, SingleBenchmark callable will not be exe- cuted, if its recovered flag is set to true. 6.2.1 Usability issues Even though resume functionality works without problems in majority of use cases, real usage showed there are some aspects, not considered during problem analysis, which can make its usage problematic. Re- strictions of its usage relate to configuration files, required to be spec- ified for the benchmarker in order to be run. The main problem arises in the following scenario. If • Based on the experience, the community (namely Geoffrey de Smet) proposed implementation of aggregation functionality, which would supplement concurrent resume mechanism by being able to merge results from different benchmarker runs. Its detailed description is included in the following chapter. • Usage transparency should be of the highest priority. 33 7 Result aggregation Consider having repository of all benchmarking runs performed. It is beneficial to be able to compare different runs with different configura- tions without the need of running them in a common benchmark. For this specific use case, aggregator application was implemented, while the storage of results from the previous part was utilized. Aggregator searches for all benchmark runs in a specific benchmarker output folder and allows for selection of SingleBenchmarks to be aggregated in a sin- gle output. The application was developed in close cooperation with OptaPlanner community1, who provided aggregation logic, and several other features, which will be listed further in the chapter (icons, ex- ception handling). Graphical user interface was developed as a part of this thesis. As the implementation part consisted of two steps (result persistence and aggregation), the process was performed in two rela- tively independent iterations. The domain model changed slightly since the first part was integrated into OptaPlanner repository2. Four core classes were renamed (SingleBenchmarkResult, SolverBenchmarkRe- sult, ProblemBenchmarkResult, PlannerBenchmarkResult). After the refactoring, the results were stored as a part of one file plannerBench- markResult.xml, instead of a single file per each SingleBenchmark. As the concept of benchmarker remained the same, old terminology will be used from this point on. Aggregator’s internal logic comprises two steps. In the first step, a new merged result (PlannerBenchmark object) is constructed out of SingleBenchmark object list by BenchmarkAggregator.aggregate() method. It is important to describe the rules, that SingleBenchmark results are grouped by. ProblemBenchmark objects are unified across all PlannerBenchmark runs. That means, if any SingleBenchmark re- sults are experienced, belonging to the same problem view, only one ProblemBenchmark instance is created. The situation is different in case of SolverBenchmarks, which are considered unique across different PlannerBenchmarks. To distinguish SolverBenchmarks with the same name belonging to different PlannerBenchmark runs, a name of cor- responding PlannerBenchmark is appended to the SolverBenchmark 1. http://www.optaplanner.org/community/team.html 2. https://github.com/droolsjbpm/optaplanner/ 34 7. Result aggregation name. Aggregator was written as a standalone Java application with Swing- based graphical user interface. Swing is a standard library for creating graphical user interfaces in Java. It originated as an alternative to its sibling technology AWT. Compared to AWT, it offers better reliability, performance and broader set of lightweight components without bind- ing to native counterparts provided by underlying operating system[27]. The last of the aforementioned properties ensures similar look over dif- ferent platforms. It offers a rich variety of components (e.g. buttons, windows, menus) which are organized into component tree with Compo- nent class residing at the top of the hierarchy. Swing components follow MVC (Model-View-Controller) design, which offers separation of their user interface (presentation layer) and data model. This implies each part can be implemented without the need to modify the other, which was successfully utilized during aggregator implementation. Compo- nent design can be modified at the runtime by setting appropriate look and feel (theme). One of the notable Swing features is event handling, which is closely connected to listener mechanism, implementing the Ob- server design pattern. A component, supporting the mechanism, allows for registration of listeners — under predefined condition (e.g. mouse click), an event is triggered, leading to notification of all registered lis- teners, which can react accordingly (e.g. update their state). The next field that deserves attention are layout managers. They are responsible for automatic positioning of components, eliminating the need to cal- culate their sizes and placement, which can be dependent on the used platform, look and feel, or simple window sizing. By default, Swing of- fers a large variety of layouts (e.g. border layout, box layout). Despite the fact, there are several libraries available, which may possibly ease the development process of Swing application (e.g. JGoodies), a deci- sion was made to use standard Swing toolkit exclusively, in order not to extend project dependency list. In order to keep source code readability and maintainability, no GUI designing tools (e.g. Netbeans Swing GUI Builder), which automatically generate code, based on visual template inserted by user, were used. Class diagram 7.2 shows the main skeleton of aggregator application. Respective parts will be used and described, as needed. 35 7. Result aggregation Figure 7.1: Basic aggregator object model 7.1 User interface The aggregator window consists of three main parts 7.2: 1. Benchmark selection area 2. Menu bar 3. Information panel 7.1.1 Benchmark selection area This is the core part of the graphical interface, containing enumer- ation of all available benchmark invocations organized into tree-like structure. On the top level, planner benchmarks are located. Each of them correspond to main benchmarking process invoked by Planner- Benchmark.benchmark() method. As it was already mentioned in the previous chapter, a single benchmarking process usually consists of sev- eral runs (single benchmarks). Every displayed item thus corresponds to a folder containing results of benchmarks located in a specified out- put folder (usually by a given planning problem). By default, they are identified by a timestamp of their generation. Second level is composed of problem benchmarks, or individual planning problem data sets, in other words. The third level contains solver benchmarks — different 36 7. Result aggregation Figure 7.2: Aggregator window parts solving approaches to tackle aforementioned planning problems. The last level contains the basic units — single benchmarking runs, iden- tified by combination of a solver and a problem. Each row contains an item with checkbox next to it, which allows multiple selection of desired SingleBenchmarks. Clicking on a checkbox, which has children present in the hierarchy allows to select/unselect all items in the sub- tree. This way, it is possible to select all runs for a particular planner benchmark/problem benchmark/solver benchmark just by one click. The color of the checkbox indicates the state of selection in the sub- tree. There are three states the checkbox can reside in: • Checked – if the item is non-leaf, then all children in the subtree defined by this item are selected. Otherwise the item represents a single benchmark and is selected • Unchecked – if the item is non-leaf, then all children in the sub- tree defined by this item are unselected. Otherwise the item rep- resents a single benchmark and is unselected • Mixed – if the item is non-leaf, then some children in the subtree 37 7. Result aggregation defined by this item are selected. It is not possible for leaf item to be switched to this state This colour scheme helps user to find information about the se- lection of the items, even if the nodes are collapsed. This required custom implementation of JCheckBox class, which was named Mixed- CheckBox. This was motivated by the fact, that standard Swing com- ponent set provides only two-state checkbox. MixedCheckBox class defines custom checkbox model, named MixedCheckBoxModel, which achieves three-state behavior by appropriate combination of its select- ed/armed/pressed attributes. Additionally, every checkbox is bound to exactly one out of four available benchmark result object types. Its label is derived from getName method, which all of the objects provide. Labels of SingleBenchmark and SolverBenchmark nodes ad- ditionally contain ranking within the PlannerBenchmark they belong to. The object binding is useful in order to determine, which objects have been selected, or eventually to decide whether an action is en- abled for given object type (e.g. renaming functionality is enabled only for PlannerBenchmark and SolverBenchmark instances). The checkbox tree is backed up by CheckBoxTree class, extending standard Swing JTree class. It keeps track of all selected nodes. Whenever a change is recorded, internal structure holding the selections is updated. It also contains logic to resolve new checkbox state, to select/unselect items in the subtree, or to collapse/expand all nodes, which will be described in the menu bar section. 7.1.2 Menu bar This bar contains several functions that enable the user to control the application. • Move up/down – after the selection of an item in the tree, it can be moved up/down in the list. This is particularly useful for ordering the benchmark runs in the way user wants it to be displayed in HTML report, as the order is maintained by processing logic. Aggregator also supports moving upper-level nodes, while selections that were previously performed in the subtree are maintained. 38 7. Result aggregation • Rename – this function allows for renaming given node (available only for PlannerBenchmark and SolverBenchmark nodes). • Switch levels – in order to ease selection process, depending on the requirements of the user, it is beneficial to be able to switch SolverBenchmark and ProblemBenchmark layers, so that the re- sults displayed in the tree are grouped by one, or the other. Previous selections are maintained. • Expand all/collapse all – this function either expands or collapses all nodes, including the children. • Generate report – core function of the application. After pressing the button, aggregate() method is invoked on the service layer (BenchmarkAggregator class), which merges selected benchmarks runs and prepares HTML report. The action is performed in a separate thread in order to keep the user interface respon- sive. Swing provides a standard mechanism for running asyn- chronous actions – SwingWorker class. This way, Event Dispatch Thread, responsible for event handling in Swing applications, is not blocked by an ongoing action, which improves user experi- ence. Status indicator shows the aggregation is performed at the moment. After the report generation is finished, user is notified about the outcome by a pop-up window 7.3 containing three options. – Show in browser – opens default system browser and dis- plays the report. – Show in files – opens the folder, where the generated report is located. – Close – closes the current window. Is close application check- box is selected, the application exits. 7.1.3 Information panel This area displays detailed information about individual items in the tree that are selected (by clicking on any item). Each level provides different information. 39 7. Result aggregation Figure 7.3: Report generation finished Planner benchmark: • Average score – global average score of all results. • Average problem scale – approximation of global problem in- stance size. Solver benchmark: • Average score – solver-scoped average score. • Total score – sum of result scores. • Average time spent – average duration of a benchmarking run. • Total winning score difference – sum of differences in scores from the best score achieved for a problem instance . Problem benchmark: • Entity count – number of planning entities for a problem. 40 7. Result aggregation • Problem scale – approximation of problem instance size for all sub-results. • Used memory – average of used memory size by loading input solutions for a problem. Single benchmark: • Score – the best score achieved by the run. • Used memory – memory used by loading a single solution. • Time spent – duration of the run. The information remains visible as long as corresponding item is selected 7.4. Figure 7.4: Information panel - selected item 7.1.4 Exception handling Aggregator contains mechanism to handle unexpected exceptions thrown during the run. The behavior is achieved by registering custom Un- 41 7. Result aggregation caughtExceptionHandler on the main application thread. By imple- menting uncaughtException() method, custom reaction to unhandled exception can be defined. In the context of aggregator, any such ex- ception results in error window being displayed containing the stack trace of the exception, so that the user gets informed about its cause, avoiding ungraceful application exit requiring its restart. 7.1.5 Real world usage As it was mentioned earlier, aggregator application offers a way, how to merge several independent benchmarker results into one. Without newly introduced functionality, result comparison is limited only to the scope of a single benchmarker invocation. This is very beneficial for comparing the results performed in different environments. One can, for example, perform benchmarking of the same problem on different platforms, or Java versions, collect the results, and merge them to create preview in form of summary document. For illustration, performance comparison of different Java runtime versions was performed using ag- gregator application, and published on official OptaPlanner website[6]. There are other use cases, where the application proves to be useful, like keeping track of the performance improvements over different Op- taPlanner versions, or just getting quick information about the results using information panel. 42 8 Usage demonstration The purpose of this chapter is to provide an overview of the outcome of this thesis, based on a sample scenario. 8.1 Problem description The problem named Machine reassignment was the main focus of Google ROADEF/EURO challenge 2011-2012 [5] and is one of OptaPlanner’s example problems. It was formulated in the following way: “The aim of this challenge is to improve the usage of a set of machines. A machine has several resources as for ex- ample RAM and CPU, and runs processes which consume these resources. Initially each process is assigned to a ma- chine. In order to improve machine usage, processes can be moved from one machine to another. Possible moves are limited by hard constraints, as for example resource capacity constraints, and have a cost. A solution to this problem is a new process-machine assignment which sa- tises all hard constraints and minimizes a given objective cost.” [4] 8.2 Scenario To demonstrate functionality of developed enhancements of the bench- marker (both result persistence and aggregation), the following steps will be taken: 1. Benchmarking of a sample Machine reassignment problem us- ing different Java runtime versions — this implies each of the benchmarking processes will have to be started separately, with corresponding Java version. Three data sets of different sizes will be used, as well as three different solver configurations, with best score problem statistics enabled. 2. Verifying, that all results got persisted (expected nine results per each planner benchmark as a result of running all combi- 43 8. Usage demonstration nations of defined solvers and problems - all included in plan- nerBenchmarkResult.xml), including one statistic file per each combination. 3. Merging the results, while utilizing aggregator application. The goal is to demonstrate usability of its functions. 8.3 Running the benchmarker The benchmarker was run with 3 different Oracle JDK versions in the following order (to demonstrate reordering functionality later): • 1.7.0_60-b19 • 1.8.0_25-b17 • 1.6.0_38-b05 Three different problem data sets were used (model name / problem size, as estimated by OptaPlanner): • model_b_1 / 500 000 • model_b_6 / 8 000 000 • model_b_10 / 250 000 000 000 Three different solver configurations were chosen: • Tabu search with entity tabu list size of 7 • Late acceptance with list size of 1000 • Late acceptance with list size of 2000 Each combination was preceded by 30 second warm-up. The test itself lasted 5 minutes, with at most 3 tests running concurrently. The test was performed on Intel R CoreTM i7-4900MQ, 16 GB of memory. Maximum JVM memory allocation limit was set using -Xmx4096m. To ease benchmarking process, a sample benchmarking application (Ma- chineReassignmentBenchmarkingApp), which comes as a part of Opta- Planner distribution, was utilized. 44 8. Usage demonstration 8.4 Data persistence verification The point of this step is to check, whether results got persisted dur- ing benchmarking process. Every time benchmarker is run, it creates a folder, identified by time stamp. The location of the folder can be specified using benchmarkDirectory attribute of benchmarker configu- ration file. Picture 8.1 shows the ouput of tree utility, which lists content of a directory in a tree structure. It shows the files, generated by the benchmarker, including both plannerBenchmarkResult.xml, holding the results of partial runs, and index.html (summary document). Figure 8.1: Tree view of the main output directory The statistics are stored in individual model folders. To demon- strate this fact, another tree output is included in picture 8.2. Due to verbosity of the output, it was performed only inside 2014-12-30- 213922/model_b_1 folder. 8.5 Result aggregation With all the results at one’s disposal, it is useful to pick a couple of them, to demonstrate ability of the aggregator to perform compari- son across independent benchmarker invocations. Running the applica- 45 8. Usage demonstration Figure 8.2: Tree view of selected problem directory tion is simple, as it can be directly started from MachineReassignment- BenchmarkingApp, described earlier in this chapter. The only require- ment is to pass --aggregator as a program argument. After launching the application, a basic window is displayed 8.3. Figure 8.3: Basic window of aggregator application It is useful to rename the benchmarks to be easily distinguishable. For this purpose rename function is used. Reordering function (move up) is applied twice to move JDK 6 benchmark to the top of the list. The result of the sequence is shown in picture 8.4. In the next step, Late Accteptance solver (lateAcceptance2000) is 46 8. Usage demonstration Figure 8.4: State of problem after renaming picked from all planner benchmarks, over all three data sets. Node signs can be used to expand the tree as needed. Generate report button is pressed, preparing summary document in the background. Once fin- ished, notification window is displayed. Exit application checkbox is unchecked, as the usage demonstration will continue with different sce- nario. Picture 8.5 shows an example aggregated chart named Average calculate count summary, clearly showing significant improvement of performed operations per second as later versions of JDK were used. Further description of results is out of scope of this thesis. Second example shows scenario, where one would like to examine only specific problem dataset. This can be potentially achieved in the same way like in the previous example. However, user would be forced to open each solver node and only after select given problem dataset, which is inconvenient mainly in cases with multiple solver configura- tions. For this reason, it is advantageous to use switch levels function, which reverses the selection order, such that problem nodes come first. The result is shown in picture 8.6. Again, the aggregation is initiated by pressing Generate report but- ton. Best score chart was selected from the document to show that all solver configurations got grouped by a single problem dataset 8.7. 47 8. Usage demonstration Figure 8.5: Average calculate count summary Figure 8.6: State of problem after level switching 48 8. Usage demonstration Figure 8.7: Aggregated best score chart 49 9 Conclusion The goal of the thesis was to implement improvements of OptaPlanner benchmarking functionality. Problems, concerning abrupt termination of benchmarking tasks, targeted by the implementation posed usabil- ity issues to users and were solved successfully. Real world usage shown that resume functionality prototype, based on the work done in the field of result persistence, did not cover all use cases regarding configuration, but motivated creation of aggregator functionality as result, which has been successfully used by OptaPlanner community. Apart from imple- mentation part of the thesis, description of OptaPlanner configuration and usage was presented, together with short introduction to the field of constraint satisfaction. 9.1 Future work There are several areas, especially in aggregator application, that could be improved in the future. Selection of single benchmarks should be maintained if user does decide to continue with aggregation after the report is generated. Cur- rently, the whole object hierarchy is reloaded from scratch, losing all previously performed selections. Improvements in provided information about individual objects in aggregator object tree need to be considered. Currently, there is no easy way to compare obtained information across several objects. Moreover, information is available only after an item is selected. More research is planned in this area. Reordering functionality only works in the scope defined by the subtree an item resides in. It may be useful to provide a tool, which would allow solver benchmarks to be reordered even if they belong to different planner benchmarks. For this reason a wizard could be introduced, which would split current selection process into two steps. The first step would not differ from the current implementation — individual single results could be selected, with aggregation initiation to follow. However a link to second step would be available, with a list containing all solver benchmarks that had been selected, with a possibility to reorder them. 50 A Source code locations A.1 Result persistence and recovery Source code: • https://github.com/mcimbora/optaplanner/tree/suspend_ resume_fixed GitHub pull request: • https://github.com/droolsjbpm/optaplanner/pull/ 33/files A.2 Aggregator GUI Source code: • https://github.com/mcimbora/optaplanner/tree/resume_ swing GitHub pull requests: • https://github.com/droolsjbpm/optaplanner/pull/ 36 • https://github.com/droolsjbpm/optaplanner/pull/ 38 • https://github.com/droolsjbpm/optaplanner/pull/ 39 • https://github.com/droolsjbpm/optaplanner/pull/ 40 • https://github.com/droolsjbpm/optaplanner/pull/ 41 • https://github.com/droolsjbpm/optaplanner/pull/ 42 51 A. Source code locations • https://github.com/droolsjbpm/optaplanner/pull/ 42 • https://github.com/droolsjbpm/optaplanner/pull/ 44 52 Bibliography [1] Choco, official page. http://choco-solver.org/. Accessed: 2014-12-15. [2] Drools documentation. http://docs.jboss.org/drools/ release/6.1.0.Final/drools-docs/html_single/ index.html. Accessed: 2014-11-13. [3] Gecode, official page. http://www.gecode.org/. Accessed: 2014-12-15. [4] Google ROADEF/EURO challenge 2011-2012, Machine reas- signment. http://challenge.roadef.org/2012/files/ problem_definition_v1.pdf. Accessed: 2014-12-09. [5] Google ROADEF/EURO challenge 2011-2012, Official page. http://challenge.roadef.org/2012/en/. Accessed: 2014-12-09. [6] How much faster is Java 8. http://www.optaplanner.org/ blog/2014/03/20/HowMuchFasterIsJava8.html. Ac- cessed: 2014-12-09. [7] IBM ILOG, official page. http://www-01.ibm.com/ software/info/ilog/. Accessed: 2014-12-15. [8] JFreeChart library. http://www.jfree.org/jfreechart/. Accessed: 2014-11-16. [9] JSR 331: Constraint Programming API. https://jcp.org/ en/jsr/detail?id=331. Accessed: 2014-12-15. [10] KIE Group, Official page. http://www.kiegroup.org/. Ac- cessed: 2014-11-13. [11] On-Line Guide to Constraint Programming. http: //ktilinux.ms.mff.cuni.cz/~bartak/constraints/. Accessed: 2014-11-21. 53 A. Source code locations [12] OptaPlanner, official page. http://www.optaplanner.org/. Accessed: 2014-12-15. [13] Optaplanner user guide. http://docs.jboss. org/optaplanner/release/6.1.0.Final/ optaplanner-docs/html_single/index.html. Ac- cessed: 2014-11-13. [14] SICStus, official page. https://sicstus.sics.se/. Ac- cessed: 2014-12-15. [15] XStream library. http://xstream.codehaus.org/. Ac- cessed: 2014-10-30. [16] S. Voß A. Goerler, F. Schulte. An Application of Late Acceptance Hill-Climbing to the Traveling Purchaser Problem. Computational Logistics, 8197:173–183, 2013. [17] R. Dechter. Constraint processing. Morgan Kaufmann Publishers, 2003. [18] T. Walsh F. Rossi, P. van Beek. Handbook of Constraint Program- ming. Elsevier, 2006. [19] J. H. M. Lee P. J. Stuckey H. Fang, Y. Kilani. Reducing Search Space in Local Search for Constraint Satisfaction. [20] P. Kenekayoro. Comparison of simulated annealing and hill climb- ing in the course timetabling problem. African Journal of Mathe- matics and Computer Science Research, 5(11):176–178, 2012. [21] R. Lawrence. The space efficiency of XML. Information and Soft- ware Technology, 46:753–759, 2004. [22] M. Pinedo. Planning and scheduling in manufacturing and ser- vices. Springer, 2005. [23] H. Rudová. Programování s omezujícími podmínkami, Lecture slides. http://is.muni.cz/el/1433/podzim2014/ PA163/um/prednaska.PDF?kod=PA163;predmet= 788667. Accessed: 2014-12-18. 54 A. Source code locations [24] Matyska L. Rudová H. Timetabling with Annotations. 1999. [25] R. Saravanan. Manufacturing Optimization through Intelligent Techniques. CRC Press, 2006. [26] E. Tsang. Foundations of Constraint Satisfaction. Academic Press Limited, 2006. [27] J. Zukowski. The Definitive Guide to Java Swing. Apress, 2005. 55
Usability Improvements of Optaplanner Benchmarker
