Soft Comput (2012) 16:527–561 DOI 10.1007/s00500-011-0754-8 ORIGINAL PAPER Metaheuristic optimization frameworks: a survey and benchmarking Jose´ Antonio Parejo • Antonio Ruiz-Corte´s • Sebastia´n Lozano • Pablo Fernandez Published online: 28 August 2011 Ó Springer-Verlag 2011 Abstract This paper performs an unprecedented com- and affordable time and cost. However, heuristics are parative study of Metaheuristic optimization frameworks. usually based on specific characteristics of the problem at As criteria for comparison a set of 271 features grouped in hand, which makes their design and development a com- 30 characteristics and 6 areas has been selected. These plex task. In order to solve this drawback, metaheuristics features include the different metaheuristic techniques appear as a significant advance (Glover 1977); they are covered, mechanisms for solution encoding, constraint problem-agnostic algorithms that can be adapted to incor- handling, neighborhood specification, hybridization, par- porate the problem-specific knowledge. Metaheuristics allel and distributed computation, software engineering have been remarkably developed in recent decades (Voß best practices, documentation and user interface, etc. A 2001), becoming popular and being applied to many metric has been defined for each feature so that the scores problems in diverse areas (Glover and Kochenberger 2002; obtained by a framework are averaged within each group of Back et al. 1997). However, when new are considered, features, leading to a final average score for each frame- metaheuristics should be implemented and tested, implying work. Out of 33 frameworks ten have been selected from costs and risks. the literature using well-defined filtering criteria, and the As a solution, object-oriented paradigm has become a results of the comparison are analyzed with the aim of successful mechanism used to ease the burden of applica- identifying improvement areas and gaps in specific tion development and particularly, on adapting a given frameworks and the whole set. Generally speaking, a sig- metaheuristic to the specific problem to solve. Based on nificant lack of support has been found for hyper-heuristics, this paradigm, there are a number of proposals which and parallel and distributed computing capabilities. It is jointly offer support for the most widespread techniques, also desirable to have a wider implementation of some platforms and languages. In this article, we coin these kind Software Engineering best practices. Finally, a wider sup- of approaches as metaheuristic optimization frameworks port for some metaheuristics and hybridization capabilities (MOFs). is needed. In addition to the advantages of having pre-implemented metaheuristics in terms of testing and reuse, using a MOF can provide a valuable benefit. They support the evaluation 1 Introduction and motivation and comparison of different metaheuristics to select the best performing one for the problem at hand. Heuristic methods have proven to be a comprehensive tool However, as the number of alternatives is extensive (we to solve hard optimization problems; they bring a balance have identified 33 different MOFs in literature) this of ‘‘good’’ solutions (relatively close to global optimum) becomes a double-edged sword and the choice of the right MOF results in a major issue. Due to the wide number of metaheuristics (and variants), each of the MOFs is focused on a particular subset; in this context, not choosing the J. A. Parejo (&) Á A. Ruiz-Corte´s Á S. Lozano Á P. Fernandez University of Sevilla, Seville, Spain right MOF leads to a no-win situation; this would imply e-mail: [email protected] further costs due to the change from one MOF to another, 123 528 J. A. Parejo et al. or the risk of obtaining a sub-optimal solution due to the new versions). Moreover, the possibility of downloading use of inappropriate metaheuristics. the benchmark as a spreadsheet and tailoring it to user A comparative framework is a useful tool to guide a needs by modifying its weights is also crucial for making it selection of the MOF that best suits a particular scenario. more relevant and applicable. However comparisons of frameworks in literature are The remainder of this article is organized as follows: either informal evaluations using author criteria or focused Section 2 defines what a metaheuristic optimization on performance (Wilson et al. 2004). Gagne` and Parizeau framework is and outlines the advantages and disadvan- (2006) present a comparison (over 6 features) of MOFs tages of using such tools. Next, Sect. 3 describes the supporting evolutionary algorithms. Voß (2002) presents a methodology used to create our comparative framework constructive discussion of various software libraries, but divided into six areas. In further sections, each area is there is a lack of a comparative analysis. Alternatively, developed in detail (Sects. 4 to 9), defining a set of char- some articles (such as Cahon et al. 2004; Di Gaspero and acteristics, its importance, metrics and data sources used Schaerf 2003) presenting a concrete MOF, include a related for its evaluation. In each section, charts and interesting work section with a comparison of specific features with results on the current support by the selected MOFs are other MOFs; however, those works present a narrow per- provided. In Sect. 10 we discuss the results obtained from a spective with a comparison of a reduced set of MOFs. global perspective, showing significant gaps and general To the best of our knowledge, no general reviews nor tendencies. Finally, in Sect. 11 we summarize and present detailed comparative studies of MOFs have been con- the main conclusions and future work. ducted in the literature. Moreover, a conceptual discussion Details about MOF assessment are provided as tables in about the desirable set of features of a MOF has not been ‘‘Appendix’’ and at http://www.isa.us.es/MOFComparison. carried out. The key point of this article is to provide a general comparative framework to guide the selection of a partic- 2 Metaheuristic optimization frameworks ular MOF and to evaluate the current MOFs found in the literature from a research perspective. In doing so, this Problem types that model real-life situations (e.g. traveling article extends the comparative framework of Gagne` and salesman problem, knapsack problem, MAX-SAT prob- Parizeau (2006) including frameworks that incorporate lem, etc.) have concrete instances that have a solution several types of metaheuristic techniques (cf. Sect. 4) and space that contains specific solutions. When those solutions presents a comparative analysis of a large set of features. are evaluated using an objective function (or a set of Specifically, this paper advances the state of the art in functions for multi-objective problems) we can define an the following: optimization problem as searching for the solution that provides the maximum (or minimum) value. 1. A general comparative framework for MOFs that can According to Glover and Kochenberger (2002)we be used to classify, evaluate and compare them. define metaheuristics as: ‘‘An iterative process that guides 2. An analysis of the current relevant MOFs in the the operation of one or more subordinate heuristics (which literature based in the comparative framework may be from a local search process, to a constructive proposed. process of random solutions) to efficiently produce quality 3. An evaluation of the current state of the art of MOFs solutions for a problem’’. An interesting concept in this from the research context that can be used: (i) to guide definition is the establishment of two distinct levels for newcomers in the area and (ii) to identify relevant gaps metaheuristic problem solving: the heuristic level that is by to MOF developers. definition highly dependent on the problem, and the It is important to highlight that the main value of this metaheuristic level based on the aforementioned level but study lies neither in comparing the rankings of two con- expressed as a problem-independent process. For instance, crete MOFs in a feature or characteristic, nor in stating when we apply simulated annealing (SA) (Kirkpatrick which MOF better fulfills the benchmark criteria. The main et al. 1983), we use three subordinate heuristics: the crea- contribution of the paper is the establishment of a general tion of an initial solution to the problem, the generation of comparison framework which clearly defines the set of similar (neighboring) solutions to another solution by some desirable features of MOFs; depicting a real ‘‘state of the criterion; and the evaluation of solutions (objective func- art’’ MOF with improvement directions and gaps in fea- tion). These heuristics are highly dependent on the specific tures support. This comparison framework has shown its problem addressed and how we encode solutions, but based value an generality, allowing the evaluation of the new on them, we can establish a general iterative algorithm that versions of assessed MOFs released during the realization has been successfully applied to a huge variety of of this study without modifications (four MOFs released problems. 123 Metaheuristic optimization frameworks 529 Thus, for each metaheuristic technique and type of 2.1 Why are MOFs valuable? problem, we have a set of subordinate heuristics that define how the metaheuristic is adapted to the problem-type at The No Free Lunch (NFL) theorem of Wolpert and Mac- hand. Note that a given problem-type may have multiple ready (1997) can be summarized as follows: ‘‘There