View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by ScholarBank@NUS METAHEURISTICS FOR NP-HARD COMBINATORIAL OPTIMIZATION PROBLEMS Dinh Trung Hoang (B.Sc, National Uni. of Vietnam) A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF ELECTRICAL AND COMPUTER ENGINEERING NATIONAL UNIVERSITY OF SINGAPORE 2008 iii ACKNOWLEDGEMENTS I would like to extend my gratitude and deepest appreciation to Dr. A. A. Ma- mun for his inspiration, excellent guidance, endless support and encourage- ment during the work. He has always made himself available for discussion whenever I encountered problems with the project. His erudite knowledge and deepest insights have been the most inspiration and made this research work a rewarding experience. I owe an immense dept of gratitude to him for having given me the curiosity about metaheuristics. Without his kindest help, this thesis and many others would have been impossible. Thanks also go to the faculties in Electrical & Computer Engineering Depart- ment in National University of Singapore, for their constant encouragement and valuable advice. Acknowledgement is extended to National University of Singapore for award- ing me the research scholarship and providing me the research facilities and challenging environment during my study time. I sincerely acknowledge all the help from all members in Mechatronics & Au- tomation lab, the National University of Singapore, in particular, my friends Dr. Tang K.Z., Mr. Trung, Mr. Zhu Zhen, Ms. Liu Jin, and Mr. Guan Feng for their kind assistance and friendship. Last but not least, I would thank my family members for their support, un- derstanding, patience and love during this process of my pursuit of a PhD., especially to my pretty and cunning sister Hang, my silly but handsome brother Hieu for all of their constant support for and sharing with me in whatever problems or happiness I faced or had since day one. During when struggling for preparing the oral defence, I was receiving strong support from iii iv my beloved Mummy who had come to Singapore twice just to help me any single thing and attended my oral defence. I am very appreciated all of what she has done to me. Also I would thank to my girlfriend Ngoc Kim for her ongoing strong and eternal love gave to me while I was stressful with indus- trial work at TECH and still trying finalizing the last version of the thesis. This acknowledgement would not complete if the great sacrifice of my Dad for his children’s further study was not recalled. This thesis, thereupon, is dedicated to them for their infinite stability margin. iv v METAHEURISTICS FOR NP-HARD COMBINATORIAL OPTIMIZATION PROBLEMS Dinh Trung Hoang National University of Singapore 2008 Abstract Combinatorial Optimization problems (COPs) are highly theoretical and of practical importance. Unfortunately, most of interesting COPs are proved to be intractable. Therefore, approximation approaches to those problems have received much intention since 1970s. During the past decades, a new kind of approximation algorithms, nowadays termed as metaheuristic, has emerged, providing a framework for solving many COPs by exploring the search space efficiently and exploiting the search history effectively. Among approximation algorithms, metaheuristic algorithms are widely recognized as one of the most practical approaches for combinatorial opti- mization problems. Some noticeable representatives of metaheuristics are Simulated Annealing (SA), Tabu Search (TS), Evolutionary Computation (EC), Ant Colony Optimization (ACO) and so on. For many combinatorial opti- mization problems the established metaheuristic algorithms are considered to be the state-of-the-art methods. In this report, we present two parts of work. One is on Ant Colony Optimization; the other is on decomposition- based hybrid metaheuristics. In particular, we propose a model of Ant algorithms that extends Graph- based Ant System (abbreviated as GBAS) model [106]. GBAS is the first and v vi most simple model which is used to study theoretical aspects related to con- vergence properties of ACO metaheuristics. All proposed to-date models for studying the convergence properties of ACO have not considered a widely- used technique which is to balance the exploration and exploitation process in almost all Ant-based algorithms. This technique is well-known in the re- search field of ACO and is called pseudo-random proportional rule or trade-off technique. To study the effectiveness of this technique in Ant-based algo- rithms from convergence perspective, an extended model of GBAS is pro- posed in one part of this report. Not only hold convergence properties as proved in GBAS, our model is also able to elucidate the practical role of this technique in Ant-based algorithms. Inspired by findings from this extended model, we suggest and experi- ment with a time-dependent approach. This approach aims at practically im- proving performance of Ant-based algorithms through a adaptively-adjusting rule for the trade-off technique. To judge the effectiveness of this time-dependent approach, we integrate it into state-of-the-art Ant-based algorithms - which are Ant Colony System (ACS), Max-Min Ant System, Best-Worst Ant System- in two different scenarios: i) use local search procedures and ii) do not use local search procedure in any algorithm. By testing on some medium-scale benchmark instances of Traveling Salesman Problem, we show experimen- tally that the performance of the Ant-based algorithms employing the adap- tively linear adjusting rule has been improved in comparison to that of the original Ant-based algorithms. A field of research on hybridization of metaheuristics with basic tech- niques in Artificial Intelligence and/or Operations Research has emerged re- vi vii cently and rapidly received attention of metaheuristics community. These hybrid metaheuristics aim at efficiently and effectively tackling large-scale real-world instances of COPs. Some findings in literature have suggested that the combination of classical artificial intelligence and operations research techniques with metaheuristics can be very beneficial for dealing with large- scale instances of some COPs. In the part of this report on hybrid meta- heuristics, we present runtime analysis of a scheme of hybridization between metaheuristics and clustering (or decomposition) methods. In particular, we prove that decomposition-based search method formed by combining a de- composition technique with a problem-solving algorithm runs faster than methods that do not utilize decomposition techniques. The speedup gained, however, is bounded and the bounds can be computed in advance. The finding of such bounds has shed some light on theoretically elucidating the runtime efficiency of decomposition-based search algorithms over the non- decomposition-based ones. This is the first work using an unified but problem- and algorithm-independent framework to evaluate the effectiveness and ef- ficiency of decomposition-based search algorithms in term of running time through the comparison to running time of alternative non-decomposition- based search algorithms. Moreover, in that part of this report, we also address concerns over a dis- advantage of decomposition-based methods, which relates to the failure of achieving optimal solutions in some scenarios. Those scenarios are simul- taneously dependent on both problem-solving methods and structure of in- stances of optimization problems. Our finding suggests that given an inex- act decomposition-based method for solving an optimization problem there vii viii probably exist some instances of the problem for which the method fails to include any optimal solutions in the search space. This means no optimal solution can be found using such a method no matter how much time any algorithmic instance of the method is given to run. To illustrate, we propose a simple inexact decomposition-based method to solve the Euclidean Traveling Salesman Problem (abbreviated as ETSP) and derive a sufficient condition on structure of ETSP instances such that if an in- stance of ETSP satisfies that condition, all of its optimal solutions will be con- tained into the search space generated by the proposed method; otherwise no optimal solution appears in that search space. However, the sufficient condition is applicable for a restricted number of subproblems, thus to make that condition more robust and applicable to large scale instances we extend it with additional assumptions on the structure of those large scale instances. The experimental results show that performance of a decomposition-based algorithm using ACS and derived from the sufficient condition is better than that of ACS on the same tests consisted of large scale clustered ETSP. viii TABLE OF CONTENTS ix TABLE OF CONTENTS Acknowledgements . iii Table of Contents . ix List of Tables . xiii List of Figures . xvi 1 Introduction 1 1.1 Background and Motivation . 1 1.1.1 Combinatorial Optimization . 2 1.1.1.1 The Optimization Problem . 2 1.1.1.2 Combinatorial Optimization . 3 1.1.2 On the Computational Complexity of Algorithms and No Free Lunch Theorem . 4 1.1.2.1 Computational Complexity of Algorithms and P vs NP 4 1.1.2.2 The No Free Lunch Theorem - a priory equivalence of search algorithms . 5 1.1.3 Exact versus approximate approaches . 7 1.1.4 Motivation . 9 1.2 Aims and Scope . 16 2 Literature Review 22 2.1 Metaheuristics - concepts, classification and characteristics . 23 2.1.1 What is a Metaheuristic ? . 23 2.1.2 Classification of Metaheuristics . 26 2.1.3 Diversification and Intensification in Metaheuristics . 33 2.2 Some state-of-the-art metaheuristics . 38 2.2.1 Population-based Approaches . 39 2.2.1.1 Evolutionary Computation . 39 2.2.1.2 Scatter Search and Path Relinking . 41 2.2.1.2.1 Scatter Search . 42 2.2.1.2.2 Path Relinking . 44 2.2.1.3 Estimation of Distribution Algorithms - EDAs . 47 2.2.1.4 Ant Colony Optimization - ACO . 48 2.2.2 Trajectory Approaches .