New Variants of Variable Neighbourhood Search for 0-1 Mixed Integer Programming and Clustering A thesis submitted for the degree of Doctor of Philosophy Jasmina Lazi´c School of Information Systems, Computing and Mathematics Brunel University c August 3, 2010 Contents Contents i List of Figures vi List of Tables viii List of Abbreviations ix Acknowledgements xi Related Publications xiii Abstract 1 1 Introduction 3 1.1 CombinatorialOptimisation . ........ 3 1.2 The0-1 MixedInteger ProgrammingProblem . ......... 11 1.3 Clustering...................................... ... 17 1.4 ThesisOverview .................................. ... 18 2 Local Search Methodologies in Discrete Optimisation 21 2.1 BasicLocalSearch ................................ .... 21 2.2 A Brief Overview of Local Search Based Metaheuristics . ............. 24 2.2.1 SimulatedAnnealing. ... 24 2.2.2 TabuSearch ................................... 26 2.2.3 Greedy Randomised Adaptive Search . ..... 28 2.2.4 GuidedLocalSearch............................. .. 29 2.2.5 IteratedLocalSearch . ... 31 2.3 VariableNeighbourhoodSearch . ........ 33 2.3.1 BasicSchemes .................................. 33 2.3.2 AdvancedSchemes ............................... 40 2.3.3 Variable Neighbourhood Formulation Space Search . ........... 40 2.3.4 Primal-dualVNS................................ 41 2.3.5 Dynamic Selection of Parameters and/or NeighbourhoodStructures . 42 2.3.6 VeryLarge-scaleVNS .. .. .. .. .. .. .. .. .. .. .. .. .. 43 i ii Contents 2.3.7 ParallelVNS ................................... 44 2.4 Local Search for 0-1 Mixed Integer Programming . ........... 44 2.4.1 LocalBranching ................................ 47 2.4.2 VariableNeighbourhoodBranching. ....... 47 2.4.3 Relaxation Induced Neighbourhood Search . ........ 49 2.5 Future Trends: the Hyper-reactive Optimisation . ............. 50 3 Variable Neighbourhood Search for Colour Image Quantisation 53 3.1 RelatedWork..................................... .. 55 3.1.1 The Genetic C-Means Heuristic (GCMH) . ..... 55 3.1.2 The Particle Swarm Optimisation (PSO) Heuristic . .......... 56 3.2 VNSMethodsfortheCIQProblem. ..... 57 3.3 ComputationalResults. ...... 59 3.4 Summary ......................................... 63 4 Variable Neighbourhood Decomposition Search for the 0-1 MIP Problem 67 4.1 TheVNDS-MIPAlgorithm .. .. .. .. .. .. .. .. .. .. .. .. ... 68 4.2 ComputationalResults. ...... 70 4.3 Summary ......................................... 81 5 Applications of VNDS to Some Specific 0-1 MIP Problems 87 5.1 The Multidimensional Knapsack Problem . ......... 87 5.1.1 RelatedWork................................... 89 5.1.2 VNDS-MIPwithPseudo-cuts . ... 92 5.1.3 ASecondLevelofDecompositioninVNDS . ..... 96 5.1.4 ComputationalResults. .... 98 5.1.5 Summary ..................................... 111 5.2 TheBargeContainerShipRoutingProblem . ......... 112 5.2.1 FormulationoftheProblem . .... 112 5.2.2 ComputationalResults. .... 118 5.2.3 Summary ..................................... 120 5.3 The Two-Stage Stochastic Mixed Integer Programming Problem .......... 120 5.3.1 Existing Solution Methodology for the Mixed Integer 2SSP ......... 123 5.3.2 VNDS Heuristic for the 0-1 Mixed Integer 2SSP . ....... 123 5.3.3 ComputationalResults. .... 125 5.3.4 Summary ..................................... 127 6 Variable Neighbourhood Search and 0-1 MIP Feasibility 129 6.1 RelatedWork: FeasibilityPump . ....... 130 6.1.1 BasicFeasibilityPump. .... 131 6.1.2 GeneralFeasibilityPump . .... 131 6.1.3 ObjectiveFeasibilityPump . ..... 132 6.1.4 Feasibility Pump and Local Branching . ....... 133 6.2 VariableNeighbourhoodPump . ...... 134 iii 6.3 Constructive Variable Neighbourhood Decomposition Search ............ 136 6.4 ComputationalResults. ...... 136 6.5 Summary ......................................... 144 7 Conclusions 147 Bibliography 151 Index 171 A Computational Complexity 175 A.1 DecisionProblemsandFormalLanguages . ......... 175 A.2 TuringMachines .................................. ... 176 A.3 Time Complexity Classes p and np ........................... 180 A.4 np-Completeness ..................................... 181 B Statistical Tests 183 B.1 FriedmanTest .................................... 183 B.2 Bonferroni-DunnTestandNemenyiTest . ......... 184 C Performance Profiles 185 iv Contents List of Figures 1.1 Structural classification of hybridisations between metaheuristics and exact methods. 10 2.1 Basiclocalsearch. ............................... ..... 22 2.2 Bestimprovementprocedure. ....... 23 2.3 Firstimprovementprocedure. ........ 23 2.4 Localsearch: stallinginalocaloptimum. ........... 24 2.5 Simulatedannealing. .. .. .. .. .. .. .. .. .. .. .. .. ...... 25 2.6 Geometric cooling scheme for Simulated annealing. .............. 26 2.7 Tabusearch. ..................................... .. 27 2.8 Greedyrandomisedadaptivesearch. ......... 29 2.9 GLS: escaping from local minimum by increasing the objective function value. 30 2.10 Guidedlocalsearch. .. .. .. .. .. .. .. .. .. .. .. .. ...... 31 2.11 Iteratedlocalsearch. ........ 32 2.12 The change of the used neighbourhood in some typical VNS solution trajectory. 34 2.13VNDpseudo-code. ................................ .... 35 2.14RVNSpseudo-code. ............................... .... 36 2.15 ThebasicVNSscheme. ............................. .... 37 2.16 TheBasicVNSpseudo-code. ...... 37 2.17GeneralVNS..................................... ... 38 2.18SkewedVNS. ..................................... .. 39 2.19 Variable neighbourhood decomposition search. .............. 40 2.20 VNSformulationspacesearch. ........ 41 2.21 Variable neighbourhood descent with memory. ............ 42 2.22 Localsearchinthe 0-1 MIP solutionspace. ........... 46 2.23 UpdatingtheneighbourhoodsizeinLB. .......... 47 2.24 NeighbourhoodupdateinVND-MIP. ........ 48 2.25 VNBshakingpseudo-code. ...... 48 2.26 VariableNeighbourhoodBranching. .......... 49 2.27 A standard local search-based metaheuristic. .............. 51 2.28 Hyper-reactivesearch. ........ 52 2.29 TheHyper-reactiveVNS. ...... 52 3.1 RVNSforCIQ. ..................................... 58 v vi List of Figures 3.2 VNDSforCIQ...................................... 59 3.3 Time performance of VNS quantisation methods . .......... 62 3.4 MSE/running time comparison between the M-Median and the M-Means solution. 63 3.5 “Lenna” quantised to 16 colours: (a) M-Median solution, (b) M-Means solution. 64 3.6 “Lenna” quantised to 64 colours: (a) M-Median solution, (b) M-Means solution. 64 3.7 “Lenna” quantised to 256 colours: (a) M-Median solution, (b) M-Means solution. 64 3.8 “Baboon” quantised to 16 colours: (a) M-Median solution, (b) M-Means solution. 65 3.9 “Baboon” quantised to 64 colours: (a) M-Median solution, (b) M-Means solution. 65 3.10 “Baboon” quantised to 256 colours: (a) M-Median solution, (b) M-Means solution. 65 3.11 “Peppers” quantised to 16 colours: (a) M-Median solution, (b) M-Means solution. 66 3.12 “Peppers” quantised to 64 colours: (a) M-Median solution, (b) M-Means solution. 66 3.13 “Peppers” quantised to 256 colours: (a) M-Median solution, (b) M-Means solution. 66 4.1 VNDSforMIPs. .................................... 69 4.2 VNDforMIPs...................................... 70 4.3 Relative gap average over all instances in test bed vs. computational time. 74 4.4 Relative gap average over demanding instances vs. computational time. 74 4.5 Relative gap average over non-demanding instances vs. computational time. 75 4.6 The change of relative gap with computational time for biella1instance. 78 4.7 Relative gap values (in %) for large-spread instances. ................ 80 4.8 Relative gap values (in %) for medium-spread instances. ............... 80 4.9 Relative gap values (in %) for small-spread instances. ............... 81 4.10 Relative gap values (in %) for very small-spread instances............... 81 4.11 Bonferroni-Dunn critical difference from the solution quality rank of VNDS-MIP . 85 4.12 Bonferroni-Dunn critical difference from the running time rank of VNDS-MIP . 85 5.1 Linearprogrammingbasedalgorithm. ......... 89 5.2 An iterativerelaxationbased heuristic. ............. 91 5.3 An iterative independent relaxation based heuristic. ................ 92 5.4 VNDS-MIPwithpseudo-cuts.. ...... 93 5.5 VNDS-MIP with upper and lower bounding and another orderingstrategy. 96 5.6 Two levels of decomposition with hyperplanes ordering. ............... 97 5.7 Flexibility for changing the hyperplanes. ............. 99 5.8 Performance profiles of all 7 algorithms over the OR librarydataset.. 110 5.9 Performance profiles of all 7 algorithms over the GK data set............. 111 5.10 Example itinerary of a barge container ship . ............ 114 5.11 VNDS-SIPpseudo-code. ...... 124 6.1 Thebasicfeasibilitypump. ....... 132 6.2 ConstructiveVND-MIP. ..... 135 6.3 The variable neighbourhood pump pseudo-code. ............ 135 6.4 Constructive VNDS for 0-1 MIP feasibility. ........... 139 List of Tables 3.1 The MSE of the VNS, GCMH and PSO heuristics . ..... 60 3.2 The MSE/CPU time of the RVNDS’, RVNDS, VNDS and RVNS algorithms . 61 3.3 The MSE/CPU time comparison between the M-Median and M-Means solutions. 62 4.1 Testbedinformation. ............................. ..... 72 4.2 VNDS1andVNDS2timeperformance. ..... 76 4.3 VNDS-MIP objective values for two different parameters settings .......... 77 4.4 Objective function values for all the 5 methods tested. ............... 78 4.5 Relative gap values (in %) for all the 5 methods tested. ............. 79 4.6 Running times (in seconds) for