Efficient Load Balancing Algorithms for Parallel and Distributed Systems A Thesis Submitted in the partial fulfillment of the requirements for the award of the degree of Doctor of Philosophy Submitted By Jahangir Alam (Registration No. 90603504) Under Supervision and Guidance of Dr. Rajesh Kumar (Professor, CSED, TU Patiala) Department of Computer Science & Engineering Thapar University, Patiala - 147004 (Punjab), India. July, 2016 iii Dedicated to My Parents, Wife and Kids Abstract Parallel and Distributed computing is essential for modern research as the demand for more and more computing power is continuously increasing. A number of aspects within parallel and distributed computing have been explored in recent years, however two of the most pertinent issues relate to the design of scalable interconnection network topology and task allocation for load balancing. This work addresses these two problems. The first part of the thesis presents the design and evaluation of a highly scal- able and economical interconnection network topology known as the STH (Seri- ally Twisted Hypercube). It begins with a brief survey of various interconnection network topologies proposed in literature followed by the design principles on which the proposed STH interconnection network will be based upon. Various properties of the proposed topology are derived and then compared with other topologies on a number of interconnection networks evaluation parameters. The second part of the thesis deals with the task allocation problem taking into account load balancing. Exploiting the full potential of a parallel and distributed system requires efficient allocation of the program tasks to the diversely capable machines within the system. If the task allocation strategy is not properly framed, machines in the system may spend most of their time waiting for each other in- stead of performing useful computations. This part of the thesis focuses on static task allocation considering load balancing in heterogeneous distributed comput- ing systems sometimes also referred to as heterogeneous multicomputer systems (HMS). The thesis first presents a brief literature survey on the proposed solutions and then classifies them according to the solution techniques. It then presents two mathematical models based on fuzzy logic to solve the task allocation problem. Finally based on the proposed models the the thesis presents two algorithms to solve the aforementioned problem. The algorithms are coded in C and the pro- posed models are verified by executing the corresponding programs with several sets of randomly generated data. Experimental results prove that the algorithms successfully allocate tasks to the machines whilst balancing the allocated task load. vi Analysis of the proposed method also proves that its complexity is low compared to similar existing approaches. Declaration The work in this thesis is based on research jointly carried out at University Women’s Polytechnic, Faculty of Engineering & Technology, Aligarh Muslim Uni- versity, Aligarh (India) and Department of Computer Science & Engineering, Tha- par University Patiala, Patiala (India). No part of this thesis has been submitted elsewhere for any other degree or qualification and it is all my own work unless referenced to the contrary in the text. Copyright c 2015 by JAHANGIR ALAM. “The copyright of this thesis rests with the author. No quotations from it should be published without the author’s prior written consent and information derived from it should be acknowledged”. vii Acknowledgements First of all, I thank ALLAH (SWT), the Lord Almighty, for giving me the health, strength, ability to complete this work and for blessing me with supportive super- visor, family and friends. I wish to express my deepest appreciation to my supervisor, Dr. Rajesh Kumar for his idea, support, enthusiasm, and patience. I have learnt an enormous amount from working with him. I would also like to thank the Hon’ble Director, Thapar University for giving me the opportunity to attend research programme at Thapar University, Patiala. I would like to express my profound gratitude to my doctoral committee mem- bers: Prof. Rakesh Sharma, Dr. Maninder Singh, Dr. Deepak Garg and Dr. Rajesh Kumar for their enlightening suggestions from time to time. I am forever indebted to my employer Aligarh Muslim University (India) and to Dr. Salma Shaheen, Principal, University Women’s Polytechnic, AMU, Aligarh for extending me the facilities required to complete this work. I express my sincere thanks to Dr. Mohhmad Athar Ali, Associate Professor, Dept. of Computer Engg., AMU Aligarh for English corrections, which helped improve all my writings. Life would be harder without the support of many good relatives and friends. Thanks to Prof. Shamim Ahmad (for being such a wonderful mentor), Mr. Far- rukh Shamim, Dr. Abdus Samad, Dr. Parveen Beg, Mr. M. Abdul Qadeer, Mr. Misbahurrahman Siddiqui, Mr. Tariq Ahmed, Mr. Wajid Ali, Mr. Rahat Mahmood, Mr. Gulsanover and Mr. Mohd. Hanzala. Thank you to all my friends at AMU Aligarh and Thapar University, Patiala. viii ix Finally, I thank to all my family members for their love, patience and uncount- able supports. Contents Abstract iv Declaration vi Acknowledgements vii 1 Introduction 1 1.1 Parallel Processing . .1 1.2 Need for Parallel Processing . .2 1.3 Taxonomy of Computer Architectures . .3 1.3.1 Single Instruction Single Data (SISD) . .3 1.3.2 Single Instruction Multiple Data (SISD) . .4 1.3.3 Multiple Instruction Single Data (MISD) . .5 1.3.4 Multiple Instruction Multiple Data (MIMD) . .6 1.3.5 Hybrid Architecture . 10 1.4 Parallel and Distributed Systems Addressed in This Thesis . 11 1.5 Motivations . 11 1.5.1 Scalable and Economical Interconnection Network . 12 1.5.2 Static Load Balancing Task Allocation . 13 1.5.3 Heterogeneous Multicomputer Systems . 14 1.5.4 No Task Duplication . 15 1.5.5 Problem Complexity . 16 1.6 Contributions . 16 1.6.1 An Economical and scalable topology for PDS . 16 x Contents xi 1.6.2 Routing and Broadcasting Procedures for ST H Topology . 17 1.6.3 Load Balancing Task Allocation Models . 17 1.6.4 Fuzzy Expected Time to Compute (F ET C) Matrix Genera- tion Algorithm . 18 1.6.5 Efficient Allocation Algorithms . 18 1.7 Thesis Organization . 18 2 Interconnection Networks and Hybrid Topologies 22 2.1 Criteria Used for Classification of INs . 22 2.1.1 Mode of Operation . 23 2.1.2 Control Strategy . 23 2.1.3 Switching Techniques . 24 2.1.4 Topology . 24 2.2 Topology Based Classification for INs . 24 2.3 Dynamic Interconnection Networks . 25 2.3.1 Bus Based Dynamic Interconnection Networks . 25 2.3.2 Switched Based Interconnection Networks . 27 2.3.3 Blocking and Non-Blocking Networks . 37 2.4 Static Interconnection Networks . 37 2.4.1 Completely Connected Networks . 40 2.5 Limited Connection Networks . 41 2.6 Hybrid Interconnection Networks . 45 2.7 Cross Product as Net. Synthesizing Operator . 46 2.7.1 Topological Properties of Product Networks . 47 2.7.2 Significance of Cross Product . 48 2.7.3 Routing on Product Networks: . 49 2.7.4 Broadcasting on Product Networks . 49 2.7.5 Performance Issues of Product Networks . 51 2.8 Network Topology and Load Balancing . 51 Contents xii 3 The Design and Analysis of STH Interconnection Network 54 3.1 Related Work . 54 3.2 Preliminaries and Graph Theoretic Definitions . 57 3.3 Design of Scalable Twisted Hypercube . 58 3.3.1 Topological Properties of STH . 62 3.3.2 Routing on ST H Network . 66 3.3.3 Broadcasting On ST H Network . 72 3.4 ST H Analysis and Comparative Study . 72 4 Taxonomy of Task Allocation Models and Related Work 90 4.1 The Task Allocation Problem . 91 4.2 Task Allocation and Load Balancing . 94 4.3 Taxonomy of Task Allocation Models . 98 4.3.1 Exact Algorithms . 98 4.3.2 Approximate Algorithms . 98 4.3.3 Mathematical Programming Techniques . 100 4.3.4 Graph Theoretic Techniques . 104 4.3.5 State Space Search Techniques . 108 4.3.6 Heuristic Techniques . 110 5 Fuzzy Load Balancing Task Allocation Models 120 5.1 Related Work . 121 5.2 Problem Statement . 122 5.3 Brief Overview of Fuzzy Logic . 123 5.3.1 Fuzzy Sets . 123 5.3.2 Fuzzy Numbers . 123 5.3.3 Triangular Fuzzy Numbers . 124 5.3.4 Linguistic Variables . 124 5.3.5 Defuzzification . 126 5.4 Fuzzy Load Balancing Task Allocation Model - I . 127 5.4.1 Assumptions . 127 Contents xiii 5.4.2 Task Execution Time and Execution Cost . 128 5.4.3 Task Precedence Constraints and Priorities . 130 5.4.4 Communication Cost . 133 5.4.5 Load Balancing . 135 5.4.6 System Reliability . 135 5.4.7 The Task Allocation Algorithm . 136 5.4.8 Experimental Setup and Implementation of FLBTA - I . 138 5.4.9 Performance Evaluation of F LBT A − I ........... 165 5.4.10 Load Balancing . 167 5.4.11 Comparative Study . 167 5.5 Fuzzy Load Balancing Task Allocation Model - II . 170 5.5.1 Selecting Best Machine to Execute the Task . 172 5.5.2 Link Reliability . 174 5.5.3 Modified Allocation Algorithm . 175 5.5.4 Experimental Setup and Implementation of FLBTA - II . 175 6 Conclusions and Future Directions 185 6.1 Summary of Contributions . 186 6.1.1 Literature Reviews . 187 6.1.2 Development of Mathematical Models . 187 6.1.3 Algorithms . 188 6.2 Concluding Remarks . 189 6.3 Future Research . 190 6.3.1 Task Partitioning . 190 6.3.2 Estimation on Communication Cost . 190 6.3.3 Designing Suitable Multicomputer Architecture . 191 6.3.4 Improving the Allocation Models for Network Contention . 191 6.3.5 Designing a Dynamic Task Allocator . 191 List of Publications . 192 References 193 List of Figures 1.1 SISD architecture . .4 1.2 SIMD architecture .