A Distributed Implementation of the Simplex Method Gavriel Yarmish Department of Computer and Information Science Technical Report TR-CIS-2001-04 08/13/2001 A Distributed Implementation of the Simplex Method DISSERTATION Submitted in Partial Fulfillment of the Requirements for the degree of DOCTOR OF PHILOSOPHY (Computer & Information Science) at the POLYTECHNIC UNIVERSITY by Gavriel Yarmish March 2001 Approved: Department Head Date Copy No.__________ ii Copyright 2001 by Gabriel Yarmish All rights reserved. iii Major: Computer Science ____________________ Richard Van Slyke Professor of Computer & Information Science ___________ Date ____________________ Alex Delis Associate Professor of Computer & Information Science ___________ Date ____________________ Torsten Suel Assistant Professor of Computer & Information Science ___________ Date Minor: Financial Engineering ____________________ Frederick Novomestky Industry Associate Professor of Management ___________ Date iv Microfilm or other copies of this dissertation are obtainable from: UMI Dissertations Publishing Bell & Howell Information and Learning 300 North Zeeb Road P.O. Box 1346 Ann Arbor, Michigan 48106-1346 v Gavriel Yarmish was born in the United States. He was awarded his BS in Computer Science, Magna Cum Laude, from Touro College in 1991 and his MA in Computer Science from Brooklyn College in 1993. He is a member of Tau Beta Pi, the engineering honor society, and has taught Computer Science and Mathematics since 1995. The research presented in this thesis was completed between 1994 and 2001. vi This Dissertation is dedicated to my family, all of whom have lent encouragement and support during the time spent on this research and before. vii Acknowledgments I wish to express my sincerest thanks to the chairman of my dissertation committee, Dr. Richard Van Slyke, for working with me throughout this long enterprise. We thank the staff at the Computer Science Department in Polytechnic University. Jeff Damens, our system administrator, helped install and troubleshoot MINOS and MPI. He has also responded quickly to all network-related problems. Torsten Suel provided input regarding various communication models. I also wish to thank Joel Wein for the use of his lab during the early stages of my research and Boris Aronov for his help and concern throughout the progress of this study. R. N. Uma helped to explain the setup of MPI in the computing labs. I wish also to acknowledge my good friend Jacob Maltz for his help with UNIX shell scripting and for other troubleshooting throughout the empirical studies. I wish also to acknowledge my committee members, Alex Delis, Frederick Novomestky, and Torsten Suel for taking the time to critique the dissertation. viii Abstract A Distributed Implementation of the Simplex Method by Gavriel Yarmish Advisor: Richard Van Slyke Submitted in Partial Fulfillment of the Requirements for the degree of DOCTOR OF PHILOSOPHY (Computer & Information Science) March 2001 The Simplex Method, the most popular method for solving Linear Programs (LPs), has two major variants. They are the revised method and the standard, or full tableau method. Today, virtually all serious implementations are of the revised method because it is more efficient for sparse LPs which are the most common. However, the full tableau method has advantages as well. First, the full tableau can be very effective for dense problems. Second, a full tableau method can easily and effectively be extended to a coarse grained distributed algorithm. While dense problems are uncommon in general, they occur frequently in some important applications such as digital filter design, text categorization, image processing and relaxations of scheduling problems. We implement two full tableau algorithms. The first, a serial implementation, is effective for small to moderately sized dense problems. The second, a simple extension of the first, is a distributed algorithm, which is effective for large problems of all densities. ix We developed performance models that predict running times per iteration for the serial version of our method, the parallel version of our method and the revised method for problems of different sizes, aspect ratios and densities. We also developed methods for choosing the number of processors to optimize the tradeoff between computation and communication in distributed computations. We tested our algorithms on practical (Netlib) and synthetic problems. x Table of Contents 1. Introduction ------------------------------------------------------------------------------------ 1 2. Related Work ---------------------------------------------------------------------------------- 4 2.1 Interior point methods 4 2.2 Parallel implementations for the simplex method 4 3. Review of the Simplex Method ------------------------------------------------------------- 6 3.1 A short review of linear algebra 6 3.2 Definition of a Linear Program 7 3.3 Short description of the full tableau simplex method 10 3.4 Short description of the revised method 13 3.5 Running time comparison of the revised method and the full tableau method 16 4. Motivation for a serial and distributed full tableau implementation -------------21 4.1 Why the method is applied to full tableau 21 4.1.1 No pricing out or column reconstruction 21 4.1.2 Alternate column choice rules can easily be used 21 4.1.3 The inverse does not have to be kept in each processor 23 4.1.4 Dense problems do not gain by use of the revised 23 4.2 Distributed computing 23 4.2.1 Why use distributed over MPP – no need for a supercomputer 23 4.2.2 Why coarse grain parallelism is used 24 5. Models and Analysis-------------------------------------------------------------------------25 5.1 Synchronized parallel pivots 25 5.2 Short sketch of one distributed pivot step 27 5.2.1 Choice of four models of parallel communication 28 5.2.2 Basic explanation of communication parameters 29 5.2.3 Analyzing basic communication operations 31 5.3 The Experimental Environment 34 5.3.1 Description of lab(s) 34 5.4 Estimates of computation parameters ucol, urow and upiv 34 5.5 Estimates of communication parameters s, g and L 36 5.6 The performance model 40 5.7 The optimum number of processors 41 5.8 The optimum number of processors with a new column division scheme 43 5.9 Estimated parallel running time for each communication model 45 5.10 Running time estimates of the revised (MINOS), serial (retroLP) and parallel Full-tableau algorithms (dpLP) 46 5.11 Advantage of the Steepest Edge column choice rule 55 5.12 Memory requirements of revised and full tableau method 57 5.13 Asymptotic (computation/communication ratio change) analysis 59 5.14 Sensitivity to s, π and p 62 6. Implementation Choices --------------------------------------------------------------------72 6.1 Distributed programming software 72 6.2 Sockets 73 6.3 Reasons for choice of both sockets and MPI 73 6.4 The specific commands used 74 xi 6.5 A brief description of MINOS, a revised simplex implementation 75 7. Computational Experiments---------------------------------------------------------------78 7.1 Problems used for experimentation 78 7.1.1 Synthetic linear programs 78 7.1.2 Netlib Problems 83 7.2 Validation of Performance Models 86 7.2.1 Computation verification 86 7.2.2 Communication verification (using regression for coefficients) 93 7.2.3 Analysis of the revised program (MINOS) expression 98 7.2.4 Revised vs. retroLP and dpLP 101 7.3 Total Time Comparisons 105 8. Summary, Applications and Future Work ------------------------------------------- 110 8.1 Summary 110 8.2 Applications with dense matrices 111 8.3 Future work 112 Appendix A. Form of Linear Program input: LPB and MPS---------------------- 115 A.1 Preprocessing into LPB format 115 A.2 The MPS format 119 Appendix B. Program description: retroLP and dpLP------------------------------ 123 B.1 retroLP - the serial implementation 124 B.2 dpLP – the parallel implementation 125 References ----------------------------------------------------------------------------------------- 128 xii Figures Figure 3.1- Full tableau vs. the revised methods......................................................... 17 Figure 3.2-n at which the revised method overtakes the tableau method (m=1,000) . 20 Figure 5.1- time per iteration as n increases................................................................ 45 Figure 5.2 – Time per iteration (m=1,000).................................................................. 49 Figure 5.3-Time per iteration as a function of density................................................ 51 Figure 5.4-Time per iteration as a function of aspect ratio ......................................... 52 Figure 5.5- Time per iteration as a function of m........................................................ 52 Figure 5.6 – Time per iteration as a function of p....................................................... 53 Figure 5.7 – Time per iteration as a function of p....................................................... 54 Figure 5.8 – Memory (in Megabytes) ......................................................................... 59 Figure 5.9 – Asymptotic speedup when a unit computation = 10-7 . s and g move together................................................................................................................ 61 Figure 5.10 - Asymptotic speedup time when s = 3.4*10-7 and g = 1.7*10-6 . Computation is changing..................................................................................... 62 Figure 5.11 – Time per iteration