Beowulf Distributed Processing and the United States Geological Survey 1 By Brian G. Maddox Open-File Report 02-15 1 Mid-Continent Mapping Center, Rolla, MO 65401 U.S. Department of the Interior U.S. Geological Survey 1 Contents Introduction ..........................................................................................................................................1 Description and History of Distributed Parallel Processing........................................................1 Comparison With Other Techniques................................................................................................3 Approach ..............................................................................................................................................4 Testing...................................................................................................................................................7 Results...................................................................................................................................................8 Discussion ..........................................................................................................................................10 Future Work........................................................................................................................................12 Conclusion ..........................................................................................................................................13 References .........................................................................................................................................14 Figures Figure 1 MCMC Beowulf Cluster....................................................................................................5 Figure 2. Runtimes on various processor architectures ............................................................8 Figure 3. Number of nodes vs. runtime. ........................................................................................9 Figure 4. Estimated runtimes. .......................................................................................................10 i Introduction In recent years, the United States Geological Survey’s (USGS) National Mapping Discipline (NMD) has expanded its scientific and research activities. Work is being conducted in areas such as emergency response research, scientific visualization, urban prediction, and other simulation activities. Custom-produced digital data have become essential for these types of activities. High-resolution, remotely sensed datasets are also seeing increased use. Unfortunately, the NMD is also finding that it lacks the resources required to perform some of these activities. Many of these projects require large amounts of computer processing resources. Complex urban-prediction simulations, for example, involve large amounts of processor-intensive calculations on large amounts of input data. This project was undertaken to learn and understand the concepts of distributed processing. Experience was needed in developing these types of applications. The idea was that this type of technology could significantly aid the needs of the NMD scientific and research programs. Porting a numerically intensive application currently being used by an NMD science program to run in a distributed fashion would demonstrate the usefulness of this technology. There are several benefits that this type of technology can bring to the USGS’s research programs. Projects can be performed that were previously impossible due to a lack of computing resources. Other projects can be performed on a larger scale than previously possible. For example, distributed processing can enable urban dynamics research to perform simulations on larger areas without making huge sacrifices in resolution. The processing can also be done in a more reasonable amount of time than with traditional single-threaded methods (a scaled version of Chester County, Pennsylvania, took about fifty days to finish its first calibration phase with a single-threaded program). This paper has several goals regarding distributed processing technology. It will describe the benefits of the technology. Real data about a distributed application will be presented as an example of the benefits that this technology can bring to USGS scientific programs. Finally, some of the issues with distributed processing that relate to USGS work will be discussed. Description and History of Distributed Parallel Processing A few definitions are in order to understand what parallel processing is. The distinction must first be made between concurrent processing and parallel processing. Concurrent processing is defined as referring to “environments in which the tasks we define can occur in any order. One task can occur before or after another, and some or all tasks can be performed at the same time” (Nichols 1998, p. 16). Parallel processing is then defined to “specifically refer to the simultaneous execution of concurrent tasks on different processors” (Nichols 1998, p. 16). What this means is that in parallel processing, tasks must be able to run correctly at the same time on multiple processors. 1 In parallel processing, tasks are truly running on multiple processors simultaneously. This is different from a common misconception that operating systems such as Microsoft Windows or the various UNIX variants allow multiple applications to run at the same time on a single processor. What happens in such cases is that the operating system divides the processor’s time up into various units. It then schedules things so that each running application gets one of these units one after the other. These units are switched so quickly that the operating system gives the illusion that the programs are running simultaneously. In parallel processing, multiple processors are being used at the same time, giving multiple tasks the ability to be truly running simultaneously on a computer. This speeds up the “wall clock” execution time of a program, since it can run many parts of itself at the same time. It also allows a program to do things that it would not ordinarily be able to do. For example, an accounting program could divide its yearly total work by running a half-year on two processors at once instead of doing the whole year on a single processor. The task would take less total time, as it would be doing multiple calculations simultaneously. Distributed parallel processing can then be defined as the simultaneous execution of concurrent tasks across multiple processors on multiple computers. This is typically done by a series of machines connected by means of a local area network. The main difference is that instead of running on multiple processors in the same machine, the multiple processors are in different machines. It is still a form of parallel processing, though, just on a larger level. It is also more difficult, as communication between different nodes must be synchronized and some nodes may process data at different rates than others. The idea of parallel processing has long been studied in computer science. It can be traced back as far as 1958, when John Cocke and Dr. Daniel Slotnick discussed the idea of parallelism in numerical calculations in an IBM research memo (Wilson, 1994). Dr. Slotnick was granted a patent for parallel processing in the mid-1960s after construction of the ILLIAC IV, the world’s first supercomputer (Computers in Your Future, 2000). Today, the theories of parallel processing are taught in many computer science programs throughout the country. A Beowulf cluster is a specialized distributed processing cluster that has several distinguishing features. Thomas Sterling and Donald Becker originated the idea at the Center of Excellence in Space Data and Information Services in 1994. As defined in the Mailing List FAQ (Sitaker and others, 1999), it is “a kind of high-performance massively parallel computer built primarily out of commodity hardware components, running a free- software operating system like Linux or FreeBSD, interconnected by a private high-speed network. It consists of a cluster of PCs or workstations dedicated to running high- performance computing tasks. The nodes in the cluster don’t sit on people’s desks; they are dedicated to running cluster jobs. It is usually connected to the outside world through only a single node.” Many clusters are constructed out of upgraded, surplus computers. A Beowulf cluster generally has a low construction cost because few, if any, specialized components are used. Because of the low cost of Beowulf clusters, and because they are now recognized by the high-performance computer community, many organizations are using them to satisfy 2 their high-end computing requirements. As an example, the NOAA Forecast Systems Laboratory has constructed a 256-node Beowulf cluster for weather predictions. Over the next three years it will be upgraded to use over one thousand processors (NOAA, 2000). The Los Alamos National Laboratory has their Avalon cluster that consists of one hundred forty Alpha workstations (Avalon, 2000). Many other organizations are using clusters for activities that range from computing galactic-collision simulations to serving information on the World Wide Web. Comparison With Other Techniques To better understand distributed parallel processing, we must compare it with other techniques, such as parallel processing and SETI@home type processing. Although these techniques