Optimizing Job Placement on the Cray XT3 Deborah Weisser, Nick Nystrom, Chad Vizino, Shawn T. Brown, and John Urbanic Pittsburgh Supercomputing Center ABSTRACT: The Cray XT3’s SeaStar 3D mesh interconnect, which at PSC is configured as a 3D torus, offers exceptional bandwidth and low latency. However, improperly scheduled jobs produce contention that undermines otherwise scalable applications. Empirical observations indicated that production jobs were being badly fragmented, seriously degrading their performance. Recently we have quantified the impact of non-contiguous partitions on a variety of communication-intensive applications, including production codes, which are representative of PSC’s workload. We are investigating different combinations of job scheduling strategies which automatically map processors to jobs or accommodate user specifications for required or desired layouts. A flexible placement policy supports multiple protocols to accommodate PSC’s varied workload and users’ needs while mitigating the effects of fragmentation. KEYWORDS: Cray XT3, processor allocation, job placement, job scheduling, optimization 1. Introduction different job layout protocols: the default job placement protocol, and an optimized protocol which assigns jobs to The Cray XT3’s SeaStar 3D mesh interconnect, connected cabinets of processors in row-major order. The which is configured on BigBen1 at PSC as a 3D torus, results are compelling: communication-intensive jobs offers exceptional bandwidth and low latency. However, representative of PSC’s workload clearly show the effects improperly placed jobs produce contention that of fragmentation under the default protocol, and their undermines otherwise scalable applications. If multiple efficiency increases substantially when they are run on communication-intensive jobs are running concurrently, compact, convex connected subsets of processors. performance will degrade when messages for one job are In addition to the optimized protocol used in the routed through processors owned by another job. experiments described in Section 3, we are evaluating Empirical observations indicated that production jobs different job placement protocols, discussed in Section 4, were being badly fragmented, substantially degrading to minimize contention, maintain compactness, and their performance. The default processor-to-job minimize fragmentation. We are investigating assignment policy, which assigns jobs to cabinets in combinations of different job scheduling strategies to numerical order, is far from optimal because numerically accommodate PSC’s varied workload and users’ priorities sequential cabinets are not necessarily directly connected and to reduce the effects of fragmentation. We are also exploring the use of pairwise communication profiling to each other, and even the connected subsets of 2 processors assigned to a job are not typically compact. information obtained from CrayPat to automatically This protocol increases latency, and it also increases job generate job placements optimized for particular fragmentation, seriously undermining performance. applications. Recently we have quantified the impact of non- The results and techniques described in this paper, contiguous job placements, described in Section 3, on a obtained on BigBen and described in the context of the several communication-intensive applications which are Cray XT3, can be generalized to other 3D mesh representative of PSC’s workload. We experimented with computers. CUG 2006 Proceedings 1 of 9 Related Work The cabinets are connected to form a 3D torus. The x-dimension cables connect cabinets within a row to each A polynomial-time approximation scheme for other; every cabinet is connected to 2 other cabinets. The assigning processors to jobs to minimize the average y-dimension cables connect cages within a cabinet and are pairwise distance is described in Bender et al.3 The internal to a cabinet, and the z-dimension cables connect 1 algorithm guarantees a placement within 2 − 2d of pairs of cabinets between rows. The wiring diagram for optimal and was implemented on the Cray T3D. The Cray BigBen is shown in Figure 1. There are additional y- and T3E job placement protocol assigned cubes of connected z-dimension connections which are internal to a cage and processors to jobs (or an approximation if a perfect cube do not appear in the wiring diagram. This topic is was not available).4 discussed in more detail in Section 4.2. 2. Machine Organization Note that the x-dimension cables connect each cabinet to 2 others, forming a folded torus to minimize the maximum cable length. Thus along each row the PSC’s Cray XT3 has 2090 processors, of which 2068 cabinets are physically located in the following order: are configured as compute processors (the remaining 22 are service I/O processors, serving as login processors, 10 – 9 – 8 – 7 – 6 – 5 – 4 – 3 – 2 – 1 – 0 Lustre OSTs, and network interfaces). BigBen is They are connected by x-dimension cables in the physically arranged in 2 rows of 11 cabinets. Each following order: cabinet contains 3 cages, and each cage contains 8 blades. Each compute blade contains 4 Opteron processors, 0 – 2 – 4 – 6 – 8 – 10 – 9 – 7 – 5 – 3 – 1 – 0 – … yielding up to 32 compute processors per cage and up to 96 compute processors per cabinet5. Figure 1. The BigBen wiring diagram shows which cabinets (and cages) are connected. CUG 2006 Proceedings 2 of 9 Sequentially numbered cabinets are not necessarily 3. Quantifying Job Placement Strategies directly connected to each other. Table 1 is an adjacency matrix of the number of hops along the x-axis between 3.1 Placement Based on Default Placement Policy pairs of cabinets. The default scheduling policy assigns free processors In the next section we will examine the relation to a job by processor number, issuing free processors between two job placement protocols, cabinet numerically from low to high. The processors are connectivity, and application performance. numbered by cabinet, alternating rows; i.e. the lowest- numbered processors are in row 0 cabinet 0, then row 1 Table 1. Number of hops messages must traverse between cabinet 0, then row 0 cabinet 1, etc. As indicated in Table cabinets within each row. Cabinets are numbered 1, numerically sequential processors may be in cabinets sequentially along each row; however, to avoid long which are separated from each other by up to 5 hops. wires, logical connectivity leapfrogs cabinets as described We use screen captures of PSC’s 3D Monitor6 to in the text. display a typical mix of jobs on BigBen in various ways. cabinet 0 1 2 3 4 5 6 7 8 9 10 The processors are color-coded by job. Free, disabled, 0 service, and down processors are designated by white, 1 1 2 1 2 red, yellow, and gray blocks, respectively, and users’ jobs 3 2 1 3 are designated by a range of colors from brown to pink. 4 2 3 1 4 5 3 2 4 1 5 Figure 2 shows the distribution of jobs on the 6 3 4 2 5 1 5 processors of BigBen, where the processors are shown in 7 4 3 5 2 5 1 4 their physical locations within cabinets. The seven jobs in 8 4 5 3 5 2 4 1 3 Figure 2 were assigned to processors using the default job 9 5 4 5 3 4 2 3 1 2 placement algorithm, which places jobs on numerically 10 5 5 4 4 3 3 2 2 1 1 sequential cabinets. From the figure, there is little Figure 2. The distribution of jobs by color on the processors of BigBen, where the processors are shown in their physical locations within cabinets. The assignment of processors to jobs was done with the default placement protocol. CUG 2006 Proceedings 3 of 9 Figure 3. The processors of BigBen, colored by job, arranged by processor connectivity. Processors are adjacent only if they are directly connected. fragmentation when using cabinet numbering as the algorithms. We used actual production codes and a metric: jobs are clustered together in nearby cabinets. communication-intensive benchmark. If we examine which processors are actually Toward our goal of optimizing performance on our connected to each other, the inherent fragmentation of production system, we strove to simulate realistic this policy becomes clear. Figure 3 shows processor conditions, neither idealistically good nor pathologically connections; processors are adjacent only if they are bad, and gather results on important, representative user directly connected to each other. It is clear that the default codes. We validated our experiments with the default job placement protocol systematically fragments jobs, placement policy by comparing results to actual sandwiching jobs around one another. In the case of the production runs under the default placement policy. pink job, shown in Figure 3 and in isolation in Figure 4, The applications we chose to use for our experiments interprocessor communication may very well occur are DNSmsp7, a direct numerical simulation code for through processors owned by another job, in this case, the analysis of turbulence, NAMD8, a scalable molecular brown job. Additionally, the pink job will bear the effects dynamics code for simulations of large biomolecular of communication traffic generated by the tan job which systems, and PTRANS, a parallel matrix transpose code surrounds it. When the mix of jobs is communication- which is part of the HPCC benchmark9. These intensive, as is the norm on the XT3 because of its high- applications were chosen because they are all performance interconnect, this arrangement is clearly less communication-intensive, and they cover a wide range of than ideal. In the next section, we quantitatively communication patterns. NAMD sends many small point- demonstrate the effects of fragmentation on the to-point messages, and the communication pattern is performance of important, representative communication- irregular. DNSmsp sends many relatively small messages intensive applications. in a regular communication pattern. PTRANS sends many large all-to-all messages. Additionally, NAMD and DNS 3.2 Quantifying the Effects of Fragmentation are production codes important to PSC’s workload.