Hybrid Messaging Passing in Shared-Memory Clusters∗

Vincent W. Freeh† Jin Xu‡ David K. Lowenthal§

Abstract An increasingly popular choice for high-performance machines is clusters of SMPs. Such clusters are popular in department-wide as well as national laboratory settings. On clusters of SMPs, there are two different memory models: shared memory, within a node, and distributed memory, between nodes. It is more difficult to write an efficient program using two memory models, i.e., combining OpenMP and MPI, than one. Therefore, several systems provide a hybrid message passing system, in which an IPC mechanism is used for local messaging and a network protocol (i.e., TCP) is used for remote messaging. This allows messaging passing programs to run much more efficiently on shared-memory clusters. The dual to this hybrid message passing program is shared-variable program supported by a software distributed shared memory. In theory, the shared-variable program is easier to create; however, in practice such a program is intolerably slow. While clearly reducing the overhead of messaging, a hybrid message passing system does not take full advantage of the shared memory model. This paper describes the Optimizing Hybrid Messaging (OHM) package that allows for highly efficient local messaging in a dis- tributed memory program. Our prototype implementation of OHM is written in Java. It re- duces message passing overhead between processes on the same shared-memory node, and it has performance comparable to a multi-threaded version.

1 Introduction

High-speed networking and low cost make distributed-memory clusters very attractive high-per- formance computers. Furthermore, clusters have proven to be more scalable than shared-memory multicomputers. However, small-scale parallelism is quite usable and cost effective. Consequently, networks of shared-memory multiprocessors, with up to eight processors per node, are emerging as the dominant architecture for high-performance computers. This is because in terms of most raw performance metrics, shared-memory clusters are a highly efficient design point. However, this architecture is problematic in terms of software and programming cost, because it has a hybrid memory model consisting of both shared and distributed memory. Compared to other architectures,

∗This research supprted by NSF grants CCR-9876073 and CCR-9733063 and a grant from DARPA HPCS. †Department of Computer Science, North Carolina State University, Raleigh, NC 27965, [email protected] ‡Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, IN 46556, [email protected] §Department of Computer Science, University of Georgia, Athens, GA, 30602, [email protected]

1 a hybrid architecture presents additional challenges for developing efficient programs as well as tuning and porting them. There are three basic programming models for a hybrid machine. The first is shared memory across all processors and nodes with software distributed shared memory (SDSM). The second uses message passing between processors and nodes, and the third uses message passing between nodes and shared memory within nodes. None of these three is ideal; SDSMs are inefficient between nodes and message passing is inefficient within a node. Although hybrid systems can be efficient, the hybrid model requires a user to use two different programming models. This paper addresses the challenge of efficiently programming shared-memory clusters through the Optimized Hybrid Messaging (OHM) system, which provides several optimized local messaging mechanisms that decrease the cost of message passing. Also, it presents results in which the performance of OHM programs nearly equivalent to multi-threaded programs on a single SM node. Many high-performance messaging systems provide optimizations for SM clusters. For exam- ple, the major MPI implementations use efficient IPC for local messaging instead of a network protocol (which demonstrates the importance of SM clusters and utility of efficient local mes- saging). These implementations provide a short cuts for messages that are destined for another process on the same SM node. However, this is partial a solution because, while it increases ef- ficiency, it does not take full advantage of the opportunities available for message passing using shared memory. Our prototype implementation of OHM uses Java primarily because of the ease of prototyping within a Java virtual machine. There are two major components. First, is a message passing class that is implemented as a “drop-in” replacement for Java’s native1 messaging methods. The second component provides and manages shared memory buffers that are used to hold message headers, queues, and data. At first blush, OHM can be viewed as a multiplexer between local and remote messaging methods. If the source and destination are located on the same machine, then OHM uses specialized, highly efficient local message passing. But when the source and destination are located on different machines, then OHM uses the native messaging methods. In this respect, OHM is similar to systems such as LAM MPI. OHM distinguishes itself from these by providing several mechanisms to pass messages for efficiency. The next section discusses general hybrid message passing. Section 3 introduces a prototype OHM system created as Java class. The next section shows that our OHM prototype is significantly faster than other message passing mechanisms and that it provides excellent SM performance. Then the paper presents related work and conclusions.

2 Hybrid Message Passing

In a hybrid computer, there are two types of messages. A remote message is between processes on different nodes, and a local message is between processes on the same node. Typically, a remote mechanism also supports local communication, but not efficiently. For example, a straightforward implementation of message passing in a hybrid computer uses TCP/IP for both local and remote messages. While it is sufficient, this way incurs unnecessary overhead for local message pass- ing. In contrast, a hybrid message passing system uses separate mechanisms for local and remote

1The native package is java.io.net, which supports communication with TCP sockets.

2 messages. A simple hybrid message passing mechanism uses TCP/IP for remote messages and interpro- cess communication (i.e., AF UNIX on Unix machines) for local messages. It is easy to determine whether two communicating processes are local or remote when a connection is being established. Therefore, the proper methods can be dynamically bound to send and receive methods when the connection is created. Thus, the mechanism that establishes a connection between two processes serves as a clearing house for specific, and efficient, implementations of message passing. As a result the inherent overhead of hybrid messaging is negligible. This simple form of hybrid message passing is supported in some MPI implementations.

2.1 Shared Memory Message Passing Messages are typically passed using copy semantics [LH89a]. In copy semantics, the send op- eration copies message data from an application buffer to an intermediate buffer. Similarly, the receive operation copies data from an intermediate buffer to an application buffer. Because source, destination, and intermediate buffers are all independent, copy semantics provides the simplest message passing interface and semantics. But not the most efficient. Other message passing semantics exist, but are not as widespread and are more difficult to use. For example, MPI provides the Isend and Irecv interface, which can be more efficient than copy semantics. However, significant effort may be necessary to transform a program from copy semantics to this more efficient form. In particular, because message data is shared (not copied), the programmer must explicitly verify that a buffer is no longer shared before modifiying it. In MPI, this involves the insertation of Wait statements before the appropriate subsequent uses of message data. Simple hybrid message passing uses generic IPC. While this is more efficient than using TCP/IP, it still has significant overhead. There are two primary sources of overhead: (i) cross- ing the user-kernel boundary and (ii) copying data. This overhead is a consequence of using a kernel-level, general-purpose mechanism. Both of these sources of overhead are eliminated in OHM. Local message passing can be performed very efficiently because memory (and hence message data) can be shared between processes. In the common case, OHM will exchange messages without a system call or other intervention from the kernel. This user-to-user transfer can be performed with very little overhead. Additionally, when the message data originates in shared memory, message passing can often be accomplished without copying buffers. As described above, the standard method of local send and receive involves two copies. The sender copies from a private buffer into a shared buffer, and the receiver copies from the shared buffer into a private buffer. A 1-copy send method allocates a shared memory buffer, then copies message data from a private buffer into it. It then inserts a descriptor into a queue associated with this connection, indicating that the message was sent. Similarly, a 1-copy receive method removes the descriptor from the queue and copies data from the shared buffer into a private buffer. The sending and receiving processes are free to manipulate the private buffers without affecting program correctness. When 0-copy versions of send and receive are available in addition to 1-copy versions, there are four ways to pass messages: two-copy, zero-copy, and two ways with a single copy. A 0-copy

3 send requires that the message data begin in shared memory. A 0-copy receive must be able use a reference to a shared buffer or a private buffer. Additionally, in both cases, the message buffer must be guarded against writing while it is shared. (In this paper, 0-copy and 1-copy refer to the underlying send or receive mechanism, and two-copy, single-copy, and zero-copy refer to the resultant message passing achieved by pairing a send with a receive.) With zero-copy message passing, the message data begins in a shared buffer on the sender side and the receiver accesses the data directly from the same shared-memory buffer. Zero-copy mech- anisms eliminate the copies between shared and private buffers. Of course, appropriate actions must be taken to provide copy semantics with zero-copy message passing. Zero-copy mechanisms are described in the next section. There are two ways to perform single-copy messaging. Sender-copy is a straightforward op- timization on the receiver side (1-copy send and 0-copy receive). The receiver realizes that the sender has already copied the message data into a shared buffer. Because the sender copied the data, it makes no claim on the data; consequently, the receiver is free to use the buffer. Even though the message data is in a shared buffer the receiver is the exclusive owner. The other single-copy mechanism, receiver-copy, occurs when the receiver chooses not to use the shared buffer (0-copy send and 1-copy receive). To the sender, receiver-copy is indistinguish- able from zero-copies, described above. The effect is that the 1-copy receive immediately copies message data and decrements the reference count for the shared buffer.

2.2 0-copy Mechanisms The primary optimization is to avoid the most expensive operation—copying message data— especially for large messages. Copying can be eliminated on the sender or receiver side (or both). OHM provides copy semantics message passing because it is the typical and easier-to-use form of message passing. Furthermore, efficient mechanisms exist to reduce the cost of copy semantics. Therefore, OHM uses such mechanisms, which logically copy message buffers. With zero-copy the buffer is shared. Therefore, a write by one process is visible to the other. This violates copy semantics. The usual mechanism that provides zero-copy efficiency yet retains copy semantics is copy-on-write (COW). When a process tries to write to a shared buffer, it will first copy the buffer, then write to this new buffer to which is has exclusive access. The copy is avoided when either (1) no writing occurs or (2) the writing occurs after the buffer is no longer needed by all other processes. In order to perform zero-copy message passing, the message data must originate in shared memory. This means that the object(s) that makes up the message data is allocated in shared memory and not private memory. In COW, because both sending and receiving processes share the same buffer, writing to it must be guarded in order to preserve copy semantics. The buffer can be guarded explicitly by inserting a check inline before a write to a protected memory buffer. Alternatively, the buffer can be guarded implicitly by using the mprotect system call, which traps when a process writes to protected memory. In both cases, a copy is effected when a process tries to write to a shared buffer.

4 2.3 User-Level Synchronization The second overhead of generic message passing is the cost of a system call. This can be elimi- nated by direct user-to-user synchronization. The expected use of OHM is tightly-coupled, SPMD programs. In such programs, there are no spurious or unanticipated receives. So while data may become available before it is needed, a receive will be issued, usually within a small amount of time. Additionally, the send process rarely gets too far ahead of the receiver; therefore, the maxi- mum number of buffers in a message queue tends to be small ( the maximum is almost always 1). Finally, there is usually one worker process for every processor, which means that a busy wait is not unduely costly. The straightforward implementation of user-level synchronization uses receiver polling. If the message has already been sent, then the receiver will see this and immediately receive the message. In which case, the send and receive were performed without a system call and without delay. When a receive is issued before the corresponding send has been completed, the receive must block. The process can block using a system-provided mechanism, or it can delay in user code. The former is heavy weight and expensive. The latter is inexpensive provided the wait interval is relatively short. An efficient solution for this type of environment is to block (e.g., busy wait) at the user-level for a small period of time before employing system-level blocking (e.g., semaphore). Short delays are expected to be the overwhelmingly common case. A sender must obtain a free element in the message queue before sending. Consequently, unless message queues are unbounded, a sender will have to block if it gets too far ahead of the receiver. Because this case is not likely given the anticipated environment, a heavy-weight system mechanism seems most appropriate.

2.4 Discussion There are tradeoffs between different message passing mechanisms. As usual, these tradeoffs are in terms of time and space. In large messages, the time is dominated by the cost of copying message data. Therefore, as a message gets larger the two-copy mechanism becomes costly. A one-copy mechanism obviously suffers less from this cost, but it still quickly becomes inefficient. Therefore, for large messages, a zero-copy is paramount. The other operations needed for sending and receiving that do not grow with message size. These fixed costs are least in the two-copy case and greatest in the zero-copy case. The reduced copy mechanisms must perform additional operations, such as guarding the buffer and setting reference counters, in order to provide copy semantics. Additionally, there are hidden costs with reduced copy mechanisms. First, the object that is the message data must be in shared memory before the send. Therefore, the sender must build it in shared memory. In the general case, this may not be very difficult (short of allocating all objects in shard memory). Moreover, allocating from shared memory is more costly than from private, and it increases the demand on shared memory. A second hidden cost arises in cases where there is significant interference between sender and receiver resulting in many buffers being copied on write. In such a situation, the lazy copy on write is more expensive than an eager copy on send (1-copy send). The space tradeoff is straightforward. Each copy requires an additional buffer. Thus the two- copy mechanism (which requires three buffers) has the greatest space demand. However, there

5 are two areas of memory in OHM: private and shared. Shared memory has a higher cost than private because a lock must be acquired and buffers might need to be guarded. Because the two- copy mechanism only uses a shared buffer for intermediate storage (with at short lifetime), it has little shared memory pressure. This re-enforces the general understanding about message passing: Avoid copying big messages. The final tradeoff is source program complexity. Copy semantics is trivial to support with two- copy. Using an implicit COW mechanism, it can be supported with any reduced copy mechanism but requires OS support. The explicit COW mechanism (which inserts checks into the source code) is much cheaper than implicit. However, it cannot always be done and requires analysis of the source code. Another part of our research that is beyond the scope of this paper is an analyzer and transformer that modifies Java bytecode. It automatically creates more efficient hybrid message passing programs from a standard message passing prgoram. Although the prototype described in this paper can be used stand alone, it was created to be a backend for this transformer.

3 OHM Implementation

This section describes a prototype implementation of Optimized Hybrid Messaging (OHM) in Java. We create a new Java class, Hybrid, that is a drop-in replacement for the native Java com- munications class, java.io.net. Thus converting from a native Java program to a Hybrid program can be as trivial as modifying one import statement. While this will create a hybrid program that is more efficient, OHM has more to offer—OHM is intended to be the target of a compiler or program optimizer. Therefore, the Hybrid interface has additional methods. This research uses Java for several reasons, which are explained below. But before discussing the implementation, we first describe some important characteristics of Java. As Java is an object- oriented language, a class is unit of encapsulation in Java. Therefore, is it natural to implement our prototype as a class. Hybrid consists of both Java and C code. The C code is the lower-level code that interfaces the directly with the operating system. This C code is incorporated into the Hybrid class (and seamlessly merged with the Java code) using JNI, the Java Native Interface [JNI]. Java source is compiled into bytecode, which is then executed by a Java VM (virtual machine). An OHM program creates a Java VM (JVM) for each processor; therefore, a multiprocessor node will have more than one JVM executing concurrently. The Hybrid class provides efficient mes- saging between these processes, executing a distributed-memory program on a shared-memory node. OHM creates a section of the virtual memory in each JVM that is shared by all processes. This is contrasted with a shared-memory program that uses a multi-threaded JVM. In the latter case, there is one JVM process with multiple threads that communicate using the memory of the process—which is shared among the threads. The JVM for OHM is slightly modified to provide shared-memory. Each JVM maps the same shared-memory heap into its address space. This heap contains all the data associated with the local message passing mechanisms. This includes message queues (for passing messages) and message buffers. Our prototype is implemented in Java 1.2.2. The Hybrid class has been ported to Linux 2.4 and Solaris (SunOS) 5.8. Because Java is (mostly) architecture and OS independent there are no changes in the Java code. At present there are no differences in the C code because the operating

6 systems are both Unix-based. However, it is expected that the minor differences in the implemen- tations of sockets between Linux and Solaris will change this.

3.1 Shared-Memory Message Passing Hybrid provides streaming, point-to-point communication, just as Java sockets. Hybrid communi- cation consists of two elements: queues and buffers. Each bi-directional message channel between two processes has a pair of queues—one for each direction of communication. A queue holds message headers that include a reference to a buffer holding the message data. In this way a send inserts into the queue and a receive removes from it. A queue in Hybrid is a fixed size array of message headers. We use a fixed size for speed and simplicity. The downside is an increased potential of blocking a sender when the queue is full. In theory there could be such blocking, but in practice a reasonable SPMD program will never have a large number of pending messages in a queue. Because there is a queue for each direction, for each queue there is exactly one process that inserts all messages and exactly one process that removes all messages. This allows for highly efficient synchronization. Each queue has a head and a tail index. The sender increments the tail index (modulo the queue size) when it inserts. Because the receiver only ever reads this index, there is no need for the sender to acquire a lock before incrementing. Similarly, the receiver exclusively modifies the head index. Consequently, in the common case, messages are passed between processes without a system call. A send blocks when the queue is full and a receive blocks when it is empty. The queue is empty when the head equals the tail; it is full when the head equals one more than the tail (modulo queue length). Neither situation should be considered bad. A sender that gets too far ahead likely needs to be throttled back. Our prototype uses busy-wait blocking between polling. Each time a process detects a blocking situation, it waits a small amount of time before re-checking. For an SPMD program with one process per processor, there is essentially no benefit in a blocked process yielding the processor because there is no other application thread that can run. Therefore, this implementation will continue to wait and poll until the blocking condition is eliminated. The second element in OHM is a shared-memory message buffer, which is a chunk of dynam- ically allocated shared memory. OHM creates one section of shared memory per node that is used by all JVMs running on that node. The alternative is to create distinct shared memories between each pair of JVMs. The advantage of the alternative is that interference is reduced as only two 1 processes ever access this memory. The disadvantage is that it creates 2 p(p − 1) distinct shared memories, where p is the number of processes, which greatly complicates the implementation and increases copying because buffers cannot be shared between more than two processes. Our imple- mentation trades off (potentially) increased interference for simplicity and (potentially) decreased copying. OHM uses a two-level free list to manage the allocation of shared memory. There is one global free list per node and a local free list for each process. Processes acquire large chunks of shared memory from the global free list. Individual buffers are allocated out of the local free lists. Most allocations and deallocations are done to the local free lists, which do not require locking. There are very few expensive global operations, which require locking. Locking of the global free list is

7 done using semaphores provided by the operating system. In the program tested in this paper the global free list is accessed only during the initialization of the program. This is because the programs have “balanced” communication. That is they send and receive equal amounts of data. Therefore, in the steady-state situation shared memory buffers will be enqueued to and dequeued from local free lists. Such balance is not uncommon in SPMD programs. But in unbalanced programs, the global free list will be accessed throughout the program’s lifetime. In such a program, the overhead of managing shared memory buffer will be greater than experienced in the programs described in this paper. However, this cost will be low because most operations will be to the local free list. The native Java classes that establish streaming communication are Socket and ServerSocket. A matching pair of instances of these objects are mated to form a TCP channel. The Hybrid implementation determines the destination of the process on the other end of the channel. If the process is on a different node, then a native Java channel is created in which the approriate methods are dynamically bound in this object. Otherwise, a Hybrid channel is created with the appropriate method bindings. In Hybrid, a communication channel consists of a pair of queues, one each for incoming and outgoing messages.

3.2 Message Passing Mechanisms OHM provides both one- and zero-copy send and receive mechanisms. The 1-copy send method copies a message into a shared buffer. The method prototype and its basic use is shown below.

Socket S;