A Survey of Process Migration Mechanisms Jonathan M. Smith Computer Science Department Columbia University New York, NY 10027 ABSTRACT We define process migration as the transfer of a sufficient amount of a process's state from one machine to another for the process to execute on the target machine. This paper surveys proposed and implemented mechanisms for process migration. We pay particular attention to the designer's goals, such as performance, load-balancing, and relia- bility. The effect of operating system design upon the ease of implementation is discussed in some detail; we conclude that message-passing systems simplify designs for migration. 1. Introduction TIME Source Destination An image is a description of a computation which can be executed by a computer. A process is an image in some state of execution. At any given time, the state of the process can be represented as two com- ponents: the initial state (the image ) and the changes ~ ~State which have occurred due to execution. We note that transfer-.. the complete state of a computation may include infor- mation which is inaccessible, for example data kept in tables internal to the operating system. 1 Process migration is the transfer of some (signi- ficant) subset of this information to another location, so Figure 2: Flow of Execution of Migrating Process that an ongoing computation can be correctly contin- ued. This is illustrated in Figure 1: Note that we have illustrated the case where the process does not begin execution on the destination host until all of its state is transferred; in fact, it is pos- Source state J Destination sible for the "migrating" process to begin execution on System transfer - [ System the destination almost immediately. This could be done by transferring relevant registers and setting up an address space; the remainder of the state would be Figure 1: Process Migration sent later or demand-paged as needed. The advantage The flow of execution of the process which is of complete state transfer is that the source machine being transferred is illustrated in Figure 2: can release resources immediately upon completion of the transfer. We can define constraints on the computation so that the subset transferred is sufficient to provide correct computation with respect to the constraints. 28 If we transfer the state of a process from one asymptotes which define the window of machine to another, we have migrated the process. opportunity for balancing, and they point out Process migration is most interesting in systems where that the more communication necessary for the involved processors do not share main memory, as the distributed system to perform the balanc- otherwise the state transfer is trivial. A typical ing, the worse the performance becomes. environment where process migration is interesting is Several algorithms to performing the load autonomous computers connected by a network. balancing have been suggested, for example, Process migration is desirable in a distributed the Drafting Algorithm of Ni [Ni1985a] and system for several reasons, among them: the algorithm used in the MOS system2, developed by Livny and Melman • Existing facilities may not provide sufficient [Livny1982a]. Since a small subset of the power; process migration can provide a simple processes running on a multiprocessing sys- solution to the problems that exist as a result tem often account for much of the load, a of the weakness in the facilities. For example, small amount of effort spent off-loading such several existing implementations of distributed processes may yield a big gain in perfor- file systems do not provide transparent access mance. Empirical data gathered by Leland to remote files; that is, the behavior of a and Ott [Leland1986a], and Cabrera remote f'fle can be differentiated from the [Cabrera1986a] strongly support this. behavior of a local f'de - they have different • Whenever the process performs data reduc- semantics. Since the usual semantics are tion 3 on some volume of data larger than the defined on the local processor, moving an process's size, it may be advantageous to active process to the relevant processor pro- move the process to the data. For example, a vides a mechanism by which semantically distributed database system may be asked to correct remote access (e.g., with exclusive- perform a join operation on a remotely- access "lock" mechanisms) can be accom- located dataset of size N. If the join is plished. expected to reduce the data volume by a signi- • Long running processes may need to move in ficant amount and the database is sufficiently order to provide reliability, or perhaps more large, it may be advantageous to move the accurately, fault-tolerance, in the face of a local process object to the remote file object certain class of faults about which advance rather than fetching the remote f'de object for notice can be achieved 1. For example, the processing by the local process object. operating system may deliver to the process a Another example is a process which performs notification that the system is about to shut statistical analysis of a large volume of data. down. In this case, a process which wants to In the case where the process object is moved, continue should either migrate to another pro- our data transfer consists of cessor or ensure that it can be restarted at size(process)+size(processed data); while the some later time on the current processor. data transfer in the case of the remote file • Moving processes may serve as a tool for object being transferred to our site is balancing the load across the processors in a size(remote file). distributed system. This load-balancing can • The resource desired is not remotely accessi- have a great effect on the performance of a ble. This is particularly true of special- system. Eager, Lazowska, and Zahorjan purpose hardware devices. For example, it [Eager1986a] indicate that there is an oppor- may be difficult to provide remote access to tunity for performance increases. They derive facilities to perform Fast Fourier Transforms or Array Processing; or this access may be l.) Surprisingly,there are many such situations. For example, an application which requires a dedicatednode to ensure real- 2.) MOS is mentioned in a later section of this paper, but the time response, such as robot control, may cause a notice to be discussion does not provide details on the load balancing algo- posted to other computations, which can then choose alternate rithm. nodes to continuetheir execution. Anotherexample is present- 3.) That is, it analyzesand reduces the volumeof data by gen- ed above. erating some result. 29 sufficiently slow to prohibit successful accom- the following significant differences: plishment of real-time objectives. • There may be multiple copies of the file The next sections discuss several systems which group, each on different nodes of the system. have been designed to support process migration. • A given copy of thefile group may be incom- While some of the systems are operational and others plete; that is, it may contain only a subset of are only paper designs, the basis for inclusion was an the files contained in the entirety of the file innovative solution to some problem. The implementa- group. tion status is noted in each section. For example, a tape drive attached to a particular 2. LOCUS LOCUS node might be accessed via the name /dev/tapel2 ; the name resolution process (which may The LOCUS operating system [Walker1983a] include several remote references in the course of developed at UCLA provides a facility to migrate traversing the directory tree) would result in a handle processes; the LOCUS developers refer to this facility which would be used in (perhaps remotely) controlling as "network" tasking. This development is described the device. Note that multiple copies of whatever is in Butterfield and Popek [Butterfield1984a]. the result of the resolution process cannot exist in this The LOCUS operating system is based on case, since there is in fact only one copy of the physi- UNIX, and attempts to provide semantics equivalent to cal device available. a single machine UNIX system. Kernel modifications UNIX file system semantics are such that the provide transparent access to remote resources, as well following is true: as enhanced reliability and availability. m Name interpretation is done in order to return The version of UNIX upon which LOCUS is an object which can be used to access the based has very limited interprocess communication named resource, the file descriptor. facilities. The major mechanism is the pipe, a one-way link connecting two processes, a reader and a writer ; -- All access is performed by means of the file the kernel provides synchronization between processes descriptor, e.g. data transfer. by suspending execution of the writer(reader) and -- The name space is distinct from the object awakening the reader(walter) when the pipe is space, in the following sense: full(empty). Pipes have the unfortunate characteristic • objects may have several names, via links that they must connect related processes, where related • objects exist which have no name. An is in the sense of common ancestry in a family tree obvious example is the pipe ; a less obvi- defined by fork() system calls. This deficiency is ous but still common example is provided remedied in other versions of UNIX [Presotto1985a, by this sequence of 2 system calls,.the first Joy1982a]. Other mechanisms for communication of which obtains a descriptor for the file exist, such as signals but these are clumsy, carry little "name" and the second which removes data (about 4 bits per signal), and depend on privilege "name" from the namespace. or inheritance relationships between the communicat- ing processes. fd = open( "name" ); LOCUS addresses the access of remote unlink( "name" ); resources through modification of the file system inter- The descriptor remains valid, thus this a face.