G2-P2P: A Fully Decentralised Fault-Tolerant Cycle-Stealing Framework Richard Mason Wayne Kelly Centre for Information Technology Innovation Queensland University of Technology Brisbane, Australia Email: {r.mason, w.kelly}@qut.edu.au Abstract present in this paper is the first that we know of that attempts to support generic cycle stealing over a fully Existing cycle-stealing frameworks are generally decentralised P2P network. based on simple client-server or hierarchical style ar- The benefits of utilizing a fully decentralised P2P chitectures. G2:P2P moves cycle-stealing into the network are obvious – unlimited scalability and no “pure” peer-to-peer (P2P), or fully decentralised single point of failure; however the challenges to arena, removing the bottleneck and single point of achieving this goal are also great. The basic problem failure that centralised systems suffer from. Addition- is that the peers that make up a cycle stealing net- ally, by utilising direct P2P communication, G2:P2P work may vary considerably during the lifetime of a supports a far broader range of applications than the computation running on the network. In a file sharing master-worker style that most cycle-stealing frame- environment, this is not a major problem; it simply works offer. means that searching for a particular song may tem- G2:P2P moves away from the task based program- porarily return no hits. With a long running parallel ming model typical of cycle-stealing systems to a dis- application, however, the consequences of losing even tributed object abstraction which simplifies commu- part of the applications state in most cases will be nication. It uses a distributed hash table based over- catastrophic – requiring the entire application to be lay network to provide an efficient method of referenc- completely rerun. ing application objects while still allowing volunteer Clearly, if a peer node decides to leave the net- machines to come and go from the network. Most work, its current application state and tasks must be importantly, G2:P2P provides a sophisticated fault taken on by some other node, i.e. they must be able tolerance mechanism to ensure applications execute to migrate. For any message passing style frame- correctly. This mechanism is entirely automated, re- work, whether it be based on communicating pro- quiring no special effort on the application developer’s cesses (such as MPI or PVM), or based on commu- part. The framework is implemented as an extension nicating distributed objects (such as Java RMI (Sun to .NET’s Remoting framework, providing a familiar Microsystems 2004), .NET Remoting (Obermeyer & model for application programmers and an easy up- Hawkins 2001) or Gardens (Roe & Szyperski Febru- grade path for existing .NET sequential applications. ary 1998)), there is the need for some form of re- mote reference or addressing scheme. That is, we Keywords: Pure Peer-to-Peer, P2P, kCycle Stealing, need a way of referring to the entities that we wish Fault Tolerance, .NET to communicate with. In traditional message passing systems, these entities reside permanently on a given 1 Introduction physical node. The remote references include the ad- dress of this physical node, so routing messages to Many cycle-stealing applications and frameworks these entities is relatively trivial. have been developed (SETI@home 1999-2004, Kelly, A number of schemes have been proposed for han- Roe & Sumitomo June 2002, Cappello, Christiansen, dling remote references to entities that may migrate. Ionescu, Neary, Schauser & Wu 1997). These systems The original Gardens system (developed at QUT), for are often loosely referred to as peer-to-peer (P2P) example, required all machines that held remote ref- systems, as compute cycles are contributed by peer erences to be informed whenever a migration took nodes. The peer nodes in these cycle stealing systems place. This required a global synchronization and are, however, almost always coordinated by a dedi- so limited scalability. Other systems, such as Java- cated central server node (or nodes) and support only Party (Philippsen & Zenger 1997) use a forwarding master-worker style programming models, or simple scheme, that is, the remote references are not actu- variations thereof. The central server in such systems ally updated, messages are simply send to the ma- often becomes a bottleneck and represents a single chine where the object used to reside, which is ex- point of failure. pected to forward them to the object’s new home. The idea of fully decentralised P2P networks has This requires nodes that have departed from the P2P been studied for some time, however, it has largely network to continue to participate (albeit, to a lesser been applied to file sharing style applications such extent) by agreeing to forward such messages. Our as the notorious music sharing networks (Kan 2001, approach allows entities to migrate without requir- Sharman Networks 2004). The framework that we ing global synchronization and without requiring de- parted nodes to continue to participate in any way. Copyright c 2005, Australian Computer Society, Inc. This pa- We do this by using a physical location independent per appeared at Australasian Workshop on Grid Computing address scheme, i.e. an addressing scheme that does and e-Research, Newcastle, Australia. Conferences in Research not include the physical address of the entity. and Practice in Information Technology, Vol. 44. Paul Cod- dington and Andrew Wendelborn, Ed. Reproduction for aca- The idea for our addressing scheme comes from demic, not-for profit purposes permitted provided this text is modern P2P networks, such as Pastry (Rowstron included. & Druschel 2001a), Tapestry (Zhao, Huang, Strib- ling, Rhea, Joseph & Kubiatowicz January 2004), is implemented in the form of such a custom trans- Chord (Stoica, Morris, Karger, Kaashoek & port channel. Application programmers can there- Balakrishnan 2001) and CAN (Ratnasamy, Francis, fore implement parallel applications using our G2- Handley, Karp & Shenker 2000), that implement a P2P framework in much the same way they would fully decentralised distributed hash table (DHT). Pas- any other .NET Remoting application. By exposing try and Tapestry assign a random n-bit identifier to our framework via such a “COTS” interface, together each physical node currently participating in the net- with .NETs support for a wide range of “COTS” pro- work. Data to be stored in the hash table in then gramming languages makes G2-P2P widely accessible hashed and stored on the physical node whose ID and simple to use. is numerically closest (modulo 2n). The beauty of .NET Remoting channels are generally responsi- this approach is that as nodes come and go from the ble for routing messages to remote objects based on network, the hash table entries will be automatically the object’s URI. These URIs are assigned when an mapped (migrated) to new physical nodes. The refer- object is created and generally consist of: a proto- ences to these entries (i.e. the keys) don’t need to be col, a machine address and the ID of the object. In updated. The major technical innovation is the use our case, we define a custom protocol G2P2P. An ex- of a fully decentralised routing algorithm that guar- act machine address is not given as the object may be antees the delivery of lookup messages to the appro- held on different physical machines throughout its life priate physical node in at most O(log N) hops, where time. We instead identify only a logical network name N is the number of physical nodes currently partic- together with a GUID to identify the object. The ipating in the network. Our approach mirrors that network name doesnt refer to the address of an actual of Pastry and Tapestry, but we could equally have machine, but rather a changing collection of machines chosen to follow other modern DHT implementations that are working together to form a P2P network. In- such as Chord and CAN which achieve the same effect cluding such a network name in the URIs allows mul- using slightly different approaches. tiple P2P network instances to coexist. An example For our cycle-stealing framework (G2-P2P) we URI might be g2p2p://plas.qut.edu/fccf281d-47bd- have opted to provide a message passing paradigm 45c7-8f2b-a48d462d171b. Only the network name is in the form of a distributed object model; however, chosen by the application programmer, the GUIDs our basic approach could equally have been applied are auto generated by the G2-P2P framework. to distributed process models such as MPI. We assign .NET Remoting uses a configuration file to list the a 128-bit GUID to each physical node when it joins types of objects that should be created remotely; and the network. Our objects are also assigned and refer- for each of those types, the protocol to use and the enced via a 128-bit GUID. Since we are (randomly) URL of the machine that should host such objects. generating the keys ourselves, there is no need to hash In our case, the URL is the name of a P2P network them; we simply use the object’s GUID directly to instance rather than a specific physical machine. Af- map it to the physical node whose GUID is numeri- ter having read this standard configuration file, the cally closest. The random generation of GUIDs and .NET runtime then knows to convert any attempts associated mapping algorithm will tend to generate to create instances of such types into remote activa- a roughly even number of objects per physical node tions. The end result of such a remote activation is in the same way that traditional hash table buckets a local transparent proxy object (containing the re- tend to have the same number of elements on average. mote object’s URI) which to the client looks exactly In Section 3.5 we describe how this probabilistic load like an instance of the type that it was originally try- balancing approach can be refined.