Object Replication in a Distributed System by Mark C Little 1 L.[. UN :i vE:Fs ir'' L i'RY Y' 1 51448 2 1QL LQ2 Ph.D. Thesis September 1991 The University of Newcastle upon Tyne Computing Laboratory Abstract Abstract A number of techniques have been proposed for the construction of fault—tolerant applications. One of these techniques is to replicate vital system resources so that if one copy fails sufficient copies may still remain operational to allow the application to continue to function. Interactions with replicated resources are inherently more complex than non—replicated interactions, and hence some form of replication transparency is necessary. This may be achieved by employing replica consistency protocols to mask replica failures and maintain consistency of state between functioning replicas. To achieve consistency between replicas it is necessary to ensure that all replicas receive the same set of messages in the same order, despite failures at the senders and receivers. This can be accomplished by making use of order preserving reliable communication protocols. However, we shall show how it can be more efficient to use unordered reliable communication and to impose ordering at the application level, by making use of syntactic knowledge of the application. This thesis develops techniques for replicating objects: in general this is harder than replicating data, as objects (which can contain data) can contain calls on other objects. Handling replicated objects is essentially the same as handling replicated computations, and presents more problems than simply replicating data. We shall use the concept of the object to provide transparent replication to users: a user will interact with only a single object interface which hides the fact that the object is actually replicated. The main aspects of the replication scheme presented in this thesis have been fully implemented and tested. This includes the design and implementation of a replicated object invocation protocol and the algorithms which ensure that (replicated) atomic actions can manipulate replicated objects. Acknowledgements Acknowledgements Firstly I would like to thank my supervisor, Professor Santosh Shrivastava, who suggested this area of research and with whom I have had numerous discussions over the years. I would also like to thank Dr. Graham Parrington and Dr. Stuart Wheater for reading and commenting upon the numerous drafts of this thesis. Their efforts are greatly appreciated. I would like to thank my fellow members of the Computing Laboratory, in particular Dan McCue, Dr. Paul Ezhilchelvan and Xavier Rousset, for many useful comments and discussions I have had on this work. Finally I would like to thank my family for their support and encouragement which they gave me during my studies. Without them it would not have been possible. Financial support for much of the work described in this thesis was provided by a Research Studentship from the Science and Engineering Research Council and Esprit Project 2267 (Integrated Systems Architecture). 1 : Introduction. 1 1 .1: Dependability and Fault Tolerance............................ 2 1 .2: Replication............................................... 2 1.2.1: Increased Availability....................................... 3 1.2.2: Increased Performance..................................... 4 1.2.3: Design Diversity........................................... 4 1.2.4: Replica Groups............................................ 4 1.2.5: Replica Consistency Protocols............................... 5 1.2.6: Further Aspects of Replication................................ 6 1 .3: Transparency............................................. 7 1 .4: Contributions of the Thesis.................................. 8 1 .5: Structure of the Thesis..................................... 9 2: Basic Fault-Tolerance Techniques. ...... 10 2.1: Object-Oriented Programming............................... 10 2.2: Atomic Actions............................................ 11 2.2.1: Action Primitives........................................... 12 2.2.2: Failure Atomicity........................................... 13 2.2.3: Concurrency in an object-based system....................... 13 2.2.4: Objects and Actions........................................ 15 2.3: Distributed Objects......................................... 15 2.3.1: Remote Procedure Call..................................... 16 2.3.2: Groups.................................................. 17 2.3.3: Multicast Communication................................... 19 2.4: Summary................................................ 20 3: Principles of Object Replication. •. ...... 22 3.1: Replication and Failure Modes............................... 22 3.1.1: Failure Classification....................................... 22 3.1.2: Fault Classification......................................... 24 3.2: Replication Overview....................................... 24 3.3: Active Replication and Passive Replication...................... 26 3.3.1: Fail-Silent Processors...................................... 26 3.3.2: Passive Replication........................................ 27 3.3.2.1: DetermInism and Message Collation............................... 28 3.3.2.2: Primary Backups............................................... 29 3.3.2.3: Retained Results............................................... 29 3.3.2.4: Failure Detection ............................................... 30 3.3.2.5: Primary Functionality............................................ 30 3.3.3: Active Replication.......................................... 32 3.3.3.1: The State Machine............................................. 33 3.3.3.2: State Machine and Fault-Tolerance................................ 33 3.3.3.3: Operation Semantics . 35 3.3.4: Communications Requirements.............................. 36 3.3.4.1: ActIve Replication.............................................. 36 3.3.4.2: Passive Replication............................................ 37 3.3.5: Using Active Replicated Services............................. 38 3.3.5.1: Increased Performance.......................................... 38 3 .4: Replication and Failure Masking.............................. 39 3.4.1: Active Replication.......................................... 40 3.4.1.1: Permanent Omission Failures..................................... 40 3.4.1.2: Value and Omission Failures..................................... 41 3.4.1.3: Timing Failures................................................ 41 3.4.1.4: Arbitrary Failures............................................... 42 3.4.2: Passive Replication........................................ 43 3.4.2.1: Permanent Omission Failures..................................... 43 3.4.2.2: Other Failures................................................. 43 3 .5: Summary................................................ 44 4: ReplicaGroupCommunication..... 46 4.1: Remote Object Invocation................................... 48 4.1.1: One-to-Many Communication............................... 49 4.1.1.1: Unordered and Unreliable....................................... 49 4.1.1.2: FIFO Muiticast................................................. 50 4.1.1.3: Atomic multicast............................................... 50 4.1.1.4: Causal multicast............................................... 51 4.1.1.5: Totally ordered muiticast......................................... 52 4.2: Multicasts and Latency..................................... 52 4.3: Review of an Existing Multicast Protocol....................... 53 4.3.1: Psync................................................... 54 4.3.1.1: Conversations and Context Graphs................................ 54 4.3.1.2: Dealing with Network and Host Failures............................ 56 4.3.1.3: Total Ordering................................................. 57 4.4: Multicasts and Replication................................... 58 4.5: The rel/REL Family of Multicast Protocols...................... 59 4.5.1: The rel/RELmjc Protocol.........................................60 4.5.1.1: Other Delivery Properties........................................63 4.5.1.2: Protocol Analysis and Performance................................64 4.6: Implementation........................................... 66 4.6.1: rel...................................................... 67 4.6.1.1: Timings...................................................... 68 4.6.2: rel/REL RPC.............................................. 69 4.6.2.1: TImings...................................................... 70 4.7: Enhancements for Replicated Procedure Calls.................. 72 4.7.1: Optimizations to the RPC.................................... 72 4.7.2: Timeouts................................................. 73 4.7.3: The Proposed Solution..................................... 74 4.7.4: Estimation of the timeout period............................... 76 4.7.4.1: Slow Replicas . 77 4.7.4.2: Example Figures............................................... 78 4.7.5: Flow Control Problem...................................... 78 4.7.6: The Proposed Solution..................................... 79 4.8: Overview of Existing Systems................................ 81 4.8.1: One-to-Many Communication............................... 82 4.8.1.1: The V System................................................. 82 4.8.1.2: The Andrew System............................................ 83 4.8.2: Many-to-Many Communication.............................