Enforcing Determinism for the Consistent Replication of Multithreaded CORBA Applications P. Narasimhan, L. E. Moser, P. M. Melliar-Smith Department of Electrical and Computer Engineering University of California, Santa Barbara, CA 93106 [email protected] , [email protected], [email protected] ministic. Unfortunately, many practical CORBA ap- Abstract plications use nondeterministic mechanisms suchaslo- cal timers, pro cessor-sp ec i c functions and, in partic- In CORBA-based applications that depend on object ular, multithreading. replication for fault tolerance, inconsistencies in the states of the replicas of an object can arise when con- The Eternal system [7] addresses this challenge in current threads within those replicas perform updates its provision of fault tolerance for CORBA applica- in di erent orders. By imposing a single logical thread tions, while ensuring lowoverheads, transparency and of control on every replicated multithreaded CORBA interop erability.To ensure strong replica consistency, client or server object, and by providing deterministic the Eternal system exploits reliable totally ordered scheduling of threads and operations across the replicas multicasts for the communication of op erations b e- of each object, the Eternal system achieves consistent tween replicated ob jects, and implements mechanisms object replication. The Eternal system does this trans- for the detection and suppression of duplicate op er- parently, with no modi cation to the application, the ations and the transfer of state to new, recovering ORB, or the concurrency model employed by the ORB. and backup replicas. This pap er fo cuses on the mech- anisms that the Eternal system employs to guaran- tee consistent replication in the face of one sp eci c source of nondeterminism, namely,multithreading in 1 Intro duction the ORB or the application. Distributed ob ject systems, based on standards such as the Ob ject Management Group's Common Ob- 2 Replicating Multithreaded ject Request Broker Architecture CORBA [9], pro- vide applications with valuable features such as Applications language transparency,interop erability and lo cation Many commercial ORBs are multithreaded, and mul- transparency. Unfortunately, most distributed ob ject tithreading can yield substantial p erformance advan- standards, including CORBA, make no provision for tages. Unfortunately, the sp eci cation of multithread- other desirable features, such as fault tolerance, that ing in the CORBA 2.2 standard [9] do es not place any current and future CORBA-based applications will guarantee on the order of op erations dispatched by need. The Ob ject Management Group OMG has rec- amultithreaded ORB. In particular, the sp eci cation ognized the need to provide fault tolerance for CORBA of the Portable Ob ject Adapter POA, whichisakey applications by issuing a Request for Prop osals RFP comp onent of the CORBA standard, provides no guar- [10] for fault-tolerant CORBA. antee ab out how the ORB or the POA dispatches re- A ma jor requirement of the OMG's RFP is that quests across threads. The ORB may dispatch several strong replica consistency must b e maintained as op- requests for the same ob ject within multiple threads erations are p erformed that change the states of the at the same time. replicas. However, strong replica consistency requires In addition to ORB-level threads, the CORBA ap- that the b ehavior of the application ob jects is deter- plication may itself b e multithreaded, with the thread This research has b een supp orted by the Defense Advanced Re- scheduling having b een determined by the application search Pro jects Agency in conjunction with the Oce of Naval programmer. The application programmer must en- Research and the Air Force Research Lab oratory, Rome, under sure correct sequencing of op erations and must prevent Contracts N00174-95-K-0083 and F3602-97-1-0248, resp ectively. the application are lo cated on the same typ e of plat- thread hazards. Careful application programming can form, .e., same vendor's ORB, same op erating sys- ensure thread-safe op erations within a single replica of tem, same typ e of workstation same pro cessing sp eed, an ob ject; however, it do es not guarantee that threads same amount of memory, etc. In addition, we assume and op erations are dispatched in the same order across deterministic b ehavior of the op erating system. Thus, all of the replicas of the ob ject. Making the application any replica inconsistency that arises can b e attributed programmer resp onsible for concurrency control and to the multithreading of the ORB or the application. ordering of dispatched op erations in replicated ob jects is unacceptable for maintaining strong replica consis- 2.1 Active Replication tency in a fault-tolerant system. For active replication, strong replica consistency Several di erent concurrency mo dels [13] are sup- means that, at the end of each op eration invoked on p orted by current commercial ORBs. These include the replicas, each of the replicas of an ob ject have the thread-p er-request, where one thread is spawned for same state. The example shown in Figure 1 illustrates each new invo cation on an ob ject, and thread-p er- the problem of replica inconsistency when the only ob ject, where a single thread executes all invo cations supp ort for consistent replication is that provided for on an ob ject. Most practical CORBA applications single-threaded ORBs. consist of servers that contain multiple ob jects, each In the gure R and R are active replicas of the 1 2 having p ossibly multiple threads. Ob jects that are co- same MT-domain. The ORB at each replica receives lo cated within the same server pro cess may access and messages in the same order, but dispatches two threads up date shared data; thus, irresp ective of the threading T and T to p erform op erations on the replicas. The 1 2 mo del used by the ORB, multiple threads may exist two threads are simultaneously active within each within each server. Because a server may itself assume replica, and can access and up date shared data. Sup- the role of a client, we do not distinguish b etween the p ose that threads T and T issue up date op erations 1 2 problem of multithreading for clients and servers. A and B , resp ectively, on the shared data and that We use the term MT-domain to refer to any op eration B A is executed b efore op eration A B CORBA client or server that supp orts multiple in replica R R . Even if the MT-domain is pro- 1 2 application-level or ORB-level threads that may grammed to avoid race conditions and other thread access shared data and that contains one or more hazards, the order of op erations in the two replicas of CORBA ob jects. The MT-domain abstraction is in- the MT-domain may di er and, thus, their states may dep endent of the concurrency mo del of the ORB, the b e inconsistent at the end of the up date op erations. role of the MT-domain as a client or server, and the Replica inconsistency can also arise when the ORB commercial ORB that hosts the MT-domain. initially dispatches only a single thread, which later In Sections 2.1 and 2.2, we provide examples that il- spawns other threads within the same replica. lustrate how replica inconsistency can arise for active 2.2 Passive Replication and passive replication of MT-domains. We assume that all of the mechanisms that guarantee replica con- For passive primary-backup replication, strong sistency for single-threaded ob jects are present, i.e., replica consistency means that, at the end of each messages are delivered to the ORB in a reliable totally state transfer, each of the replicas of an ob ject have ordered sequence, duplicate op erations are detected the same state. The example shown in Figure 2 illus- and suppressed, and state transfers are provided for trates the problem of replica inconsistency for passive recovering replicas. While these mechanisms suce to replication. guarantee replica consistency, they serve only to facil- In the gure MT-domain C is passively replicated, itate, but not to guarantee, replica consistency when with primary replica C and backup replica C . Ob- 2 1 either the ORB or the application is multithreaded. jects A, B , D and E not shown with which C inter- In particular, reliable totally ordered multicasts en- acts are not necessarily multithreaded or replicated. sure only that the ORBs that host the various replicas The example fo cusses on the passive replication of MT- receive the same messages in the same order. They domain C , and how C 's multithreading results in the do not guarantee that the ORBs will dispatch these inconsistency of its replicas. incoming messages onto the threads of the replicas in The invo cation I I of ob ject A B on MT- AC BC the same order. domain C requires thread T T to b e dispatched. 1 2 We assume further that other sources of nondeter- If thread T T is dispatched, MT-domain C will is- 1 2 minism, e.g., system calls such as lo cal timers that sue the nested invo cation I I to ob ject E D . CE CD return pro cessor-sp eci c information, are handled by Thus, MT-domain C invokes two di erent nested op er- other mechanisms. We also assume that all replicas of ations through its two threads, and must obtain results Actively Replicated Object Replica R1 Replica R2 2 1 1 2 A B A B Thread T1 Thread T2 Thread T1 Thread T2 Shared Data 1 1 Shared Data Dispatch of Dispatch of Thread T 2 Thread T2 1 2 Dispatch of Dispatch of Thread T2 Thread T1 Multithreaded ORB Multithreaded ORB Reliable totally ordered reliable multicast messages Figure 1: Inconsistency with active replication of multithreaded ob jects.