The Design and Implementation of the Reactor An Object-Oriented Framework for Event Demultiplexing (Part 2 of 2)
Douglas C. Schmidt [email protected]
http://www.cs.wustl.edu/schmidt/ Department of Computer Science Washington University, St. Louis 63130
An earlier version of this paper appeared in the September I/O-based and/or timer-based demultiplexing mechanisms. 1993 issue of the C++ Report. The following paragraphs describe the primary features of- fered by the Reactor framework:
1 Introduction Export Uniform Object-Oriented Interfaces: Appli- cations using the Reactor do not access the lower-level This is the second half of the third article in a series that de- I/O demultiplexing system calls directly. Instead, they in- scribes techniques for encapsulating existing operating sys- herit from a common Event Handler abstract base class tem (OS) interprocess communication (IPC) services using to form composite concrete derived classes (as illustrated object-oriented (OO) C++ wrappers. In the ®rst half of this in Figure 1). The Event Handler base class speci®es a article, a client/server application example was presented to uniform interface consisting of virtual methods that handle motivate the utilityof wrappers that encapsulate event demul- (1) synchronous input, output, and exceptions and (2) timer- tiplexing mechanisms [1]. Event demultiplexing is useful for based events. Applications create objects of these derived developing event-driven network servers that receive and classes and register them with instances of the Reactor.
process data arriving from multiple clients simultaneously. Automate Event Handler Dispatching: When events The previous article also examined the advantages and dis- occur, the Reactor automatically invokes the appropri- advantages of several alternative I/O demultiplexing schemes ate virtual method event handler(s) belonging to the pre- such as non-blockingI/O,multipleprocess or thread creation, registered derived class objects. Since C++ objects are reg- and synchronous I/O demultiplexing (via the select and istered with the Reactor (as opposed to stand-alone sub- poll system calls). routines), any context information associated with an object This article focuses on the design and implementation of is retained between invocations of its methods. This is par- an object-oriented framework called the Reactor.The ticularly useful for developing ªstatefulº services that retain Reactor provides a portable interface to an integrated col- information in between client invocations. lection of extensible, reusable, and type-secure C++ classes
Support Transparent Extensibility: The functionality select poll that encapsulate and enhance the and event of both the Reactor and its registered objects may be ex- demultiplexing mechanisms [2]. To help simplify network tended transparently without modifying or recompiling ex- programming, the Reactor combines the demultiplexing isting code. To accomplish this, the Reactor framework of synchronous I/O-based events together with timer-based employs inheritance, dynamic binding, and parameterized events. When these events occur, the Reactor automati- types to decouple the lower-level event demultiplexing and cally dispatches the method(s) of previously registered ob- service dispatching mechanisms from the higher-level appli- jects to perform application-speci®ed services. cation processing policies. Example low-level mechanisms This article is organized as follows: Section 2 describes include (1) detecting events on multiple I/O handles, (2) the primary features offered by the Reactor framework; handling timer expiration, and (3) invoking the appropriate Section 3 outlines the object-oriented design and implemen- method event handler(s) in response to events. Likewise, tation of the framework; Section 4 presents a distributed application-speci®ed policies include (1) connection estab- logging example that demonstrates how the Reactor sim- lishment, (2) data transmission and reception, and (3) pro- pli®es the development of concurrent, event-driven network cessing service requests from other participating hosts. applications; and Section 5 discusses concluding remarks.
Increase Reuse: The Reactor's demultiplexing and dispatching mechanisms may be reused by many network ap- 2 Primary Features of the Reactor plications. By reusing rather than redeveloping these mech- anisms, developers are free to concentrate on higher-level The Reactor provides an object-oriented interface that sim- application-related issues, rather than repeatedly wrestling pli®es the development of distributedapplications that utilize with lower-level event demultiplexing details. In addition,
1 subsequent bug-®xes and enhancements are transparently shared by all applications that utilize the Reactor's com- ponents. Conversely, developers that access select and : Logging : Logging : Logging REGISTERED poll directly must reimplement the same demultiplexing Handler Handler Acceptor OBJECTS LEVEL and dispatching code for every network application. More- LEVEL : Event : Event : Event over, any modi®cations and improvements to this code must Handler Handler Handler APPLICATION be replicated manually in all related applications. APPLICATION
Enhance Type-Security: The Reactor shields applica- 1: handle_input() tion developers from error-prone,low-level details associated with programming existing I/O demultiplexing system calls. :Timer : Handle These details involve setting and clearing bitmasks, handling Table : Signal
LEVEL Queue LEVEL Handlers timeouts and interrupts, and dispatching ªcall-backº meth- FRAMEWORK FRAMEWORK : Reactor ods. In particular, the Reactor eliminates several subtle causes of errors with poll and select that involve the OS EVENT DEMULTIPLEXING INTERFACE
misuse of I/O handles and fd set bitmasks.
Reactor LEVEL Improve Portability: The also shields appli- LEVEL KERNEL cations from differences between select and poll that KERNEL impede portability. As illustrated in Figure 5, the Reactor Figure 1: The Reactor Inheritance Hierarchy exports the same interface to applications, regardless of the underlying event demultiplexingsystem calls. Moreover, the Reactor's object-oriented architecture improves its own in- contains support for multi-threading. In general, a single in- ternal portability. For example, porting the Reactor from stance of the Reactor is active in a thread at any point in a select-based OS platform to a poll-based platform re- time. However, there may be multiple different instances of quired only a few well-de®ned changes to the framework. Reactor objects running in separate threads in a process. The framework provides the necessary synchronization op- In addition to simplifying application development, the erations to prevent race conditions [4]. Reactor also performs its demultiplexing and dispatching tasks ef®ciently. In particular, its event dispatching logic improves upon common techniques that use select di- 3.1 Platform-Independent Class Components rectly. For instance, the select-based Reactor uses The following paragraphs summarize the salient charac- an ACE Handle Set class (described in Section 3.2) that teristics of the three platform-independent classes in the avoids examining fd set bitmasks one bit at a time in many Reactor:theACE Time Value, ACE Timer Queue, circumstances. An article in a future issue of the C++ Report and ACE Event Handler classes: will empirically evaluate the performance of the Reactor and compare it against a non-object-oriented solution written ACE Time Value: This class provides a C++ wrap- in C that accesses I/O demultiplexing system calls directly. per that encapsulates an underlying OS platform's date and time structure (such as the struct timeval data type on UNIX and POSIX platforms). The timeval structure 3 The Object-Oriented Design and Im- contains two ®elds that represent time in terms of seconds and microseconds. However, other OS platforms use differ- plementation of the Reactor ent representations, so the ACE Time Value class abstracts these details to provide a portable C++ interface. This section summarizes the object-oriented design of the The primary methods in the ACE Time Value class are Reactor framework's primary class components, focus- illustrated in Figure 2. The ACE Time Value wrapper ing on the interfaces and strategic design decisions. Where uses operator overloading to simplify time-based compar- appropriate, tactical design decisions and certain implemen- isons within the Reactor. Overloading permits the use of tation details are also discussed. Section 3.1 outlines the OS standard arithmetic syntax for relational expressions involv- platform-independent components; Section 3.2 covers the ing time comparisons. For example, the following code cre- platform-dependent components. ates two ACE Time Value objects constructed by adding The Reactor was originally modeled after a C++ wrap- user-supplied command-line arguments to the current time, per for select called the Dispatcher that is available and then displaying the appropriate ordering relationship be- with the InterViews distribution [3]. The Reactor frame- tween the two objects: work described here includes several additional features. For example, the Reactor operates transparently on top of int main (int argc, char *argv[]) both the System V Release 4 poll interface and select, { if (argc != 3) { which is available on both UNIX and PC platforms (via cerr << "usage: " << argv[0] the WINSOCK API). In addition, the Reactor framework << " time1 time2" << endl;
2 return 1; }
ACE_Time_Value ct = // Time value structure from /usr/include/sys/time.h ACE_OS::gettimeofday (); // struct timeval { long secs; long usecs; }; ACE_Time_Value tv1 = ct + ACE_Time_Value (ACE_OS::atoi (argv[1])); class ACE_Time_Value ACE_Time_Value tv2 = ct + { ACE_Time_Value (ACE_OS::atoi (argv[2])); public: ACE_Time_Value (long sec = 0, long usec = 0); if (tv1 > tv2) ACE_Time_Value (timeval t); cout << "timer 1 is greater" << endl; else if (tv2 > tv1) // Returns sum of two ACE_Time_Values. cout << "timer 2 is greater" << endl; friend ACE_Time_Value operator + else (const ACE_Time_Value &lhs, cout << "timers are equal" << endl; const ACE_Time_Value &rhs); return 0; // Returns difference between two ACE_Time_Values. } friend ACE_Time_Value operator - (const ACE_Time_Value &lhs, const ACE_Time_Value &rhs); Methods in the ACE Time Value class are implemented to compare ªnormalizedº time quantities. Normalization ad- // Relational and comparison operators for timeval // normalized ACE_Time_Values. justs the two ®elds in a structure to use a canonical friend int operator < encoding scheme that ensures accurate comparisons. For ex- (const ACE_Time_Value &lhs, ample, after normalization, the quantity ACE Time Value const ACE_Time_Value &rhs); // Other relation operators... (1, 1000000) will compare equal to ACE Time Value private: (2). Note that a direct bitwise comparison of the non- // ... normalized class ®elds would not detect this equality. }; Figure 2: Interface for the ACE Time Value Class
ACE Timer Queue: The Reactor's timer-based mechanisms are used useful for applications that require timer support. For example, WWW servers require watch- dog timers to release resources if clients that connect do not send an HTTP request within a speci®c time interval. Like- class ACE_Timer_Queue wise, certain daemon con®guration frameworks (such as the { public: Service Control Manager in Windows NT) require ACE_Timer_Queue (void); services under their control to periodically report their cur- // True if queue is empty, else false. rent status. These ªheart-beatº messages are used to ensure int is_empty (void) const; that a service has not terminated abnormally. The ACE Timer Queue class provides mechanisms that // Returns earliest time in queue. const ACE_Time_Value &earliest_time (void) const; allow applications to register time-based objects that derive from the ACE Event Handler base class (described in the // Schedule a HANDLER to be dispatched at // the FUTURE_TIME. following bullet). The ACE Timer Queue ensures that the // and at subsequent INTERVALs. handle timeout method in these objects is invoked at int virtual schedule an application-speci®ed time in the future. The methods of (ACE_Event_Handler *handler, const void *arg, the ACE Timer Queue class illustrated in Figure 3 enable const ACE_Time_Value &future_time, applications to schedule, cancel, and invoke the timer objects. const ACE_Time_Value &interval); An application schedules an ACE Event Handler that // Cancel all registered ACE_Event_Handlers will expire after delay amount of time. If it expires // that match the address of HANDLER. then arg is passed in as the value to the event han- int virtual cancel (ACE_Event_Handler *handler); dler's handle timeout callback method. If interval does not equal ACE Time Value::zero it is used to // Cancel the single ACE_Event_Handler matching the // TIMER_ID value (returned from schedule()). reschedule the event handler automatically. The schedule int virtual cancel (int timer_id, method returns a handle to a timer that uniquely identi- const void **arg = 0); ®es this event handler in the timer queue's internal table. // Expire all timers <= EXPIRE_TIME The timer handle can be used by cancel to remove an // (note, this routine must be called manually ACE Event Handler before it expires. If a non-NULL // since it is not invoked asychronously). void virtual expire arg is passed to cancel it is set to the Asynchronous Com- (const ACE_Time_Value &expire_time); pletion Token (ACT) [5] passed in by the application when private: the timer was originally scheduled. This makes it possible to // ... }; free up dynamically allocated ACTs to avoid memory leaks. Figure 3: Interface for the ACE Timer Queue Class By default, the ACE Timer Queue is implemented as a linked list of tuples containing ACE Time Value,
3 ACE Event Handler void *, and * members. These typedef u_long Reactor_Mask; tuples are sorted by the ACE Time Value ®eld in typedef int ACE_HANDLE; ascending order of their ªtime-to-execute.º The class ACE_Event_Handler ACE Event Handler * ®eld is a pointer to the timer { object scheduled to be run at the time speci®ed by the public: ACE Time Value ®eld. The void *®eldisanargument // It is possible to bitwise "or" these values // together to instruct the Reactor to check for supplied when a timer object is originally scheduled. When // multiple I/O activities on a single handle. a timer expires, this argument is automatically passed to the enum { READ_MASK = 01, handle timeout method (described in the following bul- WRITE_MASK = 02, let). EXCEPT_MASK = 04, Each ACE Time Value in the linked list is stored in RWE_MASK = READ_MASK | WRITE_MASK | EXCEPT_MASK }; ªabsoluteº time units (such as those generated by the UNIX gettimeofday system call). However, since virtual meth- // Returns the I/O handle associated with the // derived object (must be supplied by a subclass). ods are used in the class interface, applications can rede®ne virtual ACE_HANDLE get_handle (void) const = 0; the ACE Timer Queue implementation to use alternative data structures such as delta-lists [6] or heaps [7]. Delta- // Called when object is removed from the Reactor. virtual int handle_close (ACE_HANDLE, Reactor_Mask); lists store time in ªrelativeº units represented as offsets or // Called when input becomes available. ªdeltasº from the earliest ACE Time Value at the front of virtual int handle_input (ACE_HANDLE); // Called when output is possible. the list. Heaps, on the other hand, use a ªpartially-ordered, virtual int handle_output (ACE_HANDLE); almost-complete binary treeº instead of a sorted list to reduce // Called when urgent data is available.
the average- and worst-case time complexity for inserting or virtual int handle_except (ACE_HANDLE);
n O n deleting an entry from O to lg . A heap representa- // Called when timer expires (TV stores the tion may be more ef®cient for certain real-time application // current time and ARG is the argument given // when the handler was originally scheduled). usage patterns [7]. virtual int handle_timeout (const ACE_Time_Value &tv, const void *arg = 0);
ACE Event Handler: This abstract base class speci®es an extensible interface used by portions of the Reactor // Called when signal is triggered by OS. class that control and coordinate the automatic dispatch- virtual int handle_signal (int signum); }; ing of I/O and timer mechanisms. The virtual methods Figure 4: Interface for the ACE Event Handler Class in the ACE Event Handler interface are illustrated in Figure 4. The Reactor uses application-de®ned sub- classes of the ACE Event Handler base class to im- // a Logging_Handler. plement its automated, event-driven call-back mechanisms. virtual int handle_input (ACE_HANDLE) { ACE_SOCK_Stream peer_handler; These subclasses may rede®ne certain virtual methods in the ACE Event Handler base class to perform application- this->acceptor_.accept (peer_handler); // Create and activate a Logging_Handler... de®ned processing in response to various types of events. } These events include (1) different types (e.g., ªreading,º // ... ªwriting,º and ªexceptionsº) of synchronous I/O on one or private: more handles and (2) timer expiration. // Passive-mode socket acceptor. An ACE_SOCK_Acceptor acceptor_; object of a subclass derived from ACE Event Handler }; typically supplies an I/O handle.1 For example, the fol- int main (int argc, char *argv[]) lowing Logging Acceptor class fragment encapsulates { // Event demultiplexer. a ªpassive-modeº ACE SOCK Acceptor factory provided ACE_Reactor reactor; by the SOCK SAP socket wrappers [8]. Logging_Acceptor acceptor class Logging_Acceptor : ((ACE_INET_Addr) PORT_NUM); public ACE_Event_Handler { reactor.register_handler public: (&acceptor, ACE_Event_Handler::READ_MASK); Logging_Acceptor (ACE_INET_Addr &addr) : acceptor_ (addr) { /* ... */ } // Loop ``forever'' accepting connections and // and handling logging records. // Double-dispatching hook. for (;;) virtual ACE_HANDLE get_handle (void) const { reactor.handle_events (); return this->acceptor_.get_handle (); /* NOTREACHED */ } }
// Factory that creates and activates Internally, the Reactor::register handler 1I/O handles may be omitted in derived class objects that are purely method retrieves the underlying I/O handle by invoking the timer-based. acceptor object's get handle virtual method. When
4 the Reactor::handle events method is invoked, han- method. Event handlers may also be removed via the dles of all registered objects are aggregated and passed to remove handler method. select (or poll). This OS-level event demultiplexing
Timer-based event handler methods ± call detects the occurrence of I/O-based events on these han- dles. When input events occur or output events become ACE Time Value arguments that are passed to possible, the I/O handles become ªactive.º At this time, the ACE Reactor's schedule timer method are the Reactor noti®es the appropriate derived objects by speci®ed ªrelativeº to the current time. For example, invoking the method(s) that handle the event(s).2 For in- the following code schedules an object to print ªhello stance, in the example above, when a connection request worldº every interval number of seconds, starting arrives the Reactor calls the handle input method of delay seconds into the future: the ACE Acceptor class. This method accepts the new class Hello_World : public ACE_Event_Handler connection and creates a Logging Handler (not shown) { public: that reads all data sent by the client and displays it on the virtual int handle_timeout ( standard output stream. const ACE_Time_Value &tv, const void *arg) { The ACE Timer Queue class described above handles ACE_DEBUG ((LM_DEBUG, "hello world\n")); time-based events. When a timer managed by this queue ex- return 0; pires, the handle timeout method of a previously sched- } // ... uled ACE Event Handler derived object is invoked. This }; method is passed the current time, as well as the void *ar- int main (int argc, char *argv[]) gument passed in when the derived object was originally { scheduled. if (argc != 3) ACE Event Handler ACE_ERROR_RETURN ((LM_ERROR, When any method in an object re- "usage: %s delay interval\n", turns 1, the Reactor automatically invokes that object's argv[0]), -1); handle close method. The handle close method Reactor reactor; may be used to perform any user-de®ned termination ac- Hello_World handler; // timer object. tivities (such as deleting dynamic memory allocated by the object, closing log ®les, etc.). Upon return of this call-back ACE_Time_Value delay = ACE_OS::atoi (argv[1]); ACE_Time_Value interval = ACE_OS::atoi (argv[2]); function, the Reactor removes the associated derived class object from its internal tables. reactor.schedule_timer (&handler, 0, delay, interval);
for (;;) 3.2 Platform-Dependent Class Components reactor.handle_events (); /* NOTREACHED */ The ACE Reactor class provides the central inter- } face for the Reactor framework. On UNIX platforms, However, the default implementation of the underly- this class is the only part of the framework that contains ing ACE Timer Queue stores the values in ªabsoluteº platform-dependent code (the private representation of the time units. That is, it adds the scheduled time to the cur- ACE Time Value class may differ on non-UNIX plat- rent time of day. forms, as well). Since the interface of the ACE Reactor class consists
ACE Reactor: Figure 6 illustrates the primary methods of virtual methods it is straight-forward to extend the in the Reactor class interface, which encapsulates and ex- ACE Reactor's default functionality via inheritance. tends the functionality of select and poll.Thesemeth- For example, modifying the ACE Timer Queue im- ods may be grouped into the following general categories: plementation to use one of the alternative represen- tations described in Section 3.1 requires no visible
Manager methods ± The constructor and open meth- changes to the ACE Reactor's public or private ods create and initialize objects of the ACE Reactor interface. by dynamically allocating various data structures (de-
scribed in Section 3.2.1 and 3.2.2 below). Thedestructor Event-loop methods ± and close methods deallocate these data structures. After registering I/O-based and/or timer-based objects,
I/O-based event handler methods ± Objects that are de- an application enters an event-loop that calls either of the rived from subclasses of the ACE Event Handler two Reactor::handle events methods continu- class may be instantiated and registered with an in- ously. These methods block for an application-speci®ed stance of the Reactor via the register handle time interval awaiting the occurrence of (1) synchronous I/O events on one or more handles and (2) timer-based 2 Note that the activated I/O handle is passed as an argument to the events. As events occur, the ACE Reactor dispatches handle input call-back function (though it is ignored in this case since the acceptor class instance variable keeps encapsulates the underlying the appropriate method(s) of objects that the application handle). registered to handle these events.
5 APPLICATION-DEFINED handle_input() (1) SELECT-BASED CONCRETE DERIVED CLASS handle_output() (2) POLL-BASED get_handle() REACTOR REACTOR handle_input() handle_output() A handle_exception() handle_timeout() EVENT HANDLER handle_close() REGISTERED get_handle() ABSTRACT BASE CLASS REGISTERED OBJECTS OBJECTS APPLICATION LEVEL
EVENT EVENT EVENT EVENT EVENT EVENT EVENT EVENT EVENT EVENT HANDLER HANDLER HANDLER HANDLER HANDLER HANDLER HANDLER HANDLER HANDLER HANDLER
TIME TIMETIME TIME TIMETIME VALUE VALUEVALUE VALUE VALUEVALUE TIMER QUEUE TIMER QUEUE EXCEPTION EVENT HANDLERS EVENT HANDLERS WRITE EVENT HANDLERS POLL POLL POLL POLL POLLFD VECTOR PRI IN IN OUT
LIBRARY LEVEL READ EVENT HANDLERS REACTOR REACTOR
C++ OBJECT OBJECT SELECT() INTERFACE POLL() INTERFACE
OPERATING SYSTEM LEVEL
Figure 5: The Reactor's External Interfaces and Internal Data Structures
The followingparagraphs describe the primary differences 3.2.2 Class Components for the poll-based Reactor between the poll-based and select-based versions of the ACE Reactor. Although the implementations of certain The poll interface is more general than select, allow- methods in the ACE Reactor class differ across OS plat- ing applications to wait for a wider-range of events (such forms, the method names and overall functionality remain as ªpriority-bandº I/O events). Therefore, the poll-based the same. This uniformity stems from the modularity of Reactor implementation shown in Figure 5 (2) is some- the ACE Reactor's design and implementation, which en- what smaller and less complicated than the select-based hances its reuse, portability, and maintainability. version. For example, the poll-based ACE Reactor re- quires neither the three ACE Event Handler * arrays nor the ACE Handle Set class. Instead, a single array of ACE Event Handler pointers and an array of pollfd structures are dynamically allocated and used internally to 3.2.1 Class Components for the select-based Reactor store the registered ACE Event Handler derived class ob- jects. As shown in Figure 5 (1), the select-based implementa- tion of the ACE Reactor contains three dynamically allo- cated ACE Event Handler * arrays. These arrays store pointers to the registered ACE Event Handler objects that process the reading, writing, exception, and/or timer- 4 Using and Evaluating the Reactor based events. The ACE Handle Set class provides an ef®cient C++ wrapper for the underlying fd set bitmask The Reactor framework is intended to simplify the de- data type. An fd set maps the I/O handle name-space onto velopment of distributed applications, particularly network a compact bit-vector representation and provides several op- servers. To illustrate a typical usage of the Reactor,the erations for enabling, disabling, and testing bits correspond- followingsection examines the design and implementationof ing to I/O handles. One or more fd setsispassedtothe the distributed logging application presented in [1]. This sec- select call. The ACE Handle Set class optimizes sev- tion describes the primary C++ class components in the log- eral common fd set operations by (1) using ªfull-wordº ging application, compares the object-oriented Reactor- comparisons to minimize unnecessary bit manipulations and based solution with an earlier version written in C, and dis- (2) caching certain values to avoid recalculating bit-offsets cusses the in¯uence of C++ on both the Reactor framework on each call. and the distributed logging facility.
6 Oct 2929 14:50:1314:48:13 [email protected]@22677@7@client-test::unable [email protected]@38491@7@client::unable to to fork fork in in function function spawn spawn Oct 29 14:50:28 [email protected]@18352@2@drwho::sending request to server bastille class Reactor STORAGE DEVICE { public: P1 CONSOLE CLIENT // = Initialization and termination methods. NAMED PIPE LOGGING P DAEMON enum { DEFAULT_SIZE = FD_SETSIZE }; 2 TCP P3 CONNECTION SERVER LOGGING DAEMON // Initialize a Reactor instance that may CLIENT // contain SIZE entries (RESTART indicates CRIMEE CRIMEEHOST A ZOLAHOST B // to restart system calls after interrupts). Reactor (int size, int restart = 0);
int spawn (void) { virtual int open (int size = DEFAULT_SIZE, if (fork () == -1) SERVER Log_Msg::log (LOG_ERROR, int restart = 0); "unable to fork in function spawn"); PRINTER // Default constructor. Reactor (void);
ONNECTION C NETWORK // Perform cleanup activities to close down // an instance of a REACTOR. TCP void close (void); { P1 CLIENT if (Options::debugging) CLIENT NAMED PIPE ZOLA // = I/O-based event handler methods Log_Msg::log (LOG_DEBUG, LOGGING "sending request to server %s", server_host); P2 DAEMON // Register an ACE_Event_Handler object according Comm_Manager::send (request, len); P3 // to the Reactor_Mask(s) (i.e., "reading," // "writing," and/or "exceptions"). virtual int register_handler (ACE_Event_Handler *, Reactor_Mask); Figure 7: Run-time Activities in the Distributed Logging // Remove the handler associated with the Facility // appropriate Reactor_Mask(s). virtual int remove_handler (ACE_Event_Handler *, Reactor_Mask); 4.1 Distributed Logging Facility Overview // = Timer-based event handler methods The distributed logging facility presented below was orig- // Register a handler to expire at time DELTA. // When DELTA expires the handle_timeout() inally designed for a commercial on-line transaction pro- // method will be called with the current time cessing product. The logger facility uses a client/server ar- // and ARG as parameters. If INTERVAL is > 0 // then the handler is reinvoked periodically chitecture to provide logging services for workstations and // at that INTERVAL. DELTA is interpreted symmetric multi-processors linked across local area and/or // "relative" to the current time of day. wide area networks. The logging facility combines the event virtual void schedule_timer ( ACE_Event_Handler *, demultiplexing and dispatching features of the Reactor to- const void *arg, gether with the object-oriented interface to BSD sockets and const ACE_Time_Value &delta, const ACE_Time_Value &interval = the System V Transport Layer Interface (TLI) provided by ACE_Timer_Queue::zero); the IPC SAP wrapper library (described in [8]). Logging provides an ªappend-onlyº storage service that records diag- // Locate and cancel timer. virtual void cancel_timer (ACE_Event_Handler *); nostic information sent from one or more applications. The primary unit of logging is the record. Incoming records are // = Event-loop methods appended to the end of a log and all other types of write // Block process until I/O events occur or timer access are forbidden. // expires, then dispatch activated handler(s). The distributedlogging facility is comprised of the follow- virtual int handle_events (void); ing three main components depicted in Figure 7: // Perform a timed event-loop that waits up to TV
// time units for events to occur; if no events Application Logging Interface: Application processes
// occur then 0 is returned, otherwise return
P P (e.g., P , , ) running on client hosts use the Log Msg // TV - (actual_time_waited). 1 2 3 virtual int handle_events (ACE_Time_Value &tv); C++ class to generate various types of logging records (such as LOG ERROR and LOG DEBUG). The Log Msg::log private: // ... method provides a printf-style interface. Figure 8 de- }; scribes the priority levels and data format for records ex- Figure 6: Interface for the Reactor Class changed by the application interface and the logging dae- mons. When invoked by an application, the logging interface formats and timestamps these records and writes them to a well-known named pipe (also called a FIFO), where they are
7 consumed by a client logging daemon.
Client Logging Daemon: The client loggingdaemon is a single-threaded, iterative daemon that runs on every host ma- chine participating in the distributed logging service. Each client logging daemon is connected to the read-side of the named pipe used to receive logging records from applica- tions on this machine. Named pipes are used since they are an ef®cient form of localhost-only IPC. In addition, the se- mantics of named pipes in System V Release 4 UNIX have // The following data type indicates the relative been expanded to allow ªpriority-bandº messages that may // priorities of the logging messages, from lowest be received in ªorder-of-importance,º as well as in ªorder- // to highest priority... of-arrivalº (which is still the default behavior) [9]. enum Log_Priority The complete design of the client logging daemon and { // Shutdown the logger. the application logging interface will appear in a subsequent LM_SHUTDOWN = 1, C++ Report article that presents C++ wrappers for several // Messages indicating function-calling sequence ªlocal-hostº IPC mechanisms (such as System V Release 4 LM_TRACE = 2, // Messages that contain information normally FIFOs, STREAM pipes, message queues, and UNIX-domain // use only when debugging a program stream sockets). In general, a client logging daemon con- LM_DEBUG = 3, // Informational messages tinuously receives the logging records in priority order from LM_INFO = 4, applications, converts the multi-byterecord header ®elds into // Conditions that are not error condition network-byte order, and forwards the records to the server // but that may require special handling. LM_NOTICE = 5, logging daemon (which typically runs on a remote host). // Warning messages LM_WARNING = 6,
Server Logging Daemon: The server logging daemon // Initialize the logger LM_STARTUP = 7, is a concurrent daemon that continuously collects, reformats, // Error messages and displays the incoming logging records to various external LM_ERROR = 8, devices. These devices may include printers, persistent stor- // Critical conditions, e.g., hard device errors LM_CRITICAL = 9, age repositories, or logging management consoles. The re- // A condition that should be corrected immediatey, mainder of this article focuses on the server logging daemon. // such as a corrupted system database. LM_ALERT = 10, In addition, several Reactor and IPC SAP mechanisms // A panic condition (broadcast to all users) are also illustrated and described throughout this example. LM_EMERGENCY = 11, // Maximum logging priority + 1 LM_MAX = 12 4.2 The Server Logging Daemon }; struct Log_Record The following section discusses the interface and im- { enum { plementation of the primary classes used to construct the // Maximum number of bytes in logging record server logging daemon. The logging server is a single- MAXLOGMSGLEN = 1024 threaded, concurrent daemon that runs in a single pro- }; cess. Concurrency is provided by having the Reactor // Type of logging record ªtime-sliceº its attention to each active client in a round- Log_Priority type_; // length of the logging record robin fashion. In particular, during every invocation of the long length_; Reactor::handle events method, a single logging // Time logging record generated record is read from each client whose I/O handle became long time_stamp_; // Id of process that generated the record active during this iteration. These logging records are writ- long pid_; ten to the standard output of the server logging daemon. This // Logging record data char rec_data_[MAXLOGMSGLEN]; output may be redirected to various devices such as print- }; ers, persistent storage repositories, or logging management Figure 8: Logging Record Format consoles. In addition to the main driver program shown below in Section 4.2.3, several other C++ class components appear in the logging facility architecture. The class inheritance and parameterization relationships between the various compo- nents are illustrated in Figure 9 using Booch notation [10]. To enhance reuse and extensibility, the component shown in this ®gure are designed to decouple the following aspects of the application architecture:
8 Logging_Handler ACE_SOCK_Stream - ACE_INET_Addr // A template class for handling connection requests from ACE_SOCK_Acceptor // a remote client. ACE_INET_Addr Logging Logging Handler template
RELATED Acceptor Handler ÄACE_Acceptor (void); COMPONENTS CONNECTION private: virtual ACE_HANDLE get_handle (void) const; virtual int handle_input (ACE_HANDLE); PEER PEER ACCEPTOR virtual int handle_close (ACE_HANDLE, STREAM Reactor_Mask); ACE Event Event Handler Handler FRAMEWORK
COMPONENTS // Accept connections. PEER_ACCEPTOR acceptor_;
// Performs event demuxing. Figure 9: Class Components in the Server Logging Daemon ACE_Reactor *reactor_; }; Figure 10: Class Interface for Accepting Connections
Reactor framework components ± The components in Reactor the framework discussed in Section 3 encap- 3 sulate the lowest-level mechanisms for performing the face for the ACE Acceptor class. This class inherits from I/O demultiplexing and event handler dispatching. ACE Event Handler, which enables it to interact with the Reactor framework. In addition, this template class
Connection-related mechanisms ± The components dis- is parameterized by a composite PEER HANDLER subclass cussed in Section 4.2.1 represent a set of generic tem- (which must understand how to perform I/O with clients), plates that provide reusable connection-related mech- a PEER ACCEPTOR class (which must understand how to anisms. In particular, the ACE Acceptor template accept client connections), and PEER ADDR (which is a C++ class is a general-purpose class designed to accept net- wrapper for the appropriate address family). work connections with remote clients. Likewise, the Classes instantiated from the ACE Acceptor template ACE Svc Handler template class is another general- are capable of the following behavior: purpose class designed to send and/or receive data to/from connected clients. 1. Accepting connection requests sent from remote clients;
Application-speci®c services ± The components dis- 2. Dynamically allocating an object of the cussed in Section 4.2.2 represent the application-speci®c PEER HANDLER subclass; portion of the distributed logging facility. In particu- 3. Registering this object with an instance of the lar, the Logging Acceptor class supplies speci®c Reactor. In turn, the PEER HANDLER class must parameterized types to the ACE Acceptor,which know how to process data exchanged with the client. creates a connection handling instantiation that is speci®c for the logging application. Likewise, the Figure 11 depicts the ACE Acceptor class implemen- Logging Handler class is instantiated to providethe tation. When one or more connection requests arrive, the application-speci®c functionality necessary to receive handle input method is automatically dispatched by the and process logging records from remote clients. Reactor. This method behaves as follows. First, it dy- namically creates a separate PEER HANDLER object, which In general, by adopting this highly-decoupledobject-oriented is responsible for processing the logging records received decomposition, the development and maintenance of the from each new client. Next, it accepts an incoming connec- server logging daemon was simpli®ed signi®cantly, com- tion into that object. Finally, it calls the open hook. This pared with the original approach. hook can register the newly created PEER HANDLER object with an instance of the Reactor or can spawn off a separate thread of control, etc. 4.2.1 Connection-related Mechanisms
The ACE Svc Handler Class: This parameterized type provides a generic template for processing data sent from
The ACE Acceptor Class: This parameterized type pro- vides a generic template for a family of classes that standard- clients. In the distributed logging facility, for example, the ize and automate the steps necessary to accept network con- 3This is a simpli®ed version of the ACE Acceptor. For a complete nection requests from clients. Figure 10 illustrates the inter- implementation see [11].
9 // Shorthand names #define PS PEER_STREAM #define SH SVC_HANDLER #define PL PEER_ACCEPTOR // Extract the underlying PS (e.g., for #define PA PEER_ADDR // use by accept()). template
// Performs event demuxing. The Logging Acceptor Class: To implement the dis- ACE_Reactor *reactor_; tributed logging application the Logging Acceptor class }; is instantiated from the generic ACE Acceptor template as Figure 12: Class Interface for Handling Services follows:
10 typedef ACE_Acceptor The PEER STREAM parameter is replaced with the
The Logging Handler Class: This class is created by The following event-loop drives the Reactor-based log- instantiating the ACE Svc Handler class as follows: ging server: class Logging_Handler : int public ACE_Svc_Handler
11 incoming events arrive, the Reactor handles them by au- tomatically dispatching (1) the handle input method of SERVER the Logging Acceptor and Logging Handler class. LOGGING : Reactor : Logging When connection requests arrive from client logging dae- DAEMON Handler mons, the Logging Acceptor::handle input func- tion is invoked. Likewise, when loggingrecords or shutdown : Logging : Logging messages arrive from previously connected client logging Acceptor Handler REGISTERED daemons, the Logging Handler::handle input OBJECTS function is invoked. Figure 7 portrays the entire system during execution. Log- SERVER ging records are generated from the application logging in- terface, forwarded to client logging daemons, transmitted LOGGING across the network to the server logging daemon, and ®nally RECORDS LOGGING displayed on the server logging console. The logging in- RECORDS formation that is displayed indicates (1) the time the logging CONNECTION record was generated by the application interface, (2) the host REQUEST machine the application was running on, (3) the process iden- NETWORK ti®er of the application, (4) the priority level of the logging CLIENT record, (5) the command-line name (i.e., ªargv[0]º) of the CLIENT CLIENT application, and (6) an arbitrary text string that contains the text of the logging message.
Figure 14: Run-time Con®guration of the Server Logging 4.3 Evaluating Alternative Logging Imple- Daemon mentations
This section compares the object-oriented and non-object- Logging_Acceptor acceptor oriented versions of the distributed logging facility in terms (&reactor, (ACE_INET_Addr) server_port); of several software quality factors such as modularity, exten- sibility, reusability, and portability. reactor.register_handler (&acceptor);
// Loop forever, handling client requests. for (;;) 4.3.1 Non-Object-Oriented Version reactor.handle_events ();
/* NOTREACHED */ The Reactor-based distributedloggingfacility is an object- return 0; } oriented reimplementation of an earlier functionally equiva- lent, non-object-oriented logging facility. The original ver- The main program starts by opening a logging channel sion was developed for a BSD UNIX-based commercial on- that directs any logging records generated by the server to its line transaction processing product. It was initially written in own standard error stream. The example code in Figures 11 C and used BSD sockets and select directly. Later, it was and 13 illustrates how the server uses the application logging ported to a system that provided only the System V-based interface to log its own diagnostic messages locally. Note TLI and poll interfaces. that since this arrangement does not recursively use the server The original C implementation is dif®cult to modify, ex- logging daemon there is no danger causing of an ªin®nite- tend, and port due to (1) tightly-coupledfunctionality and (2) logging-loop.º an excessive use of global variables. For example, the event The server then opens an instance of the Reactor,in- demultiplexing, service dispatching, and event processing stantiates a Logging Acceptor object, and registers this operations in the original version are tightly-coupled with object with the Reactor. Next, the server enters an end- both the acceptance of client connection requests and the re- less loop that blocks in the handle events method un- ception of client logging records. In addition, several global til events are received from client logging daemons. Fig- data structures are used to maintain the relationship between ure 14 illustrates the state of the logging server daemon af- (1) per-client context information (such as the client host- ter two clients have contacted the Reactor and become name and current processing status) and (2) I/O handles that participants in the distributed logging service. As shown identify the appropriate context record. Therefore, any en- in the ®gure, a Logging Handler object has been dy- hancements or modi®cations to the program directly affects namically instantiated and registered for each client. As the existing source code.
12 4.3.2 Object-Oriented Version 4.4 C++ Language Impact
The object-oriented Reactor-based version uses data ab- straction, inheritance, dynamic binding, and templates to (1) Several C++ language features are instrumental to both the minimize the reliance on global variables and (2) decou- design of the Reactor and the distributed logging facility ple the application policies that process incoming connec- that utilizes its functionality. For example, the data hid- tions and data from the lower-level mechanisms that per- ing capabilities provided by C++ classes improves portabil- form demultiplexing and dispatching. The Reactor-based ity by encapsulating and isolating the differences between logging facility contains no global variables. Instead, each select and poll. Likewise, the technique of register- Logging Handler object registered with the Reactor ing C++ class objects (rather that stand-alone subroutines) encapsulates the client address and the underlying I/O handle with the Reactor helps integrate application-speci®c con- used to communicate with clients. text information together with multiple method that access this information. Parameterized types are useful for increas- By decoupling the policies and mechanisms, a number ing the reusability of the ACE Acceptor class by allow- of software quality factors are enhanced. For example, the ing it to be instantiated with PEER HANDLERs other than reusability and extensibility of system components was im- Logging Handler and PEER ACCEPTORs other than proved, which simpli®ed both the initial development effort ACE SOCK Acceptor. In addition, inheritance and dy- and subsequent modi®cations. Since the Reactor frame- namic binding facilitate transparent extensibility by allowing work performs all the lower-level event demultiplexing and developers to enhance the functionality of the Reactor and service dispatching, only a small amount of additional code is its associated applications without modifying existing code. required to implement the server logging daemon described in Section 4.1. Moreover, the additional code is primar- Dynamic binding is used extensively in the Reactor. ily concerned with application processing activities (such A previous C++ Report article on the IPC SAP wrapper as accepting new connections and receiving client logging [8] discusses why avoiding dynamic binding is often advis- records). In addition, templates help to localize application- able when designing ªthinº C++ wrappers. In particular, speci®c code to within a few, well-de®ned modules. the overhead resulting from indirect virtual table dispatching The separation of policies and mechanisms in the may discourage developers from using the more modular and IPC SAP Reactor's architecture facilitates extensibility and porta- type-secure OO interfaces. However, unlike ,the Reactor bility both ªaboveº and ªbelowº its public interface. For framework provides more than just a thin OO ve- example, extending the server logging daemon's function- neer around the underlying OS system calls. Therefore, the ality (e.g., adding an ªauthenticated loggingº features) is signi®cant increase in clarify, extensibility, and modularity straight-forward. Such extensions simply inherit from the compensates for the slight decrease in ef®ciency. Moreover, Reactor ACE Event Handler base class and selectively imple- the is typically used to develop distributed sys- ment the necessary virtual method(s). Likewise, by in- tems. A careful examination of the major sources of over- stantiating the ACE Acceptor and ACE Svc Handler head in distributed systems reveals that most performance templates, subsequent applications may be produced with- bottlenecks result from activities such as caching, latency, out redeveloping existing infrastructure. On the other hand, network/host interface hardware, presentation-level format- modifying the original non-object-oriented C version in this ting, memory-to-memory copying, and process management manner, however, would require direct changes to the exist- [12]. Therefore, the additional memory reference overhead ing code. caused by dynamic binding is insigni®cant in comparison [13]. It is also possible to modify the Reactor's underlying I/O demultiplexingmechanism without affecting existing ap- To justify these claims empirically, an upcoming article plication code. For example, porting the Reactor-based in the C++ Report will present the results of a benchmark- distributed logging facility from a BSD platform to a Sys- ing experiment that measures the performance of IPC SAP tem V platform requires no visible changes to application and the Reactor. These performance results are based code. On the other hand, porting the original C version upon a distributed system benchmarking tool that measures of the distributed logging facility from sockets/select to client/server performance in a distributed environment. This TLI/poll was tedious and time consuming. It also intro- tool also indicates the overhead of using C++ wrappers for duced several subtle errors into the source code that did not IPC mechanisms. In particular, there are two functionally manifest themselves until run-time. Furthermore, in certain equivalent versions of the benchmarking tool: (1) an object- communication-intensive applications, data is always avail- oriented version that uses the IPC SAP and Reactor C++ able immediately on one or more handles. Therefore, polling wrappers and (2) a non-object-oriented version written in thesehandles vianon-blocking I/O may bemoreef®cient than C that uses the sockets, select,andpoll system calls using select or poll. As before, extending the Reactor directly. The experiment measures the performance of the to supportthis alternativedemultiplexingimplementationnot object-oriented implementation and the non-object-oriented modify its public interface. implementation in a controlled manner.
13 5 Concluding Remarks [10] G. Booch, Object Oriented Analysis and Design with Ap- plications (2 nd Edition). Redwood City, California: Ben- The Reactor is a object-oriented framework that simpli®es jamin/Cummings, 1993. the development of concurrent, event-driven distributed sys- [11] D. C. Schmidt, ªAcceptor and Connector: Design Patterns tems by making it easier to write correct, compact, portable, for Initializing Communication Services,º in Proceedings of the 1 st European Pattern Languages of Programming Con- and ef®cient applications. It accomplishes this by encapsu- ference, July 1996. lating existing operating system demultiplexing mechanisms [12] D. C. Schmidt and T. Suda, ªTransport System Architecture within an object-oriented C++ interface. In general, by sepa- Services for High-Performance Communications Systems,º rating policies and mechanisms,theReactor supports reuse IEEE Journal on Selected Areas in Communication, vol. 11, of existing system components, improves portability, and pp. 489±506, May 1993. provides transparent extensibility. [13] A. Koenig, ªWhen Not to Use Virtual Functions,º C++ Jour- One disadvantage with the Reactor-based approach is nal, vol. 2, no. 2, 1992. that it is somewhat dif®cult at ®rst to conceptualize where an application's main thread of control occurs. This is a typical problem with event-loop-driven dispatchers such as the Reactor or the higher-level X-windows toolkits. How- ever, theconfusion surrounding this ªindirect event-callbackº dispatching model typically disappears quickly after writing several applications that use this approach. The source code and documentation for the Reactor and IPC SAP C++ wrappers is available online at
http://www.cs.wustl.edu/schmidt/ACE.html. Also included with this release are a suite of test programs and examples, as well as many other C++ wrappers that encap- sulate named pipes, STREAM pipes, mmap, and the System V IPC mechanisms (i.e., message queues, shared memory, and semaphores). Upcoming articles in the C++ Report will describe the design and implementation of these wrappers.
References
[1] D. C. Schmidt, ªThe Reactor: An Object-Oriented Interface for Event-Driven UNIX I/O Multiplexing (Part 1 of 2),º C++ Report, vol. 5, February 1993. [2] W. R. Stevens, Advanced Programming in the UNIX Environ- ment. Reading, Massachusetts: Addison Wesley, 1992. [3] M. A. Linton and P. R. Calder, ªThe Design and Implemen- tation of InterViews,º in Proceedings of the USENIX C++ Workshop, November 1987. [4] D. C. Schmidt, ªTransparently Parameterizing Synchroniza- tion Mechanisms into a Concurrent Distributed Application,º C++ Report, vol. 6, July/August 1994. [5] T. H. Harrison, D. C. Schmidt, and I. Pyarali, ªAsynchronous Completion Token: an Object Behavioral Pattern for Ef®cient Asynchronous Event Handling,º in Proceedings of the 3rd Annual Conference on the Pattern Languages of Programs, (Monticello, Illinois), pp. 1±7, September 1996. [6] D.E.ComerandD.L.Stevens,Internetworking with TCP/IP Vol II: Design, Implementation, and Internals. Englewood Cliffs, NJ: Prentice Hall, 1991. [7] R. E. Barkley and T. P. Lee, ªA Heap-Based Callout Imple- mentation to Meet Real-Time Needs,º in Proceedings of the USENIX Summer Conference,pp. 213±222, USENIX Associ- ation, June 1988. [8]D.C.Schmidt,ªIPCSAP: An Object-Oriented Interface to Interprocess Communication Services,º C++ Report, vol. 4, November/December 1992. [9] UNIX Software Operations, UNIX System V Release 4 Pro- grammer's Guide: STREAMS. Prentice Hall, 1990.
14