Operating System Architecture Based on Distributed Objects

Home , Distributed computing

located in different nodes. However, A. Yu. Burtsev, L. B. Ryzhyk, Yu. O. efficient access to resources can oc- Timoshenko cur only locally. E1 enables access locality by making an object physi- OPERATING SYSTEM ARCHITECTURE BASED ON cally distributed, i.e. storing a com- DISTRIBUTED OBJECTS plete or partial copy of its state in several network nodes. The internal organization of an object allows mini- Introduction mizing interactions with remote nodes during execution of operations on ob- Today the computing power of the jects, thus neutralizing the influence networks of workstations has become of communication latencies on overall comparable to that of modern supercom- system performance. puters. The practical implementation Load balancing. The E1 operating of high-performance distributed compu- system supports load balancing, i.e. tations obviously requires a special dynamic distribution of computational software platform. At present, this payload between system nodes. platform is usually implemented by ap- Support for redundancy mechanisms. plication-level parallel programming E1 allows applications to transpar- libraries, providing software develop- ently use distributed algorithms for ers with distributed communication and redundant data storage and execution. synchronization facilities [1,2]. Our Component model support. E1 has a claim is that the problem can be ad- component architecture and supports dressed in full measure by an operat- component-oriented software developing system providing processes with ment paradigm. Both operating system transparent access to all resources of services and application software are the distributed system. Such operating developed within the framework of a system would allow users and applica- single E1 component model. The compo- tion programmers to think of the com- nent model provides the following ser- puter network as a single multiproces- vices: sor with common memory. • global naming service; This paper presents the architec- • protected object interaction fa- tural design of a distributed operat- cilities; ing system aimed to make the computing • late binding mechanism; potential of the network immediately • persistent objects service. available to application programmer. Distributed objects Fundamental Concepts The E1 architecture is based on the The proposed operating system ar- abstraction of distributed object. chitecture, called E1, is based on the Distributed object is an object, glob- following fundamental concepts: ally accessible from all nodes of dis- Single system image support. The tributed system. The important E1 single system image abstraction im- properties like efficient support for plies that all distributed system re- reliable distributed computing and sources are uniformly accessible from persistence are based on the corre- any of its nodes. For users and soft- sponding properties of distributed ob- ware developers such system looks and jects. feels like a single virtual computer. Object is an encapsulated abstrac- Its distributed nature is transparent tion, including state information and to applications. well defined access protocols. Efficient access to resources. In a The E1 distributed object is an ob- distributed environment execution ject having the following additional threads and resources they use can be properties:

1. Accessibility through interfaces. 4. Persistence. Persistence is the A distributed object exposes one or ability of an object to exist for more well defined interfaces, consist- unlimited time, irrespectively of ing of methods. The object can be ac- whether a system functions continu- cessed by means of method invocations ously. The object is at any time ac- only. cessible through its interfaces. 2. Global accessibility. Each object has a unique identifier associated The E1 distributed objects are com- with it. Any other object possessing posed of local objects (Figure 1). A this identifier can invoke the given local object is constrained to one object’s methods from any node of the node of a distributed system. Like system, provided that it has suffi- distributed objects, local objects are accessible through interfaces. cient capabilities. In a trivial case when the distrib- 3. Ability to replicate. The ob- uted object has only one replica, it ject’s state is physically distributed is identified with a single local ob- among its replicas. Object replicas ject, semantics object. Semantics ob- can exist in several nodes and must ject contains the distributed object exist in all nodes where object’s state, exposes the distributed object methods are invoked. Replication is a interfaces and implements its func- process of distribution of object’s tionality. state among its replicas and replica If the distributed object has sev- synchronization, aimed to support in- eral replicas, a copy of the semantics tegrity of the object. object is placed in each node, where

Node 1 Node 2

Semantics Replication Replication Semantics Object Object Object Object

Semantics Replication Object Object

Node 3

Object Interface Figure. 1. Distributed Object

the distributed object is represented. Below we briefly describe several The distributed object integrity is widely used classes of replication al- maintained by replication objects, gorithms. complementing the semantics objects in Client/server replication. Cli- each node. Replication objects imple- ent/server is a trivial replication ment the distributed object replica- strategy. A single copy of the object tion protocol. The distributed object state is maintained by a server rep- interfaces are locally exposed by replication objects. While processing the lica. Other replicas are clients. All distributed object invocation, repli- client invocations are forwarded to cation object can refer to the seman- the server. This strategy is in most tics object to execute necessary op- cases inefficient, since it does not erations over the local object state, provide local access to resources. An- as well as communicate with remote other disadvantage is low reliability replication objects to perform syn- due to centralized access to objects. chronization and remote execution of Master/slave replication is an ex- operations. tension of client/server strategy. Each replica stores a copy of an ob- Local objects ject state. One replica is assigned as primary. Read operations are executed A local object (Figure 2) consists locally in each node. Modifications of interface section and data section. are forwarded to the primary replica, The interface section contains refer- which executes the required operations ences to object’s method tables. The and updates all other replicas. For data section contains object’s private data that represents its state. this purpose the new object state or information about state changes is Class objects broadcasted to all secondary replicas. Active replication. Each replica Classes of local objects in E1 are stores a copy of an object state. Both described by objects of the special read operations and modifications are type – class objects. Encapsulation of performed locally in each node. To en- class properties by objects allows im- sure replica consistency modifications are broadcasted to all replicas. plementing dynamic class loading. Copy invalidation. This strategy Class objects provide the following was proposed by Li [4] as a distrib- functionality: uted shared memory coherence algo- • expose methods for creating and rithm. It provides strong consistency destroying instances of the given of an object, i.e., each read opera- class; tion returns the value written by the • store interface implementations. last modification. Release consistency. This strategy Replication relies on two synchronization primitives: acquire and release. The ac- The E1 distributed object architec- quire operation returns a copy of the ture follows the principle of policy object for exclusive use. Release op- and mechanism separation that was eration finalizes the set of opera- first proposed by the developers of tions on the object. This strategy re- Hydra operating system [3]. The oper- quires the developer to explicitly in- ating system provides the replication dicate the beginning and the end of mechanism while specific replication each critical section. Therefore, it strategies are implemented by replica- is most appropriate for objects, which tion objects. This architecture allows require mandatory access synchroniza- applying to each object the most effi- tion irrespectively of whether they cient replication algorithm that takes are used in local or distributed envi- into account its semantics. ronment. For example, an object repre-

senting a shared memory region can ex- Architectural overview pose acquire and release operations for access synchronization. The E1 operating system consists of Migration. Migration in E1 refers a microkernel and a set of distributed to the transfer of object replica be- objects acting at the user level. The tween nodes. Migration is not an inde- microkernel supports a minimal set of pendent replication strategy. It is primitives that are necessary for op- used in conjunction with other strate- erating system construction, such as: gies to improve the efficiency of ac- address spaces, threads, IPC and in- cess to resources by means of load terrupts dispatching. All operating balancing. system and application functionality is implemented by objects.

Class Object

iclass Object 1 get_interface addref c release create_object Interface Section Interface 1 get_interface addref Data releasc e Section method1

Interface 2 get_interface addref releasc e

Object 2 Interface 3 get_interface Interface Section addref releasc e

Data Section

Interface 1 Implementation get_interface addref relecase method1

Implementation Figure 2. Local Object

Application Domain 1 Application Domain 2

Thread 1 Thread 1 Replication Object 1 Object 2 Object

4 Services

Class Name Load File

Loader Server Balancing System 1

C r

System Domains o s

Memory Manager Thread Manager Object Registry s

d o

Domain Thread m

Replication a i

Object n

Replication

Object C

a l

Memory Memory Memory l Object 1 Object 2 Object 3 Scheduler

Application Level 3 2 Microkernel

Virtual Memory Management Thread Switching IPC

Figure 3. Е1 Architecture gions accessible to threads running Protection domains within the given domain. A common domain must be used by in- All objects in E1 are located tensively communicating objects which within protection domains. Domain jointly implement some functionality. boundaries are shown by dashed lines For instance, each of the three basic E1 subsystems: Memory Manager, Thread on Figure 3. Manager and Object Registry, uses a Protection domain is an object that separate domain. isolates one or more objects from It is necessary to distinguish be- other objects in the system. Domains tween domains as objects and domains implement object protection and ensure as resources. These entities are sub- that objects can communicate only by jects to the following relations: means of method invocations. Implemen- • the domain-object controls the tation of domains varies for different domain-resource; hardware architectures. For instance, • the domain-resource can be ac- on Intel x86 [5] architecture domains cessed through the domain-object; will be implemented as separate ad- • the domain-resource is a part of dress spaces. In contrast, on 64-bit domain-object’s state. processors (MIPS [6], IA-64 [7]) a single address-space memory model will be used; and a domain will be represented by a set of virtual memory re-

Crossdomain calls erties that can’t be implemented in the access control list based systems. Objects located in different do- Least privilege. It is possible to mains interact by means of crossdomain grant to each subject only those capa- calls. Crossdomain call is a protected in- bilities that it requires. Selective Access Right Delegation. vocation of an interface of an object A subject that possesses certain capa- located beyond the boundaries of the bilities is able to selectively dele- calling object’s domain. gate those capabilities to other sub- A sequence of steps involved in jects. crossdomain call is shown by arrows on Rights Transfer Control. A subject Figure 3. The calling object (Object2) should be able to receive additional transfers control to the microkernel authority only if that authority is through a special system call and sup- granted via an explicitly authorized plies the following information: an channel. identifier of the object to be invoked Endogenous Verification. It must be (Object1), required interface and possible to verify from within the method identifiers, and invocation ar- system itself that certain restric- guments. tions on rights transfer and informa- In order to complete the invocation flow are met. tion, the kernel requires the follow- In the E1 access control model ca- ing additional information: pabilities belong to distributed ob- • whether Object2 is authorized to jects. Each capability describes invoke the indicated method of Ob- owner’s rights to invoke certain in- ject1; terfaces of another object. • the identifier of the domain con- In each E1 node Object Registry taining Object1 and the required entry stores objects’ capability lists and point address. enforces access control policy. It ex- This information is stored in a poses take and grant methods that con- special system object – Object Regis- stitute the only way to pass capabili- try. The microkernel interacts with ties between objects. During each Object Registry to obtain the informa- crossdomain call Registry verifies tion about Object1’s interfaces and to whether the calling object possesses a validate the caller’s rights to per- capability to invoke the specified in- form the operation. terface of the target object. If the invoked object does not have Registry itself is a distributed a local replica yet, Registry initi- object. Its replication strategy is ates its replication to the local responsible for maintaining consistent node. information about objects and capa- Finally, the microkernel transfers bilities in different nodes. control to Object1 to execute the An object can obtain a capability call. The return from invocation also for some other object, which does not occurs through kernel. have a replica in the local node. On the first invocation of such non- Access control existent object Registry turns to Global Naming Service to locate target E1 uses a diminished take-grant ac- object’s contact points and initiate cess control model [8] proposed by its replication to the local node. Shapiro as a generalization of the classical take-grant capability model. This model provides a number of prop-

Threads Class Loader

In E1 objects are passive entities, Classes of local objects in E1 are de- i.e. an object does not have any scribed by objects of the special threads permanently associated with type – Class objects. Class object stores interface implementations, as- it. Instead, E1 supports a migrating signed to instances of the given class threads model [9] in which during a upon their creation. All local objects crossdomain call execution of the in the system are created by appropri- calling thread is transferred to the ate Class objects. Before creating an target object. As shown in [10], mi- object in a certain domain, an in- grating threads are more appropriate stance of the corresponding class must for object-oriented environment than be loaded to this domain. The loading traditional static threads. Static of Class objects to domains is per- threads result in significant overhead formed by a special system service – for the systems consisting of inter- Class Loader. acting objects of medium granularity, Conclusions since at least one thread must be associated with each object for dis- The E1 operating system can be used patching incoming messages. as an operating system for clusters of workstations. Currently most of the Memory management computing clusters are using Linux operating system plus PVM parallel pro- The E1 memory management system is gramming library as their software based on the concept of virtual memory platform. Linux was not designed to objects introduced by Mach [11]. All support parallel distributed comput- used virtual memory of a domain con- ing. In contrast, E1 provides the nec- sists of continuous page regions – essary set of primitives on the oper- virtual memory objects. Each memory ating system level rather than on li- object can be mapped to one or more brary level, which will allow to sig- domains in different nodes. Memory ob- nificantly improve performance and re- jects control the mapping of virtual liability of a distributed system. memory pages into external memory. E1 can also be used as an operating Main memory is viewed as a high per- system for distributed data processing formance cache for the data in exter- systems, e.g., distributed CAD sys- nal storage. Associated with each mem- tems. In order to support decentral- ory object is a pager object, support- ized processing of huge amount of ing the exchange of pages between ex- data, these systems require resources ternal and main memory. to be highly available across the net- Efficient distributed shared memory work. E1 encourages developing these support is an important component of systems within the framework of its distributed operating system. In E1 component model. The replication shared memory is implemented by memory mechanism will then enable efficient objects. Memory object is a unit of access to resources in distributed en- memory sharing. Memory coherence algorithm is implemented as memory ob- vironment, and persistence of objects ject’s replication strategy. By using will improve reliability of the system different replication strategies the and provide developers with a simple distributed shared memory parameters, programming model. such as consistency guarantees and ef- In general, the E1 architecture is ficiency of access, can be adjusted to rather universal and can be used in a specific tasks. By default, the copy variety of distributed environments invalidation strategy, providing including even corporate networks of strong consistency, is used. personal computers. E1 can also be used in heterogeneous environments, consisting of high-performance servers sharing their resources with low-end

clients, like personal or portable computers and handhelds.

А.Ю. Бурцев, Л.Б. Рыжик, Ю.А. Тимошенко A.Y. Burtsev, L.B. Ryzhyk, Y.A. Timoshenko АРХИТЕКТУРА ОПЕРАЦИОННОЙ СИСТЕМЫ НА OPERATING SYSTEM ARCHITECTURE BASED ОСНОВЕ РАСПРЕДЕЛЕННЫХ ОБЪЕКТОВ ON DISTRIBUTED OBJECTS Предложена архитектура операционной системы The paper presents architecture of E1 operating E1, управляющей ресурсами ЭВМ, объединенных system, aimed to control resources of a computer сетью, для наиболее полного использования вы- network. The goal of the operating system design is числительного потенциала. Архитектура E1 осно- to use the computing potential of the network in full вана на абстракции распределенного объекта, measure. The E1 architecture is based on the ab- что позволяет реализовать эффективный и надеж- straction of the distributed object, suggested by the ный доступ к ресурсам вычислительного комплек- authors. The distributed objects allow implementing са из всех его узлов, скрывая при этом распреде- efficient and reliable access to all resources of a ленную природу комплекса. distributed system, while hiding their distributed nature.

1. Sunderam V.S PVM: A Framework for Parallel 7. Intel Corporation. IA-64 Architecture Soft- Distributed Computing // Journal of Concur- ware Developer’s Manual. Volume 2: IA-64 rency: Practice and Experience. – 1990. – System Architecture, Revision 1.1. – 2000. December. – P. 315–339. 8. Shapiro J., Smith J., Farber D. EROS: A 2. MPI: A Message-passing interface standard Fast Capability System // 17th ACM // International J. Supercomputing Applica- Symposium on Operating System Principles tions. – 1994. (SOSP’99). – Charleston, USA, 1999. 3. Wulf W.A., Levin R., Harbison S.P. HY- 9. Ford B., Lepreau J. Evolving Mach 3.0 to DRA/C.mmp An Experimental Computer System. use migrating threads // Technical Report – N.-Y.: McGraw-Hill, 1981. UUCS-93-022. – University of Utah, 1993. 4. Li K. Shared Virtual Memory on Loosely 10. Ford B., Lepreau J. Microkernels Should Coupled Multiprocessors: PhD thesis. – Support Passive Objects // Proc. of I- 1986. – Yale. WOOOS’93. 5. Intel Corporation. i486 Processor 11. Tevanian A. Architecture-Independent Virtual Programmer’s Reference Manual. – 1990. Memory Management for Parallel and 6. Kane G., Heinrich J. MIPS RISC Architecture. Distributed Environments: The Mach – New Jersey: Prentice Hall, 1992. Approach: PhD thesis. – Carnegie Mellon University, Department of Computer Science, 1987.