The Design of the EMPS Multiprocessor Executive for Distributed Computing
Total Page：16
File Type：pdf, Size：1020Kb
The design of the EMPS multiprocessor executive for distributed computing Citation for published version (APA): van Dijk, G. J. W. (1993). The design of the EMPS multiprocessor executive for distributed computing. Technische Universiteit Eindhoven. https://doi.org/10.6100/IR393185 DOI: 10.6100/IR393185 Document status and date: Published: 01/01/1993 Document Version: Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers) Please check the document version of this publication: • A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal. If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement: www.tue.nl/taverne Take down policy If you believe that this document breaches copyright please contact us at: [email protected] providing details and we will investigate your claim. Download date: 03. Oct. 2021 The Design of the EMPS Multiprocessor Executive for Distributed Computing The Design of the EMPS Multiprocessor Executive for Distributed Computing PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Technische Universiteit Eindhoven, op gezag van de Rector Magnificus, prof. dr. J.H. van Lint, voor een commissie aangewezen door het College van Dekanen in het openbaar te verdedigen op dinsdag 16 maart 1993 om 16.00 uur door GERARDUS JOHANNES WICHARDUS VAN DIJK geboren te Nijmegen druk: wibra dissertatiecJrukkerij. halmond. Dit proefschrift is goedgekeurd door de promotoren prof. dr. dipl. ing. D.K. Hammer en prof. dr. S.J. Mullender copromotor: dr. ir. K. Kopinga Contents 1 Introduction 1 1.1 Application areas ...................... .. 1 1.1.1 Real-time data-processing and experiment control . 1 1.1.2 Dependable Distributed Computing 4 1.2 Distributed operating systems 7 1.3 Design philosophy . 10 1.4 Development stages ...... 12 1.5 Overview............ 12 2 Design of the Hardware Architecture 15 2.1 Parallel processing architectures 16 2.1.1 Organization of system memory 17 2.1.2 Interconnection network .... 18 2.1.2.1 Hierarchical duster systems 19 2.2 EMPS Hardware architecture 20 2.2.1 The computer module 22 2.2.2 The memory module. 25 2.2.3 The system controller 26 2.2.4 The local area network controller . 26 2.2.5 Inter-node communication 26 3 Design of the Software Architecture 27 3.1 The EMPSNudeus ...... 30 3.1.1 Memory management. 32 3.1.2 Process management. .. 35 3.1.2.1 Process address 35 3.1.2.2 Process states . 36 3.1.2.3 Process switching 37 3.1.2.4 Tasks and processes 40 3.1.2.5 Memory management. 41 ii CONTENTS 3.1.3 Process synchronization. 45 3.1.4 Tune management .... 46 3.1.5 Interrupt handling .... 48 3.1.6 Interprocess communication (IPC) 50 3.1.6.1 Mailbox communication 52 3.1.6.2 Mailbox address . 54 3.1.6.3 Mailbox. 54 3.1.6.4 Port..... 56 3.2 The EMPS Kernel. ...... 58 3.2.1 Process management. 61 3.2.2 Mailbox management 62 3.2.3 Device management . 63 3.3 The EMPS operating system. 66 3.3.1 File access. 66 3.4 System initialization ..... 68 4 Implementation of Interprocess Communication (IPC) 69 4.1 Service layer ..... .... 70 4.1.1 Service layer interface 72 4.1.1.1 Service data 72 4.1.1.2 Services... 73 4.1.2 Service layer implementation. 74 4.1.2.1 Messages....... 74 4.1.2.2 Protocol for transparent mailbox access 76 4.1.2.3 Mailboxes 78 4.2 MAC layer .......... 81 4.2.1 MAC layer interface . 81 4.2.1.1 Service data 81 4.2.1.2 Services... 82 4.2.2 MAC layer implementation . 82 4.2.2.1 Protocols for datagram communication 83 4.3 Example of communication 87 4.4 Implementation issues . 89 4.5 Performance evaluation 89 5 Efficient Process Migration 95 5.1 Related work ... 96 5.2 Process migration ... 98 5.2.1 Interface..... 98 5.2.2 Implementation 99 5.2.2.1 Process context. 100 5.2.2.2 Additional mailbox services 100 CONTENTS iii 5.2.2.3 Implementationissues 103 5.2.3 Example of process migration. 103 5.3 Performance evaluation 105 5.4 Discussion . 109 6 Summary 111 A Graphical representation 115 B Used class interfaces 119 B.1 Tables. 119 B.2 Queues 120 B.3 Arrays. 120 B.4 Lists .. 121 C Software architecture 123 C1 Implementationof EMPS nucleus classes 123 C1.1 Low-level memory management . 123 C1.2 Process management. ...... 124 C1.2.1 Memory management. 125 C1.3 Process synchronization. 125 C1.4 Tune management . 126 C1.5 Interrupt handling . 127 C1.6 IPC . 128 C2 Implementation of EMPS kernel classes 129 C2.1 Process management. 129 C2.2 Mailbox management ..... 132 C2.3 Device management ...... 138 D Interprocess communication architecture 139 D.1 Implementation of the service layer 139 D.1.1 Service data . 141 D.1.2 Services ........... 143 D.1.3 Mailboxes.......... 145 D.2 Implementation of the MAC layer 146 D.2.1 Service data . 146 D.2.2 Services . 147 D.2.2.1 Intra-duster protocol 149 D.2.2.2 Inter-node protocol . 150 1 Introduction 1.1 Application areas The design of the EMPS (Eindhoven Multi-ProcessorSystem) executive for distributed computing was triggered by two different application areas, viz. real-time data-processing and control of physics experiments, and dependable distributed computing. 1.1.1 Real-time data-processing and experiment control In 1979, the Department of Physics of the Eindhoven University of Tech nology initiated the standardization of hardware and software for data acquisition and control of physics experiments. This resulted in the design of a general-purpose Physics Data Acquisition System (PhyDAS) and an interpretative development environment EPEP (Eindhoven Program Edi tor and Processor). The goal in the overall design of PhyDAS is to provide a general purpose real-time measurement system that can be used as a framework for intelligent, state-of-the-art interface modules, e.g. analog-to-digital converters (ADC), digital-to-analogconverters (DAC), scalers (SCA), pre set scalers (PSC), multi channel analyzers (MCA), and transient recorders (TRC) (See Fig. 1.1). PhyDAS has a separate measuring bus (PhyBUS) that interconnects the interface modules. By separating the computer bus and measuring bus, the interface modules are independent of the type of computer that is used in the experiment, and the real-time behavior of the system is more predictable. 1 2 Introduction file server other nodes SYSTEM BUS PHYBUs PHYPAD Figure 1.1: Typical EMPS-PhyDAS configuration. The EMPS system consists of several computer modules (C) and shared memory modules (M), interconnected by a cluster bus. A system bus interconnects several clusters into a node. Each node has a system controller (SC), an inter-node communication module (IN), and a local area network controller (LAN). A detailed description of these modules is presented in Chapter 2. A converter (CNV) interconnects the system bus of the EMPS system and the PhyBUS of the PhyDAS system. Goals for the design of PhyDAS were 1. Real-time processing: The system meets real-time demands by guar anteeing that worst-case response times are bounded. 2. Modularity, extendibility: Different interface modules can be used to construct the required hardware configuration for a given experi ment. 3. Multiprocessor configuration: To increase the performance and relia bility of the system, multiple processors can be used for controlling 1.1 Application areas 3 the interface modules and processing the data. 4. Cost-effectiveness: The life cycle of computers is generally much shorter that the life cycle of the interface modules. By separating the computer bus and the measuring bus, the use of new micropro cessor architectures is very cost-effective since the same interface modules can be used. EPEP is the name of the operating system and the programming lan guage. Goals for the design of EPEP [1] were 1. Object-based: TheimplementationofEPEP is object-basedfor reasons ofmodularity and extendibility. The programming language, which is not object-based, is a Pascal-like language. 2. Interpreter language: An interpreter language is preferred rather than a compiler-language because it is often necessary that the settings of interface modules can be adapted while running an experiment, without stopping and re-compiling the program. 3. Multitasking operating system: The motivations for structuring EPEP as a multi-process system are: • In a real-time system, processes must respond to asynchronous events.