physical structure is simplified but the Bus Systems protocol is more complex and includes For Process Control powerful error detection features. R.J. Rausch Multiple controllers The last improvement to the CAMAC CERN, Geneva, Switzerland system was made in 1979 to allow more Associate Member of EPS than one controller in each CAMAC crate, while retaining full compatibility with exis­ Buses used in large experimental faci­ which has adapted technical advances to ting modules. Two or more dataway mas­ lities to communicate between elements meet complex requirements. ters could operate in a crate, and bus mas­ of control systems have strongly influen­ tership protocols had to be specified to ced the architecture of these systems Parallel highway ensure orderly sharing of the dataway. over the past 20 years. The buses were CAMAC initially handled the input/out- This new capability (EUR-6500) allowed initially employed in conjunction with put (I/O) needs of the computer to which computing capacity to increase to solve minicomputers for data acquisition and it was connected via an electronic inter­ critical real-time problems by exporting control applications : today they support face that matched the computer's internal them out of the computer into the a variety of high-speed processors and bus to the electrical specification of the CAMAC crate. powerful control modules on a single CAMAC dataway. The EUR-4100 specifi­ entity — the data highway. cation thus describes a bus system com­ Microprocessors The standardization of data highways prising a single master, or controller, and The introduction of microprocessors became necessary when physicists and up to 23 slave modules In a crate with 25 after the original CAMAC specifications engineers realized that general purpose slots for circuit boards. The master acts as were published has greatly enhanced the functions could be implemented as stan­ an interface module between the compu­ versatility and performance of CAMAC dard modules plugged onto an agreed ter's I/O bus and the slave modules. Many equipment. A microprocessor can be used dataway. Teams from different laborato­ different CAMAC slave modules are avai­ in a module to pre-process data or to ries could then collaborate more efficient­ lable on the market (e.g. simple I/O or handle a communication protocol bet­ ly on large experiments, with each team status registers to coincidence selectors ween the module and some external de­ contributing its know-how and using com­ and triggers for complicated, high-speed vice. It can also be located in the crate patible material to assemble larger data events). controller used for sequencing and hand­ acquisition and control systems. The dataway is a parallel highway with ling data in a simple stand-alone system; Complex performance features, new 24 read lines from modules to the control­ or in a crate which is also connected to a applications and alternative approaches to ler and 24 write lines In the opposite direc­ parallel or serial highway to perform local bus systems have emerged following the tion. A station address, a sub-address and control or processing and so relieve the early successes of CAMAC, the first wide­ a function code constitute the CAMAC loads on the computer and on the high­ ly used parallel bus system. In helping a command. Two dedicated signal lines con­ way. Finally, the microprocessor can be user select the optimum system from nect each station to the controller: one located in a separate controller module among a profusion of commercial pro­ addresses the station (either alone or in called an auxiliary crate controller (ACC) ducts, it is useful to consider the impact combination with other stations) and the to act like a local controller or processor. on control system architectures of Eu­ module uses the other to inform the con­ rope's three most popular parallel bus troller that it requires attention — a very 8-Bit Microprocessor Buses systems (CAMAC, G64 and VMEbus). useful interrupt capability. The emergence of 8-bit microproces­ CAMAC Branch highway sors in 1978-79 gave rise to a demand for The standardized bus systems adven­ A single CAMAC crate connected to a inexpensive, compact, intelligent buses ture started in 1965-66 when European computer soon became insufficient and with CAMAC's modularity in applications high energy physics laboratories estab­ the relatively high price of minicomputers where CAMAC equipment was not pre­ lished the CAMAC system definition [1]. justified the simultaneous control of seve­ sent and where a few I/O signals called for Despite its early specification (in 1969), ral CAMAC crates. One could either con­ some local processing. Very simple micro­ CAMAC equipment Is being produced nect several computer-dedicated crate processor buses, usually optimized for a around the world and is still widely used controllers to the I/O bus (if this was elec­ given type of microprocessor, appeared for data acquisition. CAMAC's major ad­ trically possible) or use a single interface on the market. vantages are flexibility to build optimized between the computer and a standard Several groups in the early 1980's se­ systems and the interchangeability of parallel bus onto which several CAMAC lected the G 64 bus system from GESPAC commercial equipment. Existing control crates were branched. A second CAMAC for applications at CERN's LEP accelera­ systems can thus be easily restructured specification (EUR-4600) describes a tor. It involves a single height Eurocard and upgraded by incorporating more common vertical bus, the CAMAC branch format (100 mm x 160 mm) for the prin­ powerful modules. It also provides inde­ highway (BH) linking seven slave CAMAC ted circuit boards and a crate containing a pendence from computer manufacturers crates to a single computer. A BH driver maximum of 10 cards. The dataway uses in that the large capital investment in interface for each major type of computer 64 signal lines to provide 18 address lines, CAMAC for control systems is not lost was then developed to ensure indepen­ 16 data lines, one interrupt line and when obsolete computers are replaced. dence from manufacturers of electronic memory cycles for the microprocessors. Moreover, CAMAC is a standard bus circuits. Direct memory access (DMA) transfers system, developed "by users for users", are also specified. Serial highway The basic G64 specification has since Raymond Rausch studied physics and electro­ A CAMAC serial highway has been spe­ been enhanced to cater for new technical nic engineering at the Conservatoire National cified (EUR-6100) to extend the crate ad­ needs. The number of cards per crate has des Arts et Métiers, Paris. He has been an dressing capability and the bus length. It been extended to 16 by changing the accelerator controls engineer with CERN, has the form of a loop starting from the dataway line drivers, and a double height Geneva since 1972 after setting up data acqui­ master controller, connecting up to 62 Eurocard was introduced together with a sition systems at the linear accelerator at the crates and returning to the controller. front panel to facilitate attaching I/O cable CNRS, Orsay. Compared with the branch highway, its connectors. 40 Europhys. News 22 (1991) More than 2000 G64 bus systems are cessor and memory sub-bus; VMS provi­ SPS used in LEP. Industrial sub-systems are des a serial connection between modules The SPS control system (Fig. 1a) is connected to the LEP control system via a for communications between processors. based on minicomputers, a proprietary G64 crate in which the microprocessor The VMEbus specification aimed to real-time kernel and a software develop­ acts as a communications protocol con­ define a dataway to interconnect data pro­ ment system which became available in verter. The various systems are closely lin­ cessing, data storage, peripheral control the early 1970's. A system connecting ked to the equipment they control so they devices and network interfaces into a clo­ numerous minicomputers distributed over can provide autonomous control sequen­ sely coupled configuration. It is also inten­ the SPS site (7 km in circumference) had ces, the surveillance and monitoring of ded to allow multiprocessor configura­ to be developed, even though a standard sub-systems and alarm generation. tions capable of handling data and ad­ network protocol had not been specified Hierarchically, the G64 systems are dress paths of up to 32 bits, and to adapt and the accelerator's control hardware connected to process control assemblies the Eurocard sub-rack, an existing and was not yet available. (PCA's) that actually drive beam line widely used mechanical standard. The interface to the accelerator equip­ equipment via buses (MIL-STD-1553-B The essential VMEbus feature is a mul­ ment was implemented in CAMAC, then protocol). It is also at the level of the G64 tiprocessor bus with 2 to 21 slots for Euro­ the only international standard. CAMAC bus systems that the real-time operation cards (single or double height). The con­ crates were interfaced either directly to and synchronization of controls arise. nector of the single height card provides the computer I/O bus via dedicated crate The G64 microprocessors mostly use a 16-bit data, 24-bit address, 7 interrupt controllers or by using the standard simple software kernel and are program­ lines and all the control lines. The second CAMAC branch and serial highways with med in PASCAL or BASIC languages. The connector available on the double height associated crate controllers. addressing capability (64 Kbyte) of the card adds additional 16-bit data and 8-bit Field buses for equipment were not ava­ microprocessor used in G 64 is sufficient address. The additional 64 pins on this se­ ilable so an economic serial bus with for most simple applications. More CPU cond connector are dedicated to the VSB crates and electronic modules had to be power and memory to implement more extension. VSB is limited to a maximum of developed to connect many equipment complex functions and the need for stan­ six slots to ensure high-speed data trans­ crates to the local computer. This is the dard real-time software kernels (see page fer between up to six modules and a me­ CERN SPS/MPX system with more than 34) means that 8-bit systems are reaching chanism to arbitrate amongst processors 700 crates and 6000 modules which are their limits: 16- or even 32-bit bus sys­ requesting mastership is provided. still in operation. tems will be required. VMEbus contains all the necessary fea­ tures typically required for operating mul­ LEP VMEbus tiprocessor systems : it is widely recogni­ LEP's control system was drawn up The first formal specification of the zed and is supported by more than 200 when communication network standards VMEbus appeared in 1982 (Revision A of manufacturers. and multiprocessor bus systems were the IEEE 1014 specification). Owing to its being specified and when 16-bit micropro­ functional modules, VMEbus is essentially Impact On Control Architectures cessors with a large addressing capability a Eurocard version of Motorola's Versa- The impact of bus systems on control became commercially available [2]. bus, hence "Versabus Modèle Européen" architectures can be evaluated by compa­ Owing to the rapid evolution of integra­ (VME). VMEbus has been improved and ring CERN's SPS and LEP control systems ted circuit technology and the expansion extended into a three-bus combination that were designed in 1972-76 and in of the multiprocessor features of the VME-VSB-VMS). VSB is a fast multipro­ 1983-86, respectively. VMEbus system, all the process control functions could be grouped into separate VMEbus multiprocessor crates (see Fig. 1b). Each crate is connected to a local or a wide area network conforming to inter­ national standards (Ethernet IEEE 802.3 or token ring IEEE 802.5) using a data link module. The network protocol uses the widely recognized TCP/IP standard to­ gether with remote procedure calls (RPC) for which an international standard is being finalized. The central processor unit is based on a 32-bit microprocessor with up to 8 Mbyte of memory. It runs under a real-time opera­ ting system (currently OS-9) on which the RPC is implemented. Local interaction with the processor is done via a simple terminal linked by a RS-232-C standard cable. A display module can be inserted into the VMEbus to give locally available high resolution graphics combined with an interactive touch terminal. Local mass Fig. 1 — Control architectures for CERN accelerators, (a - left) Super Proton Synchrotron (SPS) storage of data on a hard disk is achieved designed in the mid-1970's. CAMAC crates with slotted circuit boards are interfaced either by with a plug-in unit interfaced either direct­ CAMAC branch and serial highways or through dedicated crate controllers to link the accelera­ ly to the VMEbus or to the main processor tor's devices to minicomputers distributed over the 7 km in circumference site. A multidrop bus and lines for timing signals drive equipment via MPX modules (MPX is a CERN internal standard). unit via a standard SASI or SCSI bus con­ (b - right) Large Electron Positron Collider (LEP) designed in the mid-1980's. Control functions nection system. (TIM : timing, BC : field bus connection, etc.) are provided by microprocessor boards grouped For rapid data acquisition and the trans­ together in a single VMEbus crate. The equipment field bus and the synchronization signal lines fer of large amounts of data, e.g. output constitute local distribution networks to electronic control assemblies (ECA) that drive devices. from beam instrumentation, specialized Distributed intelligence is linked by token ring or Ethernet networks. I/O modules can be plugged-in and linked Europhys. News 22 (1991) 41 directly to detectors. Some demultiplexing is acceptable in less demanding applica­ CIM For Particle Factories tions where many similar items of equip­ Large experimental facilities may seem data exchange and management. Success ment are controlled, e.g. power supplies. unusual to the average person, but they are requires : Here, a field bus controller drives a multi­ comparable to industrial machines sugges­ - A coherent and powerful communication drop cable connecting up to 30 equipment ting an industrial approach for integrating network based on international standards control assemblies (ECA's) implemented computers. A major installation such as an that links together all laboratories, offices, using the MIL-STD-1553-B standard field accelerator complex or "particle factory" is workshops, accelerators and experiments. bus. The ECA's, located close to the sub­ mostly made up of well-defined industrial Organized into interconnected islands of systems they control, consist of G 64 or components so its operation largely leans local area networks (LAN's), the overall VMEbus crates with microprocessors. on functional blocks similar to those found network allows for an independent and in industry. flexible selection of computer-based pro­ Concluding Remarks But this is not the generally accepted ducts aimed at helping workers. Standardization of bus systems has view. The problems of pioneering in statis­ - Computer-based software tools that allowed investments in design, material, tics-based research topics such as particle comply fully with international data ex­ manpower and technical know-how to be physics lead engineers and physicists to change standards. capitalized upon. Meanwhile, the diversity work at the limits of what is possible in - A single, widely accessible database of applications, users' experience and several fields (e.g. electronics, mechanics, management system (DBMS) for the collec­ contributions to international standards computing, materials, etc.). They are forced tion and retrieval of data originating from bodies have enhanced and optimized to permanently explore novel ideas and to different sources (administrative, technical, technical specifications. evaluate and apply promising new techno­ etc.) or having different natures (text, pic­ One consequence is a change-over logies as soon as they emerge from labora­ ture, sound, etc.). from the use of buses for simple input/out- tories..This feature also pushes them into Weaknesses in any of these areas result put functions to the complete integration initiating new developments until they in isolated islands of different computer- of all the operations required for data begin to feel they have to step beyond in­ based tools to aid the different groups (sys­ acquisition and process control. However, dustry. One finally becomes accustomed to tems designers, operators, etc.) and a large comparisons between bus systems used developing everything that is needed be­ reduction in the overall benefits. A correctly for controlling large accelerators suggest cause one generally achieves better results configured CIM system can help by acting that the basic requirements have not than with what is easily available. as the glue between individual tools used changed drastically in 20 years: distribu­ This has been the case on introducing for different functions in a company [5]. ted communication, local processing po­ computers into experimental facilities. CERN is an example of one research orga­ wer, remote control and surveillance, and There has been an anarchic development of nization that is currently working towards local interaction with graphics remain tools for engineering, production, control, integration. A powerful communication net­ essential. planning and administrative support activi­ work has been implemented, with Ethernet The revolution is that these functions ties. Isolation from developments else­ and token rings as wide area networks and are now implemented more quickly, eco­ where and the lack of defined goals rapidly with Ethernet and communication systems became the main problems. The software based on the M1L-1553-B standard for nomically and flexibly, and with greater industry was in the meantime developing LAN's. A DBMS has been approved for both pre-processing of data. The integration of products for a much larger community. process control and administrative activi­ complete functions in a single chip (i.e. ties. Software tools are starting to be network protocols, processors, memories, Exploiting CIM connected and although links are missing digital signal processors, peripheral con­ The spectacular progress of informatics (mainly in administrative and support are­ trollers, intelligent sensors, etc.) and the has generated considerable interest in at­ as), an ambitious project to integrate all standardization of hardware for bus sys­ tempts to model a commercial enterprise. administrative tasks is underway. tems means that more computer power Views of an experimental facility, notably a and processing functions can now be particle factory, that are developed using Conclusions packed into a single multiprocessor VME- some of the latest descriptions [1, 2] em­ Large experimental facilities in physics bus crate than in a complete 1970's style phasize two unique features : have enjoyed the advantage of an intimate front-end process control assembly requi­ - the facility does not sell, strictly speak­ contact with computers. But the complex, ring several racks of equipment. ing, the products it manufactures; slightly anarchic situation which grew up Further improvements in software, stan­ - products such as high energy particles should not divert attention from the cohe­ dardization of software tools, languages, cannot be stored or transported in a conven­ rence that is now emerging elsewhere. It is real-time kernels, communication packa­ tional way as these operations need com­ essential that managers of these centres ges, graphics packages, etc. are essential plex machines. The products are instead should now seek this coherence when to improve the efficiency of process con­ manufactured for local use by physicists. approving computer-based software pro­ trol. For instance, we dream of software Considering an experimental facility as a ducts. The application of full integration modules plugging into a standard soft­ manufacturing site nonetheless provides a based on the CIM top-down approach ap­ ware system (the "software bus") much working framework for achieving greater plied to an experimental facility could then like a VMEbus module plugs into a stan­ efficiency. A computer-integrated manufac­ serve as a model for European industry. dard VMEbus. Moreover, we are already turing (CIM) system should thus be consi­ not far from a UNIX real-time software dered for an experimental facility as reports REFERENCES kernel capable of dynamic load-sharing show that its implementation produces [1] Kosankc K. and Vlietstra J., Proc. 6th among several identical VMEbus proces­ encouraging results [3, 4] by generating Ann. ESPRIT Conf., Brussels (1989). sors in a crate and remotely communi­ major improvements in productivity, cost [2] Playoust P., Tech. de l'Ingénieur, H6050. cating with user-friendly workstations for savings, throughput and flow control. [31 Foltz M.T. and Tanik N.M., Int. J. Compu­ the operators. Although CIM is not yet clearly defined ter Applns. in Technology 3 (1990) 26. [3] it aims to integrate computers and ade­ [4] Costea I., Information Strategy, Summer REFERENCES quate software tools, which may not neces­ issue (1990). [1] EUR 11079 : Guide to Internationally Reco­ sarily exist as yet, into all aspects, including [5] Leonard R., Int. J. Computer Integrated gnized Hardware Interfaces (CEC, Brussels) management, of a manufacturing cycle Manufacturing I (1990) 13. 1988. (planning, design, etc.). A top-down ap­ [2] Altaber J. ef al., Proc.1983 Particle Accele­ proach to implementing CIM leans on two M. Rabany rators Conference, IEEE Trans. NS-30 (1983). fundamental pillars: communication and CERN, Geneva 42 Europhys. News 22 (1991)