EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH

. ’ R QE N CERN DRDC/91-45 CERN LIBRARIES, GENEVA

E6 S Pow? Illllllllllll Illlllllllilllllllillllllllllllllll ~5-' 3 c. P SCOOOOOIZZ

“' " Applications of the Scalable Coherent Interface to Q; ug Data Acqu1s1t1on at LHC

A. Bogaertsl, J. Buytaertz, R. Divia, H. Miillerl, C. Paxkman, P. Ponting CERN, Geneva, Switzerland

B. Skaali, G. Midttun, D. Wormald, J. Wikne University of Oslo, Physics Department, Norway

S. Falciano, F. Cesaroni INFN Sezione di Roma and University of Rome, La Sapienza, Italy

V.I. Vinogradov Institute of Nuclear Research, Academy of Sciences, Moscow, USSR

E.H. Kristiansen, B. Solbergz Dolphin Server Technology A.S., Oslo, Norway

A. Guglielmi Digital Equipment Corporation (DEC}, Joint Project at CERN

F-H. Worm, J. Bovier Creative Electronic Systems (CES), Geneva, Switzerland

C. Davis Radstone Technology plc, Towcester, UK

joint spokesmen Fellow at CERN Scientific associate at CERN, funded by Norwegian Research Council for Science and Humanities OCR Output

ZKLUIU We propose to use the Scalable Coherent Interface (SCI) as a very high speed interconnect between LHC detector data buffers and farms of com mercial trigger processors. Both the global 2"‘f and 3"d level trigger can be based on SCI as a reconfigurable and scalable system. SCI is a proposed IEEE standard which uses fast point-to-point links to provide computer -like services. It can connect a maximum of 65536 nodes (memories or processors), providing data transfer rates of up to 1 Gbyte / s. Scalable systems can be built using either simple SCI rings or complex switches. The interconnections may be flat cables, coaxial cables, or optical fibers. SCI protocols have been entirely implemented in VLSI, resulting in a significant simplification of data acquisition software. Novel SCI features allow efficient implementation of both data and processor driven readout architectures. In particular, a very efficient implementation of the 3"d level trigger can be achieved by combining SCI’s shared and distributed mem ory with the virtual memory of modern RISC processors. This approach avoids complex event building hardware and software. Collaboration with computer and VLSI manufacturers is foreseen to assure the production of maintainable components. The proposed studies on SCI hardware and soft ware will be made in collaboration with other LHC R&D projects to provide a basis for future, standard SCI-based data acquisition systems. OCR Output Contents

LHC Detector Readout

1.1 Introduction ......

1.2 LHC Detect ors ...... 1.3 Size of the SCI Readout System ...... 1.4 SCI Readout Node Implementation ...... 1.5 Data Streams after 2”“ Level Trigger ......

16.nerace Itft2”‘ o evererLl Tigg ......

Application of SCI to Global'“* 2and "' 3Level Triggers 2.1 Status of the SCI standard ...... 2.2 Impact of SCI on Data Acquisition Systems

2.3 Coherent Caching ...... 10

2.4 Use of SCI for the 2'“‘ Level 'Irigger 11

2.5 Use of SCI for the 3"d Level 'Trigger .11 2.6 Demonstration Systems ...... 12

Research and Development Program 14 3.1 General Purpose SCI Interface . . . 14 3.2 SCI Ringlet Test System ...... 14 3.3 Direct SCI-Computer Interface . . . 15

3.4 SCI Memory ...... 16 3.5 SCI Bridges and Interfaces ...... 16 3.6 SCI/ VME Single Board Computer . 18 3.7 Intelligent Data Controller ...... 18 3.8 Diagnostics, using a Protocol Tracer 18

39.otware Sf ...... 19

3.10 Modelling and Simulation ...... 20

Collaboration with industry 22

Budgets 23

Responsibilities 25

Timescales, Milestones 26 OCR Output 1 LHC Detector Readout

1.1 Introduction

According to ECFA studies [1], the event rate for a general purpose LHC detector, operated at a luminosity of 2 >•· 103‘cm2s‘1 will not exceed 105 Hz after the 1" level trigger. The data volume generated by such a detector is estimated as:

Inner Tracking: 1 Mbyte per 15 ns bunch, 20 million channels Calorimeter: 200 kbyte per 15 ns bunch, 200 000 channels Muon Tracking: negligible amount from up to 106 sparsely filled channels

A possible readout scheme for such a detector, according to the current understanding [1] [2], is illustrated in fig. 1. A first stage of data concentration after the 1** level trigger decision is implemented by electronics located close to the detector. Next, data is carried off the detector by point-to-point links, further concentrated in bus units and stored in data buffers. The data volume of each segment is sufficiently small that these buffers can be implemented using conventional bus systems Event data is further filtered by a 2"" level trigger which is implemented in two stages. The first stage consists of local trigger processors which have access to data of one segment of a detector only. These produce reformatted, reduced events complemented by trigger data which are stored in output buffers. The overall trigger decision is taken by global 2"d level trigger processors which can correlate the pre·processed event and trigger data of the first stage. The event rate at the input stages of the local and global 2"‘ level trigger is estimated at z 100 kHz; the output rate after the global 2**** level trigger at z 1 kHz. With an average of up to 1 ms for the global 2'"' level decision, such a reduction could be obtained by a farm of 100 processors. Final event rejection is accomplished by a 3"" level trigger, based on complete event data, further reducing the rate to z 100 Hz. Both the 3'“ level trigger and data logger can be implemented by a processor farm. There are several reasons not to use buses for the readout system after the local 2"“ level trigger [3]: the expected data rates exceed the capacity of existing buses, the required con nectivity over large distances is very problematic and event building methods based on buses [4] cannot be scaled to the size of an LHC system. We propose to implement the global 2'“’ level trigger, the 3"' level trigger and the data logger using a uniform SCI network: copying of data is avoided and events are stored only in the (local) 2"‘ level output buffers from where data can be accessed by the trigger processors. These and the data logger are all implemented as farms of commercially available computers. A significant simplification of both hardware and software can be ob tained by using the processor’s virtual memory hardware for implicit event building, whilst caches avoid repetitive data access. OCR Output iii$:`—·$<$$E¤§:§i$S<$i`i$S·i:¤:¤5?·:§;§>Q: I{2{7;:;:¥>!i¥;§,§¥$?!~K¤S$§:R?$?Z~$1?‘,{§ .,w__;_;;;; ...... 1 . . . '—2—Z—‘b1·Z·'·l` '~2~2' ' '£-1·i~Z·Z*Z*Z·Z'·Z·2·2 +2 ·2·Z-§;gm> x>¢ewa3q4§;;:;Tit? ·g;: -· ·.; ·¤: :i· :*· r1 nu. §a ;~: ·>. :;: _::Tl Dill .___; T1 Dm Cuixltrllnfl §:; . ·< ‘ Cmcaumtus :·`....»...... ,s¤ ._ »

Link1fxunDcxoc\¤rE1e¤¤¤niu

a cia.: | 5°°° SCI N°°°‘

ii? SCI Interconnect

Fum Farm Fum

|v¤m1 Mmm - IVi!¤n1 Muncry in cache | a cms at cane ciosurz rsnagga

Figure 1: LHC Detector Readout based on SCI.

1.2 LHC Detectors

The EAGLE proto-collaboration serves as a model for our application proposal. Leading candidate detectors include: a silicon preshower tracker, a liquid argon calorimeter and inner tracker. Amongst various candidates for inner tracking, the Silicon Strip [5] and Silicon Preshower Detectors [6] are considered as examples. Roughly 20 * 106 channels need to be processed and compacted for each. For both detectors a large concentration of input channels between the 1'° and 2”d level trigger is foreseen. Preshower Detector: for the Silicon Tracker Preshower ( SITP ) detector, 1" level data compaction and formatting takes place within a VLSI chip. After multiplexing of silicon-pad inputs at the VLSI level, data are further concentrated via 32 bit bus units, containing 64 such chips. At I" level trigger rate of 50 kHz, the output rate per 64 pads is rather low at 12.5 kbyte/ s: a typical 32 bit bus can combine outputs from 64 chips into one link, requiring at least 800 Kbyte/ s data throughput to the 2"‘ level local trigger stage. This stage can be implemented using standard bus units, providing an additional concentration of channels in the order of 10. This stage will require fast processors and storage of compacted data, to be transferred to the further readout system. OCR Output From 25 * 106 silicon pads less than 1000 output channels could be connected to an SCI based readout system. All these SCI nodes need to be capable of transferring the bus-resident 2"" level data to an SCI memory. Silicon Strip detector: the Si Strip Detector is parameterized for 20-100 ps 2”" level decision time at a rate of 1 kHz. An analog pipeline feeds an Analog Pulse Shaper Processor whose output is kept in both analog and digital stores during the 2"" level decision time. 128 such channels are contained in one readout block, consisting of both analog and digital buses. Further concentration by a factor of 32 could be achieved via point to point links to data concentrating bus units. Local 2”“ level trigger processors and buffers in the bus concentrators could be connected to an SCI system with less than 1000 nodes, as in the case of the SITP. Calorimeters: an electromagnetic liquid argon spectrometer will probably provide 200,000 input channels whilst a hadronic spectrometer (such as SPACAL or liquid Argon) will have around 30,000 channels. After the 1“ level decision, less than 1/4 of these channels have to be read out. Proposals to process, format and compact this data are based on VLSI and multi—chip wafer [17] techniques which can integrate local processing and storage. We expect that calorimeter channels can be concentrated in less than 1000 output channels after the 2 level local trigger, which again could be connected to an SCI readout system. Muon Chambers: these would have up to 106 channels of which less than 20,000 channels will be read out after the 1'° level decision. Though the amount of data is small, fast access to a random subset of muon channels is required, in particular for correlations by a global 2"d level trigger. An SCI system would provide a fast and uniform interconnection for such correlations.

1.3 Size of the SCI Readout System

The concentration of detector input channels into a smaller number of 2”" level output channels is based on front end VLSI chips, read out by bus units which are interconnected via point to point links. The bus units contain local 2'"’ level processors which feed an output buffer with compacted event and trigger data. The massive number of input channels is reduced to several thousand output channels across which different data streams have to be transferred to the readout system. The size of an SCI system will depend on the reduction in channels achieved by the concentrator units. 2'“l Level Data Concentrator Units: the connection points to an SCI based readout are the data buffers of the local 2'“* level data concentrator units [Fig.2]. Implementation choices taken for the previous stages ( analog or digital pipelines, local 2"" level processing and compaction techniques) have little induence on the proposed SCI readout system. The 2"d level bus units will very likely be based on conventional buses such as Fastbus or VME bus. Unidirectional links, such as I·HPPI [7] could transfer data from detector-resident VLSI channels to these bus units. We estimate that after the local 2"d level, the output channels from the SITP, the Si Microstrip, the calorimeters and the muon chambers could each contribute roughly 1000 outputs to an SCI system. A general purpose detector would therefore require as 4000 SCI OCR Output +1:

;-:|·:¤;§:§:>.

··· 2 tz;;&=&>z¤m>»>>uxs$

<1U|¤ s:~t·:¢ ·:k¤:¤:¤:·:¤:¢::2¤:>:¤:¥:2¤:¤:L·1:ht€¢:g;Ei&§EFIUlla! 2525ai25si23§2i2i!2i§2iai2i2E252i§s€§§2i2%laisis E=§·r1 mu =2=2&=;$¤=€¥5=;¥s&=&=a=sTz'=a¤;$=e=‘s‘s.’s=';Qn: .». %z?2€¥z5z%t?sPz é=E =E=§ ·:·:·<~>:·m.v·xi§·2 %?=3%-=;-wi .=>;

Lum mm Duck: m zaiii ééé¥é§2 $ :¥:- :¢:¥ ·.\Q·Z$%\\ @

Links from Detector Electronics 5 n \_ k, \ \.,- K Ke

5”°‘"’ I l ¤·‘~··~· _:;

sum T scmm

Figure 2: Data Concentrator Units nodes, if a uniform system is implemented.

We assume 5000 nodes as a rough measure of the size of our proposed SCI system. Scaling to much larger or smaller size is an important feature of SCI, providing ample freedom in the Hnal implementation. OCR Output 1.4 SCI Readout Node Implementation

A11 SCI node provides access to the data buffers of the local 2"° level trigger. Our R&D program foresees several ways to implement SCI nodes for either VMEbus or Fastbus systems. A direct connection to VLSI based buffers will be studied as well. Each SCI node can transfer event and trigger data to the data logger, the 3"" level trigger and the global 2 level trigger. The SCI node supports read and write access, allowing to implement different readout methods. Nodes can be interconnected in 4 ways:

Optical Fibers at 100 Mbyte/s: where long distances up to 10 km have to be covered.

Single ended Coaxial cables at 100 Mbyte/s: where medium sized, cheap connections are desired for distances up to 50 m.

Diferential ECL, triaxial flat cables at 1 GByte/s: where fast and short connections are required for distances of up to 15 m.

Diferential low voltage CMOS, triaxial flat cables up to 1Gbyte/s: where fast and short connections are required for distances of up to 15 m. (First implementation probably about 150-250 Mbytes/s and distances 25-30 rn)

It is likely that all four versions will be required. Therefore we propose to investigate all of these versions.

1.5 Data Streams after 2"‘ Level Trigger

The parameters for an LHC data acquisition system after the local 2”" level trigger are best seen in terms of three data streams [Fig. 3]. All 2'“* level output buffers contribute data to each of the streams. A study of these, based on compacted data, has been published by the ECFA subgroup on buses on links [2] [3] and is summarized as follows:

Global 2”d level: 108 Byte/s trigger data 3"‘l level: 108 Byte/s event data Data Logger: 108 Byte/s event data

Required Bandwith: the three data streams constitute a very large total bandwidth, each a factor of z 100 above typical LEP figures, such that conventional buses cannot be used. A system is required which can transfer and route the constituents of the above streams as small packets without blocking to their destination. Though the input bandwith is divided by thousands of input nodes, the streams routed to the trigger processors add up to a bandwith of at least 100 Mbytes / s. We expect that future computers equipped with SCI interfaces can handle this bandwith. OCR Output Local 2*** Level Processors audOutput Buffers

Datalnput Port

SCI Output Port

I10'B at ·=;:i;i:i:§gigi;i;5a‘ es/s 10°B we 10°B I ws 1¤“Byres/¤v‘§§§§§g§‘ 10‘Bytes/sv' 10‘Bytes/ov

Global 2*** Level 3¤·* Level Dam Trigger Trigger Logger (trigger data) (gym; dam) (event data)

Figure 3: Data Streams

As a rule of thumb, the maximum bandwith of the transmission medium has to be a signiHcantly higher than the peak data rate required by the system. This means that in order to transfer 100 Mbyte/ s streams on average, a system with 200Mbyte/ s to 1 Gbyte/ s peak performance is required and SCI is an adequate choice.

1.6 Interface to 2'“‘ Level Trigger

The SCI readout node requires an associated microprocessor to write local 2”d level output data to a processor farm or to transfer local data to cached SCI memory. Cheap micr0— controllers would be suiiicient for this purpose. The 2"" level processing required within a concentrator unit which is connected to SCI could be performed by the same processor if a sufficiently powerful processor is chosen. The combination of a standard SCI node interface and a. fast RISC or CISC processor is therefore part of our feasibility study to be carried out in collaboration with computer industry. OCR Output 2 Application of SCI to Global 2"" and 3"" Level Triggers

2.1 Status of the SCI standard

SCI provides computer—bus-like services but uses a network of fast point—to—point links. Com pared to backplane buses, far higher speeds are possible. The proposed IEEE P1596 standard [8], consisting of the logical and physical specifications, will very likely be finalised before the end of 1991. It has passed the first level of balloting and it will be forwarded to the IEEE Standards Board after a short review in October. New working groups have been set up to cover the following areas:

P1596.l SCI to VMEbus bridge architecture P1596.2 SCI extensions P1596.3 Low voltage differential signals P1596.4 High bandwidth memory interface P1596.5 SCI Transfer Formats

The first VLSI chips for SCI are developed by Dolphin Server Technology in Oslo. First VLSI and starter kits from Dolphin are expected by the first quarter of 1992. Their node chips cormect directly to a cable for transmission at 1 Gbyte/ s and further CMOS chips provide interfaces to memory and second level cac.hes. The starter kit consists of a VMEbus based node interface, a chip set and optionally a Diagnostic 'I`racer. The and coaxial SCI versions are combinations of the node chip and the transceiver GIGA-chips from Hewlett Packard. The latter work with various framing protocols such as HIPPI and SCI and are under investigation at CERN for applications in Data Acquisition Systems [20].

2.2 Impact of SCI on Data Acquisition Systems

Amongst the innovations of SCI, coherent caching and virtual addressing can lead to new approaches to data acquisition. These provide very efficient access to distributed memories. A uniform SCI system can be constructed from a mixture of high speed SCI switches and cheap, passive ringlets interconnected by long distance optical fibers or short coaxial cables. Within such a network, any application program can trivially access data in any other node. The underlying protocols which take care of packet (dis)assembly, buffering, routing, priorities and error recovery are entirely implemented in VLSI. SCI protocols have been carefully designed to ensure reliable transmission and data coherency in a multi processor environment. Applications written in high level languages can access data implicitly without using special libraries or drivers. There is no restriction on the direction of the data flow. Data is normally cached to speed up repeated access. Inter·processor communication is possible using shared memory and coherent caching. In view of the large size of an LHC experiment scalability is a necessity. No loss in performance should be noted by enlarging the system to its final size. Contrary to backplane buses, SCI provides scaling in terms of size and performance. Simulations [9] have shown that simple systems based on SCI rings only scale to bandwith limits of the order of a few Gbytes / s. Active SCI switches which are expected to be developed soon after the availability of SCI node chips should provide large system scaling. Another important scaling feature is OCR Output topologydndependent data access. This means that data acquisition software designed for small systems will be usable without modifications or loss of performance in large systems.

2.3 Coherent Caching

Shared data residing in SCI memory may be cached by several processors. Coherency pro tocols ensure that changes to data in one processor are reflected in the caches of the sharing processors. For this purpose, SCI memories and caches contain, apart from data, a directory of pointers and status information. The protocols have been optimized to minimize memory updates, even allowing data to How between caches without being stored in the main memory [fig. 4]. This has several advantages for data acquisition hardware and software:

no software needed to tmnsfer data: SCI coherency protocols implemented in VLSI transfer data transparently.

application software can be written natumlly: applications access data directly without using special libraries or being linked to complex data acquisition software.

low bandwidth usage: only data which is accessed is transferred.

fast data access: data is cached at both ends, which speeds up repeated access.

ejcient data transfers: SCI supports many options to optimize data transfers (such as cache to cache transfers). SCI cycles to access directory information may overlap with processing. Cache-prefetch of data improves latencies for read operations.

Processor l Sums Manory Cache Memory Directory Directory

iaézcrene a Memory ii Dm Dena

Status

Cache E Data ?

Cache Directory

Processor

Figu.re 4: Coherency Protocols, transferring data directly between two cache memories.

10 OCR Output 2.4 Use of SCI for the 2"‘ Level Trigger

We assume that local 2"‘ level trigger processing is predominately performed by processors residing in bus systems. Because of the locality of the data, there is a large degree of parallelism and a modest data rate. Each local processor compacts the events and produces a small amount of additional trigger data into an output buffer, interfaced to SCI. The main difficulty of the global 2"" level trigger is the recombination of data distributed over a large number of memories at a rate of z 100 kHz. Since this involves transporting data over large distances, latencies become a limiting factor. These can be reduced using the data driven approach where data is written by a processor of an SCI readout node to a destination processor of a trigger farm. In addition, this makes better use of available bandwith. A rapid decision (100 kHz) must be made by z 5000 nodes to select a free trigger processor. SCI provides a rich set of primitives for the required synchronization of processors. Amongst these, the cache coherency protocols are very promising because synchronization using shared memory can be very fast. The average processing time available to each trigger processor depends on the number of processors and the rejection rate. Assuming a reduction to z 1 kHz, a farm of 100 processors would dispose of z 1 ms per processor and event. We propose to investigate the data driven approach for the global 2"d level trigger using an SCI network for the data collection. Both CISC and RISC based modules developed by our industrial partners provide connections between the 2"d level output buffers and SCI. We propose to implement the global trigger processors as an SCI based farm.

2.5 Use of SCI for the 3"‘* Level Trigger

After the 2"" level trigger the event rate is reduced to a tolerable 1 kHz though the total data rate remains still enormous (1 Gbytef s). A further difficulty to be solved for LHC is the event building which is required for both the 3*** level trigger and the data recording. We propose to exploit the memory mapping capabilities of SCI to reduce the effective data rate by a factor z 10 and simplify at the same time the event building. For the 3*** level trigger, our argument is based on the observation that only a small fraction of the event data (z 10%) is used for the rejection decision This reduces the effective data rate into the trigger processors to z 100 Mbytes/ s. The data logger needs only to copy those events which pass the 3"' level trigger decision. Assuming z 100 Hz for the event rate after the 3"' level trigger decision, this results also in data rate of z 100 Mbytes/s. SCI allows data to be accessed directly from applications running on the processors, without copying data. In addition, the virtual memory hardware which is implemented on all modern processors can map the data which is distributed over many output buffers into a contiguous single event, as illustrated in Fig. 5. This avoids complex event building hardware and software. Accessing data over a large network introduces latencies which manifest themselves as delays each time the processor tries to access a data item which is not yet cached. This

11 OCR Output

OCR OutputSCI dual-ported memory node

Level 2 data access SCI memory node

Ahpnr K QQU RAM

¤<=¤§.ii§. cnc

scr gpiii CHIP

SCI Domonstrator System

·--·( ·~--- ··--—- » ‘.

Trigger processors

SCI CHIP

Bus Interface .iit.$<=¤¤

Intertaca bus

Workstation

Figure 6: SCI Demonstration System.

Fastbus interface and the necessary system software. DEC offers to contribute to the software provided that the hardware interface is designed at CERN. The Readout System Test Benches Project [12] views SCI as a good candidate for cached readout, without copying data from output buffers to trigger farms. A close collaboration already exists to test the first SCI starter kit in an environment comprising VMEbus and SUN SPARC stations running UNIX. The Scalable Data Taking System Project at a. test beam for LHC [16] has expressed in· terest in SCI for the 3'“ level trigger, and regular contacts exist. The EAST 2"" Level Trigger Project [10] is currently concentrating on I·HPPI links and switches, however is interested in understanding SCI and its possibilities.

13 OCR Output 3 Research and Development Program

The Research and Development Program carried out by the different partners is briefly described. Following the requirements of an LHC data acquisition system we need to test SCI’s cache and virtual memory system, as well as the coherency option. For low latency applications we will also test non-cached transfers. Commercial components and modules are used to build a small, scalable SCI environment to test these protocols and to develop a test bed for SCI software.

3.1 General Purpose SCI Interface

A small, VLSI-based interface is required to connect SCI to local data buffers, user logic and processors. Conceived as a. daughter board [fig. 7], Dolphin’s SCI interface can be used for both existing and new implementations. It contains the node chip, the general cache controller (GCC) and cache memory and provides an SCI connector and an external bus connection which is compliant to the MC88110 bus 1. Other bus protocols can be interfaced via protocol adapters. Dolphin can provide development tools, based on , to design such protocol adapters. Each interface requires a processor to transfer local data to SCI memory, which serves as local 2"" level output buffer. If required, the same processor can be used for local trigger processing, data formatting and data collection. RISC or CISC processors or microcontrollers can be used. The SCI interface contains sufficient dynamic memory as SCI cache under control of the GCC controller. It provides a 1 Gbyte/ s transfer rate on differential coaxial SCI cables and is compliant to the SCI transaction and cache coherence protocols. Bus Interfaces: buses like TURBOchannel or Fastbus require a bus protocol adapter which, when fully implemented, are called bus bridges. Simple adapters, which implement a subset of protocols are sufficient for data readout applications. We plan to work on simple adapters to both Fastbus and TURBOchannel.

3.2 SCI Ringlet Test System

In order to understand the functionality we plan to build a minimal SCI system which can be scaled to larger size in the future [fig. 8]. Starting with a two·node ringlet to test data transfers between SCI memory and a workstation, a third node will be added later to test data transfers from external buffers. Ringlet Components: the memory and the interface to the workstation are based on commercial cards and allow testing of SCI transactions without initial hardware develop ment. The VME bridge and SCI memory are available from Dolphin, the interface to a SPARCstation from Performance Technology [11]. User Environment: for test and diagnostics, the UNIX system and the C language are used on a SPARCstation which we have purchased for this purpose. We collaborate with the RSTB project [12] on the programming part of SPARC-VMEbus interfaces.

lprocessor bus of the Motorola MC88110 RISC processor

14 OCR Output NODE CHIP CONTROLLER CBUS

DRAM

USER LOGIC CACHE

88110 bus PROCESSOR

extemal bus

PROTOCOL ADAPTER

Figure 7: SCI interface to User logic

Cabling and Connectors: the SCI cables (ECL differential coax) will be bought from Gore [13] whilst the required IEEE 1301.1 standard connectors will be purchased from Du.Pont [14]. Testing of the SCI fiber version is foreseen at a later stage after experience with small SCI ringlets has been gained.

3.3 Direct SCI-Computer Interface

Only on a the long term need SCI-workstation interfaces to be optimized for speed, requiring fast access to the CPU and its memory system. Though we expect that commercial computers with SCI interfaces will become available in the future, we need to test the functionality of a direct SCI- Computer interface, serving as a first model of a trigger processor farm. From the leading candidates 3 we have chosen the TURBOchannel for a test of a direct SCI interface. This computer bus is sufficiently fast and simple and allows us to collaborate with the Delphi experiment on a TUB.BOchannel-based VAXstation interface.

These tests could be carried out in collaboration with INR., Moscow. °DEC’s 'Iiubochaunel (100 Mbyte/s), Sun’s SBUs (200 Mbytes/s) and the + (600-800 Mbytes/s)

15 OCR Output 3.4 SCI Memory

SCI memory is exclusively accessed and modified by SCI transactions. Such a memory is for example based on the GMC global memory controller from Dolphin, providing a maximum of 4 Gbyte per node. SCI memories can be shared between a large number of processors. For our ringlet test we need one SCI memory board.

’$:EiE’§i‘i‘:5 _..emo'?... .. ¤t¢*¢¤¤¤¢¢*[racerSCI l Nl Tzrs:;;=;=;:;.;·;-.·.-.-.-.-....,;·;.;.,..,..;.;._.;.;...;.;.;.;.;.;._..,.,., ,;,;,;,,3,,,;,;,,;,;.;.;,;,,,,,,;,,:2,,;.,:,g3,,§;§;;5§5

``" `

Bus Bridge Bus Ad¤Pt0¥‘· ....._...... _. llégléZ·I·I·FI-Z-Z·2·2-Z·?·Zyl-Z-}2{Z·2-57·Z·Z)Z·Z·Z-Z{·2·Z·CjZ;Z·l§l;lgl;l;lI Bus l l

Figure 8: SCI diagnostic environment

3.5 SCI Bridges and Interfaces

Bridges between SCI and other buses and links are required to interface processor farms to the local 2"J level trigger. Possible candidates are [fig. 9]: VMEbus and Fastbus in the front ends, linked via VICbus, HIPPI or Fastbus cable segments. Futurebus+ as well as SBus and TUR.BOchannel in the computer farms. More details are listed below. We do not propose to develop a full implementation of bus bridges within this project. VMEbus Bridge: the VME bridge is a base module for our test system, since VMEbus is a fully understood commercial system supported by a range of computer interfaces. This module from Dolphin is implemented according to the IEEE substandard P1596.1. With an option to include a Motorola 88110 RISC processor, this 6U VME board conforms to

16 OCR Output FASTBUS VMEBUS

NODE SQLSC; NODE

FUTUREBU5 TURBOCHANNEL

Figure 9: SCI bridges to various bus systems

VMEbus 32 bit protocols. It is expected that a 64 bit VMEbus version will become available later. Fastbus: an interface between Fastbus and SCI is plamied to implement uni·directional data transfers &om Fastbus to SCI, making use of Service Request on the Fastbus port. A Fastbus card with an SCI interface and a processor has an internal, fifo-type data memory with a slave port input on the Fastbus backplane. A small subset of the Fastbus slave protocol is sufficient for data transfers to SCI. Read requests from SCI are converted into Fastbus Service Requests. TURBOcharmel: simple TURBOchannel I/O request are converted into SCI read re quests. SCI responders can subsequently directly write, noncached, into TURBO channels I/O window. We are planning to use an existing TURBOchannel interface board which is based on Xilinx Logic Cell Array (LCA) technology. This board as well as software support can be provided by our collaborators from DEC. ECP-EDA provide the development of the Xilinx interface, using Verilog design tools from Dolphin. Futurebusl': a bridge will be specified by an IEEE Working group, since commercial interest for such an interface is high. Major computer manufacturers plan to use Futurebus+ for high performance future platforms. A participation in this activity is desirable for our

17 OCR Output project and contacts with IEEE are maintained. INFN of Rome have expressed interest in this development. SCI-SCI: one of the most important bridges is between SCI ringlets. Dolphin plans to design a VLSI implementation towards the end of 1992. SCI·SBus: we have no immediate plans for such an interface though the speed and the popularity of SBus justifies such an interface. We assume that one of our industrial partners will design such an interface in the future.

3.6 SCI/VME Single Board Computer

Radstone Technology propose to design an Advandced Processor Extension Interface called APEX *, containing an SCI interface and coherent cache memory [fig. 10]. The SCI/ APEX will plug into an existing 68040 processor card, the 68-42, thus providing access to a wide range of software Development can start in the middle of 1992. The resulting SCI/VME SBC is particularly attractive in smaller sub—systems which require VMEbus, a connection to an SCI network, a general purpose processor and an existing software base.

3.7 Intelligent Data Controller

Creative Electronics SA (CES) intends to study the feasibility of an VMEbus based Intelligent Data Controller (HJC) with an SCI connection [fig. 11]. The principal components of the IDC are the R4000 RISC processor from MIPS, on-board memory, an SCI interface and a connection to a VICbus or HIPPI interfaces which can be stacked via an internal bus to allow collecting data from distributed buses. The VICbus allows accessing other VMEbus crates, Fastbus, CAMAC or HIPPI. CES intends to have a. working prototype available by the beginning of 1993. Preliminary studies can be carried out using a VMEbus-SCI bridge, the RIO 8260 from CES and the HIPPI Source Daughter Module developed by the ECP division at CERN [20]. The IDC can be used as a bus concentrator with optional local 2'“* level trigger processor. ‘“" OCR Output

3.8 Diagnostics, using a Protocol Tracer

A complete test system for SCI consists of an SCI ringlet, a workstation and diagnostic devices and tools. A user interface to the SCI Protocol Tracer from Dolphin is required for easy application of diagnostics to SCI tests. It is implemented as a VME card, acting passively on an SCI ringlet. Low level libraries will be available for displaying bit patters. Higher level libraries for the portable interactive user interface (based on X-windows) for a UNIX environment are required. Diagnostic hardware, the SCI Tracer, and low level libraries will be provided by Dolphin. The University of Oslo has started to develop the user interface. It may be necessary to adapt it to other platforms, such as SUN Sparc or DECstation.

*Rndstoue specific line of plug-in modules

18 SCI

SCI CHIP

..i;.¤¤¤c‘

mm Processor bus

adaptor Adaptor HIPPI/VIC

VMEbus

Figure 11: Intelligent Data Controller.

3.10 Modelling and Simulation

Simulation is widely used by many of our partners for various stages of hardware and software developments. Available today, Dolphin provides a complete, vendor independent simulation of the SCI node chip in VERILOG. Also existing today are complete SCI based systems comprising several nodes, simulated on a cycle by cycle basis (2ns) using the C routines which are part of the standard SCI protocol specifications. The Institute of Informatics at the University of Oslo where these simulations were carried out has also developed a simulation program written in SIMULA as a research tool to study cache coherence protocols. At CERN, we have developed a general SCI Modelling Program to simulate large data acquisition systems containing thousands of nodes. A precise simulation of the data flow at a granularity of z 100 ns (the typical SCI packet size) is now possible due to the availability of cheap workstations. Several global aspects of Data Acquisition Systems must be studied to evaluate latencies, memory requirements and scalability of different topologies. Detailed studies are needed to isolate and optimize critical paths and estimate throughput under different conditions (initialisation, sustained traffic, noise). We intend to continue architectural studies all the way along the project, and make the tools available to designers and integrators of data acquisition

20 OCR Output

LHC Ddtd Acquisition System Model nodes 1 — 1083.

OCR Outputvi 0.2 0

00.16

@0.12

80.0s

>0.04

0 200 400 600 800 1 000 SCI node Id Node Input Link Bandwidth

}0.05

@0.04

90.03

:0.02

@0.01

3 0 0 200 400 600 600 1000 SCI node Id Node Bandwidth

Figure 12: Simulation of an SCI based data Acquisition System with data distributed over ¤ 1000 memory nodes. The upper plot shows the raw data traffic on the SCI links. The lower plot shows the amount of data flowing into the processors belonging to the trigger farms and data logger. systems. We have chosen MODSIM H [21], a commercial, object oriented simulation language which is now being used by many other HEP institutes.

21 OCR Output A Collaboration with industry

The European computer industry is providing a large measure of essential support for our proposal. Several companies are involved, lending their expertise as well as design and development effort. In addition to bilateral contacts with our direct partners (listed below), the Project has numerous relations with industry at large through its involvement with the IEEE SCI stan dardisation process. These include: Apple Computers (personal computer networks), Hewlett Packard (optical SCI applications and specific SCI memories) and National Semiconductor (low-voltage CMOS implementations of SCI). DOLPHIN SERVER TECHNOLOGY AS (Norway) Dolphin is the supplier of indispensable technology and expertise for the SCI. Their personnel are heavily involved in the IEEE standardisation process for the SCI itself (IEEE P1596) as well as a VMEbus to SCI bridge (IEEEP1596.1). They have taken the world-wide lead in developing the VLSI parts necessary for the initial implementation of SCI node. They are contributing one engineer who is working full-time at CERN paid by Dolphin and the Norwegian Research Coumcil for Science and Humanities. RADSTONE TECHNOLOGY plc (United Kingdom) Radstone is a major manufacturer of VMEbus boards and VMEbus-based systems. They are bringing that expertise into the project to design and construct an SCI general-purpose CISC-based processor module in VMEbus. This device will provide the means to graft SCI into existing data acquisition systems and allow the early testing of some basic architectural concepts without major investments in hardware and software. CREATIVE ELECTRONIC SYSTEM SA (Switzerland) CES is a very important supplier of VMEbus and related equipment to CERN and other research laboratories. They are bringing their expertise in VMEbus RISC microprocessor implementations and system interconnect technology to provide an intelligent data controller which will be used for the collection and high-performance processing of data.

22 OCR Output B Budgets

Our budget estimation covers one year of purchase and development of SCI test equipment allowing us to set up an SCI Ringlet Test System. Continuing future work will require an additional budget.

kCHF VME basic SCI equipment: 72 [fig. 13] Diagnostic Tracer and CPU: 45 [fig. 13] Software Development tools: 34 +15 seed money 1991 4 SCI General Purpose Interfaces: 36 Inhastructure trips and visits: 50 Computers: 23 [fig. 13] Instruments, use and purchase: 35 Cables and Connectors: TUR.BOchanne1 Interface: 40 [Hg. 14] Fastbus Interface: 25 [fig. 14] Futurebus Interface + Crate: 45 Technical Student: 40 Qotal Budget: 452

Budget Partitioning

Dolphin 55 kCHF (Partial funding of B.So1berg Associateship, Verilog tools) University Oslo 30 kCHF (TRACER. and CPU) DEC 40 kCHF (VAXstation and Xilinx board) CES 20 KCHF (IDC prototype) Radstone 20 kCHF (SCI processor prototype) INFN 45 kCHF (Futurebus+ base equipment) CERN 242 kCHF

23 OCR Output VME

¢Zf‘¤‘*E;} #:2E:¤r¤¢=¤¢¤:§:ici¢=:¤:¤:¤:*:¢:¤·¤ :;:;:;E§?"‘f[ '‘'‘" ‘ `’`‘" $55555 z; - -;

snus

EXTERNAL Bus

SUN station

_A..» H ......

Figure 13: VMEbus Modules for Rjnglet Test

sam 88110

Exsrnus TURB0 OCR Output

DEC station VAX station

F`;|7`|11'9 14- "FTTRPO¢·}·mnnn]-R(YT-F‘a

24 C Responsibilities

As shown in the timescales and milestones, responsibilities are required for various subpro jects during the development phase. For a future continuation, responsibilities need to be negotiated at a later stage.

Verilog Tools Dolphin Server Technology A.S. General Purpose SCI Interface Dolphin Server Teclmology A.S. SCI Memory Dolphin Server Technology A.S. SCI Chips Set Dolphin Server Technology A.S. SCI to VMEbus Bridge Dolphin Server Technology A.S. Diagnostic Tracer Hardware Dolphin Server Technology A.S. SCI to SCI Bridge Dolphin Server Technology A.S. Diagnostic Tracer Software University of Oslo, Physics Department rI`l1.I'bOCl'l8Il.1'l€l/ SCI Interface Sw Digital Equipment Corp. CERN Joint Project Intelligent Data Controller Creative Electronics S.A. VMEbus / SCI General Purpose Proc Radstone Technology plc Futurebus-}- Bridge INFN Rome Ringlet Test System P33, CERN SCI Test Software P33, CERN Global 2nd Level ’I`rigger Tests P33, CERN 3rd Level Trigger Tests P33, CERN Data Logger Tests P33, CERN Multi-Ringlet Test System P33, CERN Participation in Data Acq. Architecture Studies P33, CERN Modsim H Simulations P33, CERN SCI to Fastbus Interface P33, CERN Turbochannel/SCI Interface Hw P33 CERN Delphi, CERN SCI DAQ Proposal + Softw. to be negotiated

25 OCR Output

OCR OutputReferences

{1] N.E1]js, S.Citt01in and L.Ma.pe1li, CERN Signal Processing, 'I&·iggering and Data Acquisition Large Hadron Collider Workshop Aachen Oct 90, Proceedings Vol I, ECFA 90-133

[2] J .F.Renardy et al.,SCI at LHC Large Hadron Collider Workshop Aachen Oct 90, Pro ceedings Vol IH, ECFA 90-133 p. 165 ii`.

[3] Buses and Standards for LHC H. Miller Convenor Large Hadron Collider Workshop Aachen Oct 90, Proceedings Vol m, ECFA 90-133, p. 161 fl`, and: CERN/ECP 90-10 [4] H. Miiiler et al., New buses and links for Data Acquisition CERN-ECP-91-xx, Submitted to NIM Proceedings of Sth Pisa meeting o.a.dvanced Detectors [5} Development of High Resolution Si Strip detectors for Experiments at High Luminosity at LHC, CERN DRDC 91-10

[61 SITP Collaboration, A Proposal to Study a Tracking/Preshower Detector for LHC CERN/DRDC/P3 and A.Poppelt0n, Fast electron triggers from a silicon track/ preshower detector ECFA 90-133, 201 [71 High Performance Parallel Interface Mechanical, Electrical and Signalling Protocol Speciication, Proposed ANSI standard X3T9.3, Bob Morris, Intelligent Interface Inc. Chairman

[8] SCI Scalable Coherent Interface, Proposed standard IEEE P1596 D.B. Gus tavson, SLAC Chairman Specifications in Postscript or Mac format are available on ftp server: hplsci.hpl.hp.com in pub / sci

[9] A. Bogaerts et al., SCI based Data Acquisition Architectures, to be published in IEEE Real Time ’91 Conference Proceedings, IEEE Seventh Conference 25 - 28 June 1991, Jiilich, Fed. Rep. of Germany. [10] EAST collaboration, Embedded Architectures for Second-level Triggering in LHC Experiments CERN/DRDC/90-56 [11] Model PT-SBS915 SBus to VMEbus adapter from Performance Technology Inc. New York USA.

[12] RSTB collabortaion, Readout system test benches, CERN/DRDC/90-62, P15, RD12

[13] W. L. Gore & Assoc. (UK) Ltd. Pitreavie Business Park, Dimfarmline Fife Scotland, UNITED KINGDOM KY11 5PU (Information sheet on Goretex cables and its application to SCI ) [14] EIA IS 64 2mm Device Connector System, DuPont Company, Electronics Division, New York USA

27 OCR Output [15] CSR1212 is a proposed IEEE standard for Control and Status Register architecture for SCI and Futurebus, preliminary specifications in Postscript or Mac format are available on ftp server: hplsci.hp1.hp.com in pub / csr

[16] A Scalable Data Taking System at a Test Beam for LHC CERN/DRDC/90-64 P16, RD-13

[17] FERW collaboration A Digital Front-end and Readout Microsystem for Calorimetry at LHC CERN/DRDC/90~74, P 19, RD16

[18} Dolphin Server Technology A.S., Oslo, PO Box 52 Bogerud, Norway, ( SCI project) [19} Hewlett Packard, 1501 Page Mill Road 3U, Palo Alto, CA 94304 (Giga·Link project)

[201 T. Angelov et al. HIPPI Developments for CERN Experiments to be presented at the Nucl. Science Symposium, S.Fe 1991 and R.C. Walker et al. 1.5 Gbit/s Link Interface Chip Set for Computer Data Transmission Instruments and Photonetic ,... Laboratory, hpl 90105 July 1990

[21] CACI Products Company, La Jolla, CA, USA “MODSIM II”

28