RD24 Status Report 1994 EUR0PEAN ORGANIZATION FOR NUCLEAR RESEARCH CERN/DRDC/ 94-23 RD24 Status Report 9 May 1994 RD24 Status Report Application of the Scalable Coherent Interface to Data Acquisition at LHC A. Bogaerts1,R.Keyser,H.Müller1, G. Mugnai, P. Ponting, D. Samyn, P.Werner CERN, Geneva, Switzerland B. Skaali, E.H.Kristiansen2,H.Golparian,J.Wikne,B.Wu University of Oslo, Department of Physics,Norway S.Gjessing University of Oslo, Departement of Informatics,Norway S. Falciano, F. Cesaroni, G. Medici INFN Sezione di Roma and University of Rome, La Sapienza, Italy P. Creti, M. Panareo INFN Sezione di Lecce, Italy A. Sytin, A. Ivanov, A. Ekimov IHEP, Protvino, Russia E. Sanchis-Peris, V. Gonzalez-Millan, J.M. Lopez-Amengual, A.Sebastia, J. Ferrer-Prieto IFIC, Valencia, Spain F.J.Wickens, D.R.Boterill, R.W.Hatley, J.L.Leake, R.P.Middleton Rutherford Appleton Laboratory, Didcot, UK R.Hughes-Jones, S.Kolya, R.Marshall, D.Mercer University of Manchester, UK K. Løchsen, S.E. Johansen, H. Kohmann, E. Rongved Dolphin Interconnect Solutions A.S., Oslo,Norway A. Guglielmi, A. Pastore Digital Equipment Corporation (DEC), Joint Project at CERN F-H. Worm, J. Bovier, A.Lounis Creative Electronic Systems (CES), Geneva, Switzerland R.Hon, D. North, G. Stone Apple Computer, Inc. Cupertino USA E. Perea Thomson-TCS Semiconducteurs Specifiques, Orsay, France 1. joint spokesmen 2 RD24 Status Report 1994 Motivation for SCI RD24 2nd Phase Activities and Milestones Previous activities and milestones of the RD24 collaboration are reported in the DRDC reports with the same title, referenced as CERN/DRDC/91-45 (Proposal P33), CERN/DRDC/ 92-06 (Addendum to Proposal P33), CERN/DRDC/93-20 (First RD24 Status Report May 93). These documents and many others are available on the public ftp server rd24.cern.ch in the directories sci/RD24_Info/Status_92, Status_93 and Status_94. Chapter 1. Motivation for SCI The goals and motivation of the RD24 first phase remain unchanged. The Scalable Coherent Interface (SCI) IEEE standard [3]after demonstrating an impressive 500 Mbyte/s link operation (Figure 1) during RD24’s first phase [1][2], has now low-cost CMOS chips and plug- SCI Link Signals: FLAG 4 16xDATA 18 SCI CLOCK Signals 4ns 16 bit DATA * 4ns = 1/2 GB/s SCI Packets: HEADER CRC DATA 0, 16, 64, 256 bytes ADDRESS (64 bits) + COMMAND Figure 1: 1993: SCI packets at 500 Mbyte/s in boards for SBus and VMEbus available. Technology independence from low cost CMOS to high performance GaAs implementations, all conforming to the same standard, now becomes visible. The options for caching and cache coherency though not at present available, me be added in the future without affecting present investment in applications. SCI has proven existence and now awaits implementations of further VLSI components and board level products to fill a range of compatible building blocks. These products will provide bus-like services between memories and processors over high speed, optimally terminated point-to point links. SCI is not designed for wide area communication, where ATM will have its dominant role, nor for very low latency environments like a 1st level trigger, nor for massive channel electronics environments where packaging, power and cooling plays an important role. In order to interface SCI to these boundary areas a variety of general purpose and specialized bridges are needed, some of which are soon expected to become commercially available such as SCI-ATM and SCI-VME bridges. Special bridges to the ATLAS 2nd level DSPs and the CMS DPM readout units are being designed or planned in RD24. The most suitable area for SCI is to provide high speed interconnections between front end data acquisition units and event builders, and between event builders and computer farms. This also means simplification by introducing a uniform, scalable and standard system between the application (the physicist) and the data buffers after the 1st level trigger. Simplification is also due to the support SCI provides in transparently accessing memory over SCI: the application software is not concerned whether the memory is local or remote. 3 RD24 Status Report 1994 CERN RD24 Milestones and activity report Another notable feature is the natural possibility of supporting bi-directional data streams within the same SCI system. The same chips support a data driven architecture to move the data from a second level trigger processor as well as supporting a DMA type broadcast to download streams of calibration constants from the processor farms. In addition to scaling, SCI is also adaptable to changes in data acquisition and trigger architectures. SCI bandwidth starts off with a potential of 125 MByte/s link rate available today across SCSI-2 like cable assemblies or Gbit optical fiber links. Performance enhancements by factors between 2 to 8 are expected soon for coming SCI components. SCI starts where buses become ineffective due to bandwidth limitations for long distances. SCI however is implemented over thin, cheap and flexible links (Fig 2) currently 18 signals for the parallel implementation. A 16 DATA +1Clock +1FLAG -> 18 pairs + _ 2 bytes @ 62.5 MHz => 125 Mbyte/s 4.2 V 2Bytesat each Transition 3.0 V 16-19 ns DIFFERENTIAL Pseudo-ECL ( ECL + 5Volt) Connector: 50 pin Cables 18 pairs 50 OHM Nodechip A 3V Nodechip B 18 pairs 50 OHM 3V Figure 2: SCI’s thin signal path between nodechips ringlet connection between two SCI nodes requires, apart from the node chips, two SCI cable assemblies and termination resistors for one outgoing and one incoming connection. SCI chips with 4 ns clocking (1993) required very high quality cables whereas the 16 ns clocking on low cost CMOS chips allows the use of low priced and popular SCSI-2 cable assemblies. Such small and robust 50 pin connectors proposed by RD24 have been adopted by the first European vendors. AMP, a world leader in cable assemblies, considers producing these specialized for SCI. Future SCI chips will probably use the SCI LVDS [4] (Low Voltage Differential Signaling) standard running at 2ns clock rate. LVDS, due to low current and low voltage swing allows the chip manufacturers to put the link termination inside the link controllers. LVDS has been used by National Semiconductors for Quickring chips at 300 MHz over 3 m cables. Chapter 2. CERN RD24 Milestones and activity report Several activities of RD24’s 1st phase [2] in particular design work on the SCI DMA node have largely contributed to today’s commercial European starter systems using the CMOS node chips from LSI Logic.[5] These chips became available only early 1994 and then required 4 RD24 Status Report 1994 CERN RD24 Milestones and activity report revision due to a problem in the bypass FiFo. RD24 had the first corrected chips available on SBUS and VME cards at CERN in May 1994 allowing us to test functionality and performance of a first two-ringlet system in a VMEbus based test setup in the ECP Division at CERN. Our download, login and test Old GaAS tests on single Sbus ringlet RIO in 1993 PT-SBS915 VMEbus REQUESTER FIC 8234 Os/9 REQUESTER RESPONDER RESPONDER SCI 8224 DMA SBUS Interface SBUS Sun/SPARC Sun/SPARC Cbus Link Diagnosis: 3 m SCI Node Node cables for different, chipNode chip single or multiple ringlet tests. Back-Back Bridge SCI test ringlets State Analyzer HP 1661 Functionality tests SCI-SCI bridge ( 16 Id’s passed to far-side ringlet) Waveform Analyzer Lecroy 9450 Figure 3: RD24’s 1994 multi-ringlet SCI test setup at CERN previously reported tests [2] were performed in 1993 using a VME -SBUS connection and software downloaded into the RIO Interface which had a GaAS node chip connected to the R3000 bus. In 1994 the laboratory setup (Fig 3) was extended to include: • Two SCI -SBus interfaces (Dolphin) with CMOS node chips for single and multiple ringlet and node tests between two Sun Workstations • A protoptype of an SCI-SCI bridge based on two CMOS node chips • One FIC8234 VMEbus processor equipped with the CES 8224 SCI-DMA card • Two 1.2 Gbit/s Glink interfaces (Lasertron) with adapters to the Sun SCI-connector and an optical fiber SCI ringlet (not shown) • Diagnostic instruments, power supplies and VME crates from the CERN pool A second laboratory for software developments (OS9, CASCADE) has been equipped with two SUN Workstations connected into an SCI ringlet. 2.1 SCI CMOS Node chip tests (CERN) We measured latencies for all SCI subactions which make up a complete SCI transaction (Fig 4). Using a requester and responder on the same SCI node and a diagnostic software set from Dolphin [6] installed on a SPARC IPC workstation, we measured latencies for read, write 5 RD24 Status Report 1994 CERN RD24 Milestones and activity report Requester A TEST CONCEPT Requester B Responder A Responder B SCI TRACER TEST TARGET REQUESTER RESPONDER T=0 request ∆ 5 ns/m + N * bypass delays ( t1) ∆t1 = echo latency T=1 request echo Performance limit for responseless ∆ Move transactions t2= access latency T=3 response Performance limit for transactions with 1 outstanding request capability only T=3 Performance response echo ∆ t3 =∆ t1 limit for responder Figure 4: Timing measurement concept on SCI subactions and move transactions both in a closed loop and between two node chips. We used an HP 1661 state analyzer which was connected via probes to both the input and output links of the node chip under test. The probe voltage levels were adapted to Pseudo-ECL switching levels, i.e. 3.7 V. 2.1.1 Viewing of packets and performance measurements First packets (Fig. 5) from a CMOS SCI node chip on a two node ringlet show that data on the link are clocked at every transition of the clock (NCLK) whilst on the Cbus application bus, the CCLK works at 1/4 speed.