Home Search Collections Journals About Contact us My IOPscience

DDL, the ALICE data transmission protocol and its evolution from 2 to 6 Gb/s

This content has been downloaded from IOPscience. Please scroll down to see the full text. 2015 JINST 10 C04008 (http://iopscience.iop.org/1748-0221/10/04/C04008)

View the table of contents for this issue, or go to the journal homepage for more

Download details:

IP Address: 131.169.4.70 This content was downloaded on 11/01/2016 at 22:38

Please note that terms and conditions apply. 2015 JINST 10 C04008 d c EDIALAB M April 10, 2015 C. Delort, : a January 16, 2015 January 22, 2015 : : ISSA November 10, 2014 S : EVISED UBLISHED G. Simonetti, P R CCEPTED b ECEIVED A 2014, R S. Chapeland, a 1 , UBLISHINGFOR T. Kiss, a HYSICS doi:10.1088/1748-0221/10/04/C04008 a P IOPP F. Costa, a C. Ionita, ARTICLE a P and B. Von Haller UBLISHEDBY a P A. Grigore, a LECTRONICSFOR E V. Chibante Barroso, P. Vande Vyvre a a U. Fuchs, RANCE a 2014, ` ` eve 23 Switzerland ,F a, W. Carena, R. Divi ORKSHOPON : Optical detector readout concepts; Data acquisition concepts a A. Telesca, : ALICE (A Large Ion Collider Experiment) is the detector system at the LHC (Large b CERN 2015, published under the terms of the Creative Commons Attribution 3.0 W ROVENCE filippo.costa@cern.ch EPTEMBER a c License by IOP Publishing Ltd and Sissa Medialab srl. Any further distribution of this P ´ os, ´ enes, Corresponding author. 1 BSTRACT EYWORDS IXEN Ministere des affaires etrangeres er europeennes, France E-mail: Research Institute for Particle and NuclearBudapest, Physics, Hungary Wigner Research Center, Ludwig Maximiliams University of Munich, Munich, German CERN - European Organization for NuclearCERN Research, CH-1211 Gen OPICAL c b a d T 22–26 S A A K Hadron Collider) that studiesplasma. the The information behaviour sent of by the stronglyAcquisition sub-detectors and interacting composing Test Environment), ALICE matter the are ALICE and read datamode out acquisition the by optical software, DATE quark using (Data links hundreds gluon called of multi- DDLLHC, (Detector the Data bandwidth Link). ofevolution To the of cope the DDL with DDL links the protocol from will higher 2 luminosity be to of upgraded 6 Gbit/s. the in 2015. This paper will describe the C. So work must maintain attribution to the author(s) and the published article’s title, journal citation and DOI. E. D F. Carena, DDL, the ALICE data transmission protocolevolution and from its 2 to 6 Gb/s 2015 JINST 10 C04008 – 1 – : a VHDL block implemented in the FPGA of the Readout ): a mezzanine card connected to the Front End Electronics Receiver Card (RORC), a PCIX or PCIedata custom from card, the installed in detectors the and DAQ stores farm that information receives in the memory of the hosting PC. Source Interface Unit (SIU Destination Interface Unit (DIU) (FEE). • • 5.1 DDL25 verification 1.1 The ALICE Detector Data Link1 1 Introduction ALICE [1] is the heavy-ion experiment builtthe to Quark-Gluon study Plasma the in physics nucleus-nucleus collisions of at stronglyALICE the interacting consists CERN matter LHC and of (Large Hadron eighteen Collider). sions sub-detectors (Pb-Pb). and its It also primary(pA) has targets collisions. a are The substantial experiment heavy-ion physics has lead-leadanticipated been program for colli- designed with Pb-Pb to reactions. proton-proton cope (pp) with the and highest proton-ion particle multiplicities 1.1 The ALICE Detector DataThe Link ALICE Detector Data Linkdata (DDL) transfer [2] between has been the designedsion detectors to of and address the the all protocol, ALICE the calledallows requirements Data DDL1, the for bi-directional is Acquisition transmission a system of 2.125 data [3]. Gb/ssition between full the system. duplex, The front multipurpose end During first electronics optical data ver- andthe fibre taking the DAQ, link. data while the acqui- at It information each flows startthe from detector of the using a the front new opposite run end communicationnents it electronics channel. (figure1): is (FEE) The possible DDL1 to to consists send of configuration three main control compo- messages to 2 DDL2 firmware 3 DDL2: motivation and3 requirements 4 DDL23 performance 5 Implementation in different FPGAs4 6 Conclusions6 Contents 11 Introduction 2015 JINST 10 C04008 : it encapsulates the data coming from the detector – 2 – : the SIU and old release of the RORC cards are using a : a multi-mode OM2 optical fibre. The detectors are connected to the : the serializer-deserializer transmits serialized data over high speed, differential : the DDL works using 32 bit words, however the SERDES provides a data bus . DDL1 components in the data taking configuration, FEE connected to the DAQ PC. : a cyclic redundancy check verifies the data to detect possible errors during the data Framing DDL protocol and Command interface transmission. interface at 16 bit. This components receives 32 bit word and splits it intoin two 16 DDL bit data words. blocks.control signals When so the commands FEE are can react sent to from incoming the commands or RORC configuration to data. the FEE it raises the TLK2501 RX/TX state machines CRC SERDES pairs and it recovers and aligns the data on the receiving side. transceiver from Texas Instruments, TLK2501.link This and chip keep has state the machines dataInstruments to aligned. has recover been the In replaced the bysimulate latest the the release SERDES chip of available behaviour in the the soequipped RORC with FPGA. the the the The transceiver code TLK2501. state of is machines Texas backward compatible with other cards that are DAQ farm using different length of fibres up to 200 m. The physical medium Figure 1 The DDL1 is a point to point protocol, so each SIU is connected directly to one channel of the • • • • • • RORC through the optical fibre. 2 DDL firmware The DDL provides a commonDAQ data system. transfer protocol to It the encapsulatesthrough detectors the the that optical data are fibre. connected collected Its to from main ALICE the components are: FEE in a DDL frame and serializes it 2015 JINST 10 C04008 – 3 – . Main DDL firmware components and their communication paths. : it is a 32 bit bi-directional bus used to interface the user logic to the DDL. Figure 2 FEE interface Using this interface it is possible to send data and receive configuration instruction. An exact copy of the data received by the DAQ is sent to the ALICE High Level Trigger (HLT) For the DDL2 the SIU card has been replaced by VHDL coreUsing to a be VHDL implemented core in to the interface FPGA the detectors to the DAQ system it can be adapted to different • to perform online tracking andcards, data called reduction. HRORC (HLT Readout In Receiver the Card).their previous PCI-X boards period is using obsolete run the by HLT same now,DDL was so hardware protocol. using HLT installed upgraded PCI-X in the ALICE DAQ system upgrading asfirmware well of the the readout electronics (figure3). cards, benefitting from thespeed resources serial available links. on the newest FPGA, high speed4 clock and high DDL2 performance During Run2, ALICE will useThe different experiment speeds will of collect the datasame DDL using time. due both to The generations different DDL1 of detector is the electronics. running data at transmission 2.125 protocol Gb/s, at while the the DDL2 will run at higher speed. The The connections between the different DDL blocks are showed in the figure2. 3 DDL2: motivation and requirements ALICE foresees a consolidation inDue 2015 to to higher data cope throughput with generated by theand two increased the sub-detectors, Transition luminosity the Radiation Time of Projection Detector the Chamber (TRD) (TPC) speed accelerator. the of DDL up protocol to has 6 been Gb/s. maintained but at a faster 2015 JINST 10 C04008 – 4 – . DDL2 components in the data taking configuration, FEE connected to the DAQ PC. The SIU card The other detectors will use theelectronics DDL1, to however implement some the of DDL2 them for are thebeen RUN3 planning ported in to also 2018. upgrade for For their XILINX this Kintex7 reason and the Virtex7. DDL code has The DAQ and HLT system will use XILINX Virtex6. TRD will use XILINX Virtex4. TPC will use MICROSEMI SmartFusion2. The ALICE DAQ system and its readout receiver card have been thoroughly tested with the • • • • Although the DDL protocol is mainly VHDLrequires custom particular code, the attention implementation to in different the families Property timing blocks, constraints or and IP thetures cores. instantiation of of the The specific devices, user, Intellectual asboth using the used these SERDES in “FPGA (SERializer the DESerializer) functions”, code. andhavior. can the access These IP FIFO specific differences cores (First-in, increased fea- of First-out) maintaining the different the development FPGA same time code differ for and in different introduced user devices. an interface additional and effort sometimes in in be- Figure 3 has been replaced by a VHDL component. TPC is aiming at 4.25 Gb/s,the detector while hardware TRD where will the use DDL protocol 4 Gb/s. is implemented. The different linkFEE using speeds DDL2 are at defined readout by speeds, from 2.125 Gb/s up to5 5.3125 Gb/s. Implementation in different FPGAs The biggest challenge in thisfamilies. project TPC, has TRD, HLT been and DAQ the aredata, implementation using so electronics of equipped the the with DDL2 different code protocol FPGAs has in to been different collect implemented FPGA in the following devices: 2015 JINST 10 C04008 – 5 – . DDL throughput, using different link speed and varying the event size. Figure 4 MICROSEMI SmartFusion2: 2.125 Gb/s.speed. More development is needed to run at higher XILINX Virtex4,Virtex6, Kintex7, Virtex7: 2.125 Gb/s, 3.125 Gb/s, 4.25 Gb/s 5.3125 Gb/s The SIU-loopback program runs on the machine that hosts the RORC. It generates different In addition to that a data generator has been implemented in the SIU VHDL component. In For each implementation different speeds of the DDL has been successfully verified: The critical part in the DDL protocol is the implementation of the SERDES. XILINX deliv- • • pattern of data (32protocol. bit From word), the incremental SIU or dataor is alternated other sent and errors back in sends to the data it the transmission.collecting RORC to more For and the each than the FPGA 80 program SIU, implementation TB checks the using of if DDL data the has there without been DDL are errors. stressed bit flip this configuration it has been possiblesends to data collect to data the using RORC using the the DATEcheck software. DDL to protocol. The verify The data that DATE generator software the collects information it are and well perform sanity formatted. ers strong and flexiblelatest solutions product in of their MICROSEMI, Smart FPGAspecial Fusion2, to attention is access during not the the as highsary implementation. robust and speed as they its Delay serial have counterpart lines to links, and betransmitting and while channels. it inserted synchronization the between requires FIFOs the user are logic neces- and the SERDES, for both5.1 receiving and DDL2 verification The different implementations ofsoftware the framework DDL2 that protocol provides have softwareused been to tools tested verify to the using check different the componentsproblem the of during ALICE DDL the data DATE DDL taking. status. (SIU, optical These fibre programs and DIU) are in case there is a 2015 JINST 10 C04008 , , (2014) 130. A 741 S08002. 3 , Aix en Provence, JINST , 2008 Nucl. Instrum. Meth. , – 6 – 5th Conference on Electronics for LHC Experiments , The ALICE experiment at the CERN LHC The ALICE data acquisition system Topical Workshop on Electronics for Particle Physics 2014 The C-RORC PCIe card and its application in the ALICE and ATLAS experiments The ALICE detector data link ´ os et al., Snowmass, CO, U.S.A., 20–24 September 1999. proceedings of the France, 22–26 September 2014. [1] ALICE collaboration, [3] ALICE collaboration, [2] C. So [4] H. Engel et al., 6 Conclusions The DDL2 protocol has beenand successfully MICROSEMI. ported Different and speeds tested of in theXILINX different link devices. FPGA have One families, been of XILINX tested, the fromthe main 2.125 DDL challenges up protocol in to in the 5.3125 development MICROSEMI Gb/sMICROSEMI has FPGA, in reference been design, SmartFusion2. had the to Different implementation be coding of appliedhas to technique, been use following properly tested the in new technology. aunder The stable development. protocol During way Run2 at the 2.125 number Gb/s.connecting of links to The that the will targeted DAQ system use speed the TPC, for DDL2 TRD will and the be HLT. TPC, a total 4.25 Gb/s, of 240, isReferences still