Xtca for Instrumentation
Total Page:16
File Type:pdf, Size:1020Kb
xTCAxTCA forfor InstrumentationInstrumentation Tomasz Jezynski Deutsches Elektronen-Synchrotron What do we want? (DAQ, Beam Instr., Accelerators, Trigger) Modularity Scalability Robust Some additional requirements: • Serviceability (avoid front panel connection) • easy upgrade path, flexibility • Available in the next 20 years (multi-vendor) • Low latency (fast communication links) • Support modern control algorithms • Limited number of different boards • Low-cost version available (monitoring) • Reliability, operability and maintainability ... Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 2 Evolution of Standards CAMAC BCNAF 1970 - 1990 From Computer INDUSTRY PS- ISR- SPS- LEP 1980 - 2007 FASTBUS VME UA1-LEP -LHC 1980 - 1995 VME64 VME 2eSST (dual edge, UA2-LEP source synchronous data transfer ) VXS (VME Switched Serial) VPX (V.46) [PMC XMC] CompactPCI 2002 - ? From HEP ATCA ? AGATA (ESONE) XFEL From Telecom ILC, ... INDUSTRY Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 3 xTCA microTCA AdvancedTCA MTCA.0 - Micro Telecommunication PICMG 3.0 is the base specification Computing Architecture (backplane, (backplane, connectors, power, connectors, power, management, size, management, size, cooling, ...) cooling, ...) • PICMG 3.1 (Ethernet), • PICMG3.2 (InfiniBand) • PICMG 3.3 (StarFabric), • PICMG 3.4 (PCI Express), • PICMG 3.5 (RapidIO) PICMG - PCI Industrial Computer Manufacturers Group Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 4 xTCA microTCA AdvancedTCA MTCA.0 - Micro Telecommunication PICMG 3.0 is the base specification Computing Architecture (backplane, (backplane, connectors, power, connectors, power, management, size, management, size, cooling, ...) cooling, ...) • PICMG 3.1 (Ethernet), • PICMG3.2 (InfiniBand) • PICMG 3.3 (StarFabric), • PICMG 3.4 (PCI Express), AMC • PICMG 3.5 (RapidIOBlades) AMC.0 - Advanced Mezzanine Card AdvancedTCA blades can be: Base Specification (connector, power, Processors, Switches, AMC size, cooling) * AMC.1 PCIx carriers, etc. * AMC.2 Gigabit Ethernet and XAUI * AMC.3 Storage (ratified) PICMG - PCI Industrial Computer * AMC.4 Serial RapidIO Manufacturers Group Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 5 xTCA microTCA AdvancedTCA MTCA.0 - Micro Telecommunication PICMG 3.0 is the base specification Computing Architecture (backplane, (backplane, connectors, power, connectors, power, management, size, management, size, cooling, ...) cooling, ...) • PICMG 3.1 (Ethernet), • PICMG3.2 (InfiniBand) • PICMG 3.3 (StarFabric), • PICMG 3.4 (PCI Express), AMC • PICMG 3.5 (RapidIOBlades) AMC.0 - Advanced Mezzanine Card AdvancedTCA blades can be: Base Specification (connector, power, Processors, Switches, AMC size, cooling) * AMC.1 PCIx carriers, etc. * AMC.2 Gigabit Ethernet and XAUI * AMC.3 Storage (ratified) PICMG - PCI Industrial Computer * AMC.4 Serial RapidIO Manufacturers Group Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 6 xTCA microTCA m AdvancedTCA TC A ≈ AM C c arr ier AMC Blades AMC.0 - Advanced Mezzanine Card AdvancedTCA blades can be: Base Specification (connector, power, Processors, Switches, AMC size, cooling) * AMC.1 PCIx carriers, etc. * AMC.2 Gigabit Ethernet and XAUI * AMC.3 Storage (ratified) PICMG - PCI Industrial Computer * AMC.4 Serial RapidIO Manufacturers Group Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 7 ATCA crate 1,5U ZONE3 – user define ZONE2 data transport: Fabric Interface – GbE, PCIEx, ... 8U Base Interface - GbE 14U ZONE1: power distribution and slow control (I2C) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 4,5U 19’’ Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 8 ATCA crate – backplanes Node Node Dual Star Node Node Fabric A Node Fabric B Board Board Node Node Node Node Node Star Node Node Full mesh Fabric Board Node Node 8 pairs between each slot + Node EthFabric & Base Interface Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 9 Why ATCA ? AdvantagesAdvantages scalable shelf capacity up to 2.5Tb/s reliability of up to 99.999% redundant power supply 48V@200 W/slot with adequate cooling (redundant) High speed point-to-point serial connectivity via Full Mesh Backplane fits our needs very well (modular, scalable, robust). Flexible configuration of processing topology according to algorithm within shelf Shelf management for remote configuration and monitoring Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 10 Why ATCA ? • Disadvantages – Not much user experience compared to VME – Limited availability of commercial modules (CPUs etc.) – Vendor statement : We only do business with large telecommunication companies, not with Universities! Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 11 Why xTCA is attractive ? ATCA/mTCA platforms are attractive because of: • the modern serial link architecture, • high availability features and many packaging options. • Less-demanding availability applications can be met economically by scaling back speed and redundancy. • The ATCA specification was originally targeted at the Telecom industry but has gained a much wider user audience. Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 12 AGATA - Structure of the system "Advanced GAmma Tracking Array" ATCA Master GTS 1 cristal CORE Slow 6 Segments control Digitizer SEG 6 Segments DAC Digitizer SEG Core PSA Digitizer Or PrePSA ATCA Slave 6 Segments SEG Digitizer 6 Segments SEG Digitizer Power management 6 Segments Digitizer SEG DAC 6 Segments ~ 360 Mb/s Digitizer SEG pro board Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 13 AGATA – ATCA carrier board FPGA V4LX25 Sorties optiques Trigger Ethernet Connections TCLK Switch Ethernet Connections FPGA V4LX25 Ethernet et 10Gb/s Switch JTAG et setup FPGA V4FX60 Slow control Acquisition Alimentation 48V et I2C Convertisseur DCDC 48/12V Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 14 T2K - DAQ overview (Tokai to Kamioka) . First prototype with a dual star 9 AMC slots µTCA shelf + 2 MCH + 1 PwM . Data acquisition module (AMC) are controlled and read through Gbe Ethernet . µTCA offers direct Gbe connection to each AMC through the backplane . MCH (Ethernet switch) use for external interface (data uplink to event building) . Synchronization could also be made on Ethernet using IEEE 1588 (specific MCH design needed) . Dual star topology could be used to separate control/synchronization from data readout . A local AMC processor board can be added to build a compact test system Possible µP + storage module . Full height single width AMC design : . 32 ADC channels inputs on AMC front side . Gigabit Ethernet output through the backplane . Cost optimization goal Ethernet 32 analog Inputs / AMC Uplink This will result in 34 µTCA racks (9 slots) for T2K 2km Control Synchro Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 15 16 DAQ to to KALMAN KALMAN processing Fast Analog Digital Conversion 1 link perhub 1 link ( 3 MB/s per crystal)per MB/s ( 3 ( 2 Gb/s) (2 EXOGAM2 EXOGAM2 per ATCA carrierper ATCA ) EXOGAM2 EXOGAM2 EXOGAM2 EXOGAM2 2 crystals/ 1NIM board 32=> boardsNIM EXOGAM2 EXOGAM2 EXOGAM2 EXOGAM2 EXOGAM2 EXOGAM2 8 links GAMMAspectrometer Digital per carrier ATCA ( Processing (1.4 Gb/s per ADC channel) Gb/sADC per (1.4 signalsper crystal 7 differential analog crystal per 100kHz ICR < => 16=> carriers 1 link Differential Preamplifiers 1 clover / 1 carrierATCA per carrierATCA and and and 16 clovers 64= crystals EXOGAM2 The EXOGAM2 technical proposal 1 GTSsupervisor 16GTS mezzanines Global Trigger Synchronization Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 Jezynski, 1st DESY Tomasz meeting, 29 March RFTech annual Hamburg DESY - ATCA – AMC carrier blade and RTM Size: RTM – Rear 322 mm x 288 mm Transition Module Power dissipation: Up to 200W Single (redundant) Power input – 48V ATCA – DESY carrier board for LLRF system Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 17 XFEL – carrier board (DESY) • 8 ch. ADC-AMC • AMC - Vector Modulator • AMC - Timing Module • 8 ch. AMC-DAC • Down-converter Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 18 XFEL – AMC modules (DESY) Tomasz Jezynski, DESY 1st annual RFTech meeting, Hamburg 29 March 2010 19 XFEL – RTM (DESY) A D C 1 A D C 2 A D C 3 A D C 4 A D C 5 A D C 6 A D C 7 A D C 8 A D C 1 o-n 6 7 . 8 7 - 4 8 . 1 4 - 6 6 . 8 6 - 6 6 . 3 9 - 7 3 . 7 1 - 6 9 . 9 0 - 6 7 . 1 1 - 7 1 . 3 8 A D C 2 o-n 4 8 . 3 5 - 6 7 . 7 9 - 6 8 . 1 4 - 7 4 . 0 8 - 6 9 . 3 5 - 7 1 . 0 0 - 6 7 . 8 6 - 7 2 . 6 7 A D C 3 o-n 5 9 . 5 1 - 6 6 . 4 7 - 6 8 . 0 9 - 5 2 . 4 3 - 6 6 . 0 8 - 7 0 . 3 9 - 6 6 . 9 8 - 7 2 . 6 0 A D C 4 o-n 6 5 . 5 2 - 6 9 . 5 5 - 4 9 . 0 3 - 6 8 . 0 3 - 6 8 . 8 2 - 6 9 . 8 1 - 6 6 . 6 9 - 7 0 . 7 8 A D C 5 o-n 7 3 . 2 7 - 7 3 . 2 7 - 6 7 . 8 1 - 6 9 . 8 2 - 6 6 . 4 4 - 4 4 . 3 5 - 6 3 . 3 0 - 6 9 . 7 7 A D C 6 o- n 2 . 9 2 - 0 . 4 5 0 . 5 6 - 3 . 2 4 1 7 . 3 0 - 8 . 1 2 4 . 0 8 8 . 2 8 A D C 7 o-n 7 6 . 2 2 - 7 0 . 1 8 - 6 9 . 3 9 - 7 7 . 3 1 - 6 5 . 3 4 - 7 0 . 2 7 - 6 8 . 4 7 - 4 5 . 7 6 A D C 8 o-n 7 0 . 8 0 - 6 3 . 6 2 - 6 2 . 1 5 - 6 9 . 6 5 - 6 7 . 4 8 - 6 2 . 7 9 - 5 2 .