Argonne TDAQ/electronics roadmap
A. Paramonov Argonne National Laboratory High Energy Physics Division
13 October 2017 Brief Overview of Argonne HEP Electronics Group
‣ 8 Staff with variety of skills: 5 engineers, CAD Layout, Instrument Repair, 3 technicians • Provides electronics design & support for Argonne HEP experiments • Supports many projects at the laboratory ‣ Group expertise • High Speed Digital Design: Data acquisition systems, Trigger Processors, Custon-to-commodity networks interface, High Speed Communication (copper and optical links) • Front End Design: Charge Amplifiers, Digitizers, Discriminator, HV, and Switching Power Supplies, - No IC design tools available at Argonne • Reliability testing for radiation tolerance - Failure mode analysis - single event upsets - Design for radiation environement • R&D: Wireless DAQ, Digital HCAL Electronics
2 Recently completed and ongoing projects
1. ATLAS experiment at CERN ongoing and completed • FTK to Level-2 Interface Card (firmware, and boards) • FELIX (firmware) • Hardware Track trigger interface (HTT IF) (firmware) • FELIX for the HL-LHC upgrade of ATLAS (firmware and board production) • Concluded work on Tile Cal upgrade (LVPS, HV_OPTO, QIE FEB, QIE MB, project engineering) 2. proto DUNE ongoing • Front-end electronics for Photon detectors (design of the boards and firmware) 3. CTA all completed • L2 trigger (boards + firmware) 4. g-2 all completed • Trolley motor control • Field measurement electronics 5. R&D Development of wireless techniques in data and power transmission application for particle-physics detectors. 6. Electronics for quantum computers at the University of Chicago ongoing • Digital synthesizer daughter board 7. GRETINA TDAQ System
3 Recent and current projects
Synthesizer for a quantum computer
QIE front-end and HV distribition systems for ATLAS Tile Calorimeter
4 NMR Readout for Fermilab g-2 Experiment
‣ g-2 depends on two numbers: spin precession frequency & B field seen by muons • 70 ppb measure of B required to achieve experiment target accuracy • Achieved 300 parts-per-trillion! measure of B with the redesigned trolley electronics ‣ Field measured with array of 17 NMR probes as trolley moved through storage ring • Refurbish & upgrade BNL g-2 trolley • Digitize NMRs, Control NMR sequence, and readout external sensors (barcode for positional location, temperature, pressure,…) ‣ Special requirements • N2 atmosphere in trolley cylinder; vacuum outside • difficult heat dissipation; low power for components in trolley; <1-1.5W • Minimize distortion of ring magnetic field; non- magnetic components • Minimize cables going around ring
5 Trolley Electronics Upgrade for Fermilab g-2
‣ Replace obsolete microcontroller with low-power SmartFusion chip (ARM-3 + FPGA) for high speed communication ‣ Full digitization of NMR signal for consistent readout of all field probes
6 The main motion controller for the trolley
. 6U rack mount chassis will include: – Galil motion controller – Custom made interface board from Argonne • 8 motor channels have interface to Shinsei driver, 2 limit switches and 1 encoder • 8 analog channels with differential to single ended receivers connected to Galil ADC – Power supply
Motor channel
Analog channel
7 Photon detector readout for proto-DUNE
. The initial requirements for the readout system targeted high-performance . The goal is to understand performance of the Silicon Photo-multiplier photo-detectors . The challenge is to readout the SiPMs though long (~20m) cables.
8 Diagram of the photo-detector readout system
9 Photo-detector readout system status
. The boards were manufactured in the spring of 2017. – Power supplied and everything else are contended in the box. – 1 box = 12 channels. . Initial on-the desk tests to characterize the digital readout (1G Ethernet) and the analog performance. . The integration with the rest of the system is ongoing.
10 Electronics for Dynamic Quantum Computing Experiments at the University of Chicago
• Typical qubit measurement involves sending a microwave pulse to a readout resonator, recording a reflected or transmitted signal, and reducing the record to a binary decision regarding the qubit state. • FPGAs are needed for qubit control since CPUs do not give desirable latency. • Cheap and user-friendly. • 1-channel system. ADC is desired to be faster. • Multi-channel system is planned.
Red Pitaya board 125 Msps ADC AD9914 Closed-source 3.5 Gbps direct digital platform for synthesizer measurements and instrument control
11 QIE12 ASIC for ATLAS Tile Calorimeter
‣ Gated integrator ‣ Superior handling of the out-of-time pulses that shaper-based front-ends ‣ Front End Board • Designed, fabricated, assembled • 3.9fC least count (shunt on) super low noise. • 17 bits dynamic range • Analog current integrator (uses programmable shunt from QIE) • 16-bit DACs for calibration (current injection) ‣ Main board • fully passive • extensive RF shielding for switching POLs
QIE Front End Board 100 GeV electron beam
12 Overview of HVOpto Board (Cont.)
• Basic Layout Power & Data Connection to Daughter Card “A” HV Connections to PMTs - Side “A”
HV Input
Data Connector
Power Supplied By the Interface
CH00 Circuitry
CH01
CH04 CH03
CH02 Power CH05 Split Plane Daughter Board Regulators Isolates Power & Ground
Isolates “A” & “B” Circuits
CH06 CH08
CH07
CH11
CH10 CH09 Uses
Interface
Redundancy Circuitry
Power
HV Output
Data Connector of the Regulators (to next board) Power System
Power & Data Connection to HV Connections to PMTs - Side “B” Daughter Card “B”
13 ATLAS Trigger & DAQ
1. Fast Tracker for Phase-I (FTK) • Hardware global tracking @ 100kHz - pattern recognition with Associative Memory - track fit with digital signal processor - Argonne work - sytem design and performance study - FTK Lvl 2 Interface Crate (FLIC) - commission production boards at CERN Fall 2015 - System wide installation & testing - Integration with TDAQ and Detectors • Primary vertexing with FTK tracks • Lepton isolation with FTK tracks • Jet selection with FTK tracks 2. Front-End Link Exchange for Phase-I (FELIX) • Bridge between the custom fully synchronous system and Ethernet • Timing Trigger and Control (TTC) - Development environment & testbed at ANL 3. Level-1 Track Trigger (L1Track) R&D for Phase-II • FTK-like Level 1 Track Trigger • Phase-1 Lvl 1 becoming Phase-II Lvl 0 • Working on TDAQ architecture - Trigger strategy and DAQ for iTK and full calorimeter granularity - Initial design report (2016) and TDR (2017)
14 15 Production is complete. Boards are being integrated into the full FTK system
16 Hardware track trigger for the HL-LHC program of the ATLAS experiment
• The hardware track processor will be an attachment to the central trigger system.
• Will reconstruct tracks with pT>1 GeV. Regional tracking at 1 MHz. Global tracking at 100 kHz. • Argonne is working on delivering data between the commodity network and the HTT boards (HTT IF). • The HTT will use associative memory and FPGAs to identify and fit tracks. • Currently we are detailing architecture specifications for the HTT system. • Can not be replaced with contemporary commodity CPU/GPU-based system because of cost, power consumption, and space.
17 FELIX • Enabling transition from custom hardware to COTS as early as possible • Using high level switch protocols of high speed and large bandwidth • Configurable and flexible data routing and error handling, without relying on detector specific hardware • Direct low latency paths between links • Universal ATLAS-wide TTC/BUSY handling as for Run 1&2 • Command scheduling with guaranteed timing for calibration
18 FELIX as a System
FELIX Host Hardware Firmware Detector FE FELIX I/O card
Electronics GBT Software
I/O Buffer I/O
TTC PCIe Buffer BUSY TTC PCIe Gen3 16 lanes BUSY Handler SWROD/ROS CPU running FELIX Application
100 Gbs NIC 100 Gbs NIC
19 FELIX Development
TTCfx Custom FMC accepting TTC input Outputing TTC clock and CH A/B info
Hitech Global HTG- 710 2 CXP cages Virtex-7 X690T PCIe Gen 3 x 8 lanes
BNL-711 Xilinx VC-709 4 miniPOD RX & Tx 4 SFP+ connectors Kintex Ultrascale Mellanox Dual-port 40 Gb Virtex-7 X690T PCIe Gen 3 x 16 Ethernet or Infiniband PCIe Gen 3 x 8 lanes lanes 20 Future directions
• Future plan: • The work for ATLAS will continue with focus on DAQ. We will leverage our expertise in FELIX to develop interfaces (e.g. HTT IF) between the custom and commodity systems • There is a need to tailor FELIX to the inner tracker detectors (Pixel and Strip). Argonne is working on integrating FELIX with Pixel readout (RD53A and beyond). High-speed I/O and system complexity is a challenge. • The need for readout electronic for quantum computing will grow as the experiments become more sophisticated. • Electronics for next-gen South Pole Telescope? • A Discussion on DUNE Photon-Detector Electronics is starting now, and the input from ProtoDUNE will be important • Future collider experiments (e.g. for HE-LHC) will require custom electronics (front-end and DAQ). More channels and higher bandwidth than ATLAS or CMS. • Expertise needed to support future experiments • Currently, digital design is in strong demand. The demand will not go away. PCB layout for high speed serial (10… 25Gbps) is a challenge. • The future experiments (e.g. for HE-LHC) will being opportunities to design front-end electronics. The expertise in analog design needs to be maintained. • A lot of on-detector electronics requires IC design; must also interface to the off-detector DAQ 21 Summary
‣ Argonne HEP Electronics Group supports HEP Division experimental work • Ongoing support over life of project: design, fabrication, testing, commissioning, installation, maintenance ‣ Provides electronics support throughout Laboratory • Design and ongoing support for experimental TDAQ • Instrument repair ‣ Collaboration with outside labs and universities is common
22 BACKUP SLIDES
23 QIE12 Front End Board Block Diagram
24 Reference Clock Redesign
‣ Estimate 17 NMR probes at 6000 locations/run generate ~ 1GB/trolley run ‣ Reference clock accuracy << 20 ppb; expect near 1.2 ppb, observe 4.9 ppb ‣ New communication link to transfer data to DAQ computer ‣ Reduce noise with RF switch alternating between pure RF NMR data and data communications over single coax ‣ Pure 62MHz signal during 10-15ms NMR measurement; alternate with sending ~20ms high speed data
25 TDAQ Design & Support for Low Energy Nuclear Physics
Gammasphere Gamma Ray Energy Tracking In-Beam Nuclear Array (GRETINA) ‣ GRETINA: 1π HPGe crystals providing energy and position ‣ Gammasphere: 110 HPGe detectors ‣ Special purpose detectors with separate TDAQ frequently incorporated into both
26 GRETINA TDAQ System
‣ Trigger and DAQ system developed by Argonne HEP Electronics • Not large challenge by HEP standards • Provides not so typical enhanced capability for low energy field ‣ Master Digitizer • recognizes central crystal contact charge above leading edge threshold • reads hit pattern • assembles trigger info and sends to TTC: time stamp, energy, pattern ‣ Global trigger decision ➛ Routers ➛ Master Digitizers ➛ transfer time stamped data ‣ Segment information used to estimate position and energy of interaction points Interaction points used to construct tracks
4/crystal
27 Digitizer/DSP Details Trigger, Timing Control Details ‣ Samples crystal analog ➛ 14-bit ADC @ ‣ FPGA trigger algorithms 100MS/s • Multiplicity ‣ 10 analog inputs per module • Energy ‣ Raw data stored in 40μs circular buffers • Pattern • Auxiliary detector trigger
DSP performed in FPGA
28 Digital Gammasphere (DGS)
‣ Analog Gammasphere began operation in 1993 at LBNL 88-inch cyclotron ‣ Rate & deadtime limitations reduced by conversion to 10 ch GRETINA digitizers • 50k gamma/s/detector • 500k gamma/s ➛ disk ‣ GRETINA trigger module improved throughput limits — triggerless is an option ‣ Phase I — only Ge central contact recorded by DGS ‣ Phase II — add digitizers for Ge & BGO Sums ‣ Phase III — Ge central side contacts and BGO hits • Phase III fully functional at present
schematic credit: Michael Carpenter, Argonne Physics Division
29 Example of Incorporating Additional Detectors
GODDESS (Oak Ridge) • Particle-gamma measurements • Structure studies - light-ion stripping/pickup - heavy-ion transfer - charge exchange
ORRUBA • Barrel array of 24 silicon telescopes Digital Gammasphere • Resistive strip detectors ~1mm position resolution • 288 channels Other auxiliary detectors: • 90 coverage • CHICO (Rochester) • Microball (Washington U.) • Neutron Shell (Washington U.)
30 Data Acquisition Scheme
schematic credit: Steven Pain ORNL
31 Three Basic High Voltage Control System Schemes for the Upgrade
HV Remote HV Internal (HVOpto) *
HV Control Bulk HV Bulk HV
USA15 USA15
~48 Cables/Drawer Detector 1 Cable/Drawer Detector ~10K Signal Lines Total 256 Cables Total PMTs PMTs
Super Drawer Super Drawer ~12 PMTs/MD HV Control HV Control ~12 PMTs/MD ~48 PMTs/SD ~48 PMTs/SD
* 2 versions in development
32 Configuration of the HVOpto System • Basic Concept: • Keep the HV-Opto functionality, similar design, new parts, 12 ch/board • Eliminate HV-Micro, replacing with FPGA/GBT/Daughter Board Interface Current Scheme Upgrade Scheme
33 Physical Connectivity of the HVOpto in a Drawer • Connectivity of HVOpto in the Drawer • An HVOpto Board resides on the bottom side of each Mini Drawer • Each has 0n-board circuitry for controlling and monitoring the HV to each PMT • Control provided via communication with the Daughter Board • (2) Data connections to the Daughter Board Implemented as (2) 30-pin Twist-N-Flat flat cables, 2x15 connector • Fully differential communication (LVDS) New LVPS • Power for the HVOpto provided via this cable
+10V Brick +10V Brick +10V Brick +10V Brick +10V Brick Main + Return +10V Main + Return Brick +10V Brick HVOpto Main Data & +10V Board Daughter Power Brick Board Cable
Bulk HV In from Patch Panel HVOpto HV Connections to PMTs Board (1 side shown) Mini Drawer Sections Bulk HV Inter-Connections 34 Overview of HVOpto Board • Single Channel Block Diagram
MAX1329
All circuitry is directly on the front-end Very simple circuitry Control circuit based on the nice original design by the Clermont Ferrand Group Improvements for fully-differential control & monitoring Max chip handles digital and analog I/O Looks very much like an ASIC Uses SPI interface for communication through the Daughter Board Excellent isolation between HV and LV 35