Streaming Readout of the CLAS12 Forward Tagger Using Tridas and JANA2

Streaming Readout of the CLAS12 Forward Tagger using TriDAS and JANA2

Tommaso Chiarusi - INFN Sezione di Bologna

co-autors:

May 19th 2021

Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within 1 Sezione di Bologna Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 Summary

• Streaming RO is necessary for a long-term HI-LUMI upgrade of CLAS12 • Test with the Forward Tagger (FT) sub-detectors (ECAL +HODO) of CLAS12 experiment @ JLab. • Switching from triggered to triggerless data taking by dedicated f/w for readout electronics • Union of • TriDAS framework derived from “NEMO” neutrino telescope project in the Mediterranean • JANA 2 framework for online/offline analysis software • Goals: • Study SRO performance: memory + cpu use, trigger efficiency, …

• Identify the process e- H/D2/Al/Pb → (X) e- π0 → (X) e- 2γ

2 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 CLAS12 Forward Tagger @ JLab • CLAS12 installed at Jefferson Lab’s experimental HALL-B • Expansive program of nuclear physics topics: nucleon structure, quark conﬁnement, meson and baryon spectroscopy, DIS

12 GeV e- beam direction

Forward Tagger Hodo ECal

MøllerMøller cone cone

DC HTTC 3 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 Modifications of FT readout - CODA and FE software

• Front-end electronics are based on available FE electronics JLAB-FADC250 and VTP modules.

• Streaming version of CODA (without back-end):

• the front end readout software running inside the VME controllers and VTP boards was modiﬁed to stream data out freely.

• new SRO component was developed to be the intermediate translator between front-end and back-end With proper threshold settings, O(2 Gbps) without Møller cone O(<1 Gbps) with Møller cone transferred to TriDAS

4 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 TriDAS • C++11 multithread software framework • ZeroMQ, Boost, CMake, Docker • State machine driven processes • Interface to an auxiliary control system • Modular and scalable

5 ms

Selected /tagged data

5 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 What TriDAS can Do What is done with CLAS12 HMs coherently slice the timeline, relying on time-synchronisation Time Time-slice i TS 1 TS 2 TS 3 TS 4 TS 5 TS 6 … TS n TS 1 TS 2 TS 3 TS 4 TS 5 TS 6 … TS n TS 1 TS 2 TS 3 TS 4 TS 5 TS 6 … TS n

One TimeSlice

… all channels (NCH) With data from all Detector TCPUs applies L1 + L2 ﬁltering L1 Trigger : only Qth Ev Ev Ev L1 trigger ~ E deposit > 2 GeV With different seeds

Event 1 Event 2 Event n-k Event n L2 Trigger : Jana 2 Plugin

… …

Simple coincidences Multi seeds Charge over threshold Structured coincidences

Tagged ✔ (with reco-i ) Not Tagged ✗

Single-seed L1 Δt=200 ns Ev (tuneable) Post-trigger ﬁle Ev ✔ Multi-seed L1 ✔ Ev Ev Δt > 200 ns ✗ ✗ 6 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 JANA 2

• Full event reconstruction • Calibrations • Translation table • Multi-threading

• L2 plugins (tagging and filtering) • Summed energy thresholds • Single/double/Triple FT-CAL cluster • Coincidence FT+FH JANA plugins JANA • Cosmic tracking • Prescale

From ONLINE to OFFLINE • Read post trigger files recorded by TRIDAS for offline analysis • Same reconstruction algorithms (software trigger) for both online and offline analysis

7 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 Data analysis for π0 signature Run 1 - Feb 2020 Run 2 - Sept 2020 with Møller cone Clusters distribution Y (mm) Y

Run 1: X (mm) 10.4 GeV electron beam on thin Pb target in Jan/ π0 reconstructed Feb 2020 no Moeller cone, thin target, FT-CAL invariant mass Preliminary Preliminary Run 2:

10.4 GeV electron beam on H2 and D2 targets in counts Aug/Sept 2020 Al target Pb target window window Møller cone, longer target, FT-CAL + FT-HODO

Compatible results from independent analysis of recorded SRO data with AI algorithm. - See proceedings for details

Invariant Mass (MeV) 8 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 Conclusions

• Streaming Readout on-beam tests performed using the CLAS12-FT at JLAB • The full chain (FT + SRO sw + ON-LINE REC) tested with promising result. • Data taken in full streaming mode, analysis in progress (traditional and AI-supported)

• Analysis was able to extract a clean physics signal in the form of a π0 invariant mass peak • The prototype system is being used as the basis for developing a larger system planned for the entire CLAS12 detector and its future physics program.

Essential references

• CLAS 12 Forward Tagger: A. Acker, et al., NIM A163475 (2020) • CLAS12 standard DAQ: S. Boyarinov, et al., NIM A 966, 163698 (2020) • TriDAS: T. Chiarusi, et al., Journal of Physics: Conference Series 898, 032042 (2017) • JANA 2: D. Lawrence, et al. Journal of Physics: Conference Series 1525, 012032 (2020)

9 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 BACKUP SLIDES

10 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 • Streaming RO is necessary for a long-term HI-LUMI upgrade of CLAS12 • Phase 1 : increasing luminosity of 2x in 2-3 year. Phase 2: increasing luminosity of 100x 5-7 year ( wrt 1035 cm-2s-1 ) • An update of the R1 CLAS12 DC with more dense detector (e.g. GEM) is expected in Phase I. • With the current triggered technology the maximum possible event acquisition rate for CLAS12 is ~100 kHz (R~30 kHz now) upgrading several systems.

• A prototype trigger-less DAQ was successfully tested in beam conditions • It brought together existing components • FEE: FADC250 + VTP + CODA • TRIDAS • JANA2

• Used to read out CLAS12-FT • FT - ECAL • FT - Hodoscope

11 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 Computing resources for TriDAS + Jana

Sever used for HMs and TCPUs

Mem: 62G Swap: 64G N. Proc 48 model Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz

Process N. Instances Memory (GB/ CPU (%) HM) HM 5/10 2,4 <600/<900 TCPU - no 10 3 400 JANA TCPU + Jana 10 <4 850 trad. TCPU + Jana 10 4,5 1600 AI

12 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 CLAS12 FT @ JLab • Installed at Jefferson Lab’s experimental HALL-B • Expansive program of nuclear physics topics: nucleon structure, quark conﬁnement, meson and baryon spectroscopy, DIS…

12 GeV e- beam direction

Forward Tagger

13 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 FT CAD sketches

14 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 Optimised Thresholds Not optimised Thresholds • Various tests with extremely not-realistic thresholds ➝ realistic thresholds; • Passing from O(30) Gbps ➝ O(500) Mbps • Various TriDAS conﬁgurations, increasing/decreasing (optimising!) the number of TriDAS resources: HM, TCPUs • Proﬁling the TriDAS CPU and RAM usage • Optimising the integration of TriDAS and JANA

15 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 CLUSTERS FEATURES (Run 1 - no Møller cone)

● Time delay between 2 clusters ● Distributions of n. of Clusters

16 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799 February tests details .

17 Tommaso Chiarusi Supported by Italian Ministry of Foreign Affairs (MAECI) as Projects of great Relevance within vCHEP21, May 19th 2021 Italy/US Scientific and Technological Cooperation under grant n. MAE0065689 - PGR00799