Neutrino Simulation Applications for Liquid Argon-Based Detectors

Neutrino Simulation Applications for Liquid Argon-based Detectors

Robert Hatcher & Hans Wenzel Fermilab Computing Division 2016-04-22 Outline

● Overview: neutrino and related test beam experiments in Fermilab LArTPC program ● The art framework ● LArSoft: Motivation ● LArG4: The Geant4 art module ● what is special about liquid argon simulation? ● Some caveats, questions. ● Some benchmark results. ● Summary

Robert Hatcher, Hans Wenzel 2 X ν at the Intensity Frontier Neutrino Detectors Tertiary Test Beams ⎧MINOS [+] ‡ (Near & Far detectors - magnetized)

Fermilab LArIAT ‡ ✅ ArgoNeuT † ✅ (same small LAr detector in test beam / NuMI beam)

CERN DUNE proto-types ⎨ ⎧MINERνA ‡ (ﬁne grained & multi-target material) single & dual phase ✅ ✅ NOνA ‡ (Near & Far detectors - oﬀ-axis)

SBND ✅ Neutrino Beams ⎧(Short Baseline Near Detector Expt, formerly LAr1ND) present & future (and recent past) ANNIE • NuMI (Main Injector) (to study neutron production in water using BNB ν ) • LE & ME target/horn conﬁgurations ｜μBooNE ‡

• Booster Neutrino Beam ✅

miniBooNE † • LBNF under design ⎨⎧ ICARUS-T600 (to be refurbished & moved from Gran Sasso National Lab in Italy to serve as BNB Far Detector) † ran previously ✅ LArTPC ‡ currently running (Liquid Argon TPC) DUNE ✅ + ✅ Technology (Deep Underground Neutrino Experiment, formerly LBNE) (Near & Far detectors + 35 ton + test beam single & dual phase prototypes at CERN) Robert Hatcher 3 Beamline Simulations

LArIAT • x Wilson Hall

MINOS, Minerva, NOvA

μBooNE, SBND, ICARUS,

DUNE

NOνA Simulation

FLUKA11 On-Axis 15 14.6 mrad Off-Axis (NOνA)

10 MINOS

5 NOνA CC / 6E20 POT kton 0.1 GeV µ ν Robert Hatcher 4 0 5 10 15 Eν [GeV] LArTPC: experiments and R&D

* *

SBND DUNE

* SB = Short Baseline; LB = Long Baseline Robert Hatcher, Hans Wenzel 5 X General Simulation Workﬂow & Products in Neutrino Experiments Beam Flux We factorize the steps to make μ Simulation (π,K, decays) them tractable problems

(also secondaries off target Simulation of the beamline to compare with e.g. MIPP) Simulation of the detectors Different energy scales Neutrino “Truth” Even detector simulations have particle lists large variation in needs due to a Physics & kinematics variety of technology (e.g. GENIE)

General Specific “digits” “hits” (raw data Detector (energy Detector similar to real depositions) Simulation Simulation detector)

Robert Hatcher, Hans Wenzel 6 Flux Combines beamline geometry w/ hadronic Beam Flux physics (initial production is especially Simulation (π,K,μ decays) important), energy loss, and decays. Initial particles are 120 GeV or 8 GeV protons

In NuMI and LBNF/DUNE experiments the two primary tools are: G4NuMI (and derivative G4LBNF) and FLUGG

•The first is a purely Geant4 base simulation •G4LBNE uses “QGSP_BERT” physics list out-of-the-box; g4NuMI uses “FTFP_BERT” (“QGSP_BERT” prior to 2014)

•The second uses a G4 geometry but fluka for physics; glued together via “flugg”

Reference LBNE/DUNE spectra:

Robert Hatcher, Hans Wenzel 7 X Neutrino Physics

Currently in use: GENIE and NEUGEN3 (also, GIBUU, NuWro, ... MINOS is deprecating NEUGEN3 in favor of GENIE)

Beam Flux GENIE can be used stand-alone (Minerva) Simulation (π,K,μ decays) or embedded in the offline framework (NOvA/ LArSoft, eventually MINOS)

Neutrino “Truth” particle lists Physics & kinematics (e.g. GENIE)

General Specific “digits” “hits” (raw data Detector (energy Detector similar to real depositions) Simulation Simulation detector) (G4+User) Robert Hatcher, Hans Wenzel 8 X Detector Simulation

Geant4 + User Specific code to take particles exiting the nucleus (GENIE Beam Flux results) and transport them through the Simulation (π,K,μ decays) detector, creating energy loss “hits” and DAQ equivalent “digits”

Neutrino “Truth” particle lists Physics & kinematics (e.g. GENIE)

General Specific “digits” “hits” (raw data Detector (energy Detector similar to real depositions) Simulation Simulation detector) (G4+User) Robert Hatcher, Hans Wenzel 9 X TPC Working Principle from Kazuhiro Terao The LArSoft Reconstruction(microBooNE) 1. Charged particles interact in Ar X = 2.5 m • Ionize electrons • Produce scintillation light

2. Ionization e- drift toward anode 2.3 m = Y Drift Time = X position 3. Wire planes detect drift e-

Z = 10.4 m Scintillation Light detected by PMTs Charge collected by wire plane

Cathode @ -128 kV Electric Field Anode (plate) 500 V/cm (wire plane) Robert Hatcher, Hans Wenzel 10

Tuesday, July 15, 14 14 Three Wire Planes from Kazuhiro Terao The LArSoft Reconstruction(microBooNE) Picture courtesy of J. Asaadi

⊕ ⊕ =

U plane V plane Y plane 8256 wires w/ pitch = 3mm (induction) (induction) (collection) (Y, Z) = coincidence on wire

Induction Plane MC Waveform Collection Plane MC Waveform Robert Hatcher, Hans (Bi-polarWenzel pulse as e- pass through) 11 (Uni-polar pulse as e- pass through) Tuesday, July 15, 14 15 A bit about events: topologies

Events & Backgrounds: ● ν “detector” (LAr vs. cryostat), ● ν in surrounding “rock”/“dirt” ● cosmics (single μ, multi-particle) ● radiological sources ● spallation in the rock ● astrophysical (many very small energy depositions) ● nucleon decay (p → K+ν & p → K0μ+ )

ν induced events:

● “pile-up” distributed in time (10μsec vs. drift time) & space (different from colliders)

● basic ν topologies: ● CC nu-mu ● CC nu-e ● NC (inc pi0 )

Much of the physics Robert Hatcher, Hans Wenzel 12 is in distinguishing these X A bit about events: spatial resolution

ν induced events:

● “pile-up” distributed in time (10μsec vs. drift time) & space (different from colliders)

● topologies: Very fine grain, ● CC nu-mu ● CC nu-e detailed tracks ● NC (inc pi0 )

Robert Hatcher, Hans Wenzel 13 X A bit about events: reconstructed

ν induced events:

● “pile-up” distributed in time (10μsec vs. drift time) & space (different from colliders) Reconstruction should be able to pull out ● topologies: these details. But this has implication for ● CC nu-mu the size of the data and the how fine grained ● CC nu-e the simulation must be. ● NC (inc pi0 )

Robert Hatcher, Hans Wenzel 14 X The art framework

● C++ framework is supported by Fermilab’s scientific computing division(SCD). Shares a lot of DNA with frameworks developed for hadron collider experiments (Tevatron, LHC) ● Uses ROOT for I/O persistency ● Uses JSON-like FHiCL for configuration ● Fermilab Hierarchical Configuration Language ● not a programming language; no in-script computations

Robert Hatcher, Hans Wenzel 15 X LArSoft

● Built upon art based C++ toolkit to ‘simulate’ and ‘reconstruct’ events in liquid Argon TPC’s. ● Emphasis on common algorithms & services ● Defines interfaces & data products common to all LArTPC expt. ● Written by a large number of people; contributions from different institutions and experiments ● Many current and future experiments are based on liquid Argon TPC technology sharing code makes sense. ● But there are differences: ● Single vs. dual phase detectors ● Light collection ● Detector size ● Configuration and number of read out wire planes → effects 3D reconstruction and hit disambiguation → a certain amount of detector-specific information necessary

Robert Hatcher, Hans Wenzel 16 LArSoft . art Framework LArPandora LArEvent LArExamples Run/Subrun/Event Stores pattern recog Display ⎞ Event Loop & Paths

DUNE Specifc LArAna MessageService ⎬ Confguration (FHiCL) LArReco Metadata

Provenance Tracking

LArIAT Specifc LArSim ⎪ NuTools Geant4 Dynamic Library Loading (inc WireSim) (LArSoft + NOvA) I/O Handling (via ROOT) GENIE ⎪ boost

gccxml LArEvt (calibration & space SBND Specifc charge interfaces) ⎪ SQLite LArData libxml ⎬ ROOT ... User Code LArCore (inc Geometry) ⎠ gsl Robert Hatcher, Hans Wenzel 17 LArSoft Lines Of Code $ cloc /grid/fermiapp/products/larsoft/larcore/v05_00_02/source/ /grid/fermiapp/products/ larsoft/lardata/v05_03_00/source/ /grid/fermiapp/products/larsoft/nutools/v1_23_02/ source /grid/fermiapp/products/larsoft/larsim/v05_02_00/source/ /grid/fermiapp/products/ ● larsoft/larreco/v05_03_00/source/ /grid/fermiapp/products/larsoft/larana/v05_03_00/source/ /grid/fermiapp/products/larsoft/larpandora/v05_02_01/source/ /grid/fermiapp/products/ larsoft/lareventdisplay/v05_02_00/source/ /grid/fermiapp/products/larsoft/larexamples/ v05_00_07/source/ 1310 text files. 1308 unique files. ------Language files blank comment code ------C++ 701 49671 43680 205556 C/C++ Header 570 15521 26100 35249 XML 8 41 72 757 Bourne Shell 8 70 78 142 Perl 1 19 6 74 ------SUM: 1288 65322 69936 241778 241,778 ------

$ cloc /grid/fermiapp/products/larsoft/nutools/v1_23_02/source /grid/fermiapp/products/larsoft/larsim/v05_02_00/source/ Simulation ~ 21% 306 text files. of total LOC 305 unique files. ------includes Language files blank comment code ------connection to C++ 162 9496 8762 44359 GENIE & Geant4, C/C++ Header 131 2379 3573 6695 XML 2 8 18 116 G4 User Actions, Perl 1 19 6 74 Bourne Shell 3 52 63 70 as well as WireSim ------SUM: 299 11954 12422 51314 ------51,314 Robert Hatcher, Hans Wenzel 18 art / root Lines Of Code $ cloc [top of the head of the art repository, not included the 'doc' directory]

1436 text files. 1366 unique files. 497 files ignored. ------Language files blank comment code ------C++ 377 6040 4174 36555 C/C++ Header 367 6781 7366 23649 CMake 72 649 679 4042 Bourne Shell 78 504 251 1365 Perl 30 267 97 1210 C 2 96 570 1204 Bourne Again Shell 7 53 39 356 Python 1 13 32 230 XML 10 20 30 177 C Shell 1 3 5 45 MXML 1 1 0 4 sed 1 0 0 1 ------SUM: 947 14427 13243 68838 68,838 ------

$ ~/bin/cloc root

10966 text files. 10751 unique files. 1810 files ignored. ------Language files blank comment code ------C++ 3869 253948 353006 1257809 C/C++ Header 3925 93108 128715 410475 C 338 30144 33047 180459 Bourne Shell 98 2686 3406 25895 Objective C++ 80 5366 4079 18575 HTML 192 2206 159 16633 JavaScript 18 2602 994 13454 Fortran 77 4 351 540 12570 [others file types] ------SUM: 9163 397933 530773 1979673 ------1,979,673

Robert Hatcher, Hans Wenzel 19 Simulation & Reconstruction Steps Simulation: GENIE event generation Geant4 energy deposit LArG4 Geant4 extensions: model drift of electrons to wires, photon response. WireSim (model electronic response to charge arriving at wire) Reconstruction: Signal processing (inverse of wiresim) → uni-/bi- polar pulses Hit finding Clustering Tracking Vertexing Calorimetry PID Physics Object (e, γ, µ…) …. Robert Hatcher, Hans Wenzel 20 LArG4: Specifics of Liquid Ar

Wire or pad readout pitch influences stepping needs. (energy localization) Competing processes: separation of Scintillation (very prompt, 2 time constants, spectrum in the UV) and Charge → energy resolution can be improved by detecting photons. Charge attenuation (lifetime can be similar to drift time) →effective gain /Signal to Noise ratio is drift length dependent. Space charge distortions caused by slow (100x slower than electrons) moving ions. e.g. drifting ions from cosmics distort ν events in μBooNE

Robert Hatcher, Hans Wenzel 21 Detector Simulation x

As G4Steps are taken, e- drift is calculated within LArG4, not using G4 routines; Transport of photons from G4Step to PMT is sampled from a lookup library (which might originally have been generated with Geant4) Robert Hatcher, Hans Wenzel 22 Input

art: fhicl files used to define paths, control flow, and to pass parameters.

Geometry: GDML But: Uses magic words to assign sensitive volumes Voxelization done later on via a parallel world geometry, default are (300 µm)3 cubes “optimized” values — needs further justification → consequences for memory footprint, speed, visualization, verification of geometry. Also, two versions of each (w/ and w/out wires) Can this be achieved in a different manner?

Robert Hatcher, Hans Wenzel 23 LArG4: Caveats, Questions Much of the code was written by physicists with various levels of experience, often to “make it work”, and not always with maintenance or upgradability in mind. Some have moved on. Is voxelization necessary? If so what is optimal granularity? Liquid Argon specific code (NEST, evaluation of electron drift etc.) not integrated in Geant4. Should it be part of Geant4? Custom (outdated) physics list contains obsolete models not available in current Geant4 anymore. Some classes are modified versions of standard Geant4 e.g. Scintillation missing methods should be communicated to Geant4 team to avoid such code “forking” All this makes “urgently necessary” upgrading to newer Geant4 not trivial (currently 4.9.6.p04) targeting 4.10.1.p03 or higher (in progress).

Robert Hatcher, Hans Wenzel 24 Some Benchmarking Results LArIATSoft: single charged particle (=780 MeV, σ =700 MeV) Timing: art timer 1000 events (except reco: 100, only π)

Timing (π±) Module T(e±) T(μ±) sec/evt 17.8% Geant4 0.140 0.388 39.2% 0.088 14.0% (1.10%) 32.1% ROOT I/O 0.252 0.245 24.7% 0.203 32.2% (1.97%) DetSim 50.1% 0.394 0.358 36.1% 0.339 53.8% (WireSim) (3.08%)

Reco — — — — (89% TrackMaker) ~12 (93.85%) Note: Geant4 will scale linearly with complexity of events; reconstruction due to combinatorics will not! Robert Hatcher, Hans Wenzel 25 Running DUNE Simulation DuneTPC: 4 APA FarDet “workspace” (g4.9.6p03) Timing: art timer (100 events)

Module Timing

GENIE Gen 0.1268 sec / evt † 0.05%

Geant4 3.842 sec / evt 1.55%

DetSim (WireSim) 44.1075 sec / evt 17.81%

Reco 199.4674 sec / evt 80.52%

MergeAna 0.1656 sec / evt 0.07%

Note: Geant4 will scale linearly with complexity of events; reconstruction due to combinatorics will not! † Not including x-sec load time (~80s) Robert Hatcher, Hans Wenzel 26 Summary We presented an overview of art, LArSoft and LArG4 comprising a toolkit to simulate and reconstruct liquid Argon TPC’s

Much work has been done to get the “physics” right; speed and memory optimizations have not been a priority yet

Robert Hatcher, Hans Wenzel 27 Thanks

To Brian Rebel, Erica Snider, Gianluca Petrillo, and Tom Junk for their time and help in understanding more of the the LArSoft code

All mistakes and misstatements are ours.

Robert Hatcher, Hans Wenzel 28 X What Makes ν Event Simulation Different from Colliders and Test Beams ?

The beam is complex and needs to be modeled Event vertices everywhere in time and space interaction location depends on beam fluence and spatial distribution of matter timing distribution follows detailed beam structure

Robert Hatcher, Hans Wenzel 29 Beam Simulation

Geant4 geometry (C++) + [G4 or Fluka physics (fortran) ] G4 geometry is quite detailed (and good match to as-built) [fluka, if used, physics everywhere (not just target)] Record decay, initial secondary production and initial proton info now: ancestors history from initial proton to decay products 2ndary production models are active areas of study Possibly uses importance weights and thresholds

wπ = min(max(30/ptot,1) * wparent , 100) threshold = 1 GeV or not (off-axis, i.e. NOvA) μ/e + X π/K (μ) protons ν

target focussing horns Robert Hatcher, Hans Wenzel 30 decay pipe + absorber Decay Reweighting

The probability that a decay results in a neutrino ray that goes through any point depends on the relativistic boost at the decay point; the ν energy will also depend on position Near and Far detectors subtend a different angular size ➙ they see different spectra

E’, weight’

Near E, weight Far ≈800+ km ≈1km Robert Hatcher, Hans Wenzel 31 Flux Window

A fictitious parallelogram in space from whence neutrino rays emanate needs to be sized:

large enough that all (to best approximation) relevant rays that might run through the geometry pass through the window small enough to exclude rays that aren’t of interest

oops Robert Hatcher, Hans Wenzel 32 GNuMIFlux/GSimpleNtpFlux GENIE’s GNuMIFlux or GDk2Nu read an entry from ntuple of decay info pick random point on flux window (x,y,z) calculate x-y weight, energy, p4

accept/reject based on weights ( wgtx-y * wgtimportance ) (possibly) push backwards along ray to (x’,y’,z0) ⇒PathSegmentList created from this ray

cycling back to same entry won’t give same ray different window point ⇒ different weight, energy, trajectory

GENIE GSimpleNtpFlux simple ntuple format of flavor, position, direction, weight

Robert Hatcher, Hans Wenzel 33 ν Rays and Geometry

PathSegmentList StartPosition rock air Direction y detector components list y z x P2 z

P3 Flux Window

Robert Hatcher, Hans Wenzel 34 GENIE x-sections

GENIE cross-sections: ~440 MB (15574 splines) 271 for proton, 301 for neutron, 577 for each of 26 nuclei currently need only ~14 of 26: C12, N14, O16, Na23, Al27, Si28, S32, Cl35, Ar40, K39, Ca40, Ti48, Fe56, Ba137 not: Mg24, P31, V51, Mn55, Fe54, Fe57, Fe58, Ni59, Cu64, Sn119, Pb207 500 knots, [0:200] GeV spaced logarithmically 5% of points > 120; ND/FD MN flux has entries up to 104 GeV could study effect of fewer knots Loading from ROOT file (vs XML) speeds up loading, but doesn’t appear to significantly change peak memory usage Study: # knots in spline vs. accuracy

Robert Hatcher, Hans Wenzel 35 Choosing a Vertex “Outside the Box” When a “topvol” isn’t set, GENIE considers the entire geometry GeomSelectorRockBox trims the volume to the hall + minimum safety + a size proportional to the neutrino energy

Robert Hatcher, Hans Wenzel 36 How to overlay Collect events MINOS: pull from input sample files Poisson distribution: nDetPerSpill + nRockPerSnarl for a given intensity single use of detector events, randomize pulling from rock files (reuse, except once) NOvA: generate events until used fixed POTs/Spill Distribute events in time over spill interval according to intensity profile

offset truth info times (StdHep/HepMC) also offset corresponding hit times, if already propagated in GEANT if combined particle list, adjust parentage indices add any event kinematics/flux records to list for spill good to have mechanism tying kin/flux to particle list

Robert Hatcher, Hans Wenzel 37 Bulk Properties including, Reinteractions, Decays, Energy Loss and Transport

Particles to be dealt with Basic particle physics combined with range from 100 GeV to specific detector geometries. ~sub-MeV; most emphasis is on the Σ E ~ <10 GeV •Majority are Geant4 based range. •ART based systems use QGSP_BERT physics list by default, but trivially changeable at run time Particles include all species •MINOS uses Geant3 in its old fortran system, and ROOT’s VMC interface to Geant3 in the C++ of hadrons, charged based system (should also work w/ Geant4) leptons and ions. “Truth” particle lists •Generally these simulations are Only MINOS is magnetized & kinematics embedded into the experiment’s (possible DUNE NearDet offline framework option being explored), •MINOS uses “loon” (homegrown built upon ROOT) •Minerva uses Gaudi but the list of materials run •LArSoft (ArgoNeut, microBooNE, DUNE, etc) uses ART the gamut from low •NOvA uses ART density single element General “hits” liquid Argon, to complex Detector (energy admixtures of materials Simulation depositions) and densities.

Robert Hatcher, Hans Wenzel 38 X Detector Specific Simulations The stage of converting energy depositions into “raw digits” involves the specific technology of the detector. The LAr experiments can make much use of code sharing due to the common technology, but even there differences in single vs. dual phase readout and photo- detection.

Some experiments that involve light collection use Geant to do that part of simulation, either directly embedded in the framework, or as a separate simulation that provides distributions that can be sampled (mostly for speed and efficiency Specific “digits” reasons). “hits” (raw data (energy Detector similar to real depositions) Simulation detector)

Robert Hatcher, Hans Wenzel 39 X The art framework

Robert Hatcher, Hans Wenzel 40 X Modules only talk to Event (Run) store; art Glossarynever to each other

e.g. G4 x

Robert Hatcher, Hans Wenzel 41