Tracking Ghost Particles high performance computing for the IceCube Neutrino Observatory
Tyce DeYoung Department of Physics and Astronomy Michigan State University Outline
• Neutrinos and the IceCube Neutrino Observatory
• Tracking Neutrinos Through Ice
• Tracking Neutrinos Through the Earth
2 Tyce DeYoung
Neutrinos – Nature’s “Ghost Particles”
• Neutrinos are fundamental particles, but very strange
• A million times lighter than any other known particle
• Barely interact with other matter
• “Oscillate” between the three different types – an electron neutrino will spontaneously change into a tau neutrino
4 Tyce DeYoung
The Science of IceCube
I NTERNATIONAL JOURNAL OF HIGH-ENERGY PHYSICS CERNCOURIER
WELCOME V OLUME 54 NUMBER 10 D ECEMBER 2014 • Learning about our Universe CERN Courier – digital edition Welcome to the digital edition of the December 2014 issue of CERN Courier.
4HE %UROPEAN 0HYSICAL *OURNAL #As CERN’s 60th-anniversary year comes to an end, CERN Courier gives a voice to some of the organization’s4HE %UROPEAN 0HYSICAL *OURNAL pioneers who are no longer with us. WPMVNFrOVNCFSrKVMZr
Extracts from audio recordings5IF&VSPQFBO1IZTJDBM+PVSOBM$ in CERN’s archives bring to life the spirit WPMVNFrOVNCFSrKVMZrof adventure of the early CERN. One of the studies that CERN pioneered • The Earth is constantly bombarded was the measurement of the “g-2” parameter of the muon Ð an experiment that in its latest incarnation is setting up in Fermilab. The year also saw the 50th anniversary of the International Centre for Theoretical Physics in Trieste, and an interview with the centre’s%0* current director reveals interesting# 3FDPHOJ[FECZ&VSPQFBO1IZTJDBM4PDJFUZ "*! 7ATT 0 -OTYLINSKI AND 23 4HORNE - 3HARIF AND 2 3ALEEM similarities and contrasts between the two organizations. The end of the year 5IFFõFDUPG-)$KFUEBUBPO.4581%'T 4UVEZPGJOnBUJPOBSZHFOFSBMJ[FEDPTNJD also offers the traditional seasonal Bookshelf. Happy reading! by relativistic particles from2EGULAR !RTICLE 4HEORETICAL 0HYSICS beyond$IBQMZHJOHBTGPSTUBOEBSEBOEUBDIZPO TDBMBSmFMET - 3CHOTT AND - $UNFORD 2EGULAR !RTICLE 4HEORETICAL 0HYSICS 3FWJFXPGTJOHMFWFDUPSCPTPOQSPEVDUJPO To sign up to the new-issue alert, please visit: 1BSUJDMFTBOE'JFMET JOQQDPMMJTJPOTBU√ʲT 5F7 * %LLIS +! /LIVE AND * :HENG http://cerncourier.com/cws/sign-up. 2EVIEW 5IFFYUFOUPGUIFTUPQDPBOOJIJMBUJPOTUSJQ 2EGULAR !RTICLE 4HEORETICAL 0HYSICS *, %VANS ET AL To subscribe to the magazine, the e-mail new-issue alert, please visit: our solar system – where do1FDDFJo2VJOOTZNNFUSJDQVSFHSBWJUZNFEJBUJPO they ! &OWLIE AND - 2AIDAL 2EGULAR !RTICLE 4HEORETICAL 0HYSICS 1SPTQFDUTGPSDPOTUSBJOFETVQFSTZNNFUSZBUhttp://cerncourier.com/cws/how-to-subscribe. √ʲT 5F7BOE√ʲT 5F7QSPUPOoQSPUPO ' !AD ET AL TVQFSDPMMJEFST 4HE !4,!3 #OLLABORATION 2EGULAR !RTICLE 4HEORETICAL 0HYSICS 5IFEJõFSFOUJBMQSPEVDUJPODSPTTTFDUJPO PGUIFf NFTPOJO√ʲT 5F7QQ 9 .YONYI 3$ -AHARAJ AND +3 'OVINDER come from? DPMMJTJPOTNFBTVSFEXJUIUIF"5-"4EFUFDUPS )JHIFSEJNFOTJPOBMDIBSHFETIFBSGSFF 2EGULAR !RTICLE %XPERIMENTAL 0HYSICS SFMBUJWJTUJDNPEFMTXJUIIFBUnVY 2EGULAR !RTICLE 4HEORETICAL 0HYSICS NEWSPAPER -' !ARTSEN ET AL )CE#UBE #OLLABORATION !* "URAS & $E &AZIO AND * 'IRRBACH - Observation of a high-energy particle IceCube: neutrinos and more 4FBSDIGPSOPOSFMBUJWJTUJDNBHOFUJDNPOPQPMFT * SVMF ε ε BOE , πνν JO ; ; D = ¡ ® ¡ shower event from August 2011, 111 XJUI*DF$VCF BOE(¡ NPEFMTXJUI'$/$RVBSLDPVQMJOHT identified as a PeV-energy neutrino. Each WPMVNF HYSICAL 2EGULAR !RTICLE 4HEORETICAL 0HYSICS 2EGULAR !RTICLE 4HEORETICAL 0HYSICS sphere represents a digital optical module P sensor in the IceCube detector. Sphere size is a measure of the recorded . !GAFONOVA ET AL 2 'ONZÕLEZ &ELIPE ET AL number of photoelectrons. Colors 4O THE MEMORY OF 0ROF ' 'IACOMELLI $POTUSBJOJOHNVMUJ)JHHTnBWPVSNPEFMT represent arrival times of photons (red, DETECTORS PEOPLE .FBTVSFNFOUPGUIF5F7BUNPTQIFSJDNVPODIBSHF 2EGULAR !RTICLE 4HEORETICAL 0HYSICS early; blue, late). Selected for a Synopsis THE ICTP
in Physics and an Editors’∙ Suggestion.
OVNCFS EVIEW SBUJPXJUIUIFDPNQMFUF01&3"EBUBTFU [M. G. Aartsen et al., IceCube CUORE: the coldest CERN Council R 6 'UZEY - 3TRIKMAN AND - :HALOV Collaboration, Phys. Rev. Lett. 111, 2EGULAR !RTICLE %XPERIMENTAL 0HYSICS AT 50 %JTFOUBOHMJOHDPIFSFOUBOEJODPIFSFOU 021103 (2013)] heart in the selects the next - "RAMBILLA & $I 2ENZO AND - (ASEGAWA RVBTJFMBTUJD+ψQIPUPQSPEVDUJPOPOOVDMFJEDITOR: CHRISTINE PHYSICALSUTTON, CERN REVIEW LETTERS known universe director-general An interview with ∙
)JHIMPPQQFSUVSCBUJWFSFOPSNBMJ[BUJPODPOTUBOUT CZOFVUSPOUBHHJOHJOVMUSBQFSJQIFSBMJPO DIGITAL EDITION CREATED BY JESSE KARJALAINEN/IOP PUBLISHING, UK™ ™ KVMZ p5 p5 ETTERSFernando Quevedo p37 R 1 2,000Ð090,1 uy21 46total (416 2013 July 12 029902, 020401Ð (2), 111 PRL L • Understanding the natureGPS-BUUJDF2$% *** UISFFMPPQRVBSLDVSSFOUTGPS of matterDPMMJTJPOTBUUIF-)$ Contents ∙
*XBTBLJHBVHFBDUJPOBOEO 8JMTPOGFSNJPOT 2EGULAR !RTICLE 4HEORETICAL 0HYSICS G Articles published 6 JulyÐ12 July 2013 Articles published week ending 12 J ULY 2013 2EGULAR !RTICLE 4HEORETICAL 0HYSICS -FGU&WFOUEJTQMBZPGBTJNVMBUFENPOPQPMFXJUITVQFSJNQPTFECBDLHSPVOEOPJTFUIFCMBDLMJOFSFQSFTFOUTUIF 3 $ESCOTES 'ENON AND " -OUSSALLAM VOLUME 111, NUMBER 2 NPOPQPMFUSBDL UIFEJHJUBMPQUJDBMNPEVMFTBSFTIPXOBTUJOZCMBDLEPUTBOE12 July 2013 UIFDPMPSDPEFJMMVTUSBUFTUIFUJNF WWW. "OBMZUJDJUZPGηπ JTPTQJOWJPMBUJOHGPSNGBDUPST TDBMFGSPNSFEGPSFBSMZUJNFTUPCMVFGPSMBUFSUJNFT5IFSBEJJPGUIFDPMPSFETQIFSFTTDBMFXJUIUIFOVNCFS CERNGeneral Physics: Statistical andCOURIERQuantum Mechanics, Quantum Information, etc. BOEUIFτ ® ηπν TFDPOEDMBTTEFDBZ PGSFDPSEFEQIPUPFMFDUSPOT3JHIU6QQFSlimits on the fVYPGOPOSFMBUJWJTUJDNBHOFUJDNPOPQPMFTEFQFOEJOH Scalable Reconstruction of Density Matrices ...... 020401 2EGULAR !RTICLE 4HEORETICAL 0HYSICS #ONTINUATION ON THE REVERSE PAGE T.VBaumgratz,OLUME D.54Gross, NM.UMBERCramer, 10and M. D B.ECEMBERPlenio 2014POUIFTQFFEbBOEDBUBMZTJTDSPTTTFDUJPOsDBUPGUIF*$BOBMZTJTBOE*$%FFQ$PSFBOBMZTJT5IFEBTIFEMJOFT Particles, Holes, and Solitons: A Matrix Product State Approach ...... BSFMJNJUTQVCMJTIFECZUIF."$30FYQFSJNFOU"020402 EEJUJPOBMMZ UIF*DF$VCFMJNJUTBSFTIPXOBTBGVODUJPO Damian Draxler, Jutho Haegeman, Tobias J. Osborne, Vid Stojevic,PG~MLaurens XIJDIJTQSVanderstraeten,PQPSUJPOBMUPUIFBWFSBHFE$IFSFOLPWQIPUPOZJFMEQFSOVDMFPOEFDBZand Frank Verstraete 'SPN*DF$VCF$PMMBCPSBUJPO4FBSDIGPSOPOSFMBUJWJTUJDNBHOFUJDNPOPQPMFTXJUI*DF$VCF • How do neutrinos behave? Why are Bounding Temporal Quantum Correlations ...... 020403 Costantino Budroni, Tobias Moroder, Matthias Kleinmann, and Otfried Gu¬hne Quantum Teleportation of Dynamics and Effective Interactions between Remote Systems ...... 020501 Christine A. Muschik, Klemens Hammerer, Eugene S. Polzik, and Ignacio J. Cirac Secure Entanglement Distillation for Double-Server Blind Quantum Computation ...... 020502 Tomoyuki Morimae and Keisuke Fujii Faithful Solid State Optical Memory with Dynamically Decoupled Spin Wave Storage ...... 020503 Marko Lovric«, Dieter Suter, Alban Ferrier, and Philippe Goldner they so different from other particles, Quantum Frameness for CPT Symmetry ...... 020504 Michael Skotiniotis, Borzu Toloui, Ian T. Durham, and Barry C. Sanders Nonadditivity in Quasiequilibrium States of Spin Systems with Lattice Distortion ...... 020601 Takashi Mori
Gravitation and Astrophysics Observables of a Test Mass along an Inclined Orbit in a Post-Newtonian-Approximated Kerr Spacetime Including the Linear and Quadratic Spin Terms ...... 021101
like the electrons and quarks that Steven Hergt, Abhay Shah, and Gerhard Scha¬fer pages) Three-Dimensional Model of Cosmic-Ray Lepton Propagation Reproduces Data from the Alpha Magnetic Spectrometer on the International Space Station ...... 021102 Daniele Gaggero, Luca Maccione, Giuseppe Di Bernardo, Carmelo Evoli, and Dario Grasso First Observation of PeV-Energy Neutrinos with IceCube ...... 021103 M. G. Aartsen et al. (IceCube Collaboration) Limits on Spin-Dependent WIMP-Nucleon Cross Sections from 225 Live Days of XENON100 Data ...... 021301 E. Aprile et al. (XENON100 Collaboration) make up atoms? Effective Field Theory Approach to Gravitationally Induced Decoherence ...... 021302 M. P. Blencowe
(Continued Inside) Selected for a Viewpoint in Physics. Please visit http://physics.aps.org/. By suggesting a few manuscripts each week, we hope to promote reading across fields. Please see our Announcement Phys. Rev. Lett. 98, 010001 (2007). 2 Copyright 2013 American Physical Society Published by Volume 111, Number 2 • What is the “dark matter” that makes American Physical Society™ 0031-9007(20130712)111:2;1-I up a quarter of the Universe?
… for science and for the experimental accomplishment of building IceCube … 7 Tyce DeYoung Neutrino Astronomy
cosmic rays + neutrinos p π± ν 0 π γ p
cosmic rays + gamma-rays
Cosmic rays deflected γ by magnetic fields ν
VHE gamma rays are absorbed by background radiation p execution threads :!5$'1;1+$'*5'-'()*&*"' propagation steps !"#$%&'()*&*"' (between scatterings) photon absorbed +$",&)'&*'-.#*%(/*"' 0!#&-"1$'&*'"$2&'#1-3$%' new photon created (taken from the pool) #1-3$%' (%*(-,-&$'&*'"$2&'#1-3$%' threads complete their execution 1)$14'5*%' 9)$14'5*%' (no more photons) )!&' !"&$%#$1/*"' 0!#&-"1$'&*' +*#&' 6!&)'78#' -.#*%(/*"'
Figure 1: Parallel nature of the photon propagation simulation: tracking of photons entails the computationally identical steps: propagation to the next scatter, calculation of the new direction after scatter, and evaluation of intersection points of the photon track segment with the detector array. These same steps are computed simultaneously for thousands of photons.
3Simulationwithphotonpropagationcode
The direct photon simulation software is typically used in the two scenarios shown in Figure 2. In the first the in-situ light sources of the detector are simulated for calibrating the detector and the properties of the surrounding ice. It is possible to very quickly re-simulate the detector response to a variety of ice scattering and absorption coefficients finely tabulated in depth bins. This allows for these coefficients to be fit directly, by finding the combination that is a best simultaneous fit to all of the in-situ light source calibration data [2]. For the 10 meter depth bins, 200 coefficients are fitted (with scattering and absorption defined in 100 layers spanning 1 km of depth of the detector), with nearly a million possible ice parameter configurations tested in less than a week on a single GPU-enabled computer. This method is intractable with a table-based simulation, as each new parameterTracking set would Neutrinos require Through generation Ice of the new set of parametrization tables, each generation taking on the orderofaweekofcomputingtimeofa ∼ • High energy particles created when neutrinos interact in the ice emit ~160 100-CPU cluster. Cherenkov photons/cm – and IceCube tracks them over a kilometer or more In the second scenario the Cerenkov photons• And they undergo stochastic processes like bremsstrahlung which create even more created by the passing muons and cascades arephotons simulated as part of the larger simulation of• thePhotons detected by IceCube may have traveled 100’s of meters detector response to atmospheric and other fluxesthrough the ice, scattering off of muons and neutrinos. The simulation is abledust grains etc. • No analytic solution to photon to account for some effects that are difficult topropagation, depth-dependent implement with the table-based simulation, be-optical properties cause their simulation would lead to additional• Reconstruct events by maximum likelihood – need p.d.f.’s for every degrees of freedom, thus increasing the size of thephoton we detect! parametrization effort and tables many-fold. One of these is the tilt of the ice layers, i.e.,depen- 10 Tyce DeYoung dence of the ice parameters not only on the depth, Figure 2: Typical simulation scenar- but also on the xy surface coordinates [2]. An- ios: photons emitted by the detector are other effect, recently discovered, is ice anisotropy tracked as part of the calibration proce- [4], which manifests itself as a rouphly 13% vari- dure (left). Cerenkov photons emitted by a ation in scattering length depending on the orien- passing muon and cascades along its track tation. are tracked to simulate the typical IceCube events (right).
GPUHEP2014 3 Optical Properties of South Pole Ice
λabsorption ~ 100-250 m λscattering ~ 20-60 m Ice optical properties reflect climatological history of the polar cap and refreezing process of ice under tremendous pressure The Dust Logger
Designed to map out dust layers 404 nm laser as light source Dusty layers due to volcanic ash Three Dust Logger runs so far (strings 21, 50, and 66) Can measure tilt of dust layers
42k years BP 60k years BP DL Signal (A.U.) Signal DL
depth(m)
Photon Tracking with GPUs
execution threads :!5$'1;1+$'*5'-'()*&*"' propagation steps !"#$%&'()*&*"' (between scatterings)
photon absorbed +$",&)'&*'-.#*%(/*"' 0!#&-"1$'&*'"$2&'#1-3$%'
new photon created (taken from the pool) #1-3$%' (%*(-,-&$'&*'"$2&'#1-3$%'
threads complete their execution 1)$14'5*%' 9)$14'5*%' (no more photons) )!&' !"&$%#$1/*"' 0!#&-"1$'&*' +*#&' 6!&)'78#' -.#*%(/*"'
• Use graphical processing units (GPUs) to track thousands of Figurephotons 1: Parallel in natureparallel of the photon propagation simulation: tracking of photons entails the computationally identical steps: propagation to the next scatter, calculation of the new direction• When after scatter,a photon and is evaluationabsorbed ofby intersection ice or a sensor, points a of new the photonone is drawn track segment to with the detectorreplace array. it from These the same pool steps waiting are computed to be tracked simultane ously for thousands of photons. • Two independent versions, one using CUDA and one OpenCL
3Simulationwithphotonpropagationcode• Provide acceleration factors of 150x or more compared to CPUs The direct photon simulation software is typically used in the two scenarios shown in Figure 2. In the first the in-situ light sources of the detector14 are simulated for calibrating theTyce detector DeYoung and the properties of the surrounding ice. It is possible to very quickly re-simulate the detector response to a variety of ice scattering and absorption coefficients finely tabulated in depth bins. This allows for these coefficients to be fit directly, by finding the combination that is a best simultaneous fit to all of the in-situ light source calibration data [2]. For the 10 meter depth bins, 200 coefficients are fitted (with scattering and absorption defined in 100 layers spanning 1 km of depth of the detector), with nearly a million possible ice parameter configurations tested in less than a week on a single GPU-enabled computer. This method is intractable with a table-based simulation, as each new parameter set would require generation of the new set of parametrization tables, each generation taking on the orderofaweekofcomputingtimeofa ∼ 100-CPU cluster. In the second scenario the Cerenkov photons created by the passing muons and cascades are simulated as part of the larger simulation of the detector response to atmospheric and other fluxes of muons and neutrinos. The simulation is able to account for some effects that are difficult to implement with the table-based simulation, be- cause their simulation would lead to additional degrees of freedom, thus increasing the size of the parametrization effort and tables many-fold. One of these is the tilt of the ice layers, i.e.,depen- dence of the ice parameters not only on the depth, Figure 2: Typical simulation scenar- but also on the xy surface coordinates [2]. An- ios: photons emitted by the detector are other effect, recently discovered, is ice anisotropy tracked as part of the calibration proce- [4], which manifests itself as a rouphly 13% vari- dure (left). Cerenkov photons emitted by a ation in scattering length depending on the orien- passing muon and cascades along its track tation. are tracked to simulate the typical IceCube events (right).
GPUHEP2014 3
High Energy Neutrino Sky Map Tracking Neutrinos Through the Earth
• In particle physics, we have a very accurate Standard Model – but we know it is incomplete
• Neutrinos are the least well understood particles known • A better understanding of neutrinos may shed light on what lies behind the Standard Model
• Our best tool for measuring neutrino properties is their oscillations • Quantum mechanical interference phenomenon that converts one type of neutrino into another as they travel through space
18 Tyce DeYoung Tracking Neutrinos Through the Earth
• IceCube observes neutrinos over a wide range of energies and baselines νµ νµ νµ • Nearly 100,000 neutrino events ~12,700km νµ to date in the relevant energy νµ range – world’s largest neutrino data set at these energies νµ
νµ • Oscillations produce distinctive pattern of flux distortions in energy-angle space νµ νµ • Precise observations of these distortions can be used to measure neutrino mixing, differences between masses IceCube Tyce DeYoung 19 = -1 = z
νµ Tracking Neutrinos cos θ = -0.85 νµ νµ z νµ ~12,700km Through the Earth νµ cosθ
νµ ν • Neutrino flux at production is nearly isotropic, µ with falling power-law energy spectrum νµ νµ • Probability of changing flavor is an oscillatory = -0.5 function of distance and energy, with additional z effects due to coherent interactions with matter cosθ IceCube
P(να)→P(νβ)
x =
Flux x Oscillations = Oscillated Flux Measuring Neutrino Properties
• Neutrino properties can be extracted from the precise shape of the wiggles • Difference between the masses of ν1 and ν2 • How much overlap there is between the different types of neutrinos
• Use binned maximum-likelihood method to compare to data • Need to recalculate pattern repeatedly for different possible neutrino properties
21 Tyce DeYoung 1. Earth Model
¥ 12 layer and 59 layer follows closer to PREM model Matter¥ NuCraft Effects earth densityon Neutrino model follows Oscillations PREM almost exactly
12 Layer Model 59 Layer Model
• Neutrino oscillations arise from solutions to Schrödinger’s Equation Timothy C. Arlen 7 • Quantum mechanical mixtures of mass eigenstates, propagating with different frequencies
• The electron density in matter affects the energy potential, and thus the frequency • Analytic solution exists for constant density, but the Earth’s density isn’t constant
22 Tyce DeYoung High Resolution Oscillation Maps
• Rapid variation in oscillation probability means we can’t simply calculate probabilities at centers of the 800 data bins • Need to calculate average probability over the bin – subdivide real analysis bins finely enough for accuracy
• Typically use a 400 x 200 grid: 80k oscillation probabilities times up to 59 matter layers per iteration of the max-likelihood fit
Analysis level bin size
23 Tyce DeYoung Oscillation Tracking Code
• Prob3 is a code for computing neutrino oscillation probabilities based on the work of Barger et al., Phys. Rev. D22, 2718 (1980). • Analytic solution for 3 ν flavors & layers of constant matter density • Publicly available: http://www.phy.duke.edu/~raw22/public/Prob3++/ • Maintained by the T2K and SuperK collaborations
• Computes transition probability matrix for propagation through an arbitrary number of spherical shells of constant density • Good approximation for matter density of earth
• Implemented in C/C++ and optimized for speed on a CPU • Recently ported to CUDA by T. Arlen to run on iCER GPUs
24 Tyce DeYoung Speed Comparisons
4.6 sec on CPU
75x speedup!
0.062 sec on GPU Full analysis run: 1 GPU-week instead of 9 CPU-months
¥ NVIDIA K20 GPU vs. single CPU core (Intel Xeon 2.50 GHz)
25 Tyce DeYoung Verifying Accuracy
νe survival νe survival residuals ✔ Current and (Near-)Future Results
• IceCube is opening a new window on the Universe – as well as making some of the world’s best measurements of neutrino properties!
27 Tyce DeYoung MSU IceCubers: Tyce DeYoung, Kendall Mahn (faculty) J. P. de André, Joshua Hignight, Dirk Lennarz (postdocs) Garrett Neer, Devyn Rysewyk (graduate students) Dean Shooltz, Michael Nila (technical staff) Neil Patel-Murray, Tawfik Abbas, Hannah Gallamore, Ashley Richey (undergraduates)