Analysis and Optimization of the Offline Software of the Atlas Experiment at Cern
Total Page:16
File Type:pdf, Size:1020Kb
Technische Universität München Fakultät für Informatik Informatik 5 – Fachgebiet Hardware-nahe Algorithmik und Software für Höchstleistungsrechnen ANALYSIS AND OPTIMIZATION OF THE OFFLINE SOFTWARE OF THE ATLAS EXPERIMENT AT CERN Robert Johannes Langenberg Vollständiger Abdruck der von der Fakultät für Informatik der Technischen Universität München zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften (Dr. rer. nat.) genehmigten Dissertation. Vorsitzender: 0. Prof. Bernd Brügge, PhD Prüfer der Dissertation: 1. Prof. Dr. Michael Georg Bader 2. Prof. Dr. Siegfried Bethke Die Dissertation wurde am 03.07.2017 bei der Technischen Universität München einge- reicht und durch die Fakultät für Informatik am 11.10.2017 angenommen. ABSTRACT The simulation and reconstruction of high energy physics experiments at the Large Hadron Collider (LHC) use huge customized software suites that were developed during the last twenty years. The reconstruction software of the ATLAS experiment, in particular, is challenged by the increasing amount and complexity of the data that is processed because it uses combinatorial algorithms. Extrapolations at the end of Run-1 of the LHC indicated the need of a factor 3 improvement in event processing rates in order to stay within the available resources for the workloads expected during Run-2. This thesis develops a systematic approach to analyze and optimize the ATLAS recon- struction software for modern hardware architectures. First, this thesis analyzes limita- tions of the software and how to improve it by using intrusive and non-intrusive tech- niques. This includes an analysis of the data flow graph and detailed performance meas- urements using performance counters and control flow and function level profilers. Different approaches and possible improvements are evaluated. An analysis of a new low complexity track reconstruction algorithm is presented. No full analysis of the grown structure of the ATLAS reconstruction software had ex- isted so far. The performance analysis identified where optimization efforts would have the largest impact. The analysis concludes that the tracking, already being the computa- tionally most expensive part, would become more dominant with future data. Consequent- ly, the optimization efforts concentrated on the tracking, combining algorithmic and structural changes as well as replacing outdated libraries. A detailed description of the implemented changes and their impact is presented. All changes combined lead to a speedup of a factor of 4.5 for the reconstruction. Both code quality and the process to ensure code quality have been improved. For the presented parallelization study a data flow graph was created and combined with module’s runtimes. The critical path was found to consume 95% of the reconstruction time, showing that there is little potential perfor- mance improvement through of inter-module-parallelism. The analysis concludes that for the tracking most parallelism must be inside a majority of the modules to have a signifi- cant effect. Currently, efforts are focused on achieving thread-safety of all modules to start introducing parallelism, while recent efforts reduced the overhead of processing multiple events concurrently on multicore machines. The new transform-based tracking method presented is parallelizable within an event and can be applied to preselect tracks to reduce the complexity for the classic slow reconstruction. It runs up to 17.5 times faster than the comparable algorithm currently in use. CONTENTS 1 INTRODUCTION .................................................................................................................. 1 1.1 RESEARCH QUESTIONS ................................................................................................................................... 4 1.2 CONTRIBUTIONS .............................................................................................................................................. 5 1.3 STRUCTURE OF THE THESIS .......................................................................................................................... 6 2 RESEARCH ENVIRONMENT ............................................................................................. 8 2.1 CERN ................................................................................................................................................................ 8 2.2 THE LARGE HADRON COLLIDER ............................................................................................................... 10 2.3 LHC PERFORMANCE IN RUN 1 .................................................................................................................. 11 2.4 ATLAS ........................................................................................................................................................... 12 2.5 THE ATLAS INNER DETECTOR ................................................................................................................. 14 2.6 DATA TO BE RECONSTRUCTED IN ATLAS ............................................................................................... 16 2.7 TRACK RECONSTRUCTION .......................................................................................................................... 17 2.7.1 Reconstruction efficiency ................................................................................................................. 21 2.8 SOFTWARE USED BY ATLAS ...................................................................................................................... 22 2.9 THE ATHENA FRAMEWORK ....................................................................................................................... 23 2.10 COMPUTING INFRASTRUCTURE .............................................................................................................. 24 3 SOFTWARE DEVELOPMENT AND COMPUTING IN ATLAS ................................... 28 3.1 CORRELATION OF BEAM AND COLLISION SETTINGS AND PROCESSING LOAD .................................. 28 3.2 TRACK RECONSTRUCTION COMPLEXITY .................................................................................................. 30 3.3 EVENT DATA MODEL .................................................................................................................................. 31 3.3.1 CPU and Memory consumption ..................................................................................................... 31 3.4 SOFTWARE DEVELOPMENT IN ATLAS ..................................................................................................... 32 3.4.1 Building and Testing in ATLAS ...................................................................................................... 35 3.4.2 EDM Design Considerations ............................................................................................................ 35 3.5 HARDWARE EVOLUTION ............................................................................................................................. 36 3.5.1 Parallel Resources on Modern Computing Hardware ......................................................... 36 3.5.2 Parallelization on modern architectures .................................................................................. 39 3.5.3 Multi-core CPU versus GPU processing ...................................................................................... 40 3.6 SOFTWARE EVOLUTION ............................................................................................................................... 40 3.6.1 Profiling evolution ............................................................................................................................... 40 3.6.2 Parallel programming trends ........................................................................................................ 41 3.6.3 Library evolution ................................................................................................................................. 41 3.6.4 Compiler optimizations ..................................................................................................................... 42 3.6.5 Continuous development on Athena ............................................................................................ 42 3.7 CONCLUSION ................................................................................................................................................. 42 4 COMPUTATIONAL PERFORMANCE ANALYSIS ......................................................... 43 4.1 DECOMPOSED RECONSTRUCTION PERFORMANCE ................................................................................ 43 4.1.1 A general test case for performance observations ................................................................ 44 4.1.2 State of ATLAS reconstruction software before LS1 ............................................................ 45 4.1.3 Source-code efficiency and hotspots ........................................................................................... 47 4.2 OPTIMIZATION AREAS ................................................................................................................................. 49 4.2.1 Software Environment ...................................................................................................................... 49 4.2.2 Hardware Environment ...................................................................................................................