Master thesis in informatics – Software development Improvements and extensions to the configuration tools of the TaskManager HLT software in the ALICE experiment at CERN Øystein Senneset Haaland University of Bergen November 2007 Abstract This thesis concerns work done to improve and extend software that is used to configure nodes and trigger software in High-Level Trigger (HLT) under the ALICE experiment at CERN. HLT is a large computer cluster that will be used to reduce data from the ALICE experiment to a an amount manageable by storage systems. To accomplish this efficiently the processing is distributed throughout the cluster and the configuration tools are used to decide how this distribution should be done. The configuration tools are composed of a few Python programs and templates, which generates configuration files for all participating nodes by taking input from a single Extensible Markup Language (XML) file describing the entire setup. While optimization and general improvement is one part of this thesis, the emphasis will be on extending the software to incorporate functionality for creating a more structured control hierarchy in the HLT data processing framework. This thesis will present the tasks to be resolved, discuss possible solutions and describe the development process. Preface The work with this thesis has been both exciting and challenging. It has been a journey of hard work and many leassons learned. Working in the setting of a large, world wide collaboration as the ALICE experiment, has given experience with international work that can hardly be matched elsewhere for a master student. For a student with a background in informatics, getting to know the world of physics from the inside, has given new perspectives to how the world is viewed. All in all, it has been an extremely rewarding experience. I would like to thank my supervisor H˚avard Helstrup for prompt and comprehensive feedback all along the process, and for tirelessly reading and commenting drafts of my thesis. Kristin Fanebust Hetland, also at Høgskulen i Bergen, has done a great job of following-up my progress and providing useful advice. The assignment for this thesis has been given by Institutt for Fysikk og Teknologi on behalf of the ALICE experiment at CERN. The institute has therefore been where i have had my work space. At the institute i would like to thank Dieter Roehrich for giving me the opportunity to work abroad at CERN, in Heidelberg and Pader- born. I would also like to thank those with whom i have shared office – Are, Knut, Kyrre, Torstein and Øystein – and from our group – Dag, Gaute, Kenneth, Matthias and Sebastian – for inspiration, being helpful and for adding a social aspect to the experience. Further, i would like to thank our colleagues and HLT collaborators at Kirchhoff Institute of Physics in Heidelberg for invaluable impulses. In particular Timm M. Steinbeck for proposing such an exciting assignment for this thesis and for providing valuable input. Thanks also to Lisa for proofreading my English and student advisor Ida for helping out with all practical problems. Finally, I would like to thank my family, my parents in particular, for selflessly encourageing me to go my own ways and for supporting my efforts. Bergen, November 2007 Øystein S. Haaland iii Contents 1. Introduction 1 1.1. Physics background . 1 1.1.1. CERN . 2 1.1.2. LHC - Large Hadron Collider . 2 1.1.3. ALICE . 3 1.1.4. HLT . 4 1.2. The assignment . 4 1.3. Structure of the report . 6 2. Background 9 2.1. Purpose of HLT . 9 2.2. Detector structure . 9 2.3. Hardware . 11 2.3.1. H-RORC . 11 2.3.2. CHARM . 12 2.4. Existing software . 12 2.4.1. HLT - AliRoot components . 12 2.4.2. The Publisher-Subscriber framework . 12 2.4.3. TaskManager . 15 2.4.4. TMGUI.py - Task manager GUI . 20 2.4.5. HLT TPC Online display . 21 2.4.6. Configuration tools . 21 2.5. Fault tolerance in HLT software . 22 2.6. Known methods . 22 3. Configuration tools 23 3.1. Control hierarchy in HLT software . 23 3.2. Motivation . 23 3.3. Implementation . 24 3.4. New requirements . 26 3.4.1. Functional requirements . 26 3.4.2. Non-functional requirements . 27 3.5. Why interesting? . 28 v 4. Problem analysis 29 4.1. Problem definition - precisions . 29 4.2. Methods and technology . 30 4.2.1. Practices and methodologies . 30 4.2.2. Programming languages . 34 4.2.3. Software, tools, libraries . 39 4.2.4. Technology . 40 4.3. Development considerations . 42 4.3.1. Implementation guidelines . 42 4.3.2. Automated build and installation tools . 43 4.3.3. Development environment/setup . 45 5. Solution 47 5.1. Task break down . 47 5.1.1. Improve XML parsing code . 47 5.1.2. Usability improvements chain operation . 48 5.1.3. Distributed configuration creation . 49 5.1.4. A mapping program . 50 5.1.5. Avoid recompilation of Python bytecode . 50 5.1.6. Repeated creation of configuration objects . 51 5.1.7. Explore different approach . 51 5.1.8. Finalize servant/node group implementation . 52 5.2. Chosen solutions . 52 5.2.1. Improve XML parsing code . 53 5.2.2. Usability improvements chain operation . 53 5.2.3. Distributed configuration creation . 53 5.2.4. A mapping program . 53 5.2.5. Avoid recompilation of Python bytecode . 54 5.2.6. Repeated creation of configuration objects . 54 5.2.7. Explore a different approach . 54 5.2.8. Finalize servant/node group implementation . 55 6. Implementation 57 6.1. User stories . 57 6.1.1. XML parser improvements . 57 6.1.2. Usability improvements chain operation . 59 6.1.3. Distributed configuration creation . 59 6.1.4. A mapping program . 60 6.1.5. Avoid recompilation of Python bytecode . 63 6.1.6. Repeated configuration creation . 64 6.1.7. NBus prototype . 64 6.1.8. Servant/node group . 66 6.2. Structural changes . 66 6.3. Contribution . 66 vi 7. Evaluation and testing 69 7.1. Test methods . 69 7.1.1. Regression testing . 69 7.1.2. Profiling - testing for effectiveness . 69 7.2. Test results . 69 7.2.1. Rewrite of XML parsing code . 70 7.2.2. Single Node Mode . 70 7.2.3. NBus . 71 7.3. Evaluation . 71 8. Conclusion 73 8.1. Summary . 74 8.2. Further work . 74 A. Single-node mode test 77 A.1. Test setup . 77 A.1.1. Nodes . 77 A.1.2. Tools . 77 A.1.3. Infrastructure . 77 A.1.4. Notes . 77 A.2. Results . 78 A.2.1. Conclusion . 78 A.2.2. Comments . 78 B. Sample master configuration file 81 Glossary 83 Bibliography 87 vii List of Figures 1.1. Overall view of LHC experiments[1]. 3 1.2. DAQ - HLT dataflow[2]. 5 2.1. The layers of HLT. Detector data can be seen entering at the top and being split by D-RORC and sent on to HLT[2]. 10 2.2. Control and data flow in hlt software, including servant and node group concepts. 13 2.3. Example of components in Publisher-Subscriber framework[3]. 16 2.4. Internals of the TaskManager[4]. 18 2.5. TaskManager control and data flow[4]. A simple master, slave control hierarchy is depicted. Large rectangles represents nodes. Ellipses are processes. 19 2.6. Screenshot of TaskManager control GUI. 20 2.7. Screenshot of TPC Online display[1]. 21 3.1. Process of creating configuration files. 25 3.2. Class diagram of configure script. 26 4.1. All diagrams defined by the UML specification[5][6] . 38 5.1. State propagation in TaskManager hierarchy . 52 6.1. Initial structure of xml parsing elements in configuration tools. 58 6.2. Structure after refactoring and new implementation of xml parsing. 58 6.3. Configuration tools in non-distributed mode . 60 6.4. Configuration tools in distributed mode. ..