Managing the CMS Online Software integrity through development and production cycles
Luciano Orsini – CERN (PH/CMD) and Dainius Simelevicius - Vilnius University (LT) Co-author: Penelope Roberts Nottingham Trent University (GB)
1st Developers@CERN Forum, Software Quality, Reliability and Testing 28th and 29th of September 2015 IT Amphitheatre and the Training Center, CERN Switzerland(D7.0)
Outline
¤ A quick overview of the CMS DAQ
¤ The CMS Online Software
¤ The Application Development Process and Technology Environment
¤ Concluding remarks
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 2 A Quick Overview of CMS Data Acquisition System Context Diagram
Detector Front-End L1 Trigger
Detector Control Run Control and CMS DAQ Monitoring
High-Level Trigger
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 4 CMS Data Acquisition System (DAQ2)
CMS Online Software
CMS HLT Software
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 5 The CMS Online Software
¤ What is the CMSOS ¤ The software stack ¤ Some history The CMS Online Software
¤ A homogeneous platform for data acquisition in a distributed environment ¤ Fulfills performance requirements ¤ Same model and design for most of the DAQ applications ¤ Breakthrough hardware/software co-designed model ¤ Highly modular, scalable and portable ¤ Interoperability through standardized protocols for control and data flow ¤ Uniform exception handling and monitoring over the whole system ¤ Extensible to new functions and protocols without disrupting applications
¤ Across the board approach ¤ Promoting the use of emerging industry standards to distributed DAQ ¤ SOA (Service Oriented Architecture) built on Web services ¤ DAQ as peer to peer ¤ DAQ as World Wide Web
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 7 CMSOS Stack
Online Software
Detector Event Front-end External System Worksuite Specific Builders Controllers Interfaces Applications
CMS specific
Data Error and User Powerpack Job Control Monitoring Alarming Interfaces
Re-usable Applications
Executive Hardware Communication Coretools OS Abstraction Framework Access Subsystems Configuration Management Support Configuration
Core Framework Hardware Operating Networking Platforms Device Systems Infrastructures Interfaces
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 8 Some Numbers
¤ CMSOS is divided into a number of self-contained packages ¤ 172 in-house developed packages ¤ 33 external third party packages
¤ More than 100 users and application developers
¤ About 10 participating core software developers since year 1998
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 9 Development History
First beam in CMS September 10th First collisions in CMS November 23rd Detector First 7 TeV collisions in CMS, March 30th Magnet We delivered! Test on surface 5fb -1 Version in use collected
LS1 LS2 v6.x 7Tev 14Tev Software First Releases prototype v7.x TDR Vol 2 v9.x 2002 December 15th v10.x Experiment wide adoption v11.x v12.x
Feasibility Activities Achieving required performance and flexibility Development of production quality applications Improvements and maintenance DAQ Upgrade and new technologies
System 2 slices 128x128 8x8 100KHz 16x16 4 slices 256x256 128x128 32x32 64x64 8 slices 512x512 Today
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 10 The Application Development
Process and Technology Environment
¤ Aspects of motivation ¤ Institutionalize the development environment ¤ Practitioner's view of the approach Aspects of Motivation
¤ CMS consists of a set of sub-projects ¤ Similar to a coordinated set of small experiments ¤ Many scenarios like central DAQ, sub-detector DAQ, test-beams etc.
¤ Geographically dispersed participants
¤ Autonomous development and different cultures
¤ High personnel turnover
¤ High performance and robustness requirements
¤ Long lifetime and need to survive technology generations
¤ Large number of packages and functionalities
¤ Cope with urgent issues and change in priorities (e.g. operations)
¤ Etc.
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 12 What is all About
¤ A platform for managing the change, correcting defects and implementing new requirements
with Integrity Software Product Changes Product Update
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 13 Contribution to Software Integrity
Management
Analysis CMSOS Quality Assurance Design Engineering with Integrity Verification & Validation Coding Test & Evaluation Unit/Subsystem Test Configuration Management Documentation & Training Development Product Assurance
Focus on product assurance through Configuration Management (CM) where software development is disciplined by providing procedures for controlling software evolution and supporting technologies
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 14 Cultivating the Process Environment
¤ Sow the seeds in an attempt to establish a process environment for attaining and maintaining a software product integrity ¤ Set of policies, guidelines, procedures and standards defining the process for developing deliverable products (e.g. documents, code, etc.) ¤ The set of the technologies used to develop software and its related product
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 15 Institutionalizing the Development Environment
Process Technologies
IT WEB Services E-Groups
Configuration Management Plan and Procedures
Development and Building Manuals Custom Software
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 16 Inside Configuration Management
¤ Identification ¤ Traceability ¤ Planning ¤ Releasing
Identifying Software Source Repository Version and Compatibility Support Owner (SVN path) (Trac subject)
Configuration Item (CI) Standard Structure (SVN directory tree) modular design is a must
Identification (Trac component)
Baselines (SVN branches)
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 18 CMSOS Baselines ¤ The environment ensures CMSOS development is regulated by providing procedures for creating and updating baselines ¤ Baseline is an approved snapshot of the software at a given point in time ¤ An arbitrary number of new baselines can be created ¤ Several baselines can be maintained in parallel ¤ A baseline is identified by a SVN path
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 19 Parallel Baselines and Revisions
New baselines are branches of the development trunk
frozen Baseline10.0
R10.1 R10.1.2 R10.2
Development Trunk
Baseline12.0
R12.1 Baseline11.0
R11.1 R11.2
Backward compatibility is Same CI can exist in guaranteed within the same different flavors baseline
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 20 Traceability Traceability
Diffs Baseline Source Code and Versions (SVN) Developers and Experts
Enhancement Defect Discrepancy Ticket Task Administrators
Users
Support Workflow Cross-references Released Documents Watchers Todo Doing Done Products
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 22 Planning Planning the Development
¤ Based on milestones facility in Trac ¤ A milestone is linked to a baseline in SVN ¤ A milestone has a due date ¤ Milestones are used for both creation and update of baselines
¤ A number of tickets are selected according to priority and attached to milestones
¤ Work on tickets determines the progress of the milestone
¤ Work is completed when all tickets are Tickets resolved
Several milestones can be processed in parallel
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 24 Releasing Development Iterations and Release
Daily Cycle
Software Increment
Milestone
Sprint Cycle
Official RPMs
Sprint Backlog Product Increment Tickets Document Updates
Software Improved Environment
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 26 Development
Workflow Build Done Todo Doing platform
Tickets Test RPMs
SVN operations
Unit/Subsystem tests
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 27 Build and Production
Todo Doing Done Software Release Notes
Administrator
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 28 Deploying the software
¤ RPMs are installed to required machines ¤ For development, testing we use RPM and YUM directly ¤ For operations in Point 5 (.cms network) we use Puppet
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 29 Concluding Remarks Future Perspectives
¤ More automation and technology integration ¤ Integrated baseline creation ¤ Building and releasing ¤ Integrated documentation facilities
¤ Integrate SCRUM based tool
¤ Investigate new technologies and migration ¤ Jira, GIT, GITLab, Trac extensions, Perforce, etc. ¤ Pulp, Copr, Jenkins
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 31 Acknowledgements
² IT People for their support on the technologies used Many thanks for your attention
Are there any questions Backup Production Item
RPM is a unique item with a unique name CI name – Version# Major.Minor.Patch - Build# - baseline#- platform – compiler – architecture (e.g. daq-toolbox-8.4.2-1.cmsos12.slc6.gcc4_4_6.x86_64.rpm)
Yum Repository and Location (e.g. http://{…}/12/slc6x/x86_64/base/RPMS/)
Build Platform
Traceable back to baseline
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 35 SVN Repository Structure
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 36 Trac Components Extension
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 37 Package Standard Structure
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 38 Trac Roles
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 39 Trac Milestones
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 40 Trac Ticket Changeset
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 41 Trac Baselines
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 42 Trac Diff
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 43 YUM Repository
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 44 Version and Compatibilty
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 45 Abstract
The Data Acquisition system of the Compact Muon Solenoid experiment at CERN is a distributed system made of several different network technologies and computers to collect data from more than 600 custom detector Front-End Drivers. It assembles events at a rate of 100 kHz, transporting event data at an aggregate throughput of 100 GByte/s. The architecture takes advantage of the latest developments in the computing industry. For data concentration, 10/40 Gbit Ethernet technologies are used while a 56Gbps Infiniband FDR CLOS network has been chosen for the event builder with a throughput of ~4 Tbps. The CMS Online Software (CMSOS) infrastructure is a complex product created specifically for the development of large distributed data acquisition systems as well as all application components to achieve the CMS data acquisition task. It is designed to benefit from different networking technologies, parallelism available on a processing platform such as multi-core or multi-processor systems. It provides platform independent framework that builds upon industrial standards, open protocols and libraries, tools for local and remote inter-process communication, configuration and control. It is designed according to the object-oriented model, and is implemented using the C++ programming language. The processing infrastructure is made scalable by the ability to partition applications into smaller functional units that can be distributed over multiple processing units. The CMSOS is highly modular. It is organized in several interdependent packages that can be individually deployed for operations. The CMSOS product development and the maintenance such as repairing problems and deficiencies, enhancement and new requirements occur in a well-established environment. Achieving a high level of the CMSOS product quality during its lifetime is an iterative process that consists of collective activities like technical and administrative direction and surveillance, whose purpose is to maintain the integrity of the software products as these products evolve from specification through design, development and production cycles (e.g. releases, milestones, baselines, versions, etc.) A cost-effective implementation of an application development process and technology environment is described to address the product evolution with the required quality over time. The provided environment has shown benefits at accounting the complexity issues associated with the software itself, accommodating the problem of transient developers and overcoming different development cultural values. The approach based on Software Configuration Management practices attempts to achieve the right balance between the support to a visible and traceable development and the overhead due to shuffling paperwork. This work is based on the experience of the authors at CMS DAQ group and the experiences of others in different organizations who have been faced with similar concerns.
1st Forum Developers@CERN -28-29 September 2015, Amphitheatre and the Training Center, CERN Switzerland – Luciano Orsini CERN (PH/CMD) 46