RPPB05 Proceedings of ICALEPCS07, Knoxville, Tennessee, USA

APPLYING AGILE PROJECT MANAGEMENT FOR ACCELERATOR CONTROLS SOFTWARE W. Sliwinski, N. Stapley, CERN, Geneva, Switzerland

Abstract engineers, who are members of AB-CO group, develop Developing accelerator controls software is a frameworks, core components and complete systems. challenging task requiring not only a thorough knowledge Thus, the users’ community is diverse, the level of of the different aspects of particle accelerator operations, proficiency in programming varies and many people but also application of good development practices and within it work on temporary basis (such as project robust project management tools. Thus, there was a associates, fellows and technical students) for the Large demand for a complete environment for both developing Hadron Collider (LHC) start up. and deploying accelerator controls software, as well as the Requirements of the Process tools to manage the projects. As an outcome, a versatile development process was formulated, covering the Taking into account the needs of the controls controls software life cycle from the inception phase up to community and the specific nature of the controls domain, the release and deployment of the deliverables. In a list of requirements can be formulated which addition, a development environment was created characterize the properties of the anticipated development providing management tools that: standardize the process: common infrastructure for all the concerned projects; help • Maximise ease of use by providing a quick to organize work within project teams; ease the process of comprehension time (particularly significant for versioning and releasing; and provide an easy integration temporary members). of the test procedures and quality assurance reports. • Standardize development environment (source Change management and issue tracking are integrated repository; projects’ structure; management of with the development process and supported by the dependencies and 3rd party libraries; build, release dedicated tools. This approach was successfully applied and deployment management). for all the new controls software for LEIR, SPS, LHC, • Automate repetitive tasks (build and release injection lines and CNGS extraction. management; running of code coverage and test suites). INTRODUCTION • Encourage high quality (code quality; design and The Controls Group, in the Accelerators and Beams code reviews; documentation; test suites). Department (AB-CO) at CERN, is responsible for the • Provide complete software configuration controls infrastructure of all the accelerators at CERN and management solution (code identification; it provides core systems for accelerator operations, e.g.: configuration control; components’ consistency central timing, front-end systems, equipment settings review; build management; defect tracking). management, software interlocks and machine protection, • Ensure reproduction of the same artefacts at any time alarms, logging services, sequencer for hardware in the future. commissioning, access control, among others. • Provide obvious value so teams easily realize its Development of the core systems can not be done in benefits. separation and a unified infrastructure is necessary in order to streamline all the efforts and to promote reuse of Agile Practices for Controls the common components. Therefore a common When developing controls systems at CERN, there is a development environment was established for software significant dependence on face-to-face communication development particularly in the Java programming with the operators, nearby in the CCC. The work is language, aimed at providing all the necessary services distributed among several small teams (between 2 and 5 for the controls community. people) often composed of co-located staff and temporary members. ABOUT THE DEVELOPMENT PROCESS Each team is assigned to one or more projects which are managed by a project leader, who is an experienced Controls Community software engineer and maintains close collaboration with Users of the controls infrastructure comprise not only operators, who provide requirements and domain of the members of AB-CO group but also operators in the expertise. User requirements can often change and the CERN Control Centre (CCC) [1], physicists and resulting features are needed quickly, which in turn leads equipment experts. Since operators are domain specialists to the necessity of frequent product releases. Operators they often provide user requirements for the control are open to test the new releases especially if they know systems. On the other hand, experienced software

Engineering Processes, Project Management Collaboration 612 Proceedings of ICALEPCS07, Knoxville, Tennessee, USA RPPB05

that their new features are available, thus enabling further with JUnit [6]; code quality inspection (using the PMD feedback for following versions. [7] tool); code coverage reports (using Cenqua Clover Having gained operational experience in the controls [8]); source code and documentation generation; domain and having tried several available approaches to integration with the release system. All products must project management in the past, including Rational follow the same directory structure and use common Unified Process (RUP) [2], the agile approach appears to configuration files, thus for the developer, the effort be very promising in assisting teams in developing required when working with a project is greatly software in the manner mentioned above. It was perceived minimized. that agile methods provide the most benefits for our environment, avoiding the overhead related to heavy- Release weight processes, where software releases are performed The release process is handled by a separate tool – only when significant amount of changes or new features Release, which is also based on Apache Ant and supports are implemented. Agile practices are not in fact a Java, C and C++. The main responsibilities of the Release traditional methodology – it is a collection of guidelines tool include: ensuring completeness and consistency and conventions based on experience, knowledge and among components; managing the build process and tools tools. Moreover, the approach is based on reflection and used for builds; ensuring adherence to the organization's continuous improvement of the current environment, development process; managing the software distribution responding to changes in technology, development best repository; making sure every defect has traceability back practices and user requests. Users are not constrained by to the source. Release tool is composed of a client and the process but instead can concentrate more on the server, where the client is fully integrated with the development related to their domain. Common Build, but it can be also used standalone. The A complete environment for Java development was Release tool extracts product sources from the CVS to a provided and only standard Java community tools were new version directory in the distribution repository and used if possible. However in order to address some builds the product by calling build tool (e.g. Common specific requirements of the controls community two Build for Java applications). Subsequently, the new custom solutions were developed: Common Build (build version is installed in a multi-versioned repository system) and Release (release and version management without modifying any previous versions. system) [3] tools. Continuous Integration DEVELOPMENT TOOLS Projects are increasingly leveraging code from other projects. As these "common components" are maintained, What follows is an overview of the tools, which grew any update must be tested with all its dependent projects. over the time and they were setup in order to help manage A Continuous Integration (CI) server was introduced to the development. Some of the mentioned tools we are still automate this increasingly time-consuming task. Using learning how to exploit fully; others are quite mature in web-based Atlassian Bamboo [9] tool, projects can be their use. marked as dependents of a component whereby they are Versioned Source Repository re-built and their test suite re-run for verification. This minimizes the risk of compatibility problems after a new All software products are stored in a dedicated version is released and reduced the time spent by repository for accelerator controls software in the CERN developers on this task. central Concurrent Versions System (CVS) [4]. Each The CI server also produces reports based on a project's product has a well defined directory structure following tests, code coverage, code metrics and inspection. This naming conventions which reflect part of the domain that provides an open account of the health of a project’s code this product is responsible for. base, highlighting areas for improvement. More recently, Common Build the acceptance level of unit tests or of some agreed code metric rules (for example, each class file must have at Common Build is a build system, based on Apache Ant least one comment) can be set. In the future, a project will [5], providing the common tasks required for be considered complete, not only when it provides all the development. When the work on new CERN controls required artefacts, but also when it meets the quality system began in 2002, there was a need for a build tool standards. which would help to set up an easy and uniform process that could be applied to new Java development. Due to Issue Tracking the lack of a mature build tool available at that time, it According to the “lean” side of Agile, work waiting to was decided to create a custom solution based on standard be done should be kept to a minimum. Task switching and tools. Common Build provides the following set of keeping work “in process” is a “waste”. Ideally, the list of common tasks: dependencies management including 3rd bugs and feature requests waiting for completion should party libraries; compilation and packaging; unit testing be limited. Nevertheless users and developers still need a

Engineering Processes, Project Management Collaboration 613 RPPB05 Proceedings of ICALEPCS07, Knoxville, Tennessee, USA

tool to hold their bug reports, feature requests, and project particularly important for system stability as it is easier to to-do lists, and for this Atlassian Jira [10] is used. Projects test and release many smaller versions with relatively few can have a large number of stakeholders, so the web changes, than a major one. interface enables transparency between them showing The provided tools are used for everyday development what work done, work ongoing, work planned and and project management – an interesting example of this priorities. is for new feature requests. These are tracked in Jira and presented at planning meetings so users can decide on, Wikis for Collaborative Online Documentation and prioritises the backlog list. It is then obvious to all Project documentation was traditionally created as concerned what is to be worked on for the next iteration. HTML. Using a wiki has made it easier to create and update reference manuals and user tutorials, with the CONCLUSION ability to version pages and easily link between them. In general, agile practices have improved project Users are also encouraged to participate and add development in AB-CO group. As a part of this overall comments to share their experiences or improve the environment, the tools mentioned above have helped to original material. One lesson learned is that, for high reduce development time and organise work in the quality information, it is still necessary to have a process projects in a standard manner. Some agile elements were for reviewing information regularly with a vision of the found to be in place already such as good communication site content and structure. between development teams and operators. Other elements like continuous improvement still require further Source Code Searching and Inspection refinement. Finally, core tools like Common Build and Open inspection and searching of code within CVS is Release are already acknowledged to be indispensable not easy to do with a large code base, which should be a and are heavily used by all the projects. valuable team resource. Although tools like ViewVC [11] are available, the information provided is very limited. REFERENCES Cenqua Fisheye [12], a code repository search tool, has [1] D. Manglunki and P. Charrue, “The CERN Control proven to be very useful in this area. Developers can track Centre: Setting Standards for the XXIst Century”, changes as change sets, search code for duplication, or ICALEPCS’2007, Knoxville, October 2007. find how others have tackled a similar problem or used a [2] IBM Rational Unified Process: certain library. www.ibm.com/software/awdtools/rup/ [3] G. Kruk et al., “Development process of accelerator APPLYING THE PROCESS TO PROJECTS controls software”, ICALEPCS 2005, Geneva, Two example projects which employ an agile October 2005. development process are LSA [13] and LASER [14]. [4] Concurrent Versions System: The LHC Alarm Service (LASER) is an alarm and http://www.nongnu.org/cvs/ notification system. It has to be flexible due to its’ large [5] Apache Ant: http://ant.apache.org/ number of disparate users including LHC and the whole [6] JUnit: http://www.junit.org/ accelerator chain as well as the technical infrastructure. [7] PMD: http://pmd.sourceforge.net/ The LHC Software Architecture (LSA) project provides [8] Cenqua Clover: http://www.cenqua.com/clover/ homogenous application software to operate the [9] Atlassian Bamboo: accelerators. It was already successfully used from 2005 http://www.atlassian.com/software/bamboo/ onwards to operate the Low Energy Ion Ring (LEIR), [10] Atlassian Jira: SPS and its transfer lines, and LHC, replacing the existing http://www.atlassian.com/software/jira/ old software. [11] ViewVC: http://www.viewvc.org/ Both projects tried and benefited from a number of [12] Cenqua Fisheye: http://www.cenqua.com/fisheye/ agile practices. They produced a simple, limited feature, [13] G. Kruk et al., “LHC Software Architecture [LSA] – working version early on and extended it incrementally. Evolution Toward LHC Beam Commissioning”, Developers on the projects work within the same room or ICALEPCS 2007, Knoxville, October 2007. corridor to lower the barrier of communication. They aim [14] Peter Sollander et al., “Alarms Configuration to integrate features into production frequently rather than Management”, ICALEPCS 2007, Knoxville, October have multi-featured long-awaited releases. This is 2007.

Engineering Processes, Project Management Collaboration 614