A Test Driving Approach For Beamline Software Development Xiaoxu Ren, Paul Gibbons, and Bill Pulford Diamond Light Source Ltd Diamond House Harwell Science and Innovation Campus Didcot, Oxfordshire, OX11 0DE, UK Email: fX.Ren, [email protected] Abstract— For more effective and efficient development of confidence. bespoke software such as Generic Data Acquisition (GDA) for In this paper, the TDD approach used in the Diamond beamline control and data acquisition at Diamond Synchrotron, Data Acquisition and Scientific Computing Group (DASC) a set of Agile methodologies is adopted. In this paper, the test driving development approach used by the Data Acquisition is introduced. The software testing strategies for GDA in and Scientific Computing Group is introduced. The challenges different software development level are summarised. It fo- faced by the beamline software developers and the benefits cuses on the higher level system test and acceptance test. The of introducing TDD approach are summarised, followed by implementation of the testing methods on these two levels the software testing strategies for GDA in different software within the FitNesse framework and the testing environment development level. The focus of this paper is on the higher level system test and user acceptance test. The simulation environment are addressed in details. This paper is organised as follows: for the beamline software system level testing is introduced. The Seciton II presents an overview of TDD approach for beamline testing framework based on the Fit/FitNesse is presented. Both the control and data acquisition software. Testing strategies at dif- GDA system test and acceptance test are conducted on this Web- ferent software development phase are outlined. The simulated based framework. Some test cases and their usage are discussed computational environment of beamline is explained, which for demonstration. constructs a simulation environment for the beamline software system level test. In Section III, The testing framework used I. INTRODUCTION by DASC group is introduced. Both the GDA system testing Software testing is essential to ensure software quality. For and acceptance testing are developed on this framework. In bespoke software such as Generic Data Acquisition (GDA) [1] Section IV, some test cases and testing results are presented for beamline control and data acquisition at Diamond Light for demonstration. Finally, some concluding remarks are made Source, a set of Agile methodologies are adopted for more ef- based on the experience. fective and efficient software development. Among them, Test II. TEST DRIVEN DEVELOPMENT FOR BAMELINE Driving Development (TDD) is one main part of the practice. SOFTWARE Due to the rapidly changing beamline environment and con- stantly varying beamline commissioning demands, beamline Most beamline software development follows the iterative software developers often must deal with requirements that development model (Fig. 1), which means that requirements tend to evolve quickly with short notice. They have to adapt from beamline users are not fully definable before coding can the new changes into the existing software framework without start. As a result, a working version of the software is built in a the real beamline environment and deploy newly developed series of iterations that each iteration follows the requirements features with existing functionalities into real beamline in a gathering, design, coding and testing. Some common problems race with tight beam time schedule. To meet the changing associated to this kind of software development approach requirements, add more functionality within limited time and include: deploy the software in a beamline computational environment ² Difficult to test due to lack of formal documentation and with minimum impact on the existing beamline operation constant changing requirements; are among big challenges for beamline software team. A ² Changes made by software developers are difficult to TDD approach for beamline software development not only trace back to requirement/software code; helps software developers to grasp the essential requirements ² The implementation of the changes can have unintended correctly before the coding stage, but also assists them to make results on other part of software. sure the newly developed features are what scientists want, To counter these problems, test driven development is without losing the existing functionalities or introducing new essential. The functional tests should be written first before bugs. As a result, both the developers and the scientists as coding and testing. Both software developers and the user software customers will benefit from the testing framework representatives have to be involved in the testing and robust with correct functionality, improved software quality and user regression testing has to be performed. earlier and it helps the developers focus on the real beamline requirements. An acceptance test is a black-box test involving a suite of tests on the software deployed in the working beamline environment. Each test, known as a case, targets a specific beamline operational scenario. Ideally, each test case should be associated with a formal description of the required beamline condition, the activity to conduct and the expected results. A passed case should provide a matched result. If there is a Fig. 1. An iterative development model [2] correct match for each test case, the software passes its user acceptance tests. The purpose of beamline acceptance test is to provide both Four levels of software test are practiced through the GDA the DASC group and the beamline scientists with confidence development by the Diamond DASC group to insure the beam- that the beamline software will function according to their line software is being developed in the right way (verification) expectations. Referring to the iterative software development and once deployed in the beamline, the software will meet the model, acceptance tests are carried out after the deployment, user needs (validation): based on the requirement specifications as a basis for test. ² Unit testing Beamline scientists as software users of the system perform ² Integration testing these tests. At Diamond the basic test cases are derived before ² System testing coding by the DASC group, based on the mutually agreed user ² Acceptance testing requirement specification with beamline scientists. Beamline The unit and integration level testing is of the DASC scientists can also create more specific test cases before or software developer’s responsibility to ensure that GDA code after development stage if needed. Once the acceptance tests written for the component meets its specification prior to are completed successfully, the beamline scientists will sign its integration with other components, and works with other off on the deployed software release as tested to indicate its components together via interface interactions. The system readiness for practice. This tested software can then be used level testing is for checking the software functionality from an for the next beamline run with newly enhanced functionalities. end-to-end perspective. For GDA, because the real beamline Any unsatisfactory test cases will be recorded, followed by is usually not available due to tightly scheduled beam time or discussion and evolution between beamline scientists and soft- new beamline hardware commissioning, the system testing is ware developers to trace the causes and provide possible new conducted in a simulated beamline environment. solutions. All the failed tests form together new requirements The simulation environment includes: in the next iterative development cycle. ² Simulated beamline computing environment ² EPICS simulation for beamline hardware control III. FRAMEWORK FOR SYSTEM AND ACCEPTANCE The simulated beamline computing environment is build on TESTING the virtualisation concept, which essentially lets one computer do the job of multiple computers by sharing the resources Automating of testing process at all levels is crucial for of a single computer across multiple environments. For every the TDD approach. For unit testing and integration testing, Diamond beamline, there are virtual counterparts of its control there are widely accepted solutions such as JUnit for Java servers, storage servers and workstations in the simulation or PyUnit for Python/Jython development, At Diamond, both environment. Each new GDA release is tested with its base JUnit and PyUnit are used extensively by the DASC group in configuration and all in-build dummy devices before the GDA software development. For system testing and acceptance release is deployed into different beamline simulations, each testing, the Fit/FitNesse framework [3] [4] has been chosen by with customised beamline configurations and following its the DASC group as the base testing platform. As a framework own deployment rules. The system level testing conducted in of creating, organising, and running Fit tests, FitNesse pro- the simulated beamline environment can reveal errors due to vides a set of flexible tools for software testing at both system interactions across the whole beamline system, or those due and functional levels. Main reasons of choosing Fit/FitNesse to environmental issues. include: By building a virtual beamline environment for GDA system ² Flexibility and extensibility. Custom fixtures can be easily testing, an objective assessment of the software can be made in added to assist the test,