17th Int. Conf. on Acc. and Large Exp. Physics Control Systems ICALEPCS2019, New York, NY, USA JACoW Publishing ISBN: 978-3-95450-209-7 ISSN: 2226-0358 doi:10.18429/JACoW-ICALEPCS2019-WEPHA090 TESTING TOOLS FOR THE IBEX CONTROL SYSTEM T. Löhnert, F. A. Akeroyd, K. Baker, D. Keymer, A. J. Long, C. Moreton-Smith, D. Oram, ISIS Neutron and Muon Source, Didcot, UK J. R. Holt, T. A. Willemsen, K. J. Woods, Tessella, Abingdon, UK Abstract behaviour of the component being tested, which is espe- cially helpful if the original author has since moved on. At the ISIS Neutron and Muon Source [1], we are cur- Finally, having a well-designed test framework with rently in the process of replacing the legacy SECI control strong simulation capabilities goes a long way to ensure system with the next generation IBEX control system [2] the surprises when testing with a real beamline are kept to based on EPICS [3]. Since IBEX replaces a fully func- a minimum. tional control program, users and developers need to have Robust testing practices raise the confidence of the de- confidence that the new system works as well as the old velopers that new code works as intended, and that exist- one, ensuring that the migration does not cause any undue ing code continues to work when subjected to changes. It disruption to beamline operations. Automated testing is an also helps raise the confidence of users, who are generally indispensable tool to continuously ensure our software is risk-averse that the new control system they are being up to the highest standard of quality. This paper gives an given performs its function satisfactorily. overview of the various types of testing tools we utilize at In the following sections, we will explore a selection of ISIS, with a particular focus on our IOC test framework relevant IBEX components, and the tools in place to en- [4]. This framework has been developed in-house for sure their continued functionality. testing drivers with simulated devices in order to circum- vent testing limitations due to beamline/device availabil- IBEX COMPONENTS ity. INTRODUCTION For the last 5 years, we in the Experiment Control team have been developing the next-generation instrument control system called IBEX for use on beamlines in the ISIS facility. The migration to IBEX is an ongoing pro- cess, with around half of our beamlines currently running on the new system. Being such a large, complex and long- running project make IBEX prone to failure if proper care is not taken. Consider the following facts: 2019). Any distribution of this work must maintain attribution to the author(s), title of the work, publisher, and DOI. • IBEX is a complex ecosystem consisting of many © different services. It is not always obvious what im- pact changes to one component will have on another Figure 1: Overview of (selected) IBEX components. • Over the course of its lifetime, over 30 developers IBEX covers a wide range of responsibilities related to have contributed to the IBEX project, many only for experiment control, from interacting with individual bits a limited duration and sometimes early in their soft- of beamline equipment such as temperature or pressure ware engineering careers controllers, choppers, jaws etc. to managing data collec- • The IBEX system is too complex even for permanent tion and writing the final experiment data file. To this staff to be familiar with every part of it at any one end, it comprises a number of technologies and services. time, let alone have in-depth knowledge of it Figure 1 shows a schematic of the core system. On the • Devices are often fixed on the beamline they are left side, we have the server comprised of various used on. Since beam time is highly valuable, it is of- backend components: ten difficult to get sufficient time to test a new device • A number of devices which are driven through EPICS driver before it is needed in production. This can Input-Output Controllers (IOCs), which expose rele- similarly apply to re-testing if changes/updates are vant values to be read/written to over the network using later made the EPICS Channel Access protocol. Automated testing helps meet all of the above challeng- • The Instrument Control Program (ICP), which deals es. A robust set of tests ensures any unforeseen changes in with the neutron data (collection, processing and writ- behaviour are caught before they are deployed to produc- ing to file) obtained from the Data Acquisition Elec- tion beamlines. Once written, robust tests benchmark any tronics (DAE) newly developed code to forever protect it from regres- • The Block Server, which is a python process that man- sion. Tests can also act as a form of documentation for ages the beamline configuration – i.e. which devices to unknown parts of the system (in addition to other forms control, which sample environment values to log etc. of documentation), as they demonstrate the expected WEPHA090 Content from this work may be used under the terms of the CC BY 3.0 licence ( Software Technology Evolution 1295 17th Int. Conf. on Acc. and Large Exp. Physics Control Systems ICALEPCS2019, New York, NY, USA JACoW Publishing ISBN: 978-3-95450-209-7 ISSN: 2226-0358 doi:10.18429/JACoW-ICALEPCS2019-WEPHA090 On the right hand side, we have various frontend clients set of tests. As such, out of the components above, the that communicate with the server through which users can GUI client, Genie_Python, web dashboard and blockserv- interact with the system: er components all come with substantial unit test suites. Some older parts of the system (such as the ICP) are missing unit tests because they are rarely touched and we generally consider them stable, which means implement- ing unit tests has not become a sufficiently high priority. In the meantime, we use static code analysis tools to keep track of test coverage so that we remain aware of such gaps. For IOCs, we do not have what would strictly be con- sidered unit tests, since they require at least a device to test against. Devices can be arbitrarily complicated, thus mocking their behaviour is a bit more involved. Figure 2: The IBEX GUI. DEVICE TESTING • The IBEX GUI – this is an Eclipse/RCP desktop appli- A large part of the work we do is writing IOC drivers cation based on the CS Studio framework [5], and rep- for specific devices used on the beamlines. EPICS pro- resents the primary form of interaction (shown in Fig. vides meta-languages which make it easy to implement 2) device drivers, but these are bespoke and do not come • Genie_Python [6], a library of python commands used with any testing frameworks. Thus, we have developed an to interact with the control system via a command line in-house IOC testing framework which allows you to interface or through running user-written scripts write tests in python that test the behaviour of IOCs against the devices with which they communicate. • The web dashboard, a basic web interface, which pro- Devices are often fixed equipment, meaning we cannot vides a read-only view of the beamline status. confirm our drivers behave correctly without having ac- Note that there are many other services and features that cess to the beamline, which is not always easy or indeed are not included in this selection, for example those that possible. With smaller devices, we are sometimes able to manage device alarms or logging. However, this repre- test against the device in the office, but even so, we usual- sentative set of core components is useful for illustrating ly only have these available for a limited time and there- our use of various different forms of automated testing. fore they are unsuitable for use in a continually running UNIT TESTING automated test suite. Because of this, we simulate the behaviour on the de- Unit tests are a standard software engineering tool most vice side, which we can do at two different levels: The 2019).developers Any distribution of this work must maintain attribution to the author(s), title of the work, publisher, and DOI. should already be familiar with. They are used © EPICS record level, and the device level. to test the system “bottom up” by testing the behaviour of small pieces of code (e.g. individual methods) in isola- Record Simulation tion, and should complete in a matter of seconds or less. Record simulation is a concept that is built into the EP- Each unit test should be strongly focused and mock (sim- ICS framework [9]. Every one of our IOCs has a “simu- ulate) functionality outside of that which is being tested, late” field, which, when enabled, will use a set of aliased e.g. a test for a method parsing the content of a text file dummy simulation database records instead of the real should never fail on account of a file system error. ones. These simulated records bypass communication Most of the languages used in our code base come with with a real device and instead use purely virtual values unit testing frameworks (e.g. JUnit in Java, unittest in held in the IOC itself. While it is possible to link simula- Python). We are conscientious that the code we write is tion records logically (e.g. when setting a setpoint on a structured in a way that permits comprehensive testing. parameter, automatically update its readback value, too), We do this for example by using test driven development the EPICS database language is too restrictive to effec- [7], a coding approach where one first writes tests, and tively simulate complex device behaviour.