Testing and Deployment Strategies - Their Impact on Accelerator Performance

TESTING AND DEPLOYMENT STRATEGIES - THEIR IMPACT ON ACCELERATOR PERFORMANCE

J-C. Garnier, CERN, Geneva, Switzerland

Abstract REVIEW OF TESTAND DEPLOYMENT IMPACT ON LHC PERFORMANCE The Accelerator Control system is developedby the Controls Group in collaborationwith numerous software Tracking Tools developers from equipment groups toprovide equipment The accelerator control systemrelies on a multitude of experts and operation crews with general and specific con- tools to track events, faults and actions: logbook,AFT [3],[3], trol and supervision software. Thispresentation proposes central tracing service, etc. Coordination initiatives com- a review of the different strategies applied for testing,val- plete this list: Exploitation Meetings and SmoothUpgrade idation and deployment of the software products. Therisk Working Group (SUWG). and impact of these strategies on the LHC [1] performance The mostvisible and exploitabletool is the logbook and availability willbe studied, andpossible evolution to where every step of the accelerator operations are recorded. continuously improve this performance will be discussed. Entries canbe added manuallyby operators or experts, or automaticallyby software tools such asthe LHC Se- INTRODUCTION quencer. The logbookwill typicallyrecord every software deployment that was announced to the operators. The fact This paper intend to present areview of common soft- that these recordings arewritten by hand, and only when ware engineering activities which are testing and deploy- announced,proves the analysis tobe difficult as some de- ments, and how it impacted theperformance of the LHC in ployments are indeed notreferenced, and thereferenced 2016. deployments arereported invery differentways. Nonethe- Software testing is a way toprove that the software is less, if a quantitative evaluationwas impossible for 2016, a performing in the way it is required by the stakeholders. It qualitative evaluation hasbeen performed and one can see is also a way to inform both stakeholders and other devel- that deploymentshave had a negative impact on theperfor- opers of thebehavior and the quality of the software. mance of the LHC, and multiple hours of operations were Software deployment is the process of ultimately using compromised after software deployments. Therewere dif- the software in aproduction environment. This process can ferent causes: consist of few or many steps, depending on the choices of 0 lack of tests andvalidation before the deploymentto the software engineers andusers of the software. operation As any software engineering activity, numerous ways toperform them have been advertised, demonstrated, ac- I validation could onlybe performed with beams cepted or deniedby the software engineering community. o impact on dependent systemswas underestimated Thispaper focuses on Java, C++ and Python technolo- gies and on high level software, ranging from Front-End This last point is very significant,cant, as it is frequentlymen- Controller software (e.g. FESA [2]) to User Interfaces, tioned, not only during deployment and integration in op- Services and Analysis Tools. Thispapers references mul- eration, but also during the developmentphases when the tiple times to controls software or controls environment, contribution of other software teams is oftenunderesti- which means the software and infrastructureproduced by mated. the Controls Group and the Equipment Groups together. Correlating data fromthe logbookwith other tools This study was done following a top-down approach, proved to be difficultcult and extremely time consuming. A where the ultimate upper goal is the performance of theAc- deployment tracking system is clearlyrequired in orderto celerators. From the estimated impact on theperformance, supervise this activity and allowthe collection ofperfor- one could identify issues in the deployment strategies and mance metrics. It couldhelp identifying all deployments: the testing strategies. major deployments, bug fixesxes and rollbacks. The firstrst section describes the investigation performed to review the impact of control software on LHC perfor- Coordination ofofdeployments deployments mance. The second section describes the testing strategies Another source of useful information was the Smooth applied during development, release and deployment of the Upgrade Working Group. Itreviews and coordinates the control software. The third section describes the deploy- deployment of softwareproduct for each technical stop. ment strategies that areused for software in the control It evaluates therisk for operation, howthe software can system. The fourth and last section gives an outlook of be validated, and if it canbe rolledback. In addition it possible directions for the future. tries to identify the outgoing and incoming dependencies

167 of the software. In spite of all the dedication and expertise unit testable,but all software canbe written in a testable of its members, it is difficult if not impossible to oversee way. The ability totest the source code of an application all dependencies andprevent all issues, and it keepstrack alsohighly depends on the thirdparty software components of them as they raise. The amount of upgrades that are on which the application relies. In the controls environ- reported to the SUWG is very important. Thisbrings diffi- ment, different strategies havebeen appliedthrough time, culties in identifying therisk of all of them while itrelies and depending on the interests ofthe providers of these on manual analysis. The SUWG could take advantage of third party tools. Itresults in a discrepancy of the ability some software support, like a dependency tracking system to test controls software. For instance the FESA framework that wouldreport that a software A isused by software B makes it complicatedtocomplicated to write unit or integrationtests as it is and C, and that an upgrade of SoftwareA needs tobe re- very difficult to isolate the logicunder test from the FESA viewed with software B and C developers. framework. This is a common issuewith frameworksthat From this investigation, one can see that some software force theuser code to be coupledwith them. Onthe other is required to better support the deployment of software hand, whenusing a library like CMW,JAPC, Post Mortem, product in the LHC environment. Software could: it is rather simpleto add an abstractionbetween the source code of the application and the thirdparty library,which 0 helpplanning the deployments,by providing an accu- in turn makes the application source code testablewithout rate and exhaustiveunderstanding of the controls en- depending on thethird party library. vironment and the dependencies on the deployed soft- In amachine like the LHC,where the objectives are am- ware. bitious and themachine protection risks significant,cant, it is o collect metrics about deployments, improving the important to consider that core componentsused by many identification of shortcomings induced by this activ- applications should not impair thetestability ofthese appli- ity. cations. Software testing is supportedby metrics that helps un- REVIEW OF TESTING STRATEGIES derstanding thepresence and quality of the tests. Twovery Testing is a wide term and numerous type of software important metrics are the coverage of lines of code and testing can be identified.ed. The following ones are considered ofbranches. A line of code is covered if it is executed in this paper: onceby a test. Branch coverage highlights the factthat all conditions of abranch (e.g. if statement) are tested. The 0 Unit tests that focus on an isolated unit of code, typi- controls environmentregisters 654 Javaprojects. 365 of cally a function or a class. these projects have less than 50% of line of code coverage. This means that more than half ofthe softwarethat runs 0 Integration tests that validate the interactions ofmul- the controls environment is not tested. These metrics only tiple components together. concerns Javabecause itwas simple to integrate the entire 0 User Acceptance Tests to con firm that the user speci- controls software stack thanks to an homogeneous develop- fications are fulfilled.lled. ment strategy. C++ and Python remainnot monitored at the moment, except on the initiative ofthe developers. These 0 Testing on a Staging environment, which is ideally as metrics are exposedpublicly at CERNvia SonarQube [4]. close as possible to the production environment. Exposing these metricsto users of theproduct could en- courage the developerstodevelopers to provide high quality software and 0 Final Validation in production, the time devoted to this improvethe user confidencedence in theproduct. last validation is reduced if other testing strategies are While unit and integration testing are very good tools applied. While unit and integration testing are very good tools to validate the waythe source codebehaves, user accep- If writing tests has an initial costs, it alsobrings numer- tance tests are of tremendous significancecance tohighlight that ous advantages to the product,hence a valuable return on the software is acting theway it should. Combinedwith investment, particularly on long termprojects like in the automation, they guarantee that a feature stays available controls environment. They can be seen as documentation for the lifetime of the softwareproduct. These testsusu- for a function, a class or a feature. They ease the long term allyneed to run in a staging environment in orderto be maintenance of the software applications as they help en- effective. They however oftenmust rely on thirdparty sys- gineers new to theproject understand the source code and tems that arenot available in the staging environment, for prove that it behaves as expected, and make them confident instancebeams. It is usuallypossible to compensatethis in refactoring - e.g. modifying the existing code - when with simulated ormock components.A successful design fixing bugs or adding new features. The automation and ofuser acceptance tests in a staging environment is for in- repeatability of the tests also helpsunderstanding the im- stance the Orbit Feedback system,which relies on auto- pact of any modificationcation in the source code or in a third mated tests performed without beams. This testing strategy party dependency on the applicationbehavior. helped in understanding the legacy software system, and Unit testing and integration testing are two kind of tests reduced the testing time in operation withbeams, thatre- that mustbe considered at design time. All software is not quiresboth operators and experts tobe thereto validate

168 the software. Many initiatives exist to provide test environ- rollback to theprevious validversion ofthe applicationby ments, for CMW, Timing and FESA, or for FGCs. Machine modifying the link they use, either developers can modify Protection is currently implementing aproject of a testbed the PRO linkto the previous stable version. for magnetprotection and interlock equipment. Theparticularity ofthe deployment area is that it ismu- At the moment, controls and equipment groups usually table, and modificationscations of the deployedproduction GUIs all have their own staging environment,which is usually are sometimes performed inparticular situationswhere a limited to their area of concern and rarely providespos- core service or library needs all its clients tobe updated. sibilities of integration with systems from other equipment The danger in this operation is thatproduction GUIs are groups. Based on the experience gathered during numerous not tested after this modificationcation and it could leadto unex- hardware commissioning, and foreseeing the work neces- pected behaviors. sary to further automatize the machine commissioning and Some alternativeprocesses are implementedby other beam checkout, the MPE group is currentlyworking on a equipment groups.For instance,BI is relying on a single staging environment thatwill enableperforming integra- and common classpath for all their applications. It guar- tion testing on entirepowering systems, comprising FGCs, antees the compatibility of their applicationbetween each QPS, interlocks systems and controls software. others,while all applications aremodi fieded if newreleases The controls environment currently brings some limi- of core components areperformed. Inthis environment, tations to theproper implementation of staging environ- a strongvalidation step isrequired to ensurevalidity of ments: there are too many dependencies on the Technical the applications. Another example of alternativeprocess is Network infrastructure, where the operational systems are used in MPE,where the GUI deploymentprocess is com- also running. A proper staging environmentwould require posed of three steps: a release, an automatedvalidation, a correct separation between itself and the operational en- and an immutable deployment. Thisprocess guarantees vironment.Numerous attemptshad been performed in this that the application isvalidated by automateduser accep- directionbut they were always unsuccessful. The Controls tance tests before it can beused in operation, and that the Groupunder the CO3 impulse is currentlyreviewing their applicationused in operationwill neverbe modified.ed.Any numerous project andwill come back with recommenda- modificationcation even a simplemodi ficationcation of dependencies tions on providing core services to staging environment out must go through the automatedvalidation step. of the TechnicalNetwork. The more time is spent onunit testing, integrationtest- Services ing, user acceptance testing and staging, the less time is re- Java servers deployments in the controls environment quired for validation in theproduction environment. There consists of two steps: arelease, and a deployment on the is of course a lack of metrics illustrating this in our environ- server thathosts the service. In order to deploy, the de- ment. Such metric couldbe provided by a deployment ser- velopers authenticate tothe operational server and elevate vice, that would help comparing the time required to vali- their privileges to run some deployment commands and date software in operation with the software code coverage. start-up commands.With their elevatedprivileges, the developers can unintentionallyperform actionsthat might im- REVIEW OF DEPLOYMENT pact other services, there isno protection against that at the STRATEGIES moment. After deployment,previous versions of the service are kept tobe used forroll back. The fact of deploying an application consist of following Rollingback a service doesn’t rely on commands,but a process. Two types of applications must be considered on filele system and symbolic linkmanipulations. There is here: Graphical User Interfaces (GUIs) and services. Both currently no consistent solution offered formulti-layer ap- do not follow the same deployment strategies. Services can plications that consist of server, database and GUI. It can thenbe split into two categories: FECs with e.g. FESA, and be a challenge to rollback allthese at once. Java services. Here MPE and BE-CO-DS areusing Continuous Inte- gration and Continuous Delivery in order to support fast GUIsGUIS delivery of features totheir services. These implementa- In the controls environment, deploying a GUI consists of tions rely on scripts that automatethe deploymentbased on one step: arelease. A releasewill deliver the GUIproduct the authentication andprivilege elevation aforementioned. to a shared file systemwith its version number. The de- It removes the risks that ahuman does not interact anymore liveredproduct is a JNLP (JavaNetwork Launch Protocol) directly withremote machine,but theunderlying mecha- that can be started anywhere provided that a Java Runtime nism is still the same and can stillbe errorprone. Environment is available. In addition, a link must existbetween the GeneralPur- A PRO release will update a symbolic link that is usually pose Network and the TechnicalNetwork in order for de- referred by numeroususers, as it is the link to the latest velopers to deploytheirdeploy their services. A clean deployment strat- production ready GUI. When a PRO release isperformed, egy couldprovide capabilities to developersto release and the previous PRO release is aliased with the PREVIOUS deploy their applications without havingto act on them- alias. This simple mechanism ensures that eitherusers can selves on the operational infrastructure. Thiswould help

169 reducing the risks of human errors. that allow them tovalidate their software in a closed stag- Deploying a FESA class is yet anotherbut very similarpro- ing environment. Larger testbeds canbe implementedto cedure. The difference mainly consists in the fact that the validate the integration of software services together. This API of the FESA class is exposed in a database. paper illustrates that theways deployment areperformed In the current environment, there is norestrictions on are not optimal and comport some risks, andthat a techni- the deployment of operational software, whether GUIs or cal solution to improve this could actuallyprovide valuable servers. A deployment canbe performed at anytime with- metrics in order tobetter understand theperformance of out any constraints coming from thebeam presence or op- software deployments. erationplanning. Indeed responsiblepeople are aware of these constraints and coordinate asbest as possiblewith ACKNOWLEDGMENT their interested colleagues for a deployment. The risk of I would liketo express my gratitudeto Kajetan Fuchs- deploying anytime and the entanglementbetween develop- berger and Grzegorz Kruk for allowing me towork onthis ment environment and operational environment couldbe vast, motivating and challenging topic. avoided with an integrated deployment tool. REFERENCES Deployment Service [1][l] LHC Study Group et al., “The Large Hadron Collider, Thispaper identiﬁed the followingpossible areas of im- Conceptual Design”, CERN, Geneva, Switzerland, Rep. provements in the testing and deployment of softwarepack- CERN/AC/95-05, Oct.2012. ages: [2][2] M. Arruat et al., “Front-End Software Architecture”, ICALEPCS07,ICALEPCSO7, Knoxville, Tennessee, USA, 2017. . Testing out of theproduction environment in order to minimize the validation time inproduction [3][3] A.Apollonio et al., “LHC Accelerator Fault Tracker - First Experience”, IPAC2016,Busan, Korea, May 2016. 0 Simple and safe deployments and rollbacks that canbe [4][4] http://sonar.cern.ch performed by Operation Crewswhen theyjudge that there is a time window to do so

0 Tracking of the time spent tovalidate a softwareproduct in production,until it is accepted or rejected

0 Proper separation of the development environment from the operational environment

0 Tracking of dependenciesbetween services and API versions

Such a deployment service is an investment thatwould make the software deployment step safer and more trans- parent. It cannot be built andprovided at once,but should rather grow step by step. A ﬁrst milestone would simply be the tracking of the deployed versions and the incoming versions that operators could deploy and validate. A valuable extension that could come afterward is the tracking of the dependencies between the software services, and their versions. Numerous open source tools can supportparts of such a deployment service.

OUTLOOK Thispaper has explained that it could notrely on any quantitative metrics of the impact of testing and deployment strategies of control software on the acceleratorper- formance. This illustrates however a scarcity in the controls environment that canbe fulﬁlled.lled. This paper has illustrated that most of the Java software stackremains uncovered by unit and integration tests, and that such metrics are missing for otherprogramming languages. It showshowever that controls and equipment groups are equippedwith testbeds

170