16th Int. Conf. on Accelerator and Large Experimental Control Systems ICALEPCS2017, Barcelona, Spain JACoW Publishing ISBN: 978-3-95450-193-9 doi:10.18429/JACoW-ICALEPCS2017-TUPHA168 IMPROVING THROUGHPUT AND LATENCY OFD-Bus TO MEET THE REQUIREMENTS OF THE FAIR CONTROL SYSTEM Dominic Day, Alexander Hahn, Cesar Prados, Michael Reese, GSI Helmholtz Centre for Heavy Ion Research, Darmstadt, Germany Abstract Timing Network In developing the control system for the FAIR The FAIR timing system uses White Rabbit to accelerator complex we encountered strict latency and distribute high precision timing events over a dedicated throughput constraints on the timely supply of data to Ethernet-based network. The complexities of clock devices controlling ramped magnets. In addition, the synchronization, signal latencies and network topology timing hardware that interfaces to the White Rabbit are abstracted from the users of the timing system. The timing network may be shared by multiple processes on a high-level applications interact with the Data Master, single front-end computer. This paper describes the which maintains a schedule of events and distributes them interprocess communication and resource-sharing system, to Timing Receivers. Low-level applications interact with and the consequences of using the D-Bus message bus. the Timing Receivers located close to the equipment. Then our experience of improving latency and throughput Equipment that controls a logically related set of performance to meet the realtime requirements of the accelerator components is collected into a Timing Group. control system is discussed. Work is also presented on prioritisation techniques to allow time-critical services to Frontend Controllers share the bus with other components. The standard environment for the FAIR control system is the Scalable Control Unit [3]. It provides an Intel 64-bit INTRODUCTION CPU, Linux Operating System with realtime patches, an The White Rabbit based FAIR Timing System FPGA Timing Receiver connected via PCI-express and developed at GSI [1] provides FPGA-based Timing Wishbone. The timing software is also required to run on Receiver hardware for frontend computers. The SAFTlib other systems with greater processing power for Beam project (Simplified API for Timing) was designed to share Diagnostics, systems interfacing to hardware using a the resources of the Timing Receivers and provide a VME Bus, and systems with USB-connected Timing stable interface that abstracts software clients from the Receivers. The SAFTlib software aims to provide a complexity of the timing system. The design goals are to: standard interface across multiple platforms. • Share the Timing Receiver hardware resources Timing Receivers 2017). Any distribution of this work must maintain attribution to the author(s), title of the work, publisher, and DOI. Unify different underlying hardware. © • The Data Master sends event messages over the White Prevent applications creating conflicting events • Rabbit network shortly in advance of their planned Isolate applications from failures in other clients • execution time. The Timing Receiver hardware matches Monitor hardware status • events against a set of conditions. The Event-Condition- Interprocess communication between clients and the Action (ECA) unit is responsible for generating actions SAFTlib process (saftd) is via the d-bus shared message from incoming Timing Events. For equipment with hard bus. real-time requirements, hardware actions are used. These This paper describes the hardware and software send signals over a variety of bus interfaces directly to environment in which it is used and experiences in using equipment. An example is the trigger synchronization SAFTlib in a production environment. The primary focus event that is used to start waveform generators. For events is on achieving the throughput and latency necessary to that can tolerate higher latency software actions can be operate the FAIR accelerators whilst maintaining the used. For example, sequence start events signal that a flexibility and compatibility of the original SAFTlib device should be prepared for a new cycle and load data design. for a later hardware trigger, a gap event can signal that software has a period in which it may freely read the FAIR ACCELERATOR ENVIRONMENT status of equipment. The FAIR project will include a complex of Function Generators accelerators and a new control system is under development [2]. The CRYRING low-energy storage ring Central to the performance investigation is the is being used to test and evaluate the control system anticipated load from Function Generator units. These before retrofitting the existing GSI infrastructure and generate arbitrary waveforms from a set of polynomial equipping the new FAIR accelerators. coefficients and after D/A conversion control a variety of magnet power supplies. The polynomials describe the waveform in segments starting as 1ms in length. This TUPHA168 Content from this work may be used under the terms of the CC BY 3.0 licence ( Software Technology Evolution 809 16th Int. Conf. on Accelerator and Large Experimental Control Systems ICALEPCS2017, Barcelona, Spain JACoW Publishing ISBN: 978-3-95450-193-9 doi:10.18429/JACoW-ICALEPCS2017-TUPHA168 representation was chosen over a value sequence or control the Timing Receiver and listen to Events without coordinate representation to reduce the bandwidth disturbing each other. requirements. The requirement to support existing equipment operating via MIL-bus forces the use of a SAFTLIB DESIGN lower bandwidth representation. The interface to the timing receiver is managed by a The polynomial sequence is checked to ensure it will single process – saftd. This process contains the interrupt result in a valid waveform before being sent to the handler and performs all communication to the timing function generator. The function generator hardware has a hardware over the PCI-express bus. limited amount of memory and requires streaming from Clients connect to saftd via the d-bus Interprocess the saftd driver during longer output ramps. The driver Communication (IPC) standard. All clients request must load a sufficient quantity of data into the function conditions: Hardware Conditions for timing event to generator and arm it before the trigger event is received. hardware action and Software Conditions for timing event The goals during initial testing were to supply 800 to software action. Saftd checks requests do not create coefficient sets, 12 Function Generator channels in 25 ms. conflicts between processes and compiles the set of This beam preparation time is acceptable for the conditions to a format that may be sent to the ECA unit. CRYRING tests. Further development will be required to make more use of the streaming mode of operation. The timing receiver uses interrupts to signal incoming Pre-loading the entire segment is however a good test of data. The satfd interrupt handler is responsible for sorting the data delivery system. interrupts from different channels, requesting data and forwarding it to the clients. By having only a single SOFTWARE ENVIRONMENT process receive the interrupts and use the Etherbone interface contention is avoided. (It is still possible for The FAIR accelerators are controlled by custom Java- another process, such as a legacy application, to use the based applications using the LSA (LHC Software Etherbone interface and cause a conflict.) Architecture) framework [4] to manage the settings for individual devices to fulfil the needs of experiments. This Driver and Proxy Client Interface upper layer generates settings for each device and a Saftd provides an object-based interface for clients. Timing Schedule that specifies to the Data Master when Objects exist in the saftd process and can be accessed via settings are to be applied. Applications may also receive d-bus IPC. Clients create Proxy objects that manage the feedback by polling or subscribing to devices. details of communication with saftd. The interfaces are Software on the front-end computers is developed using specified in XML format similar to the D-Bus reference the Front End Support Architecture (FESA) – a implementation but specialised to the expected use cases. framework developed in collaboration with CERN [5] for A code generation tool generates C++ code for both the 2017). Anythe distribution of this work must maintain attribution todevelopment the author(s), title of the work, publisher, and DOI. of C++ software. FESA software is driver service and proxy side hiding the complexities of © developed to a set of guidelines that provide a standard interacting with the d-bus argument marshalling. interface that applications use to control and monitor Drivers are part of the saftd process. Driver developers various types of hardware. An executable FESA binary must complete the implementation behaviour defined by contains instances of a FESA class representing specific the interface definition and adapt to specific hardware. devices and exposing settings and acquisition values to applications. The details of which computer the software Client applications link to a library containing Proxy is running on and how many devices that computer is classes that handle all interactions on the message bus. responsible for is largely hidden. Clients can call methods, register for callbacks and read properties. Properties are cached locally by the proxy; A typical FESA application might have: changes to a property are sent on the bus to all registered