computer programs reductus: a stateless Python data reduction service with a browser front end ISSN 1600-5767 Brian Maranville,* William Ratcliff II and Paul Kienzle NIST Center for Neutron Research, 100 Bureau Drive, Gaithersburg, MD 20899, USA. *Correspondence e-mail: [email protected] Received 24 April 2018 Accepted 23 August 2018 The online data reduction service reductus transforms measurements in experimental science from laboratory coordinates into physically meaningful quantities with accurate estimation of uncertainties from instrumental settings Edited by Th. Proffen, Oak Ridge National and properties. This reduction process is based on a few well known Laboratory, USA transformations, but flexibility in the application of the transforms and algorithms supports flexibility in experiment design, enabling a broader range Keywords: neutron reflectometry; X-ray reflectometry; data reduction; computer of measurements than a rigid reduction scheme for data. The user interface programs. allows easy construction of arbitrary pipelines from well known data transforms using a visual data flow diagram. Source data are drawn from a networked, open data repository. The Python back end uses intelligent caching to store intermediate results of calculations for a highly responsive user experience. The reference implementation allows immediate reduction of measurements as they are recorded for the three neutron reflectometry instruments at the NIST Center for Neutron Research, without the need for visiting scientists to install additional software on their own computers. 1. Motivation The transformation of raw measurements into meaningful interpretable data with attached uncertainties (data reduc- tion) is a ubiquitous task in the experimental sciences. In the case where the workflow is well established and the commu- nity is small the most direct way to accomplish this is to develop a custom application to be installed on a limited number of dedicated computers. However, at a scientific user facility a large number of visiting researchers are using the measurement tools on a part-time basis. Thus, it is necessary to make reduction capabilities widely available and flexible. In this case a web-based application is an attractive alternative to distributing dedicated installable executables. The main benefit of a web application is the almost universal accessibility to potential users. On top of this, a centralized reduction server also benefits from the ability to update the calculation code at any time without requiring all users to update their software, as well as largely eliminating the non-trivial time cost of maintaining an installable appli- cation for more than one target platform. Users can also install the service locally, much as they would if reduction were a desktop application. This allows them to access their own data and develop reduction applications for their own instrumentation. The specific implementation described herein, reductus, was developed to provide reduction for reflectometry instruments, but the system was designed to be extensible and usable for other data processing problems and work is underway to support reduction for off-specular reflectometry, small-angle 1500 https://doi.org/10.1107/S1600576718011974 J. Appl. Cryst. (2018). 51, 1500–1506 computer programs neutron scattering and triple-axis spectrometry. The approach subtract the ‘background’ signal at the corresponding incident can be adapted to any type of computational problem, but the angle from the measured reflected intensity, and then divide level of flexibility and computational complexity supported is the subtracted value by the ‘direct-beam’ measurement with driven by the needs of a data reduction engine. the same collimation. The incident angles and wavelengths of the measurements are converted to more meaningful reci- procal space Qz. The generation of error bars for the counts is 1.1. Data reduction for reflectometry based on the fact that these are drawn from a Poisson distri- Neutron reflectometry is a technique for characterizing bution, and instrumental angular and energy resolutions are surfaces, thin films and multilayers by analyzing the intensity calculated on the basis of the collimation and optics (mono- of a reflected signal relative to the incident beam. This fraction chromator) of the measurements. of number reflected/number incident is called the reflectivity, Within the application, the process is treated as a series of and it can be modeled and interpreted by using, for example, data transformations, with data flowing from one transfor- an optical matrix formalism analogous to the one used in the mation module into the next. Corrections are chained together analysis of electromagnetic reflectivity (Born & Wolf, 1959; in a data flow diagram (Sutherland, 1966), with the final data Abele`s, 1950), but where the relevant potentials arise from set saved as the result of the reduction. This yields the interactions between the neutron and nucleus, or the neutron and the internal magnetic field of the sample, rather than electromagnetism. The theory provides a prediction for the reflectivity as a function of momentum transfer (difference of momentum for incident and reflected photon or neutron, in this case). As an illustrative example, we will describe just one of the many types of reflectometry instrument setups: one with a monochromatic beam and with no polarization analysis. In order to measure this reflectivity, an apparatus is constructed such that a collimated beam (of known energy, and known angular and energetic resolution) is made incident on a planar sample, and a neutron detector is placed after the sample to record the number of neutrons that are specularly reflected (where the angle of incidence equals the angle of reflectance) in a given time. This measurement is repeated for a number of incident and reflected angles, which correspond to a range of momentum transfers, so that the results can be compared with the prediction from theory as described above. In order to calculate the reflectivity, though, we need to divide the number of reflected neutrons at each momentum transfer by the number of incident neutrons, and so we also have to measure the ‘direct-beam’ intensity separately, with no sample in the beam, with the instrument collimation set up exactly as it is for the reflection measurement. In our case, the collima- tion is controlled by the opening of two slits in the beam path before the sample position, so setting the direct beam to the same collimation just requires opening those two slits to the same position they were in during the reflected-beam measurement. We also wish to subtract spurious detection events, such as incoherent scattering from the sample holder, so we perform another set of experiments where the sample is in the beam but the detector is not placed at the specular reflection position (the angle of reflection is intentionally set to be different from the angle of incidence). These measure- Figure 1 ments we call the ‘background’. Often two background (a) Measured signal and background counts. Each color represents an independent data file from our data acquisition system, plotted against measurements are taken, where the angle is offset from the the instrument motor that controls the incident beam angle, which in this specular condition in either the positive or negative direction case is ‘sample/angle_x’. The upper curves are the signal and the lower (later labeled bg+ and bgÀ in the text and figures.) curves are the background. (b) Data flow diagram, with the left boxes for Then in the simplest case, in order to extract the physically each node representing the input data and the right boxes representing output. The experimentalist can select the signal, Æbackground and slit meaningful reflectivity (number of neutrons reflected/number normalization for the load nodes, producing the reduced data as the of neutrons incident for a given momentum transfer) we first output from the divide node. (c) Reduced data in physical coordinates. J. Appl. Cryst. (2018). 51, 1500–1506 Brian Maranville et al. reductus 1501 computer programs estimated reflectivity versus the momentum transfer, Qz (see Once connected to the server, the web client requests a Fig. 1). listing of the public data stores, the available reduction steps Users must first identify the data files that will be read as the and a set of predefined template diagrams representing the inputs of the ‘load’ modules at the left of the diagram; then usual reduction procedures for the data. The menus and further individual modules may have control parameters, such default options are populated on the basis of this information. as a scale factor for a scaling module (needed if there is a beam attenuator in one measurement but not in the others, for 2.1. Data flow diagram example). The measured reflectivity is the output of the The user interacts with the data flow diagram in order to ‘divide’ module at the right, in physical coordinates with navigate the reduction chain: by clicking on a module within instrument-specific details removed. the diagram to bring up the parameters panel for that module, Other, more complicated setups are in common use for or by clicking on the input (left) or output (right) terminals to measuring reflectivity, including polychromatic sources display calculation results. Changes made in the parameters (multiple incident energies), polarized-beam components etc., panel are by default immediately applied to the active tem- and correction modules are included for many of these cases plate, though by un-checking the auto-accept parameters so that the user can construct a reduction pipeline that box in the data menu an additional confirmation step matches their experimental setup. (pressing the ‘accept’ button in the panel) can be imposed to avoid accidental changing of the active template. In that context, if a user makes a parameter change but then selects a different module without first pressing ‘accept’, that change is 2. Web interface lost. The user interface for reductus is a JavaScript application that The ‘clear’ button removes all values from the active runs in a browser.