Boutiques: a exible framework for automated application integration in computing platforms
Tristan Glatard1, Gregory Kiar2,3, Tristan Aumentado-Armstrong2,3, Natacha Beck2,3, Pierre Bellec4, Rémi Bernard2,3, Axel Bonnet5, Sorina Camarasu-Pop5, Frédéric Cervenansky5, Samir Das2,3, Rafael Ferreira da Silva6, Guillaume Flandin7, Pascal Girard5, Krzysztof J. Gorgolewski8, Charles R.G. Guttmann9, Valérie Hayot-Sasson1, Pierre-Olivier Quirion4, Pierre Rioux2,3, Marc-Étienne Rousseau10 and Alan C. Evans2,3
1Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada and 2McGill University, Montreal, Canada and 3Montreal Neurological Institute, Montreal, Canada and 4Centre de Recherche de l’Institut de Gériatrie de Montréal CRIUGM, Montréal, QC, Canada and 5University of Lyon, CNRS, INSERM, CREATIS, Villeurbanne, France and 6University of Southern California, Information Sciences Institute, Marina del Rey, CA, USA and 7Wellcome Trust Centre for Neuroimaging, London, UK and 8Department of Psychology, Stanford University, Stanford, California, USA and 9Center for Neurological Imaging, Department of Radiology, Brigham and Women’s Hospital, Boston, Massachusetts, USA and 10Compute Canada
Abstract container systems such as Docker1 and Singularity [3] have emerged to facilitate the sharing and migration of software by We present Boutiques, a system to automatically publish, de ning immutable, reusable execution environments. integrate and execute applications across computational We present Boutiques, a system to publish, integrate and platforms. Boutiques applications are installed through execute command-line applications across platforms (see Fig- software containers described in a rich and exible JSON ure1). In Boutiques, a command line is described using a exi- language. A set of core tools facilitate the construction, val- ble template comprising the inputs it requires and the outputs idation, import, execution, and publishing of applications. it produces. Inputs may be passed directly on the command Boutiques is currently supported by several distinct virtual line or through con guration les. They may also be inter- research platforms, and it has been used to describe dozens dependent, for instance mutually exclusive. Such formal de- of applications in the neuroinformatics domain. We expect scriptions, simply referred to as descriptors, link to a container
arXiv:1711.09713v1 [cs.SE] 8 Nov 2017 Boutiques to improve the quality of application integration image where the given application is installed. Boutiques de- in computational platforms, to reduce redundancy of e ort, scriptors allow for automatic application integration in plat- to contribute to computational reproducibility, and to foster forms, and advanced validation of input values to prevent er- Open Science. rors. Boutiques descriptors are intended to be produced by sci- enti c application developers, stored alongside their applica- Key words: Application integration; Containers; Neuroin- tion, indexed by common repositories, and consumed by exe- formatics. cution platforms. A set of core tools facilitate the construction, validation, import, execution, and publishing of Boutiques de- scriptors. Introduction The remainder of this paper describes the Boutiques system and reports on its adoption by platforms and applications in Computational platforms such as web services, portals, scien- the neuroinformatics domain, our primary eld of interest. It ti c gateways, work ow engines and virtual research environ- closes on a discussion and comparison with related systems. ments commonly integrate third-party applications to enable various types of data processing. Applications, however, are often manually and repeatedly integrated whereas automating and sharing this e ort would improve computational repro- ducibility [1,2] and contribute to Open Science. Meanwhile, 1 https://www.docker.com
1 2 |
Developer value key, a ag and ag separator, and a default value. Inputs may also be ordered lists: in this case, value keys are substi-
1. Create & publish tuted by the space-separated list of input values. tool with descriptor Scientist String Number Flag File File Boutiques Tool Container Types. Inputs may be of type , , or . Descriptor may also represent a directory. Types can be restricted to a speci c set of values or to a speci c range. Data 2. Configure data & analysis Repository Container Index Platform Groups and dependencies. Groups of inputs may be de ned with an identi er, name, and list of input identi ers. Groups may be used to improve the presentation in a graphical user inter- face, and to specify the following constraints among inputs: mutually-exclusive 3. Launch analysis (1) : only one member in the group may have a value; (2) one-is-required: at least one member in the Execution Platform group must have a value; (3) all-or-none: if any of the mem- bers have a value then all members must have a value. De- pendencies among inputs may also be de ned regardless of a
Local HPC Cloud particular group: an input may (1) require a list of inputs and (2) disable a list of inputs.
Figure 1. Publication, integration and execution of applications with Boutiques. Listing1 shows the de nition of an input in the command line exempli ed above. According to this de nition and assum- ing that the input value entered by the user is 0.3, the string System description [NUMBER_INPUT] will be replaced by -n=0.3 on the command line.
In Boutiques, applications are described with a JSON descriptor that speci es the command-line template, inputs and outputs. The descriptor may point to a container where the application { and all its dependencies are installed. It may also contain an "id": "num_input", invocation schema used for input validation (this will be cre- "name": "A number input", ated at runtime if it is not found). At runtime, the execution "type": "Number", platform builds the command line from the descriptor and the "value-key": "[NUMBER_INPUT]", values entered by the user. The platform runs the command "optional": true, line on the execution infrastructure, e.g., a server, a cluster or "command-line-flag": "-n", a cloud, within a container whenever available. To build and "command-line-separator": "=", run the command line, the platform may rely on the Boutiques "minimum":0, core tools, in particular the validator and executor, packaged "maximum":1, through the bosh command-line utility. "exclusive-minimum": true, "exclusive-maximum": false } Command-line description
The core component of the descriptor is a command-line tem- plate complying to the syntax of the bash UNIX shell, the de- Listing 1. Example of a Number-type input. fault shell on most of the Linux distributions and on OS X. The command-line template is a single string that may contain placeholders for input and output values, called value keys. It may also encompass several commands separated by bash con- | & structs such as semicolons, pipes ( ) or ampersands ( ), to fa- Output description cilitate the embedding of basic operations on the command line, for instance directory creation, input decompression or output Application outputs are les and directories that need to be de- archival. livered to the user once the execution is complete. Outputs Here is an example of a typical command-line template: need to be speci ed so that computing platforms can identify exampleTool_1 [CONFIG_FILE] [STRING_INPUT] [FILE_INPUT] | \ the les that must be saved after the execution and raise errors exampleTool_2 [FLAG_INPUT] [NUMBER_INPUT] >> [LOG].txt if they are not present. In Boutiques, output les must have a unique identi er, a The template contains ve value keys, identi ed by square name, and a path template that speci es the le or directory brackets, that will be replaced by values and le names accord- name. Path templates may include input value keys in case out- ing to the user input when the application is executed. Flags put les are named after the input values. In this case, input will also be added wherever appropriate, with customizable values may be stripped from speci c strings, e.g., le exten- separators. Value keys have to be unique but do not have to sions, before being substituted in the path template. Output comply to any particular syntax. Note the use of the | opera- les may also have a description, a command-line ag, a ag tor to chain applications, and of the » operator to redirect the separator, and a value key in case they appear on the command standard output to a le. line. They may be optional in case the le is not always pro- duced by the application, for instance when it is produced only Input description when a particular ag is activated. They may also be lists: in this case, the path template must contain a wildcard (*) match- General properties. Inputs must have a name, a unique identi- ing any string of characters and de ning the pattern used to er, and a type. They may be optional, have a description, a match the output les in the list. | 3
Listing2 shows the de nition of an output le in the com- Command-line construction mand line exempli ed before. According to this de nition and assuming that the string input value entered by the user is At runtime, a value is assigned to all the mandatory and some of foo.csv, the string [LOG] will be replaced by log-foo on the com- the optional inputs. Algorithm1 shows how the command line mand line. is constructed from the descriptor and the values entered. It substitutes all the value keys in the command line, output path templates and con guration les, and writes the con guration { les. "id": "logfile", "name": "Log file", Algorithm 1 Command-line construction "description": "The output log file", "path-template": "log-[STRING_INPUT]", # Substitute input value keys in output path templates, con gura- "value-key": "[LOG]", tion les and command line. "path-template-stripped-extensions":[".txt", ".csv"], for input in inputs do "optional": false if input has a value-key then } for output in outputs do stripped_value = input_value if input type is File then In stripped_value, remove all elements in Listing 2. Example of an output leveraging path-template path-template-stripped-extensions. search-and-replacement. end if In path-template, replace all occurrences of value-key by stripped_value. if output has a file-template then In any line of file-template, replace all occur- Con guration les rences of value-key by stripped_value. end if A large number of applications rely on con guration les rather end for than command-line options to de ne their input and output Prepend command-line-flag and parameters. As the number of parameters increases, command command-line-flag-separator to input_value. lines rapidly become long and cumbersome whereas con gura- In command-line, replace all occurrences of value-key by tion les allow for better structure and documentation. input_value. end if Con guration les may be complex though, and speci ed end for in any language. For this reason, Boutiques allows application # Substitute output value keys in con guration les and command developers to specify their own template containing input and line. output value keys. Con guration les are speci c types of out- for output in outputs do put les that must have a le template that de nes how they if output has a value-key then will be named and where they will be written. They may also for output in outputs do have a value key and a ag in case they need to be passed on if output has a file-template then the command line. Listing3 shows an example. In any line of file-template, replace all occur- rences of value-key by path-template. # Input value-key have been substituted in { path-template previously. "id": "config_file", end if "name": "Configuration file", end for "type": "File", Prepend command-line-flag and "value-key": "[CONFIG_FILE]", command-line-flag-separator to path-template. "path-template": "config.txt", In command-line, replace all occurrences of value-key by "file-template":[ path-template. "# This input is hard-coded", end if "stringInput=foo", end for "# This is an input file", # Write all con guration les. "fileInput=[FILE-INPUT]", for output in outputs do "# And here is the result", if output has a file-template then "fileOutput=[OUTPUT-FILE-NAME]", Write file-template in path-template "" # Value keys have been substituted in file-template previ- ] ously. } end if end for
Listing 3. Example of a con guration input le. The le tem- plate is de ned as an array of strings to allow for multi-line strings in JSON. Invocation schema
Rigorous input validation is an important motivation for Bou- tiques. For this purpose, Boutiques relies on an application- 4 |
speci c JSON schema, called invocation schema, to specify the tainers formats to convert container images across frameworks. input values accepted by an application. Platforms can rely on For instance, a platform dedicated to high-performance com- invocation schemas to validate inputs using any JSON validator, puting may accept descriptors referring to Docker containers to without having to develop speci c code. facilitate application integration by developers, and run these Invocation schemas, however, are complex JSON objects. images on clusters using Singularity. Basically, they must represent the properties described above Container images are de ned from their URL (rootfs) or im- in a formal way, including dependencies between inputs. List- age name in a Docker or Singularity index. Descriptors may ing4 shows an example of how dependencies between mutu- specify a working directory where the application has to be run, ally exclusive parameters are de ned in the invocation schema. they may indicate if the image has an entry point, and they may To relieve application developers from the burden of having to also report a hash to accurately identify container images and write JSON schemas, invocation schemas can be generated au- detect updates. tomatically by the bosh command-line utility. The invocation Containers were adopted because they allow for an auto- schema is stored as an optional property of the Boutiques de- mated and lightweight integration of application implementa- scriptor. tions in platforms. They are extremely useful to improve the re- producibility of analyses as variations in the software environ- ment may have an important impact on the computed results. { They also have limitations, in particular they do not specify "dependencies":{ the hardware architecture required to execute an application, "num_input":{ which can be an issue in some cases. "properties":{ "str_input":{ Resource requirements "not":{} } Boutiques descriptors may contain requirements regarding the } number of CPU cores or nodes, the amount of RAM or disk }, storage, and the total walltime expected for a typical execu- "str_input":{ tion of the application. Such properties are called “suggested "properties":{ resources” as we are well aware that the actual resource re- "num_input":{ quirements usually depend on the input data, parameters and "not":{} hardware infrastructure. } } } Custom properties } } Custom properties may be added to the Boutiques speci cation without restriction. Custom properties are grouped together in a speci c JSON object to facilitate validation. They may be useful to implement platform-speci c features but they should Listing 4. Excerpt from invocation schema showing dependen- be used with care to avoid making applications dependent on a num_input cies between two mutually exclusive parameters and particular platform or replacing existing functionality already str_input . represented in the Boutiques schema.
Core tools Work ow support Boutiques is available on Python through the PyPi package repository as “boutiques”. The Boutiques package exposes a Boutiques does not specify a particular language to build work- command-line utility to the user, bosh, which contains entry- ows from descriptors, due to the large amount of specialized points for all core functions within Boutiques. The core tools frameworks to do so. However, work ows can be both com- provided by Boutiques are, the: validator, executor (for both posed from and described as Boutiques descriptors: work ow launch and simulation), invocation schema handler, importer, engines can leverage the bosh tools to call Boutiques applica- and publisher. The tools exposed through the command-line tions from their descriptors; in turn, work ows can be de- interface are also available consistently through a Python API scribed as Boutiques descriptors. Such a “task encapsulation” by importing the Boutiques package. Though not a component model allows for a scalable and reliable execution of work ows of the Boutiques tool chain, a Jupyter Notebook tutorial exists expressed in a variety of languages, as detailed in [4]. to facilitate new users getting started with Boutiques.
Containers Validator. The Boutiques validator checks conformance of JSON descriptors to the Boutiques schema using a basic JSON valida- Applications may be installed in a container image complying tor. It also performs the following checks that cannot be easily to the Docker, Singularity or rootfs format. We intentionally implemented in JSON schema: value keys are unique among support multiple container formats as we anticipate that they inputs, input and output identi ers are unique, input and out- will be used for di erent purposes. For instance, Docker is well put value keys are all included in the command line, identi ers suited for developers and users who manipulate applications with the same value key are mutually exclusive, value keys are on their local workstations or the cloud. It is well documented, not contained within each other (which would puzzle substi- maintained and it has a rich ecosystem of tools to build and tution), output path templates are unique (to avoid results to run containers on most operating systems. Singularity is more overwrite each other), inputs of type Flag have a command- suited for users and platforms that need to run applications line ag, are optional and are not lists, the default value of on shared computing clusters. Bridges exist among these con- restricted types is part of the restriction, an input cannot both | 5
require and disable another input, required inputs cannot re- processing applications are integrated and over 340,000 pro- quire or disable other parameters, group member identi ers cessing batches have been submitted since 2010. must correspond to existing inputs and cannot appear in dif- Applications in CBRAIN are integrated as Ruby classes that ferent groups, mutually exclusive groups cannot have mem- create web forms, validate parameters and run command lines bers requiring other members, one-is-required groups should on computing resources. Boutiques is supported through a set never have required members, and all-or-none group mem- of templates that generate such classes from the application bers must not be required. descriptor. Two application integration modes are available:
Executor. The executor has two modes of operation: simulate i. The descriptor is stored in a CBRAIN plugin and the Ruby and launch. The simulate mode can generate hypothetical com- classes are generated on-the- y when CBRAIN starts. This mand lines from random values given the descriptor (and corre- mode allows CBRAIN developers to update all Boutiques ap- sponding invocation schema) for debugging purposes, or dis- plications at once by editing the templates. However, it does play the command that would be executed given a provided not allow for customization beyond the Boutiques schema. valid invocation. The launch mode can execute command line To provide more exibility, we added a custom property from a Boutiques descriptor and a set of input values repre- (cbrain:inherits-from-class) to the Boutiques descriptor to sented in JSON le complying with the invocation schema. It de ne the Ruby class that should be used as parent class for runs the command in a container provided that the required the application. framework (e.g. Docker) is installed. The executor can be used ii. Ruby classes are generated from descriptors through an by application users to run applications locally, or by platforms oine process and integrated in CBRAIN as any other appli- to generate command lines to be run on the execution infras- cation. This mode allows developers to customize applica- tructure. tions by editing the generated Ruby classes, but the result- ing applications are di cult to maintain in the long term, in Invocation Schema Handler. The invocation schema handler can particular when the descriptors are updated. create an invocation schema from a Boutiques descriptor and validate input data against it using a regular JSON validator. It We also extended CBRAIN to enable the parallelization of can be used to add invocation schemas to existing descriptors. work ows wrapped as Boutiques descriptors. Applications cbrain:can-submit-new-tasks It is used by the executor if no invocation schema is present in with the custom property may the Boutiques descriptor being deployed. submit sub-tasks by creating Boutiques invocations in their working directory. CBRAIN periodically scans working directo- Importer. The importer takes Boutiques descriptors from older ries, submits the requested sub-tasks, and writes back an in- versions and updates them to be compliant with the most re- vocation identi er in the same directory. This parallelization cent version of the schema. This tool can also create descriptors model is simple, the application only needs to write Boutiques from selected application collections, such as BIDS apps [5]. invocations and communication happens through the le sys- tem, and it is also powerful as it enables the parallelization of Publisher. As Boutiques has primarily been adopted in the neu- complex work ows such as the Niak ones described later. roinformatics community, the publisher gets a further descrip- We also introduced a new list mechanism in CBRAIN to fa- tion (such as author, website, etc.) of the described application, cilitate the iteration of Boutiques applications on large sets of and adds an index to it on NeuroLinks2, a repository containing les. CBRAIN lists are speci c les that contain references to links to neuroinformatics resources and tools. This function- other CBRAIN les. When a list is passed to a Boutiques ap- ality could be extended to new repositories for other domains, plication, the elements in the list are either concatenated in a such as Bioconductor for bioinformatics [6]. single command line (when the corresponding Boutiques input is a list), or a new command line is generated for every element in the list (when the input is not a list). Supporting lists as a Results speci c CBRAIN le type allows for improved validation. For in- stance, lists that contain references to non-existent or deleted Supported platforms les can be detected. It also allows users to edit lists using their own tools such as scripts or spreadsheet applications. The import and execution of Boutiques applications is currently supported in the platforms enumerated below. Nipype Nipype (http://nipype.readthedocs.io/en/latest)[9] is a CBRAIN work ow engine widely used in neuroinformatics. Nipype CBRAIN (http://github.com/aces/cbrain)[7] is a web platform work ows can be composed from Boutiques applications us- 3 to process data distributed into multiple storage locations on ing the Python API. As an example, we implemented NipBIDS , computing clusters and clouds. CBRAIN o ers transparent ac- a Nipype work ow to process BIDS datasets using BIDS apps cess to remote data sources, distributed computing sites, and imported as Boutiques applications. NipBIDS iterates partici- an array of processing and visualization tools within a con- pant analyses on all the subjects found in a BIDS dataset and trolled, secure environment. The CBRAIN service deployed at runs a group analysis if requested. the Montreal Neurological Institute relies on the infrastruc- ture provided by Compute Canada [8]. It currently provides SPINE 500+ collaborators in 22 countries with web access to several SPINE (http://spinevirtuallab.org), which stands for Struc- systems, including 6 clusters of the Compute Canada high- tured Planning and Implementation of New Explorations, is performance computing infrastructure (totaling more than a web-based, collaborative platform (virtual laboratory) de- 100,000 computing cores and 40 PB of disk storage) and Ama- signed to support the design and execution of experiments zon EC2. CBRAIN transiently stores about 10 million les rep- centered on speci c scienti c questions. SPINE enables dis- resenting over 50 TB distributed in 42 servers. 51 public data
3 https://github.com/big-data-lab-team/sim/tree/master/sim/other_wf_ 2 https://brainhack101.github.io/neurolinks examples/nipype 6 |
tributed data collection and management, as well as experi- was integrated in CBRAIN through Boutiques. The integration ment design, execution, and review. Boutiques will serve as uses the CBRAIN sub-tasking mechanism described earlier so SPINE’s algorithm and work ow repository, and enables un- that even the invocations processing a single subject can be equivocal referencing of speci c work ows applied to speci ed parallelized. It also allows CBRAIN to leverage the e cient datasets within an experiment, thereby describing the prove- agent model used in PSOM, as described in [16]. The integra- nance and facilitating the reproducibility of image-derived tion required some work in PSOM to facilitate its invocation measurements. Work ows in SPINE may combine human im- as a non-interactive command-line application. The result- age annotation with automated image processing algorithms. ing CBRAIN plugin is available at https://github.com/SIMEXP/ Future development will focus on extending Boutiques work- cbrain-plugins-psom. Descriptors are on Neurolinks. ow descriptors to include the identi cation and characteriza- tion of human operators and their speci c historic performance GinFizz. We integrated the GinFizz8 Nipype-based fMRI pre- on the required tasks. SPINE is currently hosted at Brigham processing pipeline in VIP with Boutiques. A few technical & Women’s Hospital in Boston, and supports several interna- issues coming from the management of users in Docker con- tional projects. tainers had to be addressed: to enable the execution in Bou- tiques we had to override the permissions of les and folders VIP in the GinFizz container. We also had to install all the GinFizz The Virtual Imaging Platform (VIP) [10] is a web portal for med- pipeline components in a single container while multiple ones ical simulation and image data analysis. VIP makes applica- were used by the application initially. tions available as services, and connects them to the biomed Virtual Organization (VO) in the European Grid Infrastructure4. Di usion imaging The biomed VO interconnects approximately 65 computing MRTrix3. A few applications from the MRtrix3 package [17]9 for sites world-wide and provides access to 130 computing clus- di usion MRI processing were also made available in VIP, as ters and 5 PB of storage. The VIP service is deployed at the well as a pipeline developed at Creatis, which combines MRtrix3 Creatis laboratory5 in Lyon and it uses the DIRAC French na- and FSL applications. tional service6 to execute jobs on EGI grid and cloud resources. As of October 2017, VIP counts more than 1,000 registered users ndmg. The NeuroData MRI to Graphs one-click connectome es- and a growing number of available applications. timation pipeline [18], developed in Python and leveraging FSL, Until recently, applications were manually integrated in VIP was exported to Boutiques and is available at https://github. as work ows written in the Gwendia [11] language and exe- com/neurodata/boutiques-tools. The ndmg pipeline is deployed cuted with the MOTEUR [12] engine. As of today, Boutiques is in CBRAIN via its Boutiques descriptor, and is available both supported through an importer tool that parses the JSON de- through Docker and Singularity container environments. scriptor and automatically generates the corresponding appli- cation work ow and the wrapper script that handles, among Image simulation other things, the execution of the command line. In VIP, ap- CreaPhase. The CreaPhase phase-contrast simulator, developed plication work ows enable (1) iterations on input lists, (2) the at Creatis, was integrated in VIP through Boutiques. The in- generation of parallel tasks, and (3) the concatenation of mul- puts had certain particularities (some needed to be enclosed tiple applications. For example, the applications used in the in simple quotes, other were vectors of variable size enclosed MICCAI challenges described below required work ows to eval- in brackets) that required the post-processing of the wrapper uate results using the metrics de ned by the challenge. Appli- script generated by the Boutiques importer. cation concatenation is handled at the importer level based on pre-de ned work ow templates. ODIN. The Odin MRI simulator [19]10 was integrated in VIP through Boutiques. Since Odin requires important amounts of computing resources it is executed by VIP on the EGI grid that Integrated applications currently does not support Docker. The Docker image was used just for compilation and testing; the Odin executable was ex- Dozens of neuroinformatics applications were integrated in tracted from the Docker image and the Odin wrapper script was CBRAIN or VIP using Boutiques. The main ones are described modi ed accordingly. below. Several Boutiques descriptors were published in Neu- rolinks through the bosh publisher. BIDS apps BIDS apps [5], an e ort for the adoption of the Brain Imaging Anatomical imaging Data Structure (BIDS) in common neuroimaging pipelines, re- FSL. Several MRI analysis applications from the FMRIB Soft- quire a standardized set of input and output parameters. We ware library (FSL [13]) were integrated in CBRAIN using Bou- developed a tool as part of the Boutiques importer to gener- tiques: BET, fsl_anat, FAST and FIRST. Descriptors are on Neu- ate a descriptor for any such BIDS app. We validated this tool rolinks. by importing BIDS apps containing the Statistical Parametric Mapping toolbox (SPM) [20] and the ndmg pipeline mentioned Anima. animaN4BiasCorrection, an ITK-based bias eld correc- above. Descriptors are on Neurolinks. tion application from the Anima7 project was made available in VIP through Boutiques. 2016 MICCAI challenges We used Boutiques to integrate 23 pipelines in the VIP plat- Functional MRI (fMRI) form in the context of two challenges organized by the MICCAI Niak. The Niak fMRI pre-processing pipeline [14], executed conference in 2016, related to the segmentation of multiple- with the Pipeline System for Octave and Matlab (PSOM) [15],
4 https://www.egi.eu 5 https://www.creatis.insa-lyon.fr 8 https://github.com/thomashirsch/ginfizz 6 https://dirac.france-grilles.fr 9 http://www.mrtrix.org/ 7 https://github.com/Inria-Visages/Anima-Public/wiki 10 http://od1n.sourceforge.net | 7
sclerosis lesions in MR images (MSSEG challenge11) and of les and deliver them to the user. tumor volumes in PET images (PETSEG challenge [21]). The pipelines were integrated in VIP and executed on 205 subjects In particular, command-line generation and advanced valida- in a few weeks only. Some pipelines had to be adjusted manu- tion features such as dependencies between inputs are embed- ally once integrated in the platform for the following reasons: ded in bosh, without requiring the platform to support the re- lated descriptor properties. i. A pipeline required a GPU, which we enabled through the nvidia-docker12 tool not supported in Boutiques although it could be a possible extension. { ii. A pipeline required more than 10 GB of data dependen- "name": "echo", cies (atlas data) which exceeded the maximum size allowed "tool-version": "1.0", for Docker containers in our setup. We solved the issue by "description": "A simple script to test output files", installing the data in a directory of the host server that we "command-line": "echo [PARAM] > output.txt", mounted in the container. "schema-version": "0.5", iii. A pipeline wrote more than 10 GB of intermediate data "inputs":[{ in a temporary directory of the container located on a 2 GB "id": "param", partition. We solved the issue by mounting a host directory "name": "Parameter", in the temporary directory. "value-key": "[PARAM]", "type": "Number" Two custom properties (vip:miccai-challenger-email and }], vip:miccai-challenge-team-name) were also added to the Bou- "output-files":[{ tiques descriptor to help post-process results in the speci c "id": "output_file", context of MICCAI challenges. "name": "Output file", "path-template": "output.txt" }] Discussion } With Boutiques, developers can integrate their applications once and execute them in several platforms. Boutiques re- moves the technological dependency to a particular platform Listing 5. A minimal Boutiques descriptor. and facilitates application migration. Although the motivating use cases were taken from neuroinformatics, our primary eld of interest, nothing prevents the system from being used in other domains. Work ow support
Boutiques descriptor Boutiques intentionally does not provide a work ow language to compose applications, as this is already possible with numer- The Boutiques descriptor speci cation allows describing a wide ous work ow engines. Work ows can be either composed from range of applications, but it is also getting increasingly com- Boutiques applications, as we illustrated with the Nipype and plex through additions such as invocation schemas and depen- MOTEUR engines, or described as Boutiques applications, as dencies among inputs. Extending the Boutiques descriptor has we demonstrated with the Niak fMRI pre-processing pipeline. two main goals: (1) validation: incorrect input values and exe- Furthermore, the CBRAIN platform has a sub-tasking mecha- cution results are more precisely detected when the application nism that allows Boutiques applications to submit tasks, which descriptor is comprehensive; (2) automation: a rich descrip- is used to parallelize work ows wrapped as Boutiques applica- tor schema reduces the need for custom application wrappers, tions. Based on the CBRAIN experience, we may specify the which is particularly useful for containerized applications. sub-tasking mechanism in Boutiques so that other platforms Nonetheless, a complex descriptor schema has a cost for ap- can bene t from it. This model is powerful because it shields plication developers and platforms, which we address as fol- the Boutiques speci cation from speci c work ow constructs lows. For developers, we maintain the set of mandatory de- and it allows a wide range of work ow engines to be described scriptor properties as small as possible so that simple applica- and used uniformly. tions can be described in a few lines only (see Listing5). For platforms, we aim at supporting as many features as possible in the bosh executor so that only the following steps need to be Reproducibility implemented in a platform, regardless of the complexity of the descriptor: Boutiques helps computational reproducibility through con- tainers and formal command-line descriptions. With Bou- i. Input entry: generate the interface to enter inputs. tiques, complete sets of applications could be easily migrated ii. Input validation: create a JSON invocation from the inter- across execution platforms, including high-performance com- face, validate it against the invocation schema. puting clusters and individual laptops, to reproduce analyses. iii. Input delivery: transfer the input les to the application The Boutiques descriptor describes the parameters and imple- execution location. mentation of the application, and the invocation schema de- iv. Execution: pass the invocation to the bosh executor to run scribes the parameter values. the application. However, reproducibility is a large problem that Boutiques v. Output delivery: from the descriptor, identify the output only partially addresses. At the command-line execution level, containers help freeze a large fraction of the software ecosys- tem but they do not shield against discrepancies arising from 11 https://portal.fli-iam.irisa.fr/msseg-challenge/overview di erent Linux kernel versions or hardware platforms. For 12 https://github.com/NVIDIA/nvidia-docker instance, containers may not execute consistently on di er- 8 |
ent CPUs complying to the x86_64 architecture (e.g. Intel tion through a strict JSON schema, (3) exible application de- and AMD) when the application is compiled with architecture- scription through a rich JSON schema. speci c ags such as GCC’s -march. In addition, important runtime parameters, for instance re- Common Work ow Language lated to multi-threading or available resources (storage, RAM), may be set by the execution platform without being speci ed The Common Work ow Language (CWL13) is the work most in the Boutiques descriptor. Such runtime parameters may im- closely related to Boutiques as it provides a formal way to de- pact reproducibility in some cases. To properly cover this is- scribe containerized applications. In particular, CWL’s Com- sue, Boutiques descriptors should be complemented by a prove- mand Line Tool Description overlaps with the Boutiques de- nance framework that captures a detailed trace of the execu- scriptor. This Section highlights the main di erences between tion. We plan to leverage the provenance format being de ned CWL and Boutiques, based on version 1.0 of the CWL Command by the NeuroImaging Data Model-Work ow [22] initiative for Line Tool Description14. According to GitHub, CWL started 6 this purpose. months before Boutiques (September 2014 vs. May 2015).
Application types Conceptual di erences The following di erences are conceptual in the sense that they So far, Boutiques has focused on the description of non- may not be easily addressed in CWL or Boutiques without interactive command-line applications. While such applica- deeply refactoring the frameworks. tions cover a large fraction of the applications involved in sci- First, CWL has a work ow language whereas Boutiques does enti c data processing, other types of programs exist such as not. In Boutiques, work ows are integrated as any other ap- web services, interactive applications, and applications with a plications, except that they may submit other invocations to graphical user interface (GUI). Such application classes could enable work ow parallelism. This fundamental di erence has be described in Boutiques through a command-line mapping. consequences on the complexity of CWL application descrip- For instance, web services may be wrapped as command-line tions and on the possibility to reuse existing work ows in Bou- applications using tools such as curl or wget. Interactive ap- tiques. The adoption of ontologies in CWL may also be another plications may also be transformed to non-interactive ones consequence (see below). through con guration les. Finally, nothing prevents a Bou- CWL imposes a strict command-line format while Bou- tiques application from popping up a window for a user to pro- tiques is more exible. CWL speci es command lines using an vide input through a GUI. This should, however, be speci ed array containing an executable and a set of arguments whereas as an extension to the descriptor since most platforms would Boutiques only uses a string template. Boutiques’ template ap- not support this feature by default. Graphical output produced proach may create issues in some cases, but it also allows devel- by applications executed in containers should also be treated opers to add simple operations to an application without having speci cally. to write a speci c wrapper. For instance, a Boutiques command line may easily include input decompression using the tar com- mand in addition to the main application command. Impor- Limitations tantly, Boutiques’ template system allows supporting con gu- ration les. A few limitations remain that should be addressed in the fu- CWL uses ontologies while Boutiques does not. Ontologies ture. First, Boutiques moves the application integration bot- allow for richer de nitions but they also have an overhead. The tleneck from integration to validation. Using Boutiques, func- main consequences are the following: tions can be automatically exported from frameworks such as Nipype and SPM, creating hundreds of richly described appli- • CWL uses a speci c framework for validation, called SALAD cations potentially usable by end-users. However, the auto- (Semantic Annotations for Linked Avro Data) whereas Bou- mated validation of such applications remains challenging. A tiques uses plain JSON schema. The main goal of SALAD is to Boutiques-speci c testing framework could be designed and allow “working with complex data structures and document potentially fed by existing frameworks, to address this issue. formats, such as schemas, object references, and names- Another limitation is related to the security of containerized paces”. Boutiques only relies on the basic types required to applications. Since containers are usually controlled by appli- describe and validate a command line syntactically. While cation developers rather than platforms, which is a good thing the use of SALAD certainly allows for higher-level valida- to reduce application integration bottlenecks, nothing prevents tion, and may simplify the composition and validation of developers to embed malicious code in their container at any complex work ows, it also introduces a substantial over- stage of the process, possibly after a platform administrator head in the speci cation, and platforms have to use the val- inspected the container. Containers are bulky le archives that idator provided by CWL. On the contrary, a regular JSON are cumbersome to inspect. Tools need to be developed to al- validator can be used in Boutiques. low for an easier characterization of container contents, for in- • CWL has a rich set of types whereas Boutiques only has sim- stance through comparison digests with respect to validated ple types. This may again be seen as a feature or as an over- base images. Singularity containers have reduced security risks head depending on the context. Boutiques tries to limit the as compared to Docker containers, but the issue of content complexity of the speci cation to facilitate its support by transparency is still not avoided. platforms where applications will be integrated.
Major di erences Related work The following di erences are major but they may be addressed by the CWL and Boutiques developers as they do not undermine Several frameworks have been developed to describe and in- the application description model. tegrate applications in various types of platforms. Bou- tiques focuses on (1) fully-automatic integration of applica- tions, including deployment on heterogeneous computing re- 13 http://www.commonwl.org sources through containers, (2) comprehensive input valida- 14 http://www.commonwl.org/v1.0/CommandLineTool.html | 9
• CWL applications have to write in a speci c set of directo- tecture (GEMLCA [28]) was used to wrap applications in grid ries called “designated output directory”, “designated tem- computing systems. Interestingly, it has been used to embed porary directory” and “system temporary directory”. Ap- work ow engines in the SHIWA platform [29], in a similar but plications are informed of the location of such directories di erent way than proposed by Boutiques. through environment variables. Having to write in speci c The recent advent of software containers requires a new directories is problematic because applications have to be generation of application description frameworks that are in- modi ed to enable that. In Boutiques, the path of output dependent from any programming language and that expose a les is de ned using a dedicated property. rich set of properties to describe command lines, as intended • CWL types are richer, not only semantically but also syn- by Boutiques. tactically. For instance, les have properties for basename, dirname, location, path, checksum, etc. • Boutiques supports various types of containers (Docker, Sin- Conclusion gularity, rootfs) while CWL supports only Docker. Both https://github.com/boutiques tools have rich requirements: for instance, they may include Boutiques is available at . We RAM, disk usage, and walltime estimate. CWL has hints, welcome feedback, issue reporting and pull requests. Bou- i.e., recommendations that only lead to warnings when not tiques adopts a bottom-up approach where new features are respected. progressively added based on feedback from applications and • In Boutiques, dependencies can be de ned among inputs, platforms. Beyond the technicalities discussed in this paper, e.g., to specify that an input may be used only when a par- the availability of a solid core of applications and platforms in ticular ag is activated. This is a very useful feature to im- the framework is key to its success, which we plan to continu- prove validation, in particular for applications with a lot of ously enhance. options. • In Boutiques, named groups of inputs can be de ned, which improves the presentation of long parameter lists for the Acknowledgments user and enables the de nition of more constraints within Pipeline integration for the MICCAI 2016 challenge was funded groups (e.g. mutually exclusive inputs). by the French National Agency for Research (ANR) through “France Life-Imaging”. We also thank Compute Canada and BIDS apps Calcul Québec for providing a computing infrastructure sup- porting Docker and Singularity containers. This research was BIDS apps [5] specify a framework for neuroimaging applica- undertaken thanks in part to funding from the Canada First Re- tions to process datasets complying to the Brain Imaging Data search Excellence Fund, awarded to McGill University for the Structure (BIDS). They share common goals with Boutiques, Healthy Brains for Healthy Lives initiative. We also thank the in particular reusability across platforms through container- developers of all the applications described with Boutiques. ization. Conceptually, however, BIDS apps and Boutiques are di erent since BIDS apps intend to standardize application in- terfaces while Boutiques intends to describe them as exibly Competing interests as possible. BIDS apps have a speci c set of inputs and out- puts, for instance the input dataset, that have to be present in The authors declare that they have no competing interests. a speci c order on the command line for the application to be valid. The speci cation adopted by BIDS apps simpli es the integration of applications in platforms as they all comply to References the same interface. However, it is also limited to the subset 1. Peng RD. Reproducible research in computational science. of neuroimaging applications that process BIDS datasets and Science 2011;334(6060):1226–1227. it does not formally describe application-speci c inputs. All 2. Stodden V, McNutt M, Bailey DH, Deelman E, Gil Y, Han- in all, BIDS apps and Boutiques complement each other: BIDS son B, et al. Enhancing reproducibility for computational apps provide a practical way to integrate neuroimaging applica- methods. Science 2016;354(6317):1240–1241. tions, while Boutiques o ers a formal description of their spe- 3. Kurtzer GM, Sochat V, Bauer MW. Singularity: Sci- ci c parameters. Boutiques descriptors can be generated from enti c containers for mobility of compute. PloS one BIDS apps using the bosh importer. 2017;12(5):e0177459. 4. Glatard T, Étienne Rousseau M, Camarasu-Pop S, Adalat Other frameworks R, Beck N, Das S, et al. Software architectures to integrate work ow engines in science gateways. Future Generation Several other frameworks have been created to facilitate the Computer Systems 2017;75(Supplement C):239 – 255. integration of command-line applications in platforms. In 5. Gorgolewski KJ, Alfaro-Almagro F, Auer T, Bellec P, Capotă neuroinformatics, many platforms de ne a formal interface M, Chakravarty MM, et al. BIDS apps: Improving ease to embed command-line applications. Among them, the Com- of use, accessibility, and reproducibility of neuroimag- 15 mon Toolkit interoperates with several platforms such as 3D ing data analysis methods. PLoS computational biology Slicer [23], NiftyView [24], GIMIAS [25], MedInria [25], MeVis- 2017;13(3):e1005209. Lab [26] and MITK workbench [27]. The framework, however, 6. Gentleman RC, Carey VJ, Bates DM, Bolstad B, Dettling M, remains tightly bound to the Common Toolkit’s C++ implemen- Dudoit S, et al. Bioconductor: open software development tation which limits its adoption, e.g., in web platforms. for computational biology and bioinformatics. Genome bi- In the distributed computing community, systems were also ology 2004;5(10):R80. proposed to facilitate the embedding of applications in plat- 7. Sherif T, Rioux P, Rousseau ME, Kassis N, Beck N, Adalat R, forms. The Grid Execution Management for Legacy Code Archi- et al. CBRAIN: a web-based, distributed computing plat- form for collaborative neuroimaging research. Frontiers in Neuroinformatics 2014;8(54). 15 http://www.commontk.org 8. Das S, Glatard T, MacIntyre LC, Madjar C, Rogers C, 10 |
Rousseau ME, et al. The MNI data-sharing and process- and Modeling of the Heart Springer; 2009. p. 417–426. ing ecosystem. NeuroImage 2016;124:1188–1195. 26. Heckel F, Schwier M, Peitgen HO. Object-oriented appli- 9. Gorgolewski K, Burns CD, Madison C, Clark D, Halchenko cation development with MeVisLab and Python. In: GI YO, Waskom ML, et al. Nipype: a exible, lightweight Jahrestagung; 2009. p. 1338–1351. and extensible neuroimaging data processing framework 27. Nolden M, Zelzer S, Seitel A, Wald D, Müller M, Franz AM, in python. Frontiers in neuroinformatics 2011;5. et al. The medical imaging interaction toolkit: challenges 10. Glatard T, Lartizien C, Gibaud B, Ferreira da Silva R, and advances. International journal of computer assisted Forestier G, Cervenansky F, et al. A Virtual Imaging Plat- radiology and surgery 2013;8(4):607–620. form for multi-modality medical image simulation. IEEE 28. Delaitre T, Kiss T, Goyeneche A, Terstyanszky G, Winter S, Transactions on Medical Imaging 2013;32(1):110–118. Kacsuk P. GEMLCA: Running legacy code applications as 11. Montagnat J, Isnard B, Glatard T, Maheshwari K, Blay- grid services. Journal of Grid Computing 2005;3(1-2):75– Fornarino M. A data-driven work ow language for grids 90. based on array programming principles. In: Workshop on 29. Terstyanszky G, Kukla T, Kiss T, Kacsuk P, Balaskó Á, Work ows in Support of Large-Scale Science (WORKS’09); Farkas Z. Enabling scienti c work ow sharing through 2009. . coarse-grained interoperability. Future Generation Com- 12. Glatard T, Montagnat J, Lingrand D, Pennec X. Flexible and puter Systems 2014;37:46–59. e cient work ow deployment of data-intensive applica- tions on grids with MOTEUR. Journal of High Performance Computing and Applications 2008;22(3):347–360. 13. Jenkinson M, Beckmann CF, Behrens TE, Woolrich MW, Smith SM. Fsl. Neuroimage 2012;62(2):782–790. 14. Bellec P, Carbonell F, Perlbarg V, Lepage C, Lyttelton O, Fonov V, et al. A neuroimaging analysis kit for Matlab and Octave. In: Proceedings of the 17th International Confer- ence on Functional Mapping of the Human Brain; 2011. p. 2735–2746. 15. Bellec P, Lavoie-Courchesne S, Dickinson P, Lerch J, Zij- denbos A, Evans AC. The Pipeline System for Octave and Matlab (PSOM): a lightweight scripting framework and ex- ecution engine for scienti c work ows. Frontiers in neu- roinformatics 2012;6(7). 16. Glatard T, Quirion PO, Adalat R, Beck N, Bernard R, Caron BL, et al. Integration between PSOM and CBRAIN for dis- tributed execution of neuroimaging pipelines. In: Meet- ing of the Organization for Human Brain Mapping Geneva, Switzerlad; 2016. . 17. Tournier J, Calamante F, Connelly A, et al. MRtrix: di u- sion tractography in crossing ber regions. International Journal of Imaging Systems and Technology 2012;22(1):53– 66. 18. Kiar G, Bridgeford E, Chandrashekhar V, Mhembere D, Burns R, Roncal WG, et al. A Comprehensive Cloud Frame- work for Accurate and Reliable Human Connectome Esti- mation and Meganalysis. bioRxiv 2017;p. 188706. 19. Jochimsen TH, Von Mengershausen M. ODIN—object- oriented development interface for NMR. Journal of Mag- netic Resonance 2004;170(1):67–78. 20. Penny WD, Friston KJ, Ashburner JT, Kiebel SJ, Nichols TE. Statistical parametric mapping: the analysis of functional brain images. Academic press; 2007. 21. Hatt M, Laurent B, Ouahabi A, Fayad H, Tan S, Li L, et al. The rst MICCAI challenge on PET tumor segmentation. Medical Image Analysis 2018;Under minor revision. 22. Ghosh S, Auer T, Flandin G, Glatard T, Gorgolewski K, Halchenko Y, et al. Capturing and Reusing Computation Details with the Neuroimaging Data Model (NIDM). In: Annual meeting of the Organisation of Human Brain Map- ping (OHBM 2017); 2017. . 23. Pieper S, Halle M, Kikinis R. 3D Slicer. In: IEEE Interna- tional Symposium on Biomedical Imaging: Nano to Macro IEEE; 2004. p. 632–635. 24. Craddock RC, Bellec P, Margules DS, Nichols BN, Pfann- möller JP, Badhwar A, et al. 2015 Brainhack Proceedings. GigaScience 2016;5(1):1–26. 25. Larrabide I, Omedas P, Martelli Y, Planes X, Nieber M, Moya JA, et al. GIMIAS: an open source framework for e cient development of research tools and clinical proto- types. In: International Conference on Functional Imaging