A Tool for Tracking Provenance and Reproducibility of Big Data Experiments
Total Page：16
File Type：pdf, Size：1020Kb
PROB: A tool for Tracking Provenance and Reproducibility of Big Data Experiments. Vlad Korolev Anupam Joshi University of Maryland, Baltimore County University of Maryland, Baltimore County Baltimore, MD, USA Baltimore, MD, USA [email protected] [email protected] Abstract In this work, we propose a tool that aids tracking of the provenance and the ability to achieve reproducibility Reproducibility of computations and data prove- of experiments that involve big data workflows. The nance are very important goals to achieve in order to development of this tool is still ongoing, therefore, the improve the quality of one’s research. Unfortunately, code is not available to the public yet. despite some efforts made in the past, it is still very hard to reproduce computational experiments with high de- gree of certainty. The Big Data phenomenon in recent 2. Background years makes this goal even harder to achieve. In this work, we propose a tool that aids researchers to improve Most of the scientific data analysis workflows have reproducibility of their experiments through automated very similar structure. Especially big-data workflows keeping of provenance records. that are based on Map-Reduce model[4]. A typical six step big-data scientific pipeline is shown on Figure 1. At step 1, the data is collected by observing some phe- 1. Introduction nomenon or performing a study of some population of the subjects. For big-data workflows, this is usually done by a third party. In the case of Gene Wide Association Reproducibility and provenance tracking is an on- Studies (GWAS) in United States, the National Center going concern in the scientific community. Historically for Biotechnology Information (NCBI) [5] would be computational sciences were particularly bad about dis- conducting the work. closing the actual data and code used to obtain published results. To quote David Donoho “Computing results are At step 2, the data is transferred to the researcher now being presented in a very loose, “breezy” way – in who considers it as raw data. For many domains espe- journal articles, in conferences, and in books. All too cially the ones related to medicine or biology, where often one simply takes computations at face value. [1].” the privacy of subjects is of very high importance, re- With the recent increase of collaboration between searchers must follow specific protocols before the data computational and biological scientific communities, it is released. Often most of the controlled distribution is especially important to demonstrate provenance of the datasets arrive encrypted and must be decrypted before- source and allow other researchers to reproduce experi- hand. Very often the agencies encrypt the dataset using ments described in the publications. In fact according to the tools not available for the operating systems that will [2] a major cause of low rate of collaboration between be used for the actual analysis of the data. This step is different research groups is attributed to difficulties in usually performed on a different machine. reproduction of experiments. Which consequently leads For big-data workflows, after the data is decrypted to the poor quality of research. Conversely it has been and converted it must be loaded onto the distributed argued that allowing for easier reproducibility of the computation cluster that will be used for actual analysis. experiments leads to the higher quality of research [3]. Such transfer happens at step 3. To address this problem we must strive for higher At step 4, the actual analysis takes place. During degree of reproducibility. Which is impossible without this step the researcher uses some software that reads the thorough tracking of the provenance of original data and input data and outputs some results. Besides the input all manipulations done to it. data, the researcher can supply different parameters to the software which will tune the course of the analysis. Hamake[9] and Taverna[10] has been around for sev- The result of this step is a derived data that was obtained eral years and were used in a variety of research projects. as a result of the performed analysis. This step is usually These systems allow the researcher to setup and exe- performed repeatedly, while the researcher fine-tunes the cute analytic workflows. All of these tools represent a parameters given to the analytic software. The software workflow as a directed acyclic graph (DAG) where the itself could be changed either by the researcher or a third nodes are either data files, parameters or preprocessing party vendor. Also, this step could consist of multiple and analysis routines. The edges connect the inputs of sub-steps where the output of one sub-step is supplied these routines to the datasets, parameters or the outputs as an input or a parameter to the subsequent sub-step. of another step. Usually these tools are more domain After the researcher have obtained the desired result oriented than traditional programming languages, thus in step 4, step 5 is performed. In this step, the data is making a job of the researcher much easier. extracted out of a distributed cluster and then visualized Software developers have been using version con- and described to produce a publication. trol systems (VCS) for considerable time [11]. Among At the final step, the derived data and/or publication the modern VCS tools there is a growing trend towards is placed into publicly accessible storage where it could distributed version control systems DVCS which do not serve as raw data for a different research. rely on a central server. The emergence of cloud based web services such as GitHub and BitBucket has led to the explosive popularity of DVCS tools. Subjects / Phenomena Expressing and recording provenance information has been a known problem. There were several attempts 1 Collection to address it such as [12]. Recently W3C has introduced Raw Data a new standard PROV[13] for tracking provenance of P the artifact. A notable tool Git2Prov[14] was developed r 2 Transfer /Preprocess to convert metadata stored in GIT version control system o v into the PROV format. Preprocessed Data e Unfortunately, we believe that none of the tools men- n a tioned above provide functionality that is comprehensive 3 Data Load n enough to track provenance and provide reproducibil- c Data loaded e ity of big-data map-reduce workflows. Such workflows onto cluster file system have their own challenges. They must run on Hadoop type distributed clusters. Whereas most of the scientific 4 Analysis / Workflows T r workflow tools mentioned above, with the exception of Derived Data a Kangaroo and Hamake, are designed to be used on a c single workstation. Also most of the tools above do not k 5 Visualize / Describe i provide the means to store and reconstruct inputs and ex- n act workflow steps. The tools that do either rely on VCS g Publications systems to store the artifacts or in case of VisTrails they assume that entire experiment including all code and 6 Distribute data files could be packaged and given to a third party. Such approach would not work with strictly controlled Distributed / Shared Data datasets that have strict distribution policies. Even for publicly available datasets such approach is not feasible. For example, GoogleBooks dataset is 2.2TB. Including a copy of such large datasets in every workflow archive will make regular exchange of the workflows infeasi- Figure 1. Typical scientific big data workflow. ble. Storing such large datasets directly in the VCS also does not work. In addition to poor performance and inefficiency of these tools with very large files, most of 3. Prior work these tools limit the maximum object size well below the file size found such datasets. Moreover, a typical map- Scientific workflow systems are widely used to reduce file systems such as HDFS, do not implement full represent and execute computation experiments. Sys- POSIX call interface, so it is not possible to use standard tems such as Kangaroo[6], VisTrails[7], NiPype[8], VCS with such file-system. Recently a new tool called Git-Annex[15] was de- provenance information. In addition to its own ontology, veloped to address some problems of storing large files the PROB tool uses several existing ontologies discussed in git repositories and allow for controlled transfers of earlier. To represent basic provenance assertions such as restricted datasets. This tool uses a storage format sim- authorship and artifacts, we use PROV W3C standard. ilar to git, but it keeps the large object store separately The Git2Prov ontology is used to represent information from the regular git-object store. The storage format of contained in Git version control system. The NiPype git-annex has been modified to allow handling of very ontology is used to express facts about data analysis large files on the order of multiple terabytes. However workflows and the execution environments. the git-annex tools still maintains full metadata related At present the PROB tool is geared towards work- to the large files in the main git repository so git still flows written in PIG language which are intended to run can record all manipulations done to these large files as Hadoop Map-Reduce clusters. The tool could be used to well as maintain the integrity of the repository through support the pipelines such as the one depicted on figure 1. the use of checksums. This is achieved by git-annex While the steps 2 through 4 of the pipeline are being exe- checking-in the links to the hashes of the large files into cuted the tool makes the record of all produced artifacts the git repository. Such scheme also allows research as well as exact operations that were performed on the groups at different organizations to share the code used data objects.