DryadLINQ for Scientific Analyses
Jaliya Ekanayake1,a, Atilla Soner Balkirc, Thilina Gunarathnea, Geoffrey Foxa,b, Christophe Poulaind, Nelson Araujod, Roger Bargad aSchool of Informatics and Computing, Indiana University Bloomington bPervasive Technology Institute, Indiana University Bloomington cDepartment of Computer Science, University of Chicago dMicrosoft Research {jekanaya, tgunarat, gcf}@indiana.edu, [email protected],{cpoulain,nelson,barga}@microsoft.com
Abstract data-centered approach to parallel processing. In these frameworks, the data is staged in data/compute nodes Applying high level parallel runtimes to of clusters or large-scale data centers and the data/compute intensive applications is becoming computations are shipped to the data in order to increasingly common. The simplicity of the MapReduce perform data processing. HDFS allows Hadoop to programming model and the availability of open access data via a customized distributed storage system source MapReduce runtimes such as Hadoop, attract built on top of heterogeneous compute nodes, while more users around MapReduce programming model. DryadLINQ and CGL-MapReduce access data from Recently, Microsoft has released DryadLINQ for local disks and shared file systems. The simplicity of academic use, allowing users to experience a new these programming models enables better support for programming model and a runtime that is capable of quality of services such as fault tolerance and performing large scale data/compute intensive monitoring. analyses. In this paper, we present our experience in Although DryadLINQ comes with standard samples applying DryadLINQ for a series of scientific data such as Terasort, word count, its applicability for large analysis applications, identify their mapping to the scale data/compute intensive scientific applications is DryadLINQ programming model, and compare their not studied well. A comparison of these programming performances with Hadoop implementations of the models and their performances would benefit many same applications. users who need to select the appropriate technology for the problem at hand. 1. Introduction We have developed a series of scientific applications using DryadLINQ, namely, CAP3 DNA Among many applications benefit from cloud sequence assembly program [4], High Energy Physics technologies such as DryadLINQ [1] and Hadoop [2], data analysis, CloudBurst [5] - a parallel seed-and- the data/compute intensive applications are the most extend read-mapping application, and Kmeans important. The deluge of data and the highly compute Clustering [6]. Each of these applications has unique intensive applications found in many domains such as requirements for parallel runtimes. For example, the particle physics, biology, chemistry, finance, and HEP data analysis application requires ROOT [7] data information retrieval, mandate the use of large analysis framework to be available in all the compute computing infrastructures and parallel runtimes to nodes, and in CloudBurst the framework needs to achieve considerable performance gains. The support process different workloads at map/reduce/vertex for handling large data sets, the concept of moving tasks. We have implemented all these applications computation to data, and the better quality of services using DryadLINQ and Hadoop, and used them to provided by Hadoop and DryadLINQ made them compare the performance of these two runtimes. CGL- favorable choice of technologies to implement such MapReduce and MPI are used in applications where problems. the contrast in performance needs to be highlighted. Cloud technologies such as Hadoop and Hadoop In the sections that follow, we first present the Distributed File System (HDFS), Microsoft DryadLINQ programming model and its architecture DryadLINQ, and CGL-MapReduce [3] adopt a more on HPC environment, and a brief introduction to Hadoop. In section 3, we discuss the data analysis var input = PartitionedTable.Get
outputs of the map tasks are first stored in local disks until later, when the reduce tasks access them (pull) via We developed a DryadLINQ application to perform HTTP connections. Although this approach simplifies the above data analysis in parallel. The DryadLINQ the fault handling mechanism in Hadoop, it adds application executes the CAP3 executable as an significant communication overhead to the external program, passing an input data file name, and intermediate data transfers, especially for applications the other necessary program parameters to it. Since that produce small intermediate results frequently. The DryadLINQ executes CAP3 as an external executable, current release of DryadLINQ also communicates it must only know the input file names and their using files, and hence we expect similar overheads in locations. We achieve the above functionality as DryadLINQ as well. Table 1 presents a comparison of follows: (i) the input data files are partitioned among DryadLINQ and Hadoop on various features supported the nodes of the cluster so that each node of the cluster by these technologies. stores roughly the same number of input data files; (ii) a “data-partition” (A text file for this application) is 3. Scientific Applications created in each node containing the names of the original data files available in that node; (iii) Dryad In this section, we present the details of the “partitioned-file” (a meta-data file for DryadLINQ) is DryadLINQ applications that we developed, the created to point to the individual data-partitions located techniques we adopted in optimizing the applications, in the nodes of the cluster. and their performance characteristics compared with Following the above steps, a DryadLINQ program Hadoop implementations. For all our benchmarks, we can be developed to read the data file names from the used two clusters with almost identical hardware provided partitioned-file, and execute the CAP3 configurations as shown in Table 2. program using the following two DryadLINQ queries.
Table 2. Different computation clusters used for the IQueryable
(Ref A) (Ref B) Although we use this program specifically for the CPU Intel(R) Xeon(R) Intel(R) Xeon(R) CAP3 application, the same pattern can be used to CPU L5420 CPU L5420 execute other programs, scripts, and analysis functions 2.50GHz 2.50GHz # CPU 2 2 written using the frameworks such as R and Matlab, on # Cores 8 8 a collection of data files. (Note: In this application, we Memory 32GB 16 GB assumed that DryadLINQ would process the input data # Disk 1 2 files on the same nodes where they are located. Network Giga bit Ethernet Giga bit Ethernet Although this is not guaranteed by the DryadLINQ Operating Red Hat Windows Server runtime, if the nodes containing the data are free System Enterprise Linux Enterprise - 64 bit during the execution of the program, it will schedule Server -64 bit the parallel tasks to the appropriate nodes; otherwise, # Nodes 32 32 the data will be accessed via the shared directories.) When we first deployed the application on the 2.1. CAP3 cluster, we noticed a sub-optimal CPU core utilization by the application, which is highly unlikely for a CAP3 is a DNA sequence assembly program compute intensive program such as CAP3. A trace of developed by X. Huang and A. Madan [4] that job scheduling in the HPC cluster revealed that the performs several major assembly steps such as scheduling of individual CAP3 executables in a given computation of overlaps, construction of contigs, node is not optimal in terms of the utilization of CPU construction of multiple sequence alignments and cores. In a node with 8 CPU cores, we would expect generation of consensus sequences, to a given set of DryadLINQ to execute 8 ExecuteCAP3() functions gene sequences. The program reads a collection of simultaneously. However, we noticed that the number gene sequences from an input file (FASTA file format) of parallel executions varies from 8 to 1 with the and writes its output to several output files and to the DryadLINQ’s scheduling mechanism. For example, standard output as shown below. The input data is with 16 data file names in a data-partition; we noticed contained in a collection of files, each of which needs the following scheduling pattern (of concurrent to be processed by the CAP3 program separately. ExecuteCAP3()s) from PLINQ, 8->4->4, which Input.fsa -> Cap3.exe -> Stdout + Other output files should ideally be 8 ->8. When an application is scheduled, DryadLINQ uses the number of data partitions as a guideline to determine the number of vertices to run. Then completes. In contrast, in Hadoop, the user can set the DryadLINQ schedules these vertices to the nodes maximum and minimum number of map and reduce rather than individual CPU cores assuming that the tasks to execute concurrently on a given node so that it underlying PLINQ runtime would handle the further will utilize all the CPU cores. parallelism available at each vertex and utilize all the CPU cores. The PLINQ runtime, which is intended to optimize processing of fine grained tasks in multi-core nodes, performs optimizations by chunking the input data. Since our input for DraydLINQ is only the names of the original data files, it has no way to determine how much time the ExecuteCAP3() take to process a file, and hence the chunking of records at PLINQ results sub-optimal scheduling of tasks. We found a workaround to this problem by changing the way we partition the data. Instead of partitioning input data to a single data-partition per node, we create data-partitions containing at most 8 (=number of CPU cores) line records (actual input file names). This way, we used DryadLINQ’s scheduler to Figure 3. Scalability of different implementations of schedule series of vertices corresponding to different CAP3 data-partitions in nodes while PLINQ always schedules 8 tasks at once, which gave us 100% CPU utilization. The performance and the scalability graphs shows Figure 2 and 3 show comparisons of performance and that all three runtimes work almost equally well for the the scalability of the DryadLINQ application, with the CAP3 program, and we would expect them to behave Hadoop and CGL-MapReduce versions of the CAP3 in the same way for similar applications with simple application. parallel topologies. 2.2. High Energy Physics
Next, we developed a high energy physics (HEP) data analysis application and compared it with the previous implementations of Hadoop and CGL- MapReduce versions. As in CAP3, in this application the input is also available as a collection of large number of binary files, each with roughly 33MB of data, which will not be directly accessed by the DryadLINQ program. We manually partition the input data to the compute nodes of the cluster and generated data-partitions containing only the file names available in a given node. The first step of the analysis requires applying a function coded in ROOT script to all the Figure 2. Performance of different implementations input files. The analysis script we used can process of CAP3 application multiple input files at once, therefore we used a Although the above approach (partitioning data) homomorphic Apply (shown below) operation in Dryad works perfectly fine with this application, it does not to perform the first stage (corresponds to the map() solve the underlying problem completely. DryadLINQ stage in MapReduce) of the analysis. does not schedule multiple concurrent vertices to a [Homomorphic] given node, but one vertex at a time. Therefore, a ApplyROOT(string fileName){..} vertex which uses PLINQ to schedule some non- IQueryable
CloudBurst is an open source Hadoop application that performs a parallel seed-and-extend read-mapping algorithm optimized for mapping next generation sequence data to the human genome and other reference genomes. It reports all alignments for each read with up to a user specified number of differences including mismatches and indels [5]. It parallelizes execution by seed, so that the reference and query sequences sharing the same seed are grouped together and sent to a reducer for further analysis. It is composed of a two stage MapReduce workflow: The first stage is to compute the alignments for each read with at most k differences where k is a Figure 4. Performance of different implementations user specified input. The second stage is optional, and of HEP data analysis applications it is used as a filter to report only the best unambiguous alignment for each read rather than the full catalog of The results in Figure 4 highlight that Hadoop all alignments. The execution time is typically implementation has a considerable overhead compared dominated by the reduce phase. to DraydLINQ and CGL-MapReduce implementations. An important characteristic of the application is that This is mainly due to differences in the storage the variable amount of time it spent in the reduction mechanisms used in these frameworks. DryadLINQ phase. Seeds composed of a single DNA character and CGL-MapReduce access the input from local disks occur a disproportionate number of times in the input where the data is partitioned and distributed before the data and therefore reducers assigned to these “low computation. Currently, HDFS can only be accessed complexity” seeds spend considerably more time than using Java or C++ clients, and the ROOT – data the others. CloudBurst tries to minimize this effect by analysis framework is not capable of accessing the emitting redundant copies of each “low complexity” The results in Figure 5 show that all three seed in the reference and assigning them to multiple implementations follow a similar pattern although reducers to re-balance the workload. However, DryadLINQ is not fast enough especially with small calculating the alignments for a “low complexity” seed number of nodes. As we mentioned in the previous in a reducer still takes more time compared to the section, DryadLINQ assigns vertices to nodes rather others. This characteristic can be a limiting factor to than cores and PLINQ handles the core level scale, depending on the scheduling policies of the parallelism automatically by assigning records to framework running the algorithm. separate threads running concurrently. However, we We developed a DryadLINQ application based on observed that the cores were not utilized completely, the available source code written for Hadoop. The and the total CPU utilization per node varied Hadoop workflow can be expressed as: continuously between 12.5% and 90% during the Map -> Shuffle -> Reduce -> Identity Map -> reduce step due to the inefficiencies with the PLINQ Shuffle -> Reduce scheduler. Conversely, in Hadoop, we started 8 reduce The identity map at the second stage is used for tasks per node and each task ran independently by grouping the alignments together and sending them to doing a fairly equal amount of work. a reducer. In DryadLINQ, the same workflow is Another difference between DryadLINQ and expressed as follows: Hadoop implementations is the number of partitions Map -> GroupBy -> Reduce -> GroupBy -> Reduce created before the reduce step. For example, with 32 Notice that we omit the identity map by doing an nodes (with 8 cores each), Hadoop creates 256 on-the-fly GroupBy right after the reduce step. partitions, since there are 256 reduce tasks. Although these two workflows are identical in terms of DryadLINQ, on the other hand, creates 32 vertices and functionality, DryadLINQ runs the whole computation assigns each vertex to a node (if we start the as one large query rather than two separate MapReduce computation with 32 partitions initially). Notice that jobs followed by one another. the partitions are created using a hash function, and The reduce function takes a set of reference and they are not necessarily the same size. Creating more query seeds sharing the same key as input, and partitions results in having smaller groups and thus produces one or more alignments as output. For each decreases the overall variance in the group size. The input record, query seeds are grouped in batches, and total time spent in the reduce function is equal to the each batch is sent to an alignment function sequentially maximum time spent in the longest reduce task. Since to reduce the memory limitations. We developed Hadoop creates more partitions, it balances the another DryadLINQ implementation that can process workload among reducers more equally. If the PLINQ each batch in parallel assigning them as separate scheduler worked as expected, this would not be a threads running at the same time using .NET Parallel problem as it would keep the cores busy and thus yield Extensions. a similar load balance to Hadoop. As a work around to this problem, we tried starting the computation with more partitions. In the 32 node example, we created 256 partitions aiming to schedule 8 vertices per node. However, DryadLINQ runs the tasks in order, so it is impossible to start 8 vertices at once in one node with the current academic release. Instead, DryadLINQ waits for one vertex to finish before scheduling the second vertex, but the first vertex may be busy with only one record, and thus holding the rest of the cores idle. We observed that scheduling too many vertices (of the same type) to a node is not efficient for this application due to its heterogeneous record structure. Our main motivation behind using the .NET Figure 5. Figure5. Scalability of CloudBurst with parallel extensions was to reduce this gap by fully different implementations. utilizing the idle cores, although it is not identical to We compared the scalability of these three Hadoop’s level of parallelism. implementations by mapping 7 million publicly Figure 6 shows the performance comparison of available Illumina/Solexa sequencing reads [10] to the DryadLINQ and Hadoop with increasing data size. full human genome chromosome1. Both implementations scale linearly, and the time gap is mainly related to the current limitations with PLINQ and DryadLINQ’s job scheduling policies explained centers. Another Apply operation, which works above. Hadoop shows a non linear behavior with the sequentially, calculates the new cluster centers for the last data set and we will do further investigations with nth iteration. Finally, we calculate the distance between larger data sets to better understand the difference in the previous cluster centers and the new cluster centers the shapes. using a Join operation to compute the Euclidian distance between the corresponding cluster centers. DryadLINQ support “unrolling loops”, using which multiple iterations of the computation can be performed as a single DryadLINQ query. In DryadLINQ programming model, the queries are not evaluated until a program accesses their values. Therefore, in the Kmeans program, we accumulate the computations performed in several iterations (we used 4 as our unrolling factor) as queries and “materialize” the value of the new cluster centers only at each 4th iteration. In Hadoop’s MapReduce model, each iteration is represented as a separate MapReduce computation. Notice that without the loop unrolling feature in DryadLINQ, each iteration would be Figure 6. Performance comparison of DryadLINQ and represented by a separate execution graph as well. Hadoop for CloudBurst. Figure 7 shows a comparison of performances of different implementations of Kmeans clustering. 2.4. Kmeans Clustering
We implemented a Kmeans Clustering application using DryadLINQ to evaluate its performance under iterative computations. We used Kmeans clustering to cluster a collection of 2D data points (vectors) to a given number of cluster centers. The MapReduce algorithm we used is shown below. (Assume that the input is already partitioned and available in the th compute nodes). In this algorithm, Vi refers to the i th th vector, Cn,j refers to the j cluster center in n th iteration, Dij refers to the Euclidian distance between i vector and jth cluster center, and K is the number of cluster centers.
K-means Clustering Algorithm for MapReduce Figure 7. Performance of different implementations do of Kmeans clustering algorithm Broadcast Cn Although we used a fixed number of iterations, we [Perform in parallel] –the map() operation changed the number of data points from 500k to 20 for each Vi millions. Increase in the number of data points triggers for each Cn,j the amount of computation. However, it was not Dij <= Euclidian (Vi,Cn,j) sufficient to load the 32 node cluster we used. As a Assign point Vi to Cn,j with minimum Dij result, the graph in Figure 7 mainly shows the overhead of different runtimes. The use of file system for each C n,j based communication mechanisms and the loading of Cn,j <=Cn,j/K static input data at each iteration (in Hadoop) and in [Perform Sequentially] –the reduce() operation each unrolled loop (in Dryad) has resulted in higher Collect all Cn overheads compared to CGL-MapReduce and MPI. Calculate new cluster centers Cn+1 Iterative applications which perform more Diff<= Euclidian (Cn, Cn+1) computations or access large volumes of data may while (Diff The DryadLINQ implementation uses an Apply ameliorate the higher overhead induced by these operation which will be performed in parallel in terms runtimes. of the data vectors, to calculate the partial cluster 4. Related Work amount of overheads in DryadLINQ and Hadoop is extremely large for this type of applications compared Various scientific applications have been to other runtimes such as MPI and CGL-MapReduce. previously adapted to the MapReduce model for the As our future work, we plan to investigate the use past few years and Hadoop gained significant attention of DryadLINQ and Hadoop on commercial cloud from the scientific research community. Kang et al. infrastructures. studied [11] efficient map reduce algorithms for finding the diameter of very large graphs and applied 6. Acknowledgements their algorithm to real web graphs. Dyer et al. described [12] map reduce implementations of We would like to thank the Dryad team at parameter estimation algorithms to use in word Microsoft Research, including Michael Isard, Mihai alignment models and a phrase based translation Budiu, and Derek Murray for their support in model. Michael Schatz introduced CloudBurst [5] for DryadLINQ applications, and Joe Rinkovsky from IU mapping short reads from sequences to a reference UITS for his dedicated support in setting up the genome. In our previous works [3][13], we have compute clusters. discussed the usability of MapReduce programming model for data/compute intensive scientific References applications and the possible improvements to the programming model and the architectures of the [1] Y.Yu, M. Isard, D. Fetterly, M. Budiu, Ú. Erlingsson, P. runtimes. Our experience suggests that most Gunda, and J. Currey, “DryadLINQ: A System for General- Purpose Distributed Data-Parallel Computing Using a High- composable applications can be implemented using Level Language,” Symposium on Operating System Design MapReduce programming model either by directly and Implementation (OSDI), CA, December 8-10, 2008. exploiting their data/task parallelism or by adopting [2] Apache Hadoop, http://hadoop.apache.org/core/ different algorithms compared to the algorithms used [3] J. Ekanayake and S. Pallickara, “MapReduce for Data in traditional parallel implementations. Intensive Scientific Analysis,” Fourth IEEE International Conference on eScience, 2008, pp.277-284. 5. Conclusions and Future Works [4] X. Huang and A. Madan, “CAP3: A DNA Sequence Assembly Program,” Genome Research, vol. 9, no. 9, pp. We have applied DryadLINQ to a series of 868-877, 1999. data/compute intensive applications with unique [5] M. Schatz, "CloudBurst: highly sensitive read mapping requirements. The applications range from simple map- with MapReduce", Bioinformatics. 2009 June 1; 25(11): only operations such as CAP3 to multiple stages of 1363–1369. MapReduce jobs in CloudBurst and iterative [6] J. Hartigan. Clustering Algorithms. Wiley, 1975. MapReduce in Kmeans clustering. We showed that all [7] ROOT Data Analysis Framework, these applications can be implemented using the DAG http://root.cern.ch/drupal/ based programming model of DryadLINQ, and their [8] http://research.microsoft.com/en- performances are comparable to the MapReduce us/downloads/03960cab-bb92-4c5c-be23-ce51aee0792c/ implementations of the same applications developed [9] Chaiken, R., Jenkins, B., Larson, P., Ramsey, B., Shakib, using Hadoop. D., Weaver, S., and Zhou, J. 2008. SCOPE: easy and We also observed that cloud technologies such as efficient parallel processing of massive data sets. Proc. VLDB Endow. 1, 2 (Aug. 2008), 1265-1276. DryadLINQ and Hadoop work well for many applications with simple communication topologies. [10] The 1000 Genomes Project, “A Deep Catalog of Human Genetic Variation”, January 2008, The rich set of programming constructs available in _http://www.1000genomes.org/page.php DryadLINQ allows the users to develop such [11] U. Kang, C. Tsourakakis, A. P. Appel, C. Faloutsos, J. applications with minimum programming effort. Leskovec, “HADI: Fast Diameter Estimation and Mining in However, we noticed that higher level of abstractions Massive Graphs with Hadoop”, CMU ML Tech Report in DryadLINQ model sometimes makes fine-tuning the CMU-ML-08-117, 2008. applications more challenging. [12] C. Dyer, A. Cordova, A. Mont, J. Lin, “Fast, Easy, and The current scheduling behavior of PLINQ hinders Cheap: Construction of Statistical Machine Translation the performance of DryadLINQ applications. We Models with MapReduce”, Proceedings of the Third would expect a fix to this problem would simply Workshop on Statistical Machine Translation at ACL 2008, increase performance of DryadLINQ applications. Columbus, Ohio. The features such as loop unrolling let DryadLINQ [13] G. Fox, S. Bae, J. Ekanayake, X. Qiu, and H. Yuan, “Parallel Data Mining from Multicore to Cloudy Grids,” perform iterative applications faster, but still the High Performance Computing and Grids workshop, 2008.