DOCSLIB.ORG

Scalable Collaborative Caching and Storage Platform for Data Analytics

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Scalable Collaborative Caching and Storage Platform for Data Analytics Scalable Collaborative Caching and Storage Platform for Data Analytics by Timur Malgazhdarov A thesis submitted in conformity with the requirements for the degree of Master of Applied Science Edward S. Rogers Sr. Department of Electrical and Computer Engineering University of Toronto c Copyright 2018 by Timur Malgazhdarov Abstract Scalable Collaborative Caching and Storage Platform for Data Analytics Timur Malgazhdarov Master of Applied Science Edward S. Rogers Sr. Department of Electrical and Computer Engineering University of Toronto 2018 The emerging Big Data ecosystem has brought about dramatic proliferation of paradigms for analytics. In the race for the best performance, each new engine enforces tight cou- pling of analytics execution with caching and storage functionalities. This one-for-all ap- proach has led to either oversimplifications where traditional functionality was dropped or more configuration options that created more confusion about optimal settings. We avoid user confusion by following an integrated multi-service approach where we assign responsibilities to decoupled services. In our solution, called Gluon, we build a collab- orative cache tier that connects state-of-art analytics engines with a variety of storage systems. We use both open-source and proprietary technologies to implement our archi- tecture. We show that Gluon caching can achieve 2.5x-3x speedup when compared to uncustomized Spark caching while displaying higher resource utilization efficiency. Fi- nally, we show how Gluon can integrate traditional storage back-ends without significant performance loss when compared to vanilla analytics setups. ii Acknowledgements I would like to thank my supervisor, Professor Cristiana Amza, for her knowledge, guidance and support. It was my privilege and honor to work under Professor Amza's supervision. I would also like to thank my examination committee members: Professor Eyal de Lara, Professor Ashvin Goel, and Professor Ashish Khisti for their valuable comments and feedback. I am truly grateful to my colleagues and lab mates: Dr. Stelios Sotiriadis, Seyed Ali Jokar, and Arnamoy Bhattacharyya for their knowledge, help, and support. Last but not least, I would like to thank my family, especially my mother Nurgul Yessetova for her understanding, love, and support. iii Contents Acknowledgements iii Contents iv 1 Introduction1 2 Background7 2.1 Analytics Engines...............................8 2.1.1 Hadoop MapReduce (HMR).....................8 2.1.2 Spark................................. 10 2.1.3 Specialized Graph Processing.................... 11 2.2 Resource Managers.............................. 12 2.3 Storage platforms............................... 13 2.3.1 Network-attached storage...................... 13 2.3.2 Storage Area Networks........................ 14 2.3.3 Distributed systems with direct-attached storage.......... 14 2.4 Distributed Cache............................... 15 2.4.1 Alluxio................................. 15 2.5 Common Solutions.............................. 16 2.5.1 Vanilla Hadoop Solution....................... 16 2.5.2 Vanilla Spark Solution........................ 17 2.6 Conclusion................................... 18 iv 3 Thesis Idea and Design 19 3.1 Thesis Idea.................................. 19 3.2 Usability Issues................................ 20 3.2.1 Case Study: Spark.......................... 20 3.2.2 HDFS................................. 22 3.3 Proposed Design............................... 23 3.3.1 Collaborative caching layer...................... 23 3.3.2 Service Decoupling and Modularity................. 27 3.3.3 Consolidated Storage Layer..................... 28 3.4 Summary................................... 30 4 Implementation 32 4.1 Components.................................. 33 4.1.1 Alluxio................................. 33 4.1.2 Server SAN.............................. 34 4.1.3 GFS2................................. 36 4.1.4 YARN................................. 36 4.2 Control and Data Flow............................ 38 4.3 Connecting storage component........................ 41 4.3.1 Server SAN to filesystem connection................ 41 4.4 Connecting GFS2 with Analytics Engines.................. 43 4.5 Cache integration............................... 46 4.5.1 GFS2 to Alluxio connection..................... 46 4.6 Spark integration............................... 49 4.7 Additional optimizations........................... 50 4.7.1 Asynchronous Delete......................... 50 4.7.2 File consistency checker....................... 51 4.8 Summary................................... 52 5 Evaluation 53 5.1 Environment Setup.............................. 53 v 5.1.1 Benchmarks.............................. 55 5.2 Comparative evaluation using Spark..................... 57 5.2.1 Spark count.............................. 57 5.2.2 Logistic Regression.......................... 58 5.2.3 PageRank............................... 59 5.2.4 Gluon job statistics.......................... 60 5.2.5 Discussion............................... 61 5.3 Comparative evaluation using Hadoop MapReduce............ 62 5.3.1 DFSIO................................. 62 5.3.2 Terasort................................ 63 5.3.3 PageRank............................... 64 5.3.4 Discussion............................... 65 5.4 Graph Processing Framework - Hama.................... 65 5.5 Conclusion................................... 66 6 Related Work 68 6.1 Caching in Analytics............................. 68 6.2 HPC and shared storage integrations.................... 70 6.3 Full-stack integrations............................ 72 6.4 Conclusion................................... 74 7 Future Work and Final Remarks 75 Bibliography 77 vi Chapter 1 Introduction Several data analytics paradigms have been recently proposed in order to accommodate the growing needs of Big Data. Each new paradigm brought with it specialization for a particular need of data analytics workloads. At the same time, each such specializa- tion had as side effect a significant departure from existing data processing paradigms. From a usability perspective, this trend makes it increasingly difficult to analyse the trade-offs of existing offerings and determine the appropriate platform support, including interfaces, environments, settings and configurations for both functionality and optimal performance. In other words, as many different paradigms have proliferated to facilitate various data management needs, they have made usability and platform management and integration itself a growing concern. For example, the initial MapReduce offerings, such as Apache Hadoop[8], came with a departure from traditional approaches to data processing. Relational data access typ- ically used SQL-based interfaces to data maintained by consolidated storage back-ends. Newer data analytics systems, such as Hadoop, not only introduced a new Java-based data processing language; they also required that data reside in a distributed fashion, on compute nodes, which formed a separate data silo for data analytics. Spark[61] came later with yet another data processing language, Scala, and also with an even more pro- nounced decoupling from persistent data storage concerns. Both paradigms imply that input data and intermediate data is stored in a distributed fashion on new commodity distributed file systems, such as, HDFS[52]. Moreover, both Apache Hadoop and Spark 1 Chapter 1. Introduction 2 had their own data caching techniques with the only commonality the data locality and distributed file system principles. On the other hand, Apache Hama[50] and Giraph[22] have been recently introduced for better support of graph-based data analytics as compared with Apache Hadoop and Spark. The BSP[56] data processing paradigm, which they proposed, strays from the data locality principle used in all former data analytics paradigms. This makes typi- cal performance enhancements for distributed data analytics, such as, network traffic avoidance and effective caching difficult or impossible. In this work, we propose a scalable, unified, caching and storage platform for data analytics, called Gluon. Our unified platform provides performance, robustness and ease of use for any data analytics paradigm currently in use with little or no modifications. Gluon comes with two essential services for integration of platform support for all types of data analytics. First, our Gluon caching layer supports global collaborative caching across the memories of all participating compute (and storage) nodes. Second, Gluon supports full integration of the collaborative caching service with traditional consolidated storage back-end services. With Gluon we emphasize the principle of data locality for in-memory data on any compute node. At the same time, we take full advantage of fast remote memory access when opportunities for memory availability in collaborating nodes exist. Such opportu- nities may be present due to a variety of reasons. For example, compute nodes may be temporarily idle due to imperfect load balancing, such as created by fault-induced strag- glers, or skewed workloads. Furthermore, unused memory may be available on back-end storage nodes, which can be leveraged by compute nodes. Whenever data would be normally evicted from the local in-memory cache on any compute node, Gluon has the capability to push the data to be evicted to a remote node. Conversely, Gluon fetches remote in-memory data on-demand from collaborative nodes upon subsequent local access. We currently opt for disjoint caching
Load more

Recommended publications
  • Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks
    Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks The MIT Faculty has made this article openly available. Please share how this access benefits you. Your story matters. Citation Haoyuan Li, Ali Ghodsi, Matei Zaharia, Scott Shenker, and Ion Stoica. 2014. Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks. In Proceedings of the ACM Symposium on Cloud Computing (SOCC '14). ACM, New York, NY, USA, Article 6 , 15 pages. As Published http://dx.doi.org/10.1145/2670979.2670985 Publisher Association for Computing Machinery (ACM) Version Author's final manuscript Citable link http://hdl.handle.net/1721.1/101090 Terms of Use Creative Commons Attribution-Noncommercial-Share Alike Detailed Terms http://creativecommons.org/licenses/by-nc-sa/4.0/ Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks Haoyuan Li Ali Ghodsi Matei Zaharia Scott Shenker Ion Stoica University of California, Berkeley MIT, Databricks University of California, Berkeley fhaoyuan,[email protected] [email protected] fshenker,[email protected] Abstract Even replicating the data in memory can lead to a signifi- cant drop in the write performance, as both the latency and Tachyon is a distributed file system enabling reliable data throughput of the network are typically much worse than sharing at memory speed across cluster computing frame- that of local memory. works. While caching today improves read workloads, Slow writes can significantly hurt the performance of job writes are either network or disk bound, as replication is pipelines, where one job consumes the output of another. used for fault-tolerance. Tachyon eliminates this bottleneck These pipelines are regularly produced by workflow man- by pushing lineage, a well-known technique, into the storage agers such as Oozie [4] and Luigi [7], e.g., to perform data layer.
    [Show full text]
  • Using Alluxio to Optimize and Improve Performance of Kubernetes-Based Deep Learning in the Cloud
    Using Alluxio to Optimize and Improve Performance of Kubernetes-Based Deep Learning in the Cloud Featuring Alibaba Cloud Container Service Team Case Study What’s Inside 1 / New Trends in AI: Kubernetes-based This article presents the collaborative work of Deep Learning in the Cloud Alibaba, Alluxio, and Nanjing University in tackling the problem of Artificial Intelligence and Deep Learning 2 / Container and Data Orchestration model training in the cloud. We adopted a hybrid Based Architecture solution with a data orches-tration layer that connects 3 / Training in the Cloud - The First Take at private data centers to cloud platforms in a Alluxio Distributed Cache containerized environment. Various perfor-mance bottlenecks are analyzed with detailed optimizations of 4 / Performance Optimization of Model each component in the architecture. Our goal was to Training on the Cloud reduce the cost and complexity of data access for Deep 5 / Summary and Future Work Learning training in a hybrid environment, which resulted in over 40% reduction in training time and cost. © Copyright Alluxio, Inc. All rights reserved. Alluxio is a trademark of Alluxio, Inc. WHITEPAPER 1 / New trends in AI: Kubernetes-Based Deep Learning in the Cloud Background The rising popularity of artificial intelligence (AI) and deep learning (DL), where artificial neural networks are trained with increasingly massive amounts of data, continues to drive innovative solutions to improve data processing. Distributed DL model training can take advantage of multiple technologies, such as: • Cloud computing for elastic and scalable infrastructure • Docker for isolation and agile iteration via containers and Kubernetes for orchestrating the deployment of containers • Accelerated computing hardware, such as GPUs The merger of these technologies as a combined solution is emerging as the industry trend for DL training.
    [Show full text]
  • Alluxio Overview
    Alluxio Overview What’s Inside 1 / Introduction 2 / Data Access Challenges 3 / Benefits of Alluxio 4 / How Alluxio Works 5 / Enabling Compute and Storage Separation 6 / Use Cases 7 / Deployment © Copyright Alluxio, Inc. All rights reserved. Alluxio is a trademark of Alluxio, Inc. WHITEPAPER 1 / Introduction Alluxio is an open source software that connects analytics applications to heterogeneous data sources through a distribut- ed caching layer that sits between compute and storage. It runs on commodity hardware, creating a shared data layer abstracting the files or objects in underlying persistent storage systems. Applications connect to Alluxio via a standard interface, accessing data from a single unified source. Application Interface: Alluxio API / Hadoop Compatible / S3 Compatible / REST / FUSE Storage Interface: Any Object / File Storage WHITEPAPER / 2 2 / Data Access Challenges Organizations face a range of challenges while striving to extract value from data. Alluxio provides innovation at the data layer to abstract complexity, unify data, and intelligently manage data. This approach enables a new way to interact with data and connect the applications and people doing the work to the data sources, regardless of format or location. Doing so provides a solution to a range of challenges, for example: · Lack of access to data stored in storage silos across different departments and locations, on-premise and in the cloud · Difficulty in sharing data with multiple applications · Each application and storage system has its own interface and data exists in a wide range of formats · Data is often stored in clouds or remote locations with network latency slowing performance and impacting the freshness of the data · Storage is often tightly coupled with compute making it difficult to scale and manage storage independently / 3 3 / Benefits of Alluxio Alluxio helps overcome the obstacles to extracting value from data by making it simple to give applications access to what- ever data is needed, regardless of format or location.
    [Show full text]
  • Comparison of Spark Resource Managers and Distributed File Systems
    2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom) Comparison of Spark Resource Managers and Distributed File Systems Solmaz Salehian Yonghong Yan Department of Computer Science & Engineering, Department of Computer Science & Engineering, Oakland University, Oakland University, Rochester, MI 48309-4401, United States Rochester, MI 48309-4401, United States [email protected] [email protected] Abstract— The rapid growth in volume, velocity, and variety Section 4 provides information of different resource of data produced by applications of scientific computing, management subsystems of Spark. In Section 5, Four commercial workloads and cloud has led to Big Data. Traditional Distributed File Systems (DFSs) are reviewed. Then, solutions of data storage, management and processing cannot conclusion is included in Section 6. meet demands of this distributed data, so new execution models, data models and software systems have been developed to address II. CHALLENGES IN BIG DATA the challenges of storing data in heterogeneous form, e.g. HDFS, NoSQL database, and for processing data in parallel and Although Big Data provides users with lot of opportunities, distributed fashion, e.g. MapReduce, Hadoop and Spark creating an efficient software framework for Big Data has frameworks. This work comparatively studies Apache Spark many challenges in terms of networking, storage, management, distributed data processing framework. Our study first discusses analytics, and ethics [5, 6]. the resource management subsystems of Spark, and then reviews Previous studies [6-11] have been carried out to review open several of the distributed data storage options available to Spark. Keywords—Big Data, Distributed File System, Resource challenges in Big Data.
    [Show full text]
  • Accelerate Big Data Processing (Hadoop, Spark, Memcached, & Tensorflow) with HPC Technologies
    Accelerate Big Data Processing (Hadoop, Spark, Memcached, & TensorFlow) with HPC Technologies Talk at Intel® HPC Developer Conference 2017 (SC ‘17) by Dhabaleswar K. (DK) Panda Xiaoyi Lu The Ohio State University The Ohio State University E-mail: [email protected] E-mail: [email protected] http://www.cse.ohio-state.edu/~panda http://www.cse.ohio-state.edu/~luxi Big Data Processing and Deep Learning on Modern Clusters • Multiple tiers + Workflow – Front-end data accessing and serving (Online) • Memcached + DB (e.g. MySQL), HBase, etc. – Back-end data analytics and deep learning model training (Offline) • HDFS, MapReduce, Spark, TensorFlow, BigDL, Caffe, etc. Network Based Computing Laboratory Intel® HPC Developer Conference 2017 2 Drivers of Modern HPC Cluster Architectures High Performance Interconnects - Accelerators / Coprocessors InfiniBand high compute density, high Multi-core Processors <1usec latency, 100Gbps Bandwidth> performance/watt SSD, NVMe-SSD, NVRAM >1 TFlop DP on a chip • Multi-core/many-core technologies • Remote Direct Memory Access (RDMA)-enabled networking (InfiniBand and RoCE) • Solid State Drives (SSDs), Non-Volatile Random-Access Memory (NVRAM), NVMe-SSD • Accelerators (NVIDIA GPGPUs and Intel Xeon Phi) Tianhe – 2 Titan Stampede Tianhe – 1A Network Based Computing Laboratory Intel® HPC Developer Conference 2017 3 Interconnects and Protocols in OpenFabrics Stack for HPC (http://openfabrics.org) Application / Middleware Application / Middleware Interface Sockets Verbs Protocol Kernel Space TCP/IP TCP/IP
    [Show full text]
  • Alluxio: a Virtual Distributed File System by Haoyuan Li A
    Alluxio: A Virtual Distributed File System by Haoyuan Li A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the Graduate Division of the University of California, Berkeley Committee in charge: Professor Ion Stoica, Co-chair Professor Scott Shenker, Co-chair Professor John Chuang Spring 2018 Alluxio: A Virtual Distributed File System Copyright 2018 by Haoyuan Li 1 Abstract Alluxio: A Virtual Distributed File System by Haoyuan Li Doctor of Philosophy in Computer Science University of California, Berkeley Professor Ion Stoica, Co-chair Professor Scott Shenker, Co-chair The world is entering the data revolution era. Along with the latest advancements of the Inter- net, Artificial Intelligence (AI), mobile devices, autonomous driving, and Internet of Things (IoT), the amount of data we are generating, collecting, storing, managing, and analyzing is growing ex- ponentially. To store and process these data has exposed tremendous challenges and opportunities. Over the past two decades, we have seen significant innovation in the data stack. For exam- ple, in the computation layer, the ecosystem started from the MapReduce framework, and grew to many different general and specialized systems such as Apache Spark for general data processing, Apache Storm, Apache Samza for stream processing, Apache Mahout for machine learning, Ten- sorflow, Caffe for deep learning, Presto, Apache Drill for SQL workloads. There are more than a hundred popular frameworks for various workloads and the number is growing. Similarly, the storage layer of the ecosystem grew from the Apache Hadoop Distributed File System (HDFS) to a variety of choices as well, such as file systems, object stores, blob stores, key-value systems, and NoSQL databases to realize different tradeoffs in cost, speed and semantics.
    [Show full text]
  • Intel Select Solutions for HPC & AI Converged Clusters
    Solution Brief Intel® Select Solutions High Performance Computing and AI March 2021 Intel Select Solutions for HPC & AI Converged Clusters Intel Select Solutions deliver the compute-intensive resources needed to run artificial-intelligence (AI) workloads on existing high-performance computing (HPC) clusters. Enterprises are looking to simulation and modeling, artificial intelligence (AI), and big data analytics to help them achieve breakthrough discoveries and innovation. They understand these workloads benefit from a high-performance computing (HPC) infrastructure, yet they might still believe that separate HPC, AI, and big data clusters are the best choice for running these workloads. Contributing to this belief are two challenges. The first challenge is a fundamental difference in how workloads request resources and how HPC systems allocate them. AI and analytics workloads request compute resources dynamically, an approach that isn’t compatible with batch scheduling software used to allocate system resources in HPC clusters. The second challenge is the pattern of using computing systems based on graphics processing units (GPUs) as dedicated solutions for AI workloads. Enterprises might not realize that adding these workloads to an existing HPC cluster is feasible without the use of GPUs. Enterprises can deliver the compute infrastructure needed by AI workloads, with high levels of performance and cost-effectiveness, without adding the complexity of managing separate, dedicated systems. What they need is the ability to run HPC, big data analytics, and AI workloads within the same HPC infrastructure. They also need optimized resource scheduling that helps save time and reduce computing costs. Creating a converged platform to run HPC, AI, and analytics workloads in a single cluster infrastructure supports breakthrough innovation.
    [Show full text]
  • Best Practices
    Cloud Container Engine Best Practices Issue 01 Date 2021-09-07 HUAWEI TECHNOLOGIES CO., LTD. Copyright © Huawei Technologies Co., Ltd. 2021. All rights reserved. No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd. Trademarks and Permissions and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders. Notice The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied. The information in this document is subject to change without notice. Every effort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied. Issue 01 (2021-09-07) Copyright © Huawei Technologies Co., Ltd. i Cloud Container Engine Best Practices Contents Contents 1 Checklist for Deploying Containerized Applications in the Cloud...............................1 2 Cluster.......................................................................................................................................
    [Show full text]
  • Crystal: a Unified Cache Storage System for Analytical Databases
    Crystal: A Unified Cache Storage System for Analytical Databases Dominik Durner∗ Badrish Chandramouli Yinan Li Technische Universität München Microsoft Research Microsoft Research [email protected] [email protected] [email protected] ABSTRACT 1.1 Challenges Cloud analytical databases employ a disaggregated storage model, These caching solutions usually operate as a black-box at the file or where the elastic compute layer accesses data persisted on remote block level for simplicity, employing standard cache replacement cloud storage in block-oriented columnar formats. Given the high policies such as LRU to manage the cache. In spite of their sim- latency and low bandwidth to remote storage and the limited size of plicity, these solutions have not solved several architectural and fast local storage, caching data at the compute node is important and performance challenges for cloud databases: has resulted in a renewed interest in caching for analytics. Today, • Every DBMS today implements its own caching layer tailored to each DBMS builds its own caching solution, usually based on file- its specific requirements, resulting in a lot of work duplication or block-level LRU. In this paper, we advocate a new architecture of across systems, reinventing choices such as what to cache, where a smart cache storage system called Crystal, that is co-located with to cache, when to cache, and how to cache. compute. Crystal’s clients are DBMS-specific “data sources” with • Databases increasingly support analytics over raw data formats push-down predicates. Similar in spirit to a DBMS, Crystal incorpo- such as CSV and JSON, and row-oriented binary formats such as rates query processing and optimization components focusing on Apache Avro [6] – all very popular in the data lake [16].
    [Show full text]
  • A Solution for Bigdata Integrate with Ceph Inwinstack / Chief Architect Thor Chin Agenda
    A solution for BigData integrate with Ceph inwinSTACK / Chief Architect Thor Chin Agenda Why we need Ceph? Why we need to integrate Ceph with Big Data? How to integrate Ceph with Big Data? What’s Alluxio? Alluxio User List Architecture for Ceph integrate with Big Data Example Why we need Ceph? Distributed storage system - Fault tolerant , no SPoF X86 Commodity hardware - Saving you costs, giving you flexibility Large scale – Incremental expansion - 10s to 1000s of nodes Unified storage platform - Scalable object , Block , File system. Open source – No vender lock-in Automatically balance the file system Data security - with 2 or more copies in different physical store media. Why we need to integrate Ceph with Big Data? Separate Application and Data Better Architecture to solve Hadoop Name Node HA Issue Scalable storage solution High Performance How to integrate Ceph with Big Data? Through an Interface Alluxio: A Virtual Distributed Storage System What’s Alluxio Unified Access Thriving Community High Performance Alluxio is the unified access point Start from UC Berkeley Open Alluxio's memory-centric for your data. Applications only Source Project – Tachyon since architecture enables data access need to connect with Alluxio to 2013 Apr. released 16 versions orders of magnitudes faster than access data in all your under and rename to Alluxio release 1.0 existing solutions. storage systems. version in 2016 Feb. Fastest growing open source project in big data history with more than 200 contributors from over 50 organizations.
    [Show full text]
  • High Performance File System and I/O Middleware Design for Big Data on HPC Clusters
    High Performance File System and I/O Middleware Design for Big Data on HPC Clusters Dissertation Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By Nusrat Sharmin Islam, M.Sc. Graduate Program in Computer Science and Engineering The Ohio State University 2016 Dissertation Committee: Dr. Dhabaleswar K. (DK) Panda, Advisor Dr. Ponnuswamy Sadayappan Dr. Radu Teodorescu Dr. Xiaoyi Lu c Copyright by Nusrat Sharmin Islam 2016 Abstract Hadoop MapReduce and Spark are the two most popular Big Data processing frame- works of the recent time and Hadoop Distributed File System (HDFS) is the underlying file system for MapReduce, Spark, Hadoop Database, HBase as well as SQL query en- gines like Hive and Spark SQL. HDFS along with these upper-level middleware is now extensively being used on High Performance Computing (HPC) systems. The large-scale HPC systems, by necessity, are equipped with high performance interconnects like Infini- Band, heterogeneous storage devices like RAM Disk, SSD, HDD, and parallel file systems like Lustre. Non-Volatile Memory (NVM) is emerging and making its way into the HPC systems. Hence, the performance of HDFS, and in turn, the upper-level middleware and applications, is heavily dependent upon how it has been designed and optimized to take the system resources and architecture into account. But HDFS was initially designed to run over low-speed interconnects and disks on commodity clusters. As a result, it cannot efficiently utilize the resources available in HPC systems. For example, HDFS uses Java Socket for communication that leads to the overhead of multiple data copies and offers little overlapping among different phases of operations.
    [Show full text]
  • An Architecture for Fast and General Data Processing on Large Clusters
    An Architecture for Fast and General Data Processing on Large Clusters Matei Zaharia Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No. UCB/EECS-2014-12 http://www.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-12.html February 3, 2014 Copyright © 2014, by the author(s). All rights reserved. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission. An Architecture for Fast and General Data Processing on Large Clusters by Matei Alexandru Zaharia A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the GRADUATE DIVISION of the UNIVERSITY OF CALIFORNIA, BERKELEY Committee in charge: Professor Scott Shenker, Chair Professor Ion Stoica Professor Alexandre Bayen Professor Joshua Bloom Fall 2013 An Architecture for Fast and General Data Processing on Large Clusters Copyright c 2013 by Matei Alexandru Zaharia Abstract An Architecture for Fast and General Data Processing on Large Clusters by Matei Alexandru Zaharia Doctor of Philosophy in Computer Science University of California, Berkeley Professor Scott Shenker, Chair The past few years have seen a major change in computing systems, as growing data volumes and stalling processor speeds require more and more applications to scale out to distributed systems.
    [Show full text]