Performance Evaluation of a Distributed Storage Service in Community Network Clouds
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Globalfs: a Strongly Consistent Multi-Site File System
GlobalFS: A Strongly Consistent Multi-Site File System Leandro Pacheco Raluca Halalai Valerio Schiavoni University of Lugano University of Neuchatelˆ University of Neuchatelˆ Fernando Pedone Etienne Riviere` Pascal Felber University of Lugano University of Neuchatelˆ University of Neuchatelˆ Abstract consistency, availability, and tolerance to partitions. Our goal is to ensure strongly consistent file system operations This paper introduces GlobalFS, a POSIX-compliant despite node failures, at the price of possibly reduced geographically distributed file system. GlobalFS builds availability in the event of a network partition. Weak on two fundamental building blocks, an atomic multicast consistency is suitable for domain-specific applications group communication abstraction and multiple instances of where programmers can anticipate and provide resolution a single-site data store. We define four execution modes and methods for conflicts, or work with last-writer-wins show how all file system operations can be implemented resolution methods. Our rationale is that for general-purpose with these modes while ensuring strong consistency and services such as a file system, strong consistency is more tolerating failures. We describe the GlobalFS prototype in appropriate as it is both more intuitive for the users and detail and report on an extensive performance assessment. does not require human intervention in case of conflicts. We have deployed GlobalFS across all EC2 regions and Strong consistency requires ordering commands across show that the system scales geographically, providing replicas, which needs coordination among nodes at performance comparable to other state-of-the-art distributed geographically distributed sites (i.e., regions). Designing file systems for local commands and allowing for strongly strongly consistent distributed systems that provide good consistent operations over the whole system. -
Big Data Storage Workload Characterization, Modeling and Synthetic Generation
BIG DATA STORAGE WORKLOAD CHARACTERIZATION, MODELING AND SYNTHETIC GENERATION BY CRISTINA LUCIA ABAD DISSERTATION Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science in the Graduate College of the University of Illinois at Urbana-Champaign, 2014 Urbana, Illinois Doctoral Committee: Professor Roy H. Campbell, Chair Professor Klara Nahrstedt Associate Professor Indranil Gupta Assistant Professor Yi Lu Dr. Ludmila Cherkasova, HP Labs Abstract A huge increase in data storage and processing requirements has lead to Big Data, for which next generation storage systems are being designed and implemented. As Big Data stresses the storage layer in new ways, a better understanding of these workloads and the availability of flexible workload generators are increas- ingly important to facilitate the proper design and performance tuning of storage subsystems like data replication, metadata management, and caching. Our hypothesis is that the autonomic modeling of Big Data storage system workloads through a combination of measurement, and statistical and machine learning techniques is feasible, novel, and useful. We consider the case of one common type of Big Data storage cluster: A cluster dedicated to supporting a mix of MapReduce jobs. We analyze 6-month traces from two large clusters at Yahoo and identify interesting properties of the workloads. We present a novel model for capturing popularity and short-term temporal correlations in object re- quest streams, and show how unsupervised statistical clustering can be used to enable autonomic type-aware workload generation that is suitable for emerging workloads. We extend this model to include other relevant properties of stor- age systems (file creation and deletion, pre-existing namespaces and hierarchical namespaces) and use the extended model to implement MimesisBench, a realistic namespace metadata benchmark for next-generation storage systems. -
File System Design Approaches
File System Design Approaches Dr. Brijender Kahanwal Department of Computer Science & Engineering Galaxy Global Group of Institutions Dinarpur, Ambala, Haryana, INDIA [email protected] Abstract—In this article, the file system development design The experience with file system development is limited approaches are discussed. The selection of the file system so the research served to identify the different techniques design approach is done according to the needs of the that can be used. The variety of file systems encountered developers what are the needed requirements and show what an active area of research file system specifications for the new design. It allowed us to identify development is. The file systems researched fell into one of where our proposal fitted in with relation to current and past file system development. Our experience with file system the following four categories: development is limited so the research served to identify the 1. The file system is developed in user space and runs as a different techniques that can be used. The variety of file user process. systems encountered show what an active area of research file 2. The file system is developed in the user space using system development is. The file systems may be from one of the FUSE (File system in USEr space) kernel module and two fundamental categories. In one category, the file system is runs as a user process. developed in user space and runs as a user process. Another 3. The file system is developed in the kernel and runs as a file system may be developed in the kernel space and runs as a privileged process. -
A Decentralized Cloud Storage Network Framework
Storj: A Decentralized Cloud Storage Network Framework Storj Labs, Inc. October 30, 2018 v3.0 https://github.com/storj/whitepaper 2 Copyright © 2018 Storj Labs, Inc. and Subsidiaries This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 license (CC BY-SA 3.0). All product names, logos, and brands used or cited in this document are property of their respective own- ers. All company, product, and service names used herein are for identification purposes only. Use of these names, logos, and brands does not imply endorsement. Contents 0.1 Abstract 6 0.2 Contributors 6 1 Introduction ...................................................7 2 Storj design constraints .......................................9 2.1 Security and privacy 9 2.2 Decentralization 9 2.3 Marketplace and economics 10 2.4 Amazon S3 compatibility 12 2.5 Durability, device failure, and churn 12 2.6 Latency 13 2.7 Bandwidth 14 2.8 Object size 15 2.9 Byzantine fault tolerance 15 2.10 Coordination avoidance 16 3 Framework ................................................... 18 3.1 Framework overview 18 3.2 Storage nodes 19 3.3 Peer-to-peer communication and discovery 19 3.4 Redundancy 19 3.5 Metadata 23 3.6 Encryption 24 3.7 Audits and reputation 25 3.8 Data repair 25 3.9 Payments 26 4 4 Concrete implementation .................................... 27 4.1 Definitions 27 4.2 Peer classes 30 4.3 Storage node 31 4.4 Node identity 32 4.5 Peer-to-peer communication 33 4.6 Node discovery 33 4.7 Redundancy 35 4.8 Structured file storage 36 4.9 Metadata 39 4.10 Satellite 41 4.11 Encryption 42 4.12 Authorization 43 4.13 Audits 44 4.14 Data repair 45 4.15 Storage node reputation 47 4.16 Payments 49 4.17 Bandwidth allocation 50 4.18 Satellite reputation 53 4.19 Garbage collection 53 4.20 Uplink 54 4.21 Quality control and branding 55 5 Walkthroughs ............................................... -
Alpha Release of the Data Service
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 777533. PROviding Computing solutions for ExaScale ChallengeS D5.2 Alpha release of the Data service Start / 01 November 2017 Project: PROCESS H2020 – 777533 Duration: 36 Months Dissemination1: Public Nature2: R Due Date: 31 January 2019 Work Package: WP 5 Filename3 PROCESS_D5.2_Alpha_release_of_the_Data_service_v1.0.docx ABSTRACT During the first 15 months of its implementation, PROCESS has progressed from architecture design based on use cases’ requirements (D4.1) and through architecture validation again based on use cases (D4.2) towards initial implementations of computing services (D6.1) and data services - effort presented here in the deliverable D5.2. In D5.2 we provide an initial demonstrator of the data services, which works in cooperation with the computation services demonstrated in D6.1. The demonstrator is based on the design of the PROCESS data infrastructure described in D5.1. The implementation and initial integration of the infrastructure are based on use case requirements, formulated here as custom application-specific services which are part of the infrastructure. The central, connecting component of the data infrastructure is LOBCDER. It implements a micro-infrastructure of data services, based on dynamically provisioned Docker containers. Additionally to LOBCDER and use case-specific services, the data infrastructure contains generic data and metadata- handling services (DISPEL, DataNet). Finally, Cloudify integrates the micro-infrastructure and the orchestration components of WP7. 1 PU = Public; CO = Confidential, only for members of the Consortium (including the EC services). 2 R = Report; R+O = Report plus Other. -
Hands-On Linux Administration on Azure
Hands-On Linux Administration on Azure Explore the essential Linux administration skills you need to deploy and manage Azure-based workloads Frederik Vos BIRMINGHAM - MUMBAI Hands-On Linux Administration on Azure Copyright © 2018 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Vijin Boricha Acquisition Editor: Rahul Nair Content Development Editor: Nithin George Varghese Technical Editor: Komal Karne Copy Editor: Safis Editing Project Coordinator: Drashti Panchal Proofreader: Safis Editing Indexer: Mariammal Chettiyar Graphics: Tom Scaria Production Coordinator: Deepika Naik First published: August 2018 Production reference: 1310818 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78913-096-6 www.packtpub.com mapt.io Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. -
Tahoe-LAFS Documentation Release 1.X
Tahoe-LAFS Documentation Release 1.x The Tahoe-LAFS Developers January 19, 2017 Contents 1 Welcome to Tahoe-LAFS! 3 1.1 What is Tahoe-LAFS?..........................................3 1.2 What is “provider-independent security”?................................3 1.3 Access Control..............................................4 1.4 Get Started................................................4 1.5 License..................................................4 2 Installing Tahoe-LAFS 5 2.1 First: In Case Of Trouble.........................................5 2.2 Pre-Packaged Versions..........................................5 2.3 Preliminaries...............................................5 2.4 Install the Latest Tahoe-LAFS Release.................................6 2.5 Running the tahoe executable.....................................8 2.6 Running the Self-Tests..........................................8 2.7 Common Problems............................................9 2.8 Using Tahoe-LAFS............................................9 3 How To Run Tahoe-LAFS 11 3.1 Introduction............................................... 11 3.2 Do Stuff With It............................................. 12 3.3 Socialize................................................. 13 3.4 Complain................................................. 13 4 Configuring a Tahoe-LAFS node 15 4.1 Node Types................................................ 16 4.2 Overall Node Configuration....................................... 16 4.3 Connection Management........................................ -
Porting FUSE to L4re
Großer Beleg Porting FUSE to L4Re Florian Pester 23. Mai 2013 Technische Universit¨at Dresden Fakult¨at Informatik Institut fur¨ Systemarchitektur Professur Betriebssysteme Betreuender Hochschullehrer: Prof. Dr. rer. nat. Hermann H¨artig Betreuender Mitarbeiter: Dipl.-Inf. Carsten Weinhold Erkl¨arung Hiermit erkl¨are ich, dass ich diese Arbeit selbstst¨andig erstellt und keine anderen als die angegebenen Hilfsmittel benutzt habe. Declaration I hereby declare that this thesis is a work of my own, and that only cited sources have been used. Dresden, den 23. Mai 2013 Florian Pester Contents 1. Introduction 1 2. State of the Art 3 2.1. FUSE on Linux . .3 2.1.1. FUSE Internal Communication . .4 2.2. The L4Re Virtual File System . .5 2.3. Libfs . .5 2.4. Communication and Access Control in L4Re . .6 2.5. Related Work . .6 2.5.1. FUSE . .7 2.5.2. Pass-to-Userspace Framework Filesystem . .7 3. Design 9 3.1. FUSE Server parts . 11 4. Implementation 13 4.1. Example Request handling . 13 4.2. FUSE Server . 14 4.2.1. LibfsServer . 14 4.2.2. Translator . 14 4.2.3. Requests . 15 4.2.4. RequestProvider . 15 4.2.5. Node Caching . 15 4.3. Changes to the FUSE library . 16 4.4. Libfs . 16 4.5. Block Device Server . 17 4.6. File systems . 17 5. Evaluation 19 6. Conclusion and Further Work 25 A. FUSE operations 27 B. FUSE library changes 35 C. Glossary 37 V List of Figures 2.1. The architecture of FUSE on Linux . .3 2.2. The architecture of libfs . -
Paratrac: a Fine-Grained Profiler for Data-Intensive Workflows
ParaTrac: A Fine-Grained Profiler for Data-Intensive Workflows Nan Dun Kenjiro Taura Akinori Yonezawa Department of Computer Department of Information and Department of Computer Science Communication Engineering Science The University of Tokyo The University of Tokyo The University of Tokyo 7-3-1 Hongo, Bunkyo-Ku 7-3-1 Hongo, Bunkyo-Ku 7-3-1 Hongo, Bunkyo-Ku Tokyo, 113-5686 Japan Tokyo, 113-5686 Japan Tokyo, 113-5686 Japan [email protected] [email protected] [email protected] tokyo.ac.jp tokyo.ac.jp tokyo.ac.jp ABSTRACT 1. INTRODUCTION The realistic characteristics of data-intensive workflows are With the advance of high performance distributed com- critical to optimal workflow orchestration and profiling is puting, users are able to execute various data-intensive an effective approach to investigate the behaviors of such applications by harnessing massive computing resources [1]. complex applications. ParaTrac is a fine-grained profiler Though workflow management systems have been developed for data-intensive workflows by using user-level file system to alleviate the difficulties of planning, scheduling, and exe- and process tracing techniques. First, ParaTrac enables cuting complex workflows in distributed environments [2{5], users to quickly understand the I/O characteristics of from optimal workflow management still remains a challenge entire application to specific processes or files by examining because of the complexity of applications. Therefore, one low-level I/O profiles. Second, ParaTrac automatically of important and practical demands is to understand and exploits fine-grained data-processes interactions in workflow characterize the data-intensive applications to help workflow to help users intuitively and quantitatively investigate management systems (WMS) refine their orchestration for realistic execution of data-intensive workflows. -
The Quantcast File System
The Quantcast File System Michael Ovsiannikov Silvius Rus Damian Reeves Quantcast Quantcast Google movsiannikov@quantcast [email protected] [email protected] Paul Sutter Sriram Rao Jim Kelly Quantcast Microsoft Quantcast [email protected] [email protected] [email protected] ABSTRACT chine writing it, another on the same rack, and a third on a The Quantcast File System (QFS) is an efficient alterna- distant rack. Thus HDFS is network efficient but not par- tive to the Hadoop Distributed File System (HDFS). QFS is ticularly storage efficient, since to store a petabyte of data, written in C++, is plugin compatible with Hadoop MapRe- it uses three petabytes of raw storage. At today’s cost of duce, and offers several efficiency improvements relative to $40,000 per PB that means $120,000 in disk alone, plus ad- HDFS: 50% disk space savings through erasure coding in- ditional costs for servers, racks, switches, power, cooling, stead of replication, a resulting doubling of write through- and so on. Over three years, operational costs bring the put, a faster name node, support for faster sorting and log- cost close to $1 million. For reference, Amazon currently ging through a concurrent append feature, a native com- charges $2.3 million to store 1 PB for three years. mand line client much faster than hadoop fs, and global Hardware evolution since then has opened new optimiza- feedback-directed I/O device management. As QFS works tion possibilities. 10 Gbps networks are now commonplace, out of the box with Hadoop, migrating data from HDFS and cluster racks can be much chattier. -
Extendable Storage Framework for Reliable Clustered Storage Systems by Sumit Narayan B.E., University of Madras, 2002 M.S., University of Connecticut, 2004
Extendable Storage Framework for Reliable Clustered Storage Systems Sumit Narayan, Ph.D. University of Connecticut, 2010 The total amount of information stored on disks has increased tremendously in re cent years with data storage, sharing and backup becoming more important than ever. The demand for storage has not only changed in size, but also in speed, reli ability and security. These requirements not only create a big challenge for storage administrators who must decide on several aspects of storage policy with respect to provisioning backups, retention, redundancy, security, performance, etc. but also for storage system architects who must aim for a one system fits all design. Storage poli cies like backup and security are typically set by system administrators for an entire file system, logical volume or storage pool. However, this granularity is too large and can sacrifice storage efficiency and performance - particularly since different files have different storage requirements. In the same context, clustered storage systems that are typically used for data storage or as file servers, provide very high performance and maximum scalability by striping data across multiple nodes. However, high num ber of storage nodes in such large systems also raises concerns for reliability in terms of loss of data due to failed nodes, or corrupt blocks on disk drives. Redundancy techniques similar to RAID across these nodes are not common because of the high overhead incurred owing to the parity calculations for all the files present on the file system. In the same way, data integrity checks are often omitted or disabled in file systems to guarantee high throughput from the storage system. -
Collocated Data Deduplication for Virtual Machine Backup in the Cloud
UNIVERSITY OF CALIFORNIA Santa Barbara Collocated Data Deduplication for Virtual Machine Backup in the Cloud A Dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science by Wei Zhang Committee in Charge: Professor Tao Yang, Chair Professor Jianwen Su Professor Rich Wolski September 2014 The Dissertation of Wei Zhang is approved: Professor Jianwen Su Professor Rich Wolski Professor Tao Yang, Committee Chairperson January 2014 Collocated Data Deduplication for Virtual Machine Backup in the Cloud Copyright c 2014 by Wei Zhang iii To my family. iv Acknowledgements First of all, I want to take this opportunity to thank my advisor Professor Tao Yang for his invaluable advice and instructions to me on selecting interesting and challenging research topics, identifying specific problems for each research phase, as well as prepar- ing for my future career. I would also like to thank my committee members, Professor Rich Wolski and Professor Jianwen Su, for their pertinent and helpful instructions on my research work. I also want to thank members and ex-members of my research group, Michael Agun, Gautham Narayanasamy, Prakash Chandrasekaran, Xiaofei Du, and Hong Tang, for their incessant support in the forms of technical discussion, system co-development, cluster administration, and other research activities. I owe my deepest gratitude to my parents, for their love and support, which give me the confidence and energy to overcome all past, current, and future difficulties. Finally and most importantly, I would like to express my gratitude beyond words to my wife, Jiayin, who has been encouraging and inspiring me since we met in love.