DOCSLIB.ORG

Cloudpools and Google Cloud Architectural Overview, Considerations, and Best Practices

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Cloudpools and Google Cloud Architectural Overview, Considerations, and Best Practices Technical White Paper Dell EMC PowerScale: CloudPools and Google Cloud Architectural overview, considerations, and best practices Abstract This white paper provides an overview of Dell EMC™ PowerScale™ CloudPools software in OneFS™ 9.1.0.0. It describes its policy-based capabilities that can reduce storage costs and optimize storage by automatically moving infrequently accessed data to Google Cloud. April 2021 H17993.3 Revisions Revisions Date Description October 2019 Initial release June 2020 Updated best practices October 2020 Updated CloudPools operations April 2021 Updated best practices Acknowledgments Author: Jason He ([email protected]) Dell EMC and the authors of this document welcome your feedback on this white paper. This document may contain certain words that are not consistent with Dell's current language guidelines. Dell plans to update the document over subsequent future releases to revise these words accordingly. This document may contain language from third party content that is not under Dell's control and is not consistent with Dell's current guidelines for Dell's own content. When such third party content is updated by the relevant third parties, this document will be revised accordingly. The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any software described in this publication requires an applicable software license. Copyright © 2019 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. [4/25/2021] [Technical White Paper] [H17993.3] 2 Dell EMC PowerScale: CloudPools and Google Cloud | H17993.3 Table of contents Table of contents Revisions............................................................................................................................................................................. 2 Acknowledgments ............................................................................................................................................................... 2 Table of contents ................................................................................................................................................................ 3 Executive summary ............................................................................................................................................................. 5 Audience ............................................................................................................................................................................. 5 1 CloudPools solution architectural overview .................................................................................................................. 6 1.1 PowerScale ......................................................................................................................................................... 6 1.1.1 SmartPools ......................................................................................................................................................... 7 1.1.2 SmartLink files .................................................................................................................................................... 7 1.1.3 File pool policies ................................................................................................................................................. 7 1.2 Google Cloud ...................................................................................................................................................... 9 1.2.1 Cloud metadata object ........................................................................................................................................ 9 1.2.2 Cloud data object ................................................................................................................................................ 9 1.3 CloudPools operations ....................................................................................................................................... 9 1.3.1 Archive ................................................................................................................................................................ 9 1.3.2 Recall ................................................................................................................................................................ 11 1.3.3 Read ................................................................................................................................................................. 11 1.3.4 Update .............................................................................................................................................................. 12 2 CloudPools 2.0 ........................................................................................................................................................... 14 2.1 AWS signature v4 authentication support ........................................................................................................ 14 2.2 NDMP and SyncIQ support .............................................................................................................................. 15 2.3 Non-disruptive upgrade support ....................................................................................................................... 15 2.4 Snapshot efficiency .......................................................................................................................................... 15 2.4.1 Scenario 1 ......................................................................................................................................................... 17 2.4.2 Scenario 2 ......................................................................................................................................................... 17 2.4.3 Scenario 3 ......................................................................................................................................................... 18 2.4.4 Scenario 4 ......................................................................................................................................................... 18 2.4.5 Scenario 5 ......................................................................................................................................................... 19 2.5 Sparse files handling ........................................................................................................................................ 20 2.6 Quota management .......................................................................................................................................... 20 2.7 Anti-virus integration ......................................................................................................................................... 21 2.8 WORM integration ............................................................................................................................................ 21 3 Best practices for PowerScale storage and Google Cloud ........................................................................................ 22 3.1 PowerScale configuration ................................................................................................................................. 22 3 Dell EMC PowerScale: CloudPools and Google Cloud | H17993.3 Table of contents 3.1.1 CloudPools settings .......................................................................................................................................... 22 3.1.2 File pool policy .................................................................................................................................................. 23 3.1.3 Other considerations ........................................................................................................................................ 23 3.2 Google Cloud configuration .............................................................................................................................. 25 3.3 Protecting SmartLink files ................................................................................................................................. 25 3.3.1 SyncIQ .............................................................................................................................................................. 25 3.3.2 NDMP ............................................................................................................................................................... 26 3.4 Performance ..................................................................................................................................................... 27 4 Reporting .................................................................................................................................................................... 28 4.1 CloudPools network stats ................................................................................................................................. 28 4.2 Query network stats by CloudPools account ...................................................................................................
Load more

Recommended publications
  • Google Cloud Issue Summary Multiple Products - 2020-08-19 All Dates/Times Relative to US/Pacific
    Google Cloud Issue Summary Multiple Products - 2020-08-19 All dates/times relative to US/Pacific Starting on August 19, 2020, from 20:55 to 03:30, multiple G Suite and Google Cloud Platform products experienced errors, unavailability, and delivery delays. Most of these issues involved creating, uploading, copying, or delivering content. The total incident duration was 6 hours and 35 minutes, though the impact period differed between products, and impact was mitigated earlier for most users and services. We understand that this issue has impacted our valued customers and users, and we apologize to those who were affected. DETAILED DESCRIPTION OF IMPACT Starting on August 19, 2020, from 20:55 to 03:30, Google Cloud services exhibited the following issues: ● Gmail: The Gmail service was unavailable for some users, and email delivery was delayed. About ​ 0.73% of Gmail users (both consumer and G Suite) active within the preceding seven days experienced 3 or more availability errors during the outage period. G Suite customers accounted for 27% of affected Gmail users. Additionally, some users experienced errors when adding attachments to messages. Impact on Gmail was mitigated by 03:30, and all messages delayed by this incident have been delivered. ● Drive: Some Google Drive users experienced errors and elevated latency. Approximately 1.5% of Drive ​ users (both consumer and G Suite) active within the preceding 24 hours experienced 3 or more errors during the outage period. ● Docs and Editors: Some Google Docs users experienced issues with image creation actions (for ​ example, uploading an image, copying a document with an image, or using a template with images).
    [Show full text]
  • System and Organization Controls (SOC) 3 Report Over the Google Cloud Platform System Relevant to Security, Availability, and Confidentiality
    System and Organization Controls (SOC) 3 Report over the Google Cloud Platform System Relevant to Security, Availability, and Confidentiality For the Period 1 May 2020 to 30 April 2021 Google LLC 1600 Amphitheatre Parkway Mountain View, CA, 94043 650 253-0000 main Google.com Management’s Report of Its Assertions on the Effectiveness of Its Controls Over the Google Cloud Platform System Based on the Trust Services Criteria for Security, Availability, and Confidentiality We, as management of Google LLC ("Google" or "the Company") are responsible for: • Identifying the Google Cloud Platform System (System) and describing the boundaries of the System, which are presented in Attachment A • Identifying our service commitments and system requirements • Identifying the risks that would threaten the achievement of its service commitments and system requirements that are the objectives of our System, which are presented in Attachment B • Identifying, designing, implementing, operating, and monitoring effective controls over the Google Cloud Platform System (System) to mitigate risks that threaten the achievement of the service commitments and system requirements • Selecting the trust services categories that are the basis of our assertion We assert that the controls over the System were effective throughout the period 1 May 2020 to 30 April 2021, to provide reasonable assurance that the service commitments and system requirements were achieved based on the criteria relevant to security, availability, and confidentiality set forth in the AICPA’s
    [Show full text]
  • Building a Cloud-Enabled File Storage Infrastructure
    F5 White Paper Building a Cloud-Enabled File Storage Infrastructure A cloud-enabled infrastructure can help your organization seamlessly integrate cloud storage and maximize cost savings, while also offering significant benefits to your traditional file storage environments. by Renny Shen Product Marketing Manager White Paper Building a Cloud-Enabled File Storage Infrastructure Contents Introduction 3 What Makes a Cloud? 3 Types of Cloud Storage 4 What Makes Cloud Storage Different? 4 Accessing Files Remotely over the Network 5 Accessing Files on Object-Based Storage 5 Unique Cost Structure 6 Where Clouds Make Sense 7 Fitting the Cloud into a Tiered Storage Framework 7 Expanding the Parameters for Tiering with the Cloud 8 Defining Cloud-Enabled 9 Integrating Different Types of Storage 10 Non-Disruptive File Migration 11 Automated Storage Tiering 11 Benefits of a Cloud-Enabled Infrastructure 12 Reduced Storage Costs 12 Reduced Backup Times and Costs 13 Reduced Operational Costs 13 The F5 Cloud Storage Model 13 Creating a Private Cloud 15 Conclusion 18 2 White Paper Building a Cloud-Enabled File Storage Infrastructure Introduction Cloud storage offers enterprise organizations the opportunity to bring constantly rising file storage costs and management burden under control. By moving appropriate types of files to the cloud, organizations can reduce not only the amount of storage capacity that they need to purchase, but also the operational overhead involved in managing it. In addition, the cloud enables storage capacity to be increased on demand, while charging organizations only for the amount of storage that is actually utilized. Cloud storage will bring many changes to the way enterprises manage storage.
    [Show full text]
  • Hybrid Cloud Storage with Cloudian Hyperstore® and Amazon S3
    SOLUTION BRIEF Hybrid Cloud Storage with Cloudian HyperStore® and Amazon S3 NEW DATA STORAGE CHALLENGES With the popularity of rich media, the nt,%and%the%proli.eraon%o.%mobile%de8ices2%there%has%bee digiN5aon%o.%conte n% e:ponenNal%growth%in%the%amount o.%unstructured%data that IT%is%tasked%with%managing?and%the%rate%o.%this%growth%is% only%accelerang.%TradiNonal%networked%storage%approaches2%such%as%SAC%and%CAS2%.ail%to%meet the%per.ormance%and% throughput demands% o.% this% new% generaon% o.% IT% as% they% lack% the% scalability2% De:ibility% and% agility% which% modern% business%reEuires.%In%.act,%whole%legacy%storage%systems%are%breaking%down.%Fackups%and%restores%take%longer.%Migraon% to%new%storage%systems%is%labor%intensi8e.%And%the%list goes%on%and%on.%As%a result,%many%enterprise%organi5aons%are% mo8ing%to%hybrid%IT%en8ironments2%combining%the%De:ibility%and%scale%o.%the%cloud%with%the%security%and%control%o.%their% on-­‐premises%IT%capabiliNes.%In%this%respect,%Iloudian%and%Amazon%are%changing%the%game.%% HYBRID CLOUD SOLUTION BENEFITS Iloudian% HyperStore% soKware% pro8ides% cost-­‐eLecN8e% SbOect storage% has% changed% the% li8es% o.% enterprise% pri8ate% cloud% storage% which% deploys% on% commodity% customers2% sa8ing% them% Nme% and% money% while% hardware%housed%within%enterprise%IT%data centers%and% increasing% per.ormance% and% peace% o.% mind.% Tey% seamlessly% integrates% with% the% oL-­‐premises% Amazon% benePts%o.%using%Iloudian%and%Amazon%to%build%hybrid% cloud% in.rastructure.% IT% managers% can% dynamically% cloud9%"nclude: control%bi-­‐direcNonal%data Nering%to%and%.rom%Amazon% • Impro8ed% security% and% per.ormance% by% using% on-­‐ S3% and% Nlacier% at the% bucket le8el2% 8ia bucket li.ecycle% premises%cloud%storage%.or%criNcal%business%content rules.%Fucket rules%can%also%be%set up%to%automacally% and% the% most cost eLecN8e% and% highly% reliable% oL-­‐ e:pire%obOects%aer%a predetermined%amount o.%Nme%or% premises%cloud%storage%.or%long%term%data archi8al.% specific date.
    [Show full text]
  • Security Services Using ECDSA in Cloud Computing
    Volume 4, Issue 5, May 2014 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Security Services using ECDSA in Cloud Computing S.Sathish* D.Sumathi P.Sivaprakash Computer Science and Engineering Computer Science and Engineering Computer Science and Engineering Jaisriram Group of Institutions, India PPG Institute of Technology, India PPG Institute of Technology, India Abstract— Cloud computing security is the set of control-based technologies and policies designed to comply to the rules and regulations framed by the provider team to support and protect information, data applications and infrastructure associated with cloud computing use. Cloud computing security process should address the issues faced by the cloud users. Cloud Service Provider needs to incorporate the maintenance activity in order to provide the customer's data security, privacy and compliance with necessary regulations. The Elliptic Curve Digital Signature Algorithm (ECDSA) is a public key cryptosystem used for creation and verification of digital signatures in securing data uploaded by the cloud users. Information security concerns have been focused so that identifying unwanted modification of data, deletion of data is identified. Keywords— Cloud Computing,ECDSA,Crptography,RSA, I. INTRODUCTION Cloud computing is internet-based computing, where by shared resources, software and information are provided to computers and other devises on demand. It is a culmination of numerous attempts at large scale computing with seamless access to virtually limitless resources. Cloud Computing providers offer their services according to three fundamental models, namely Infrastructure-as-a- Service (IaaS), Platform-as-a-Service (PaaS), and Software-as-a-Service (SaaS) which is illustrated in figure 1.
    [Show full text]
  • Overview of Cloud Storage Allan Liu, Ting Yu
    Overview of Cloud Storage Allan Liu, Ting Yu To cite this version: Allan Liu, Ting Yu. Overview of Cloud Storage. International Journal of Scientific & Technology Research, 2018. hal-02889947 HAL Id: hal-02889947 https://hal.archives-ouvertes.fr/hal-02889947 Submitted on 6 Jul 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Overview of Cloud Storage Allan Liu, Ting Yu Department of Computer Science and Engineering Shanghai Jiao Tong University, Shanghai Abstract— Cloud computing is an emerging service and computing platform and it has taken commercial computing by storm. Through web services, the cloud computing platform provides easy access to the organization’s storage infrastructure and high-performance computing. Cloud computing is also an emerging business paradigm. Cloud computing provides the facility of huge scalability, high performance, reliability at very low cost compared to the dedicated storage systems. This article provides an introduction to cloud computing and cloud storage and different deployment modules. The general architecture of the cloud storage is also discussed along with its advantages and disadvantages for the organizations. Index Terms— Cloud Storage, Emerging Technology, Cloud Computing, Secure Storage, Cloud Storage Models —————————— u —————————— 1 INTRODUCTION n this era of technological advancements, Cloud computing Ihas played a very vital role in changing the way of storing 2 CLOUD STORAGE information and run applications.
    [Show full text]
  • Rclone VFS and Mergerfs Setup
    Rclone VFS and MergerFS Setup This guide is for advanced users only and it serves as a guide for you to use rclone and mergerFS. The files here are the recommended settings for our slots and will subject to change whenever there are new configurations that are appropriate for the slots. Furthermore, USB is not responsible for any data loss or application errors due to this setup should you proceed and will not provide official support for it due to the large volume of variables and different configurations possible with rclone and mergerFS. You may visit the community discord server or the software's respective forums for assistance. Please make yourself aware of the Ultra.cc Fair Usage Policy. It is very important not to mount your Cloud storage to any of the premade folders, this creates massive instability for both you and everyone else on your server. Always follow the documentation and create a new folder for mounting. It is your responsibility to ensure usage is within acceptable limits. Ignorance is not an excuse. Please do not mount to any of the default directories such as: files media bin .apps .config www /homexx/username/ or any pre-created directory found on your Ultra.cc Slot This section will teach you how to set up a rclone VFS mount and MergerFS on Ultra.cc slots, and it assumes the following: You have a working rclone setup, especially correctly configured remotes of your preferred cloud storage provider. In this tutorial, we'll be using Google Drive. If you use another cloud storage provider, change the flags that are appropriate to your setup and visit rclone documentation for more information.
    [Show full text]
  • Sebastian Witowski
    I made a website! Now what? Sebastian Witowski 1 Disclaimer There are many great tools at CERN (OpenShift). This presentation is not about those tools. 2 This presentation is about external tools 3 Free to use Open source Good value/money Good service Supports FOSS/education 4 Hosting 5 Cloud hosting Physical server vs VPS: Billed for real usage (per second on AWS) Easy to scale 6 Hosting providers AWS (EC2 and Amazon Lightsail) Free tier for 1 year (2x t2.micro, 1 DB, 5GB storage, etc.) Complex pricing model DigitalOcean From $5/month (512MB RAM, 20GB storage, 1TB transfer) Additional storage from $0.10/GB/month Linode From $5/month (1GB RAM, 20GB storage, 1TB transfer) Additional storage from $0.10/GB/month Heroku PaaS: more expensive, but easier to use (less congurable) 1 dyno free forever (512 MB RAM, 10k PostgreSQL rows) 7 Server parameters Storage - 20GB is plenty (use AWS S3 for more) Transfer - 1TB is usually more than enough CPU - 1 CPU is ne for simple website RAM - the more the better (caching, DB, etc.) 8 Cloud computing 9 When a VPS is not enough AWS Free tier for 1 year (VPS, DB, S3, Lambda and more) More features than competition Longest time on market == more 3rd party libraries Google Cloud $300 free credits to spend during rst 12 months "Always free" tier with VPS (1 f1-micro, 30GB storage), DB (1GB NoSQL), storage (5 GB), messaging, logging, serverless functions, but also NLP, speech and vision API Azure 30 days free trial with $200 free credits, some services free for 12 months and some free forever Good support for Windows applications 10 Static websites 11 When a VPS is too much GitLab pages Supports any static site generator Free Continuous Integration Slightly more complicated setup GitHub pages Supports only Jekyll (to use other static site generators, you need to precompile les locally) Easy, out-of-the-box setup Both are free and offer separate websites for projects and organizations.
    [Show full text]
  • Comparison of Availability Between Local and Cloud Storage Leviathan Security Group
    Comparison of Availability Between Local and Cloud Storage Leviathan Security Group limitless innovation. no compromise. Introduction The U.S. National Institute for Standards and Technology denes cloud computing as having ve essential characteristics: on-demand self- service for provisioning, network access, the use of virtual machines to provide servers, rapid elastic provisioning and release of servers, and metering capability on the service provided.1 This paper seeks to answer whether, from the perspective of availability of data, especially in the face of natural or manmade threats to data and access, cloud computing is able to provide a greater benet than local data storage. For the purposes of this discussion, local data storage will include arrangements where data is kept on business premises or in a nearby datacenter, such as a colocation facility or other rigidly-provisioned resource. We will rst examine the signicant dierentiator that cloud computing provides in this space, and then examine dierent scales of threats to availability to see whether local data storage or cloud data storage fares better against known historical threats. 2 The Cloud Dierentiator: Geographic Redundancy At its heart, cloud computing and cloud storage are exactly the same as computing and storage provided through a datacenter or SOHO server; the cloud is made of computers. The dierence between cloud- based storage and local storage is that using cloud-based storage allows replication between separate geographic regions. Indeed, the key feature of
    [Show full text]
  • File Gateway for Hybrid Cloud Storage Architectures AWS Whitepaper File Gateway for Hybrid Cloud Storage Architectures AWS Whitepaper
    File Gateway for Hybrid Cloud Storage Architectures AWS Whitepaper File Gateway for Hybrid Cloud Storage Architectures AWS Whitepaper File Gateway for Hybrid Cloud Storage Architectures: AWS Whitepaper Copyright © Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. File Gateway for Hybrid Cloud Storage Architectures AWS Whitepaper Table of Contents Abstract ............................................................................................................................................ 1 Abstract .................................................................................................................................... 1 Introduction ...................................................................................................................................... 2 File Gateway Architecture .................................................................................................................... 3 File to Object Mapping ............................................................................................................... 3 Read/Write Operations and Local Cache ......................................................................................
    [Show full text]
  • The Cloud Storage Standard
    Cloud Data Management Interface (CDMI) The Cloud Storage Standard Mark Carlson, SNIA TC and Oracle Chair, SNIA Cloud Storage TWG SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies and individual members may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. CDMI – The Cloud Storage Standard 2 © 2010 Storage Networking Industry Association. All Rights Reserved. Abstract The SNIA has published the CDMI Cloud Storage standard for implementation by cloud storage vendors as well as Public and Private clouds. This tutorial will overview the features of the new standard and explain how interoperability between clouds is achieved. Now that the standard is available, what should you be requiring from your cloud vendors? How can you expect this standard to roll out in implementations? Learning Objectives How this cloud storage standard can achieve interoperability and what that means to you.
    [Show full text]
  • Economic and Social Impacts of Google Cloud September 2018 Economic and Social Impacts of Google Cloud |
    Economic and social impacts of Google Cloud September 2018 Economic and social impacts of Google Cloud | Contents Executive Summary 03 Introduction 10 Productivity impacts 15 Social and other impacts 29 Barriers to Cloud adoption and use 38 Policy actions to support Cloud adoption 42 Appendix 1. Country Sections 48 Appendix 2. Methodology 105 This final report (the “Final Report”) has been prepared by Deloitte Financial Advisory, S.L.U. (“Deloitte”) for Google in accordance with the contract with them dated 23rd February 2018 (“the Contract”) and on the basis of the scope and limitations set out below. The Final Report has been prepared solely for the purposes of assessment of the economic and social impacts of Google Cloud as set out in the Contract. It should not be used for any other purposes or in any other context, and Deloitte accepts no responsibility for its use in either regard. The Final Report is provided exclusively for Google’s use under the terms of the Contract. No party other than Google is entitled to rely on the Final Report for any purpose whatsoever and Deloitte accepts no responsibility or liability or duty of care to any party other than Google in respect of the Final Report and any of its contents. As set out in the Contract, the scope of our work has been limited by the time, information and explanations made available to us. The information contained in the Final Report has been obtained from Google and third party sources that are clearly referenced in the appropriate sections of the Final Report.
    [Show full text]