DOCSLIB.ORG

WP: ECS Overview & Architecture

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
WP: ECS Overview & Architecture Technical White Paper ECS Overview and Architecture Abstract This document provides a technical overview and design of the Dell EMC™ ECS™ software-defined cloud-scale object storage platform. August 2021 h14071.19 Revisions Revisions Date Description December 2015 Initial release May 2016 Updated for 2.2.1 September 2016 Updated for 3.0 August 2017 Updated for 3.1 March 2018 Updated for 3.2 September 2018 Updated for Gen3 Hardware February 2019 Updated for 3.3 September 2019 Updated for 3.4 February 2020 Updated ECSDOC-628 changes May 2020 Updated for 3.5 November 2020 Updated for 3.6 February 2021 Updated for 3.6.1 August 2021 Updated for 3.6.2 and compression mechanism Acknowledgements This paper was produced by the following: Author: Zhu, Jarvis The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any software described in this publication requires an applicable software license. This document may contain certain words that are not consistent with Dell's current language guidelines. Dell plans to update the document over subsequent future releases to revise these words accordingly. This document may contain language from third party content that is not under Dell's control and is not consistent with Dell's current guidelines for Dell's own content. When such third-party content is updated by the relevant third parties, this document will be revised accordingly. Copyright © 2015–2021 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. [9/2/2021] [Technical White Paper] [h14071.19] 2 ECS Overview and Architecture | h14071.19 Table of contents Table of contents Revisions............................................................................................................................................................................. 2 Acknowledgements ............................................................................................................................................................. 2 Table of contents ................................................................................................................................................................ 3 Executive summary ............................................................................................................................................................. 5 1 Introduction ................................................................................................................................................................... 6 1.1 Audience ............................................................................................................................................................. 6 1.2 Scope .................................................................................................................................................................. 6 2 Value of ECS ................................................................................................................................................................ 7 3 Architecture .................................................................................................................................................................. 9 3.1 Overview ............................................................................................................................................................. 9 3.2 ECS portal and provisioning services ............................................................................................................... 10 3.3 Data services .................................................................................................................................................... 12 3.3.1 Object ............................................................................................................................................................... 12 3.3.2 HDFS ................................................................................................................................................................ 13 3.3.3 NFS ................................................................................................................................................................... 16 3.3.4 Connectors and gateways ................................................................................................................................ 16 3.4 Storage engine ................................................................................................................................................. 17 3.4.1 Storage services ............................................................................................................................................... 17 3.4.2 Data .................................................................................................................................................................. 17 3.4.3 Data management ............................................................................................................................................ 19 3.4.4 Data flow ........................................................................................................................................................... 21 3.4.5 Write optimizations for file size ......................................................................................................................... 22 3.4.6 Space reclamation ............................................................................................................................................ 22 3.4.7 SSD metadata caching ..................................................................................................................................... 23 3.4.8 Cloud DVR ........................................................................................................................................................ 24 3.5 Fabric ................................................................................................................................................................ 24 3.5.1 Node agent ....................................................................................................................................................... 24 3.5.2 Lifecycle manager............................................................................................................................................. 25 3.5.3 Registry ............................................................................................................................................................. 25 3.5.4 Event library ...................................................................................................................................................... 25 3.5.5 Hardware manager ........................................................................................................................................... 25 3.6 Infrastructure ..................................................................................................................................................... 25 3.6.1 Docker .............................................................................................................................................................. 25 4 Appliance hardware models ....................................................................................................................................... 27 3 ECS Overview and Architecture | h14071.19 Table of contents 4.1 EX-series .......................................................................................................................................................... 27 4.2 Appliance networking ....................................................................................................................................... 29 4.2.1 S5148F - front-end public switches .................................................................................................................. 29 4.2.2 S5148F - back-end private switches ................................................................................................................ 30 4.2.3 S5248F - front-end public switches .................................................................................................................. 31 4.2.4 S5248F - back-end private switches ................................................................................................................ 31 4.2.5 S5232 – aggregation switch ............................................................................................................................. 32 5 Network separation .................................................................................................................................................... 33 6 Security ....................................................................................................................................................................... 34 6.1 Authentication ................................................................................................................................................... 34 6.2 Data services authentication
Load more

Recommended publications
  • Irods and Spectrum Archive
    Integration of iRODS with IBM Spectrum Archive™ Enterprise Edition A flexible tiered storage archiving solution Version 1.1 (02/17/2020) Nils Haustein IBM European Storage Competence Center Mauro Tridici Euro-Mediterranean Center on Climate Change (CMCC) Contents Introduction ............................................................................................................................................ 3 Executive summary ............................................................................................................................. 4 Tiered storage file systems ..................................................................................................................... 4 IBM Spectrum Scale with IBM Spectrum Archive ............................................................................... 5 Challenges with tiered storage file systems ....................................................................................... 6 iRODS ...................................................................................................................................................... 6 Integration of iRODS with IBM Spectrum Archive .................................................................................. 7 Typical workflows ............................................................................................................................... 8 Preventing transparent recalls ............................................................................................................ 9 Solution
    [Show full text]
  • Irods User Group Meeting 2016 Proceedings
    iRODS User Group Meeting 2016 Proceedings © 2016 All rights reserved. Each article remains the property of the authors. 8TH ANNUAL CONFERENCE SUMMARY The iRODS User Group Meeting of 2016 gathered together iRODS users, Consortium members, and staff to discuss iRODS-enabled applications and discoveries, technologies developed around iRODS, and future development and sustainability of iRODS and the iRODS Consortium. The two-day event was held on June 8th and 9th in Chapel Hill, North Carolina, hosted by the iRODS Consortium, with over 90 people attending. Attendees and presenters represented over 30 academic, government, and commercial institutions. Contents Listing of Presentations .......................................................................................................................................... 1 ARTICLES iRODS Audit (C++) Rule Engine Plugin and AMQP .............................................................................................. 5 Terrell Russell, Jason Coposky, Justin James - RENCI at UNC Chapel Hill Speed Research Discovery with Comprehensive Storage and Data Management .......................................... 21 HGST | PANASAS | iRODS Integrating HUBzero and iRODS ............................................................................................................................ 23 Rajesh Kalyanam, Robert A. Campbell, Samuel P. Wilson, Pascal Meunier, Lan Zhao, Elizabett A. Hillery, Carol Song - Purdue University An R Package to Access iRODS Directly ...........................................................................................................
    [Show full text]
  • Ios Forensics
    UNIVERSITY OF PIRAEUS Department of Digital Systems MSc in “Digital Systems Security” iOS Forensics Postgraduate student: Nikolaos Anagnostopoulos MTE 1202 Supervisor: Christos Xenakis Assistant Professor, University of Piraeus Academic Year: 2013-1014 Σ ΙΩ Α ΙΡ Ε Π ΙΟ Μ Η Τ Σ Ι Π Ε Ν Α Π THIS PAGE WAS INTENTIONALLY LEFT BLANK Table of Contents Acknowledgements ........................................................................................................ 1 Abstract .......................................................................................................................... 1 Introduction .................................................................................................................... 2 Digital Forensics ............................................................................................................ 3 The current state............................................................................................................ 4 Smartphones operating system market share ............................................................. 4 Smartphones Security ................................................................................................Σ . 5 Mobile Device Forensics ...............................................................................................Ω 6 Evidence of great interest ................................................................Ι ........................... 6 Live forensics ................................................................................................Α
    [Show full text]
  • Free License Key for Jalbum
    Free License Key For Jalbum Free License Key For Jalbum 1 / 4 2 / 4 Download jAlbum (64-bit) for Windows PC from FileHorse. 100% Safe and Secure ✓ Free Download 64-bit Latest Version 2021. ... This product is not just about presenting images, it's just as good at handling videos. ... Open Source software is software with source code that anyone can inspect, modify or .... Download JAlbum for Mac to create web photo albums with slideshows of your images. JAlbum has had ... Key Details of JAlbum. Create web ... During the installation of JAlbum for Mac the user is prompted to buy a product license. Support is .... Publishing your album to the web is easy; use your free Jalbum account or any other hosting provider. ... Fixed "Invalid code" error when activating license Choose the ad free Standard License for non-commercial purposes for just $12 ... code after completing your purchase to sign up at the jAlbum.net website to .. Full Cracked PC Software Product keys, License Key, Serial Free, ... jAlbum 23.2 (64-bit) Crack + License Key Latest 2021 jAlbum Crack into you can make web ... Groschengrab Deluxe 1.31 Cracked.epub EaseUS Data Recovery Wizard Pro 11 Activation Code, EaseUS Data ... Final Cut Library Manager 3.51 Mac Crack Free Download, ... jAlbum 15.2 Serial Key,. Check out this link on youtube it works ! Feb 6, 2013 Computers & Internet. 2 Answers. Adobe photoshop elements 11 serial number free. Telecharger Ali Mini Upgrade 6.10 Gratuit Star Timesl [FULL] Engineering Probability And Statistics D.k Murugesan 3 / 4 ReFX.Nexus.Dance.Orchestra.Expansion.Pack-DYNAMiCS.rar Compare the best jAlbum alternatives in 2021.
    [Show full text]
  • Cyverse Austria—A Local, Collaborative Cyberinfrastructure
    Mathematical and Computational Applications Communication CyVerse Austria—A Local, Collaborative Cyberinfrastructure 1, 1, , 2 3 Konrad Lang y , Sarah Stryeck * y , David Bodruzic , Manfred Stepponat , Slave Trajanoski 4 , Ursula Winkler 2 and Stefanie Lindstaedt 1,5,* 1 Institute for Interactive Systems and Data Science, Graz University of Technology, 8010 Graz, Austria; [email protected] 2 Server and Storage Systems, University of Graz, 8010 Graz, Austria; [email protected] (D.B.); [email protected] (U.W.) 3 Central Information Technology, Graz University of Technology, 8010 Graz, Austria; [email protected] 4 Core Facility Computational Bioanalytics, Medical University of Graz, 8010 Graz, Austria; [email protected] 5 Know-Center GmbH, 8010 Graz, Austria * Correspondence: [email protected] (S.S.); [email protected] (S.L.); Tel.: +43-316-873-30677 (S.S.); +43-316-873-30600 (S.L.) These authors contributed equally to this work. y Received: 25 May 2020; Accepted: 22 June 2020; Published: 24 June 2020 Abstract: Life sciences (LS) are advanced in research data management, since LS have established disciplinary tools for data archiving as well as metadata standards for data reuse. However, there is a lack of tools supporting the active research process in terms of data management and data analytics. This leads to tedious and demanding work to ensure that research data before and after publication are FAIR (findable, accessible, interoperable and reusable) and that analyses are reproducible. The initiative CyVerse US from the University of Arizona, US, supports all processes from data generation, management, sharing and collaboration to analytics.
    [Show full text]
  • Swiftstack and Cyberduck Summary Cyberduck Is an Open Source Client for Browsing Various Storage Back-Ends Including Openstack Swift
    SOLUTION BRIEF SwiftStack and Cyberduck Summary Cyberduck is an open source client for browsing various storage back-ends including OpenStack Swift. Combining Cyberduck with SwiftStack Object Storage enables users to quickly store and retrieve files in a cloud storage system. Like storage technology, Cyberduck has evolved in recent decades. Initially a tool for remotely browsing and facilitating transfers of local storage in stand-alone computers and servers (via FTP), functionality was added to support browsing network file shares (via SFTP and WebDAV) as centralized “filers” became the de facto standard for storage architecture. In recent years, however, storage administrators have “SwiftStack is the simplest way to deploy, found that NAS/SAN-based architectures cannot scale manage, and monitor Swift object storage to the capacities required by today’s applications and and undoubtedly offers the lowest TCO of users—resulting in multiple silos of storage. In addition, any private-cloud object storage on the these legacy disk-based systems are burdened with a market today—never mind the high-cost, very high cost per TB—making them impractical for highly distributed configurations spanning multiple low-scale legacy NAS vendors. With datacenters and geographies. Cyberduck, you have a free tool to view the contents of your SwiftStack object store, and To address today’s storage needs, object storage it only takes 30 seconds to set it up!” technologies have been introduced to enable private and public clouds using predominantly two APIs—OpenStack Chris Nelson, Director of Systems Engineering, Swift’s Swift API and Amazon’s S3 API, and Cyberduck SwiftStack now supports browsing cloud storage using both APIs.
    [Show full text]
  • Operators Manual
    ____________ _______________________________ OPERATORS MANUAL For use with Software Version 4.05.370 Image Server MAXX 6T January 2008 Copyright 2008, 360 Systems All rights reserved Printed in the United States of America 900-105-0002-07 6T UM ____________________________________________________________________________________ Page 2 MAXX 6T Owner’s Manual TABLE OF CONTENTS LIST OF FIGURES............................................................................................... 7 PREFACE............................................................................................................ 8 Software and Operations Manual Revisions................................................ 8 Typographical Conventions........................................................................ 8 SAFETY NOTICES .............................................................................................. 9 Safety Terms and Symbols.......................................................................... 9 General Safety Caution............................................................................... 9 Personal Injury Precautions ........................................................................ 9 PRODUCT DAMAGE PRECAUTIONS ............................................................. 10 Product Registration ................................................................................. 10 Product Improvements and Upgrades....................................................... 10 Trademarks .............................................................................................
    [Show full text]
  • Foss-2020-Spring
    CyVerse Documentation Release 0.2.0 CyVerse Jan 15, 2021 Contents 1 Expected outcomes: 3 2 Funding and Citations 155 i ii CyVerse Documentation, Release 0.2.0 Learning Center Home Foundational Open Science Skills (FOSS) is a novel, camp-style training designed to prepare principal investigators and their lab teams, both new and established, to meet the growing expectations of funding agencies, publishers, and research institutions for scientific reproducibility and data accessibility. Workshop Level There are no pre-requisites for FOSS, but the course will cover a lot of material in a short time. Participants who have limited computational experience should try to view the Software Carpentry Core Lessons before attending. Contents 1 CyVerse Documentation, Release 0.2.0 2 Contents CHAPTER 1 Expected outcomes: • Become familiar with productivity software for organizing your lab group, communications, and research • Learn how to scale out your computation from laptop to cloud and high performance computing (HPC) systems • Learn how to manage data for open science and reproducibility By working through an example project relevant to their interests, participants will practice open science skills using CyVerse, GitHub, R or Python, and other resources. At the end of the week, students will present a plan for how to integrate open science into their labs. Learning Center Home 1.1 Before you arrive Please endeavor to complete the pre-workshop setup before arriving at FOSS. The workshop will be run under a Code of Conduct. Please familiarize yourself with it prior to your arrival. There will be a webinar one week before camp to help answer questions about the setup, logistics, and materials.
    [Show full text]
  • Admin Tools for Wordpress Nicholas K
    Admin Tools for WordPress Nicholas K. Dionysopoulos Davide Tampellini Admin Tools for WordPress Nicholas K. Dionysopoulos Davide Tampellini Copyright © 2017-2021 Akeeba Ltd Abstract This book covers the use of the Admin Tools for WordPress site security plugin for WordPress™ -powered web sites. Both the free Admin Tools Core and the subscription-based Admin Tools Professional editions are completely covered. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the appendix entitled "The GNU Free Documentation License". Table of Contents 1. Getting Started .......................................................................................................................... 1 1. What is Admin Tools? ....................................................................................................... 1 1.1. Disclaimer ............................................................................................................. 1 1.2. The philosophy ....................................................................................................... 2 2. Server environment requirements ......................................................................................... 2 3. Installing Admin Tools ......................................................................................................
    [Show full text]
  • Introduction to Linux
    Introduction to Linux Augustine Abaris Research Computing Services Information Services & Technology Topics for Today ● Research Computing Services ● Linux Overview ● Linux Interaction - Shell and Commands ● I/O redirection (pipes, etc.) ● Navigating the file system ● Processes and job control ● Editors ● Creating and Running Code Research Computing Services Research Computing Services (RCS) A group within Information Services & Technology at Boston University provides computing, storage, and visualization resources and services to support research that has specialized or highly intensive computation, storage, bandwidth, or graphics requirements. Three Primary Services: ● Research Computation ● Research Visualization ● Research Consulting and Training RCS Team and Expertise Our Team Consulting Focus: ● Scientific Programmers ● Bioinformatics ● Systems Administrators ● Data Analysis / Statistics ● Service Management Specialists ● Molecular modeling ● Research Facilitators ● Geographic Information Systems ● Special Initiatives (Grants) ● Scientific/Engineering Simulation ● [email protected] ● Visualization Me ● Lead Systems Programmer/Administrator ● 20+ years of systems administration experience ● 15+ years at BU, contributed to design and deployment of SCC and 2 predecessor HPC clusters ● Contact: [email protected] You ● Who has experience programming? ● Using Linux? ● Using compute clusters? ● Using the Shared Computing Cluster (SCC)? Linux What is an operating system? What is Linux? ● Operating System ● Originated in early-90s ● Free ● Open Source Where is Linux? Why Linux ● Free and open-source. ● Powerful for research datacenters ● Personal for desktops and phones The most common OS used ● Universal by BU researchers when ● Community (and business) driven. working on a server or computer cluster Connecting Let’s use Linux Local System Remote Server Connection Protocols and Software Remote Connections: Remote Graphics: Data Transfer: Secure SHell X-Windowing Secure File Transfer Protocol (SSH) (X, X-Win) (SFTP) Other protocols too, but let’s start with these.
    [Show full text]
  • VSC HPC Tutorial for University of Antwerp Mac Users
    VLAAMS SUPERCOMPUTER Innovative Computing CENTRUM for A Smarter Flanders HPC Tutorial Last updated: September 29 2021 For Mac Users Authors: Franky Backeljauw5, Stefan Becuwe5, Geert Jan Bex3, Geert Borstlap5, Jasper Devreker2, Stijn De Weirdt2, Andy Georges2, Balázs Hajgató1,2, Kenneth Hoste2, Kurt Lust5, Samuel Moors1, Ward Poelmans1, Mag Selwa4, Álvaro Simón García2, Bert Tijskens5, Jens Timmerman2, Kenneth Waegeman2, Toon Willems2 Acknowledgement: VSCentrum.be 1Free University of Brussels 2Ghent University 3Hasselt University 4KU Leuven 5University of Antwerp 1 Audience: This HPC Tutorial is designed for researchers at the University of Antwerp and affiliated institutes who are in need of computational power (computer resources) and wish to explore and use the High Performance Computing (HPC) core facilities of the Flemish Supercomputing Centre (VSC) to execute their computationally intensive tasks. The audience may be completely unaware of the UAntwerpen-HPC concepts but must have some basic understanding of computers and computer programming. Contents: This Beginners Part of this tutorial gives answers to the typical questions that a new UAntwerpen- HPC user has. The aim is to learn how to make use of the HPC. Beginners Part Questions chapter title What is a UAntwerpen-HPC exactly? 1 Introduction to HPC Can it solve my computational needs? How to get an account? 2 Getting an HPC Account How do I connect to the UAntwerpen- 3 Connecting to the HPC infrastructure HPC and transfer my files and pro- grams? How to start background jobs? 4 Running batch jobs How to start jobs with user interaction? 5 Running interactive jobs Where do the input and output go? 6 Running jobs with input/output data Where to collect my results? Can I speed up my program by explor- 7 Multi core jobs/Parallel Computing ing parallel programming techniques? Troubleshooting 8 Troubleshooting What are the rules and priorities of 9 HPC Policies jobs? FAQ 10 Frequently Asked Questions The Advanced Part focuses on in-depth issues.
    [Show full text]
  • Introduction to Working in the Linux Shell Environment September 17, 2012
    Introduction to Working in the Linux Shell Environment September 17, 2012 Note: Do not attempt to copy/paste out of this document. Commands will likely not work! Some software you will need (Note: these are examples of free software, not endorsements): In order to From Mac or Linux From Windows Connect to submit (ssh) Terminal PuTTY Move files to/from submit FileZilla, CyberDuck (Mac) FileZilla, CyberDuck Edit text files TextWrangler (Mac) Notepad++ Getting around in Linux: • File paths (directories or folders): /, /home/magitz/, /scratch/hpc/magitz/ • pwd, cd, ls (Where am I, change directory, list directory) • cp, mv, rm (copy, move, delete) • more, less, head, tail, cat (examine files) • nano, vim (text editors in Linux) Making things easier: • Tab completion • history: redo something that you did before without retyping • man: getting help, also –h or --help flag Learning by doing: 1. Connect to HPC Center: ssh [email protected] a. Type your password and hit return (no characters display while you type). 2. Where are you when you login? pwd 3. What files are there? ls 4. Make a directory: mkdir test_script 5. Now what’s there? ls –l a. Linux commands usually have flags to change how they work b. man, –h or --help often give you help 6. Change into test_script directory: cd test_script or cd tes<tab> 7. Copy a sample file here (.): cp /project/bio/training/2012-09-17/simple.pbs . 8. Check that the copy worked: ls 9. Delete that file: rm simple.pbs a. That file is now GONE! Not in your recycle bin or trashcan, but gone! * *We may be able to retrieve some files, so if it’s an important file, let us know ASAP.
    [Show full text]