DOCSLIB.ORG

SWIFT Documentation Documentation Release 1.0

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
SWIFT Documentation Documentation Release 1.0 SWIFT Documentation Documentation Release 1.0 SWIFT Support <[email protected]> Aug 22, 2021 Contents 1 About SWIFT 3 2 Getting access to SWIFT 5 3 How to use SWIFT 7 4 Advanced usecases 11 i ii SWIFT Documentation Documentation, Release 1.0 Welcome to the documentation for using the SURF Object Store service at SURF which is based on Openstack SWIFT. The information on these pages will help you get started with SWIFT. We welcome your comments at [email protected] or your contribution to help us improve the documentation. Need help? Do you need help with this? We are more than willing to assist! Just contact us at [email protected]. Contents 1 SWIFT Documentation Documentation, Release 1.0 2 Contents CHAPTER 1 About SWIFT There is a growing demand for general purpose Online Storage capabilities that may be used for the storage of large quantities of unstructured data in a shared multi-tenant environment. SURFsara’s Openstack SWIFT storage cluster is an all round online cloud storage system which can meet this demand and that is characterised by its scalability, durability and availability. SWIFT is ideal for storing unstructured data that can grow without bound. Openstack SWIFT will provide apart from it’s own native protocol also the S3 protocol, which is the defacto standard for addressing object storage. The S3 REST API was developed by Amazon for providing access on their cloud storage system. This opens up a wealth of possibilities for this storage system. SWIFT is an object storage system which means that it does not have a directory-type structure like a normal file system has but it organises its data in so-called containers that contain objects. There is no tree-like structure with files and directories. There are only containers with objects in them. 3 SWIFT Documentation Documentation, Release 1.0 4 Chapter 1. About SWIFT CHAPTER 2 Getting access to SWIFT In this page you will find general information about getting access to the SWIFT storage cluster. Contents • Getting access to SWIFT If you are interested to use the SWIFT storage cluster, please contact the SURFsara helpdesk([email protected]). There are two ways to get access. • Local SWIFT account through the OpenStack Keystone AuthN service. This is what we recommend at the moment unless you have a special reason to use one of the other methods below. • SWIFT account through the OpenStack keystone AuthN service using you user name and password that is in SURFsara Central User Administration (CUA) system. This is handy for users that use other SURFsara services like the national super computer or the Lisa compute cluster. It is also possible to obtain an access key and secret key for S3 access. 5 SWIFT Documentation Documentation, Release 1.0 6 Chapter 2. Getting access to SWIFT CHAPTER 3 How to use SWIFT In this page you will find documentation about the different SWIFT clients that are available. SWIFT can be addressed with its own native protocol and with the S3 protocol. More detailed information on the compatibility of the SWIFT S3 implementation with AWS S3 may be found at: https://docs.openstack.org/swift/latest/s3_compat.html Contents • How to use SWIFT – Authentication – Command line clients – GUIs – Mounted file systems * S3QL * S3FS * Rclone * Goofys – Backups * Duplicity – Synchronisation * Rclone – Libraries and APIs * Python Library * REST API 7 SWIFT Documentation Documentation, Release 1.0 * Boto3 3.1 Authentication In order to use the SWIFT service you need a user name and a password. SWIFT supports so-called v3 authentication. For v3 authentication, you have projects. Unless we agree upon something else, your project name is the same as your user name. In v3 authentication there are domains for both users and projects. If you have an account in SURFsara’s Central User Administration (CUA) system, then both your project domain as your user domain is CuaUsers. When you only have a local account, then both domains are equal to Default. In order to be able to use the S3 protocol, you need to get to create so-called EC2 credentials, i.e. an access key and a secret key. How this works is described in EC2 credentials for S3. 3.2 Command line clients • Swift – Python SWIFT client. – Curl – rclone • S3 – S3cmd – AWS S3 client (awscli) – S5cmd, for parallel file transfers – rclone 3.3 GUIs • Cyberduck 3.4 Mounted file systems 3.4.1 S3QL It is possible to mount SWIFT object storage as a file system with s3ql both using swift’s native protocol and S3. S3QL has features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival. The s3ql page has more information. 8 Chapter 3. How to use SWIFT SWIFT Documentation Documentation, Release 1.0 3.4.2 S3FS It is possible to mount SWIFT object storage as a file system with s3fs using the S3 protocol. The s3fs page has more information. 3.4.3 Rclone Using rclone mount you can use rclone to mount SWIFT using the native SWIFT protocol as well as S3. 3.4.4 Goofys Goofys is a tool written in golang to mount S3 object stores. Information may be found at the goofys page. 3.5 Backups 3.5.1 Duplicity Duplicity is a backup tool that amongst others supports cloud storage systems. More information about this can be found at the duplicity page. 3.6 Synchronisation 3.6.1 Rclone Rclone is the rsync for cloud storage. Here is more information on how to use rclone on the rclone page. 3.7 Libraries and APIs 3.7.1 Python Library The python library page gives you information on the python-swiftclient library. 3.7.2 REST API SWIFT offers a REST API. Information about this API and some examples are described at the API page. 3.7.3 Boto3 Boto3 is the AWS SDK for python developed by Amazon. It contains of course an S3 part to access object stores. At boto3 has information on how to use it. 3.5. Backups 9 SWIFT Documentation Documentation, Release 1.0 10 Chapter 3. How to use SWIFT CHAPTER 4 Advanced usecases Information about more advanced usecases 4.1 Delegating access using S3 credentials Information on this usecase may be found at the delegating access page. 4.2 Multi-user buckets Information on this usecase may be found at the multi-user buckets page. 4.3 Serving Static Web Pages SWIFT offers the possibility to serve data in containers as a static web site. The staticweb page has more. 4.4 Uploading large files When you want to upload some large files you may want to have a look at this page. 4.5 Uploading many small files When you want to upload a folder with many small files, then you may want to have a look at this page. • search 11.
Load more

Recommended publications
  • Openstack Designate
    OpenStack Designate Stephan Lagerholm Graham Hayes What is OpenStack? OpenStack is a free open standard cloud computing platform, mostly deployed as infrastructure-as-a-service (IaaS) in both public and private clouds where virtual servers and other resources are made available to users. The software platform consists of interrelated components that control diverse, multi-vendor hardware pools of processing, storage, and networking resources throughout a data center. Users either manage it through a web-based dashboard, through command-line tools, or through RESTful web services. OpenStack began in 2010 as a joint project of Rackspace Hosting and NASA. As of 2012, it was managed by the OpenStack Foundation (Source: Wikipedia) 2 Designate • Designate started as a project to maintain DNS infrastructure for OpenStack Users. It was an ecosystem project that was in production at both HP’s and Rackspace’s clouds. During 2015, Designate was moved into OpenStack Foundation and in 2017 it became a registered trademark. • Designate is providing API, CLI and a Graphical User interface so that OpenStack Users can setup and make changes to DNS data. The zones are thereafter exposed to secondary DNS servers via Zone Transfers. • Officially Bind 9.X and PowerDNS 4.X is supported although other DNS servers are known to work too. Most resource Record Types such as A, AAAA, PTR, CNAME, NS, MX, etc are supported 3 Producer Producer Backend Producer Producer Producer Worker Customer Facing API Central DNS Servers Standard XFR Secured by TSIG Nova / DB Mini
    [Show full text]
  • 1) Installation 2) Configuration
    rclone 1) Installation........................................................................................................................................1 2) Configuration...................................................................................................................................1 2.1) Server setup..............................................................................................................................1 2.2) Client setup...............................................................................................................................2 2.3) Server setup - part 2..................................................................................................................2 2.4) Client verification.....................................................................................................................3 2.5) rclone - part 1............................................................................................................................3 2.6) rclone - part 2............................................................................................................................4 3) Backup configuration.......................................................................................................................5 4) Usage................................................................................................................................................5 1) Installation https://rclone.org/install/ Script installation To install rclone on Linux/macOS/BSD
    [Show full text]
  • Ovirt and Openstack Storage (Present and Future)
    oVirt and OpenStack Storage (present and future) Federico Simoncelli Principal Software Engineer, Red Hat January 2014 1 Federico Simoncelli – oVirt and OpenStack Storage (present and future) Agenda ● Introduction ● oVirt and OpenStack Overview ● Present ● oVirt and Glance Integration ● Importing and Exporting Glance Images ● Current Constraints and Limitations ● Future ● Glance Future Integration ● Keystone Authentication in oVirt ● oVirt and Cinder Integration 2 Federico Simoncelli – oVirt and OpenStack Storage (present and future) oVirt Overview ● oVirt is a virtualization management application ● manages hardware nodes, storage and network resources, in order to deploy and monitor virtual machines running in your data center ● Free open source software released under the terms of the Apache License 3 Federico Simoncelli – oVirt and OpenStack Storage (present and future) The oVirt Virtualization Architecture 4 Federico Simoncelli – oVirt and OpenStack Storage (present and future) OpenStack Overview ● Cloud computing project to provide an Infrastructure as a Service (IaaS) ● Controls large pools of compute, storage, and networking resources ● Free open source software released under the terms of the Apache License ● Project is managed by the OpenStack Foundation, a non-profit corporate entity established in September 2012 5 Federico Simoncelli – oVirt and OpenStack Storage (present and future) OpenStack Glance Service ● Provides services for discovering, registering, and retrieving virtual machine images ● RESTful API that allows querying
    [Show full text]
  • IBM Power Systems and Openpower Solutions for Hybrid Cloud Optimized Infrastructure and Solutions for Data and Computational Services in the Cloud
    IBM Systems Power Systems Solution Brief IBM Power Systems and OpenPOWER Solutions for Hybrid Cloud Optimized infrastructure and solutions for data and computational services in the Cloud Technology plays in increasingly critical role in the ability of organiza- Highlights tions in all industries and all regions of the world to compete and succeed. Ubiquitous access to the internet from anywhere is erasing traditional ●● ●●IBM® Power Systems™ and boundaries and empowering customers, partners, and even things—to OpenPOWER servers with POWER8® processor technology are purpose-built connect, respond, engage, compare and buy. There is a digital transfor- for more efficient processing of data and mation sweeping the planet and organizations that do not make bold analytic workloads in the cloud moves to adapt quickly and intelligently, will undoubtedly fall behind ●● ●●Delivers the broadest choice of cloud- and fail to grow. ready infrastructure and solutions to suit client’s needs and workloads: scale up, The challenge and responsibility of responding effectively to these scale out, converged, proven Reference Architectures, hybrid and public cloud changing dynamics rests solidly with today’s technology leaders. Without services strong leadership to chart a strategic path, organizations will flounder and will see their business decline. IT needs to adopt flexible infrastructure ●● ●●OpenStack based cloud management from IBM, ISV partners and open source strategies and agile development methods to deal with the: Linux distributions provide extensible, scalable and resilient solutions for public ●●●Explosion in amount of data that needs to be rapidly analyzed and and private clouds managed ●●●Security threats that are constant and constantly changing ●●●Need to reach global markets and supply chains ●●●Demand for fast and error-free client’ s engagement experiences IBM Systems Power Systems Solution Brief Organizations need a flexible infrastructure to enable growth and innovation while lowering overall IT costs.
    [Show full text]
  • 14. Comparison of Cloud Management Platforms
    14. Comparison of cloud management platforms Kimmo Ahokas Aalto University School of Science [email protected] Abstract tion cloud computing is divided into three different service models, namely Software as a Service (SaaS), Platform as a Cloud computing allows fast and efficient resource provi- Service (PaaS) and Infrastructure as a Service (IaaS). In this sioning within data centers. In large companies this can paper we are only interested in IaaS service model, which lead to significant savings, thus creating market for complete is defined as "The capability provided to the consumer is cloud platforms. In addition to commercial products, sev- to provision processing, storage, networks, and other fun- eral open source cloud platforms exist. This paper compares damental computing resources where the consumer is able four cloud management platforms and identifies the factors to deploy and run arbitrary software, which can include op- affecting future success of each of the platforms. We also es- erating systems and applications. The consumer does not timate the future development of the cloud platform market. manage or control the underlying cloud infrastructure but has control over operating systems, storage, and deployed KEYWORDS: cloud platform, IaaS, CloudStack, Open- applications; and possibly limited control of select network- Stack, OpenNebula, Eucalyptus, VMware ing components (e.g., host firewalls)." [10] Cloud management platform is a software system that 1 Introduction controls the allocation of physical resources on the data cen- ter. In the IaaS model users can launch virtual machines us- Cloud computing has rapidly changed the way in which re- ing the management console, which causes the platform to sources in data centers can be provisioned.
    [Show full text]
  • Scaling a Game-Sharing Platform Introduction
    Level Up and SIMMER.io Down Scaling a Game-sharing Platform Introduction Much like gaming, starting a business means a lot of trial and error. In the beginning, you’re just trying to get your bearings and figure out which enemy to fend off first. After a few hours (or a few years on the market), it’s time to level up. SIMMER.io, a community site that makes sharing Unity WebGL games easy for indie game developers, leveled up in a big way to make their business sustainable for the long haul. When the site was founded in September 2017, the development team focused on getting the platform built and out the door, not on what egress costs would look like down the road. As it grew into a home for 80,000+ developers and 30,000+ games, though, those costs started to encroach on their ability to sustain and grow the business. We spoke with the SIMMER.io development team about their experience setting up a multi-cloud solution—including their use of the Bandwidth Alliance between Cloudflare and Backblaze B2 Cloud Storage to reduce egress to $0—to prepare the site for continued growth. © 2021 Backblaze Inc. All rights reserved. 500 Ben Franklin Ct. San Mateo, CA 94401 USA How to Employ a Multi-cloud Approach for Scaling a Web Application In 2017, sharing games online with static hosting through a service like AWS S3 was possible but certainly not easy. As one SIMMER.io team member put it, “No developer in the world would want to go through that.” The team saw a clear market opportunity.
    [Show full text]
  • Initial Definition of Protocols and Apis
    Initial definition of protocols and APIs Project acronym: CS3MESH4EOSC Deliverable D3.1: Initial Definition of Protocols and APIs Contractual delivery date 30-09-2020 Actual delivery date 16-10-2020 Grant Agreement no. 863353 Work Package WP3 Nature of Deliverable R (Report) Dissemination Level PU (Public) Lead Partner CERN Document ID CS3MESH4EOSC-20-006 Hugo Gonzalez Labrador (CERN), Guido Aben (AARNET), David Antos (CESNET), Maciej Brzezniak (PSNC), Daniel Muller (WWU), Jakub Moscicki (CERN), Alessandro Petraro (CUBBIT), Antoon Prins Authors (SURFSARA), Marcin Sieprawski (AILLERON), Ron Trompert (SURFSARA) Disclaimer: The document reflects only the authors’ view and the European Commission is not responsible for any use that may be made of the information it contains. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 863353 Table of Contents 1 Introduction ............................................................................................................. 3 2 Core APIS .................................................................................................................. 3 2.1 Open Cloud Mesh (OCM) ...................................................................................................... 3 2.1.1 Introduction .......................................................................................................................................... 3 2.1.2 Advancing OCM ....................................................................................................................................
    [Show full text]
  • A Comparative Study of Current Open-Source Infrastructure As a Service Frameworks
    A Comparative Study of Current Open-source Infrastructure as a Service Frameworks Theo Lynn, Graham Hunt, David Corcoran, John Morrison and Philip Healy Irish Centre for Cloud Computing, Dublin 9, Ireland Keywords: Cloud Computing, Open Source, IaaS, Openstack, Cloudstack, Opennebula, Eucalyptus. Abstract: With the growth of cloud computing in recent years, several commercial and open source IaaS frameworks have emerged. The development of open source IaaS solutions offers a free and flexible alternative to commercial cloud services. The main contribution of this paper is to provide a qualitative comparative of current open-source IaaS frameworks. Existing research papers examining open source IaaS frameworks have focused on comparing OpenStack with a small number of alternatives. However, current research fails to adequately compare all major open source frameworks in a single study and notably lacks the inclusion of CloudStack. Our research paper provides the first overview of the five main open source cloud IaaS frameworks – OpenStack, CloudStack, OpenNebula, Eucalyptus and Nimbus. As such, this review provides researchers and potential users with an up to date and comprehensive overview of the features of each solution and allows for an easy comparison between the open source solutions. 1 INTRODUCTION the freedom to modify the source code and build a cloud that is pluggable and open to extensions while Cloud Computing technologies have seen significant reducing costs and avoiding vendor lock-in (Zhang adoption in recent years by enterprises, researchers et al., 2013; Wen et al. 2012; Bist et al., 2013). The and individuals. It is forecast that the Cloud development of open source solutions is of particular Computing industry will reach a market size of $241 importance for the further proliferation of private billion by 2020 (Reid and Kilster, 2011).
    [Show full text]
  • Scaling Openstack Clouds Using Peer-To-Peer Technologies
    Scaling OpenStack Clouds Using Peer-to-peer Technologies Master’s thesis in Computer Systems and Networks XIN HAN Department of Computer Science and Engineering CHALMERS UNIVERSITY OF TECHNOLOGY UNIVERSITY OF GOTHENBURG Gothenburg, Sweden 2016 Master’s thesis 2016:NN Scaling OpenStack CloudsUsing Peer-to-peer Technologies XIN HAN Department of Computer Science and Engineering Chalmers University of Technology And University of Gothenburg Gothenburg, Sweden 2016 Scaling OpenStack CloudsUsing Peer-to-peer Technologies XIN HAN © XIN HAN, 2016. Supervisors: Vincenzo Gulisano, Department of Computer Science and Engineering Joao Monteiro Soares, Ericsson Fetahi Wuhib, Ericsson Vinay Yadhav, Ericsson Examiner: Magnus Almgren, Department of Computer Science and Engineering Master’s Thesis 2016:NN Department of Computer Science and Engineering Chalmers University of Technology and University of Gothenburg SE-412 96 Gothenburg Telephone +46 31 772 1000 Cover: Illustration of P2P OpenStack cloud system. Each cloud is a standalone OpenStack cloud instance. Cloud instances are federated as one using P2P tech- nologies. Gothenburg, Sweden 2016 iv Scaling OpenStack Clouds Using Peer-to-peerTechnologies XIN HAN Department of Computer Science and Engineering Chalmers University of Technology and University of Gothenburg Abstract OpenStack is an open-source software platform for cloud computing, mostly de- ployed as an infrastructure-as-a-service (IaaS) and has a user base in industry and academia to date. Despite its popularity, OpenStack still has drawbacks in terms of scalability of number of compute nodes (metal machines) in a single cloud instance. More precisely, a single standard OpenStack cloud instance does not scale well and fails to handle user request once its number of compute nodes reaches a particular amount.
    [Show full text]
  • Faq Cloud Sync
    FAQ CLOUD SYNC 1 What is Cloud Sync? NetApp® Data Fabric Cloud Sync is a simple replication and synchronization service. This software-as-a-service (SaaS) offering enables you to transfer and synchronize NAS data to and from cloud or on-premises object storage. The SMB/CIFS or NFS server can be the NetApp Cloud Volumes Service, a NetApp system, or a non-NetApp system. Cloud Sync supports these sources and targets: • CIFS • NFS • Amazon S3 • Amazon EFS • Azure Blob • IBM Cloud Object Storage • NetApp StorageGRID® Webscale appliance After your data is synchronized, it remains available for use in the target server and is updated during the next synchronization cycle. © 2019 NetApp, Inc. All Rights Reserved. | 1 2 Why should I use Cloud Sync? Cloud Sync enables you to perform data migration, data transformation, and data synchronization in a fast, efficient, and secure way. Key benefits of using Cloud Sync are: Fast. Cloud Sync transfers data in parallel processes. This speeds throughput to 1TB in four hours (in the default configuration), and up to 10x faster than in-house developed or traditional tools (such as rsync or Robocopy). Efficient. After the initial synchronization, only changes since the last synchronization are transferred. Data that hasn’t changed isn’t re-replicated, which makes updates faster. Cost-effective. Cloud Sync pricing is based on hourly usage, not on capacity. Compatible. Cloud Sync supports any NFS or CIFS servers, Amazon or private S3 buckets, Azure Blob, IBM Cloud Object Storage, Amazon EFS. Secure. Data is not transferred to our service domain; it remains in your protected environment.
    [Show full text]
  • Introduction to Openstack Private Cloud
    Introduction to OpenStack Private Cloud In this guide In this e-guide: Get your head in the cloud: Intro to OpenStack components As the private cloud market is heating up, there are two clear Navigating the OpenStack players leading the way: VMware and OpenStack. However, services maze due to the VMware tax, along with rising performance issues, Overcome the most pressing many IT professionals are flocking to the attractive, lower-cost OpenStack security challenges alternative. Getting more PRO+ exclusive content In this exclusive e-guide, find out what sets OpenStack apart from its competitors, how to find out which OpenStack cloud service is right for your organization, and the steps to take to overcome its common security challenges. Page 1 of 17 In this guide Get your head in the cloud: Intro to Get your head in the cloud: OpenStack components Intro to OpenStack components http://searchservervirtualization.techtarget.com/tip/Get-your-head-in-the-cloud- Navigating the OpenStack Intro-to-OpenStack-components services maze Overcome the most pressing As more and more businesses consider the different options for a private cloud OpenStack security challenges platform, it's important to know what distinguishes open source OpenStack from Getting more PRO+ exclusive content other popular cloud computing software on the market. In this article, I will attempt to explain what OpenStack is and what it is not, and explore some essential OpenStack components without getting bogged down in technical details. The OpenStack ecosystem OpenStack is very similar to Amazon EC2 in that both platforms allow users to provision VMs from a dashboard or API.
    [Show full text]
  • HPE Digital Learner Google Cloud for Developers (Intermediate) Content Pack
    Content Pack data sheet HPE Digital Learner Google Cloud for Developers (Intermediate) Content Pack This self-paced eLearning Content Pack provides the required HPE Content Pack CP020 number training to help transition a traditional IT architect into a cloud architect utilizing the Google public cloud environment. This Content Pack 20 Hours length is a comprehensive intermediate training series that includes Content Pack Category 2 many core areas required to plan, design and implement category modern cloud infrastructures, applications and data in the Learn more View now Google public cloud. This training is intended for IT personnel who need to rapidly uplift their IT/cloud skills as part of an ongoing business and operational strategy to migrate IT to Google public cloud. Why HPE Education Services? • IDC MarketScape leader 5 years running for IT education and training* Audience Content Pack objectives • Recognized by IDC for leading with This Content Pack provides the information global coverage, unmatched technical • IT professionals including managers, expertise, and targeted education engineers and developers evaluating or necessary needed to transition a traditional consulting services* implementing application environments on IT architect into a cloud architect capable of • Key partnerships with industry leaders Google Cloud Platform working within a typical Google public cloud OpenStack®, VMware®, Linux®, Microsoft®, environment. This training series provides • Data professionals who are responsible ITIL, PMI, CSA, and SUSE
    [Show full text]