SWIFT Documentation Documentation Release 1.0 SWIFT Support <[email protected]> Aug 22, 2021 Contents 1 About SWIFT 3 2 Getting access to SWIFT 5 3 How to use SWIFT 7 4 Advanced usecases 11 i ii SWIFT Documentation Documentation, Release 1.0 Welcome to the documentation for using the SURF Object Store service at SURF which is based on Openstack SWIFT. The information on these pages will help you get started with SWIFT. We welcome your comments at [email protected] or your contribution to help us improve the documentation. Need help? Do you need help with this? We are more than willing to assist! Just contact us at [email protected]. Contents 1 SWIFT Documentation Documentation, Release 1.0 2 Contents CHAPTER 1 About SWIFT There is a growing demand for general purpose Online Storage capabilities that may be used for the storage of large quantities of unstructured data in a shared multi-tenant environment. SURFsara’s Openstack SWIFT storage cluster is an all round online cloud storage system which can meet this demand and that is characterised by its scalability, durability and availability. SWIFT is ideal for storing unstructured data that can grow without bound. Openstack SWIFT will provide apart from it’s own native protocol also the S3 protocol, which is the defacto standard for addressing object storage. The S3 REST API was developed by Amazon for providing access on their cloud storage system. This opens up a wealth of possibilities for this storage system. SWIFT is an object storage system which means that it does not have a directory-type structure like a normal file system has but it organises its data in so-called containers that contain objects. There is no tree-like structure with files and directories. There are only containers with objects in them. 3 SWIFT Documentation Documentation, Release 1.0 4 Chapter 1. About SWIFT CHAPTER 2 Getting access to SWIFT In this page you will find general information about getting access to the SWIFT storage cluster. Contents • Getting access to SWIFT If you are interested to use the SWIFT storage cluster, please contact the SURFsara helpdesk([email protected]). There are two ways to get access. • Local SWIFT account through the OpenStack Keystone AuthN service. This is what we recommend at the moment unless you have a special reason to use one of the other methods below. • SWIFT account through the OpenStack keystone AuthN service using you user name and password that is in SURFsara Central User Administration (CUA) system. This is handy for users that use other SURFsara services like the national super computer or the Lisa compute cluster. It is also possible to obtain an access key and secret key for S3 access. 5 SWIFT Documentation Documentation, Release 1.0 6 Chapter 2. Getting access to SWIFT CHAPTER 3 How to use SWIFT In this page you will find documentation about the different SWIFT clients that are available. SWIFT can be addressed with its own native protocol and with the S3 protocol. More detailed information on the compatibility of the SWIFT S3 implementation with AWS S3 may be found at: https://docs.openstack.org/swift/latest/s3_compat.html Contents • How to use SWIFT – Authentication – Command line clients – GUIs – Mounted file systems * S3QL * S3FS * Rclone * Goofys – Backups * Duplicity – Synchronisation * Rclone – Libraries and APIs * Python Library * REST API 7 SWIFT Documentation Documentation, Release 1.0 * Boto3 3.1 Authentication In order to use the SWIFT service you need a user name and a password. SWIFT supports so-called v3 authentication. For v3 authentication, you have projects. Unless we agree upon something else, your project name is the same as your user name. In v3 authentication there are domains for both users and projects. If you have an account in SURFsara’s Central User Administration (CUA) system, then both your project domain as your user domain is CuaUsers. When you only have a local account, then both domains are equal to Default. In order to be able to use the S3 protocol, you need to get to create so-called EC2 credentials, i.e. an access key and a secret key. How this works is described in EC2 credentials for S3. 3.2 Command line clients • Swift – Python SWIFT client. – Curl – rclone • S3 – S3cmd – AWS S3 client (awscli) – S5cmd, for parallel file transfers – rclone 3.3 GUIs • Cyberduck 3.4 Mounted file systems 3.4.1 S3QL It is possible to mount SWIFT object storage as a file system with s3ql both using swift’s native protocol and S3. S3QL has features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival. The s3ql page has more information. 8 Chapter 3. How to use SWIFT SWIFT Documentation Documentation, Release 1.0 3.4.2 S3FS It is possible to mount SWIFT object storage as a file system with s3fs using the S3 protocol. The s3fs page has more information. 3.4.3 Rclone Using rclone mount you can use rclone to mount SWIFT using the native SWIFT protocol as well as S3. 3.4.4 Goofys Goofys is a tool written in golang to mount S3 object stores. Information may be found at the goofys page. 3.5 Backups 3.5.1 Duplicity Duplicity is a backup tool that amongst others supports cloud storage systems. More information about this can be found at the duplicity page. 3.6 Synchronisation 3.6.1 Rclone Rclone is the rsync for cloud storage. Here is more information on how to use rclone on the rclone page. 3.7 Libraries and APIs 3.7.1 Python Library The python library page gives you information on the python-swiftclient library. 3.7.2 REST API SWIFT offers a REST API. Information about this API and some examples are described at the API page. 3.7.3 Boto3 Boto3 is the AWS SDK for python developed by Amazon. It contains of course an S3 part to access object stores. At boto3 has information on how to use it. 3.5. Backups 9 SWIFT Documentation Documentation, Release 1.0 10 Chapter 3. How to use SWIFT CHAPTER 4 Advanced usecases Information about more advanced usecases 4.1 Delegating access using S3 credentials Information on this usecase may be found at the delegating access page. 4.2 Multi-user buckets Information on this usecase may be found at the multi-user buckets page. 4.3 Serving Static Web Pages SWIFT offers the possibility to serve data in containers as a static web site. The staticweb page has more. 4.4 Uploading large files When you want to upload some large files you may want to have a look at this page. 4.5 Uploading many small files When you want to upload a folder with many small files, then you may want to have a look at this page. • search 11.