Sohonet Media Storage Swift CLI User Guide Media Storage
Total Page:16
File Type:pdf, Size:1020Kb
Sohonet Media Storage Swift CLI User Guide media storage Sohonet Media Storage Swift CLI User Guide Based on python-Swift client version 1.6.0 Overview Sohonet media storage is built using Openstack Swift. OpenStack Swift is a scalable object storage system - it is not a file system in the traditional sense, it is made up of containers and objects. Authentication Before you can perform operations on the object store you first need to authenticate. Any operation on the object store must be accompanied by a valid token passed using the HTTP header X-Auth-Token . Authentication is performed against an authentication URL and the end result of a successful authentication request is a valid X-Auth- Token and the storage endpoint URL, which is the URL where object store operations are sent. There are two flavors of authentication available today, v1.0 and v2.0. v2.0 is also known as Keystone, and it is recommended for all future deployments, in place of v1.0. Containers and Objects A container is a storage compartment for your data and Sohonet provides a way for you to organize your data. You can Contract #1 think of a container as a folder in Windows® or a Storage directory in UNIX®. The primary difference between a Storage Administrator Users container and these other file system concepts is that Tenant (Account) containers cannot be nested. You can, however, create #1 an unlimited number of containers within your account. Data must be stored in a container so you Container Container must have at least one container defined in your account prior to uploading data [Fig. 1]. Folder Folder File File File Folder Object Object Object File File Object Object Figure 1 Sohonet Storage Relationship with Tenants, Containers and Users T+44 (0) 20 7292 6900 [email protected] www.sohonet.com Sohonet Limited, 5 Soho Street, London, W1D 3DG United Kingdom Copyright © Sohonet Ltd. 2013. All rights reserved 12/11/2013 14:10 Confidential – Internal Document Page 1 of 9 Sohonet Media Storage Swift CLI User Guide media storage The only restriction on container names is that they cannot contain a forward slash (/) and must be less than 256 bytes in length. Please note that the length restriction applies to the name after it has been URL encoded. For example, a container name of Course Docs would be URL encoded as Course%20Docs and therefore be 13 bytes in length rather than the expected 11. Although you cannot nest directories in OpenStack Swift, you can simulate a hierarchical structure within a single container by adding forward slash characters (/) in the object name. To navigate the pseudo-directory structure, you may use the delimiter query parameter. see below for more detail. An object is the basic storage entity and any optional metadata that represents the files you store in the OpenStack Swift system. When you upload data to OpenStack Swift, the data is stored as-is (no compression or encryption) and consists of a location (container), the object's name, and any metadata consisting of key/value pairs. For instance, you may chose to store a backup of your digital photos and organise them into albums. In this case, each object could be tagged with metadata such as Album : Caribbean Cruise or Album : Aspen Ski Trip. The only restriction on object names is that they must be less than 1024 bytes in length after URL encoding. For example, an object name of C++final(v2).txt should be URL encoded as C%2B%2Bfinal%28v2%29.txt and therefore be 24 bytes in length rather than the expected 16. Swift Client Availability and Installation This document is based on python-swiftclient version 1.6.0. The Python command line tool is available via pip (recommended), easy_install and Github. $ pip install python-swiftclient==1.6.0 $ pip install python-keystoneclient https://github.com/openstack/python-$ $ swiftclient Storing Authentication Details To avoid authentication details showing in your command history, you can add environment variables to the .bash_profile file for your user. Swiftclient uses certain environment variables detailed in the man pages, here is an example: $ vi ~/.bash_profile export OS_USERNAME=<your username> export T+44 (0) 20 7292 6900 [email protected] www.sohonet.com Sohonet Limited, 5 Soho Street, London, W1D 3DG United Kingdom Copyright © Sohonet Ltd. 2013. All rights reserved 12/11/2013 14:10 Confidential – Internal Document Page 2 of 9 Sohonet Media Storage Swift CLI User Guide media storage OS_PASSWORD=<your API key> export OS_TENANT_NAME=<your tenant name> export OS_AUTH_URL=https://<currently lax or lon>.identity.sohonet.com/v2.0/ The above stops you from having to specify the -A (authentication server), --os-tenant-name, -U (user) and -K (API Key) switches. NOTE: You could stop other regular users reading your bash profile like so: $ chmod 600 ~/.bash_profile Typical Commands List Containers To list the contents of a container the swift list command can be used:- $ swift list dcc_container martin_container $ swift list -l 3463 30790914721 2013-04-24 19:03:15 dcc_container 6 2172229836 2013-04-24 23:16:59 martin_container $ swift list --lh 3463 28G 2013-04-24 19:03:15 dcc_container 6 2.0G 2013-04-24 23:16:59 martin_container List Container Content To list specific contents of a container the swift list container folder/file command can be used:- $ swift list martin_container myfiles/test.zip $ swift list -l martin_container T+44 (0) 20 7292 6900 [email protected] www.sohonet.com Sohonet Limited, 5 Soho Street, London, W1D 3DG United Kingdom Copyright © Sohonet Ltd. 2013. All rights reserved 12/11/2013 14:10 Confidential – Internal Document Page 3 of 9 Sohonet Media Storage Swift CLI User Guide media storage 24746156 2013-04-24 23:29:58 myfiles/test.zip $ swift list --lh martin_container 23M 2013-04-24 23:29:58 myfiles/test.zip Advanced Listing It is more efficient to define as much as possible the file range you are looking for as this will be filtered on the server sided rather than doing a list filtering on the client side using GREP. $ Swift list container –p main/folder1/folder2] List all files that begin with Catalog $ Swift list container Catalog List all folders/files that begin with Cata: $ Swift list container Cata List all folders/files within containers that begin with Cata in folder/subfolder. $ Swift list container “folder/subfolder/Cata” Uploading Files can be uploaded using the swift upload target_container localfiles command. $ Swift upload container File.* The File.* is a shell based assembly of all the files beginning with File. $ Swift upload container *ile.* Create Container To create a new container use swift post new_container_name command $ swift post martin_new_container T+44 (0) 20 7292 6900 [email protected] www.sohonet.com Sohonet Limited, 5 Soho Street, London, W1D 3DG United Kingdom Copyright © Sohonet Ltd. 2013. All rights reserved 12/11/2013 14:10 Confidential – Internal Document Page 4 of 9 Sohonet Media Storage Swift CLI User Guide media storage Upload a File Specific files can be uploaded using swift upload target_container specific_file command. $ swift upload martin_new_container test1.zip NOTE: For files over 5GB in size, please see segmented upload, got to Upload Segmented Files (Large File Support) Download a File Files can be downloaded using the swift download container file command. $ swift download martin_new_container test1.zip Upload a Directory/Folder Complete local/networked directories can be uploaded using swift upload container folder folder/file command. $ swift upload martin_new_container test test/test2.zip test/test1.zip NOTE: Because this is an object store, directory structures are pseudo-hierarchical. For files over 5GB in size, please see segmented upload. Download a Directory To download a directory use the swift download container –p sourcefolder command. NOTE: Because this is an object store, directory structures are pseudo-hierarchical. The -p switch here specifies a prefix. $ swift download martin_new_container -p test/ Upload Changes to a Directory Synchronisation between the source and destination/target folders can be achieved using the –c comparison switch when using swift upload –c container folder command. $ swift upload -c martin_new_container test <no output signifies nothing has changed> T+44 (0) 20 7292 6900 [email protected] www.sohonet.com Sohonet Limited, 5 Soho Street, London, W1D 3DG United Kingdom Copyright © Sohonet Ltd. 2013. All rights reserved 12/11/2013 14:10 Confidential – Internal Document Page 5 of 9 Sohonet Media Storage Swift CLI User Guide media storage $ touch test/test2.zip $ swift upload -c martin_new_container test test/test2.zip Show Container or Object Information To show information and meta data associated with the file object use the swift stat command. $ swift stat Account: AUTH_ d16c6e2d8d23hjksdhfgkds8935076c64c2 Containers: 3 Objects: 1101 Bytes: 937347336 Meta Temp-Url-Key: key X-Timestamp: 1366829142.94203 Content-Type: text/plain; charset=utf- 8 Accept-Ranges: bytes Other command examples: $ swift stat --lh $ swift stat martin_container $ swift stat martin_container myfiles/test.zip Advanced Object Information Similar to the previous section you can be specific about container and filename(s) $ swift stat container filename Upload Segmented Files (Large File Support) Using segmented files can improve performance in some circumstances. Required for files over 5GB, please read the Openstack documentation in order to understand how this works. There are two standards for large objects, DLO and SLO. $ swift upload --use-slo -S 1073741824 martin_container testdir/test.file Above example to upload a Static Large Object (SLO). The -S specifies use of segmentation and the segmentation size (1GB in this example). T+44 (0) 20 7292 6900 [email protected] www.sohonet.com Sohonet Limited, 5 Soho Street, London, W1D 3DG United Kingdom Copyright © Sohonet Ltd.