Sohonet Media Storage Swift CLI User Guide Media Storage

Sohonet Media Storage Swift CLI User Guide media storage

Sohonet Media Storage Swift CLI User Guide

Based on python-Swift client version 1.6.0

Overview Sohonet media storage is built using Openstack Swift. OpenStack Swift is a scalable object storage system - it is not a file system in the traditional sense, it is made up of containers and objects.

Authentication

Before you can perform operations on the object store you first need to authenticate. Any operation on the object store must be accompanied by a valid token passed using the HTTP header X-Auth-Token . Authentication is performed against an authentication URL and the end result of a successful authentication request is a valid X-Auth- Token and the storage endpoint URL, which is the URL where object store operations are sent.

There are two flavors of authentication available today, v1.0 and v2.0. v2.0 is also known as Keystone, and it is recommended for all future deployments, in place of v1.0.

Containers and Objects

A container is a storage compartment for your data and Sohonet provides a way for you to organize your data. You can Contract #1 think of a container as a folder in Windows® or a Storage directory in UNIX®. The primary difference between a Storage Administrator Users container and these other file system concepts is that Tenant (Account) containers cannot be nested. You can, however, create #1 an unlimited number of containers within your account. Data must be stored in a container so you Container Container must have at least one container defined in your account prior to uploading data [Fig. 1].

Folder Folder

File File File Folder Object Object Object

File File Object Object

Figure 1 Sohonet Storage Relationship with Tenants, Containers and Users

12/11/2013 14:10 Confidential – Internal Document Page 1 of 9

Sohonet Media Storage Swift CLI User Guide media storage

The only restriction on container names is that they cannot contain a forward slash (/) and must be less than 256 bytes in length. Please note that the length restriction applies to the name after it has been URL encoded. For example, a container name of Course Docs would be URL encoded as Course%20Docs and therefore be 13 bytes in length rather than the expected 11.

Although you cannot nest directories in OpenStack Swift, you can simulate a hierarchical structure within a single container by adding forward slash characters (/) in the object name. To navigate the pseudo-directory structure, you may use the delimiter query parameter. see below for more detail.

An object is the basic storage entity and any optional metadata that represents the files you store in the OpenStack Swift system. When you upload data to OpenStack Swift, the data is stored as-is (no compression or encryption) and consists of a location (container), the object's name, and any metadata consisting of key/value pairs. For instance, you may chose to store a backup of your digital photos and organise them into albums. In this case, each object could be tagged with metadata such as Album : Caribbean Cruise or Album : Aspen Ski Trip.

The only restriction on object names is that they must be less than 1024 bytes in length after URL encoding. For example, an object name of C++final(v2).txt should be URL encoded as C%2B%2Bfinal%28v2%29.txt and therefore be 24 bytes in length rather than the expected 16.

Swift Client

Availability and Installation

This document is based on python-swiftclient version 1.6.0. The Python command line tool is available via pip (recommended), easy_install and Github.

$ pip install python-swiftclient==1.6.0 $ pip install python-keystoneclient https://github.com/openstack/python-$

$ swiftclient

Storing Authentication Details

To avoid authentication details showing in your command history, you can add environment variables to the .bash_profile file for your user. Swiftclient uses certain environment variables detailed in the man pages, here is an example:

$ vi ~/.bash_profile export OS_USERNAME= export

12/11/2013 14:10 Confidential – Internal Document Page 2 of 9

Sohonet Media Storage Swift CLI User Guide media storage

OS_PASSWORD= export OS_TENANT_NAME= export OS_AUTH_URL=https://.identity.sohonet.com/v2.0/

The above stops you from having to specify the -A (authentication server), --os-tenant-name, -U (user) and -K (API Key) switches.

NOTE: You could stop other regular users reading your bash profile like so:

$ chmod 600 ~/.bash_profile

Typical Commands

List Containers

To list the contents of a container the swift list command can be used:-

$ swift list dcc_container martin_container

$ swift list -l

3463 30790914721 2013-04-24 19:03:15 dcc_container 6 2172229836 2013-04-24 23:16:59 martin_container

$ swift list --lh

3463 28G 2013-04-24 19:03:15 dcc_container 6 2.0G 2013-04-24 23:16:59 martin_container

List Container Content

To list specific contents of a container the swift list container folder/file command can be used:-

$ swift list martin_container myfiles/test.zip

$ swift list -l martin_container

12/11/2013 14:10 Confidential – Internal Document Page 3 of 9

Sohonet Media Storage Swift CLI User Guide media storage

24746156 2013-04-24 23:29:58 myfiles/test.zip

$ swift list --lh martin_container

23M 2013-04-24 23:29:58 myfiles/test.zip

Advanced Listing

It is more efficient to define as much as possible the file range you are looking for as this will be filtered on the server sided rather than doing a list filtering on the client side using GREP.

$ Swift list container –p main/folder1/folder2]

List all files that begin with Catalog

$ Swift list container Catalog

List all folders/files that begin with Cata:

$ Swift list container Cata

List all folders/files within containers that begin with Cata in folder/subfolder.

$ Swift list container “folder/subfolder/Cata”

Uploading

Files can be uploaded using the swift upload target_container localfiles command.

$ Swift upload container File.*

The File.* is a shell based assembly of all the files beginning with File.

$ Swift upload container *ile.*

Create Container

To create a new container use swift post new_container_name command

$ swift post martin_new_container

12/11/2013 14:10 Confidential – Internal Document Page 4 of 9

Sohonet Media Storage Swift CLI User Guide media storage

Upload a File

Specific files can be uploaded using swift upload target_container specific_file command.

$ swift upload martin_new_container test1.zip

NOTE: For files over 5GB in size, please see segmented upload, got to Upload Segmented Files (Large File Support)

Download a File

Files can be downloaded using the swift download container file command.

$ swift download martin_new_container test1.zip

Upload a Directory/Folder

Complete local/networked directories can be uploaded using swift upload container folder folder/file command.

$ swift upload martin_new_container test test/test2.zip test/test1.zip

NOTE: Because this is an object store, directory structures are pseudo-hierarchical. For files over 5GB in size, please see segmented upload.

Download a Directory

To download a directory use the swift download container –p sourcefolder command.

NOTE: Because this is an object store, directory structures are pseudo-hierarchical. The -p switch here specifies a prefix.

$ swift download martin_new_container -p test/

Upload Changes to a Directory

Synchronisation between the source and destination/target folders can be achieved using the –c comparison switch when using swift upload –c container folder command.

$ swift upload -c martin_new_container test

12/11/2013 14:10 Confidential – Internal Document Page 5 of 9

Sohonet Media Storage Swift CLI User Guide media storage

$ touch test/test2.zip $ swift upload -c martin_new_container test test/test2.zip

Show Container or Object Information

To show information and meta data associated with the file object use the swift stat command.

$ swift stat

Account: AUTH_ d16c6e2d8d23hjksdhfgkds8935076c64c2 Containers: 3 Objects: 1101 Bytes: 937347336 Meta Temp-Url-Key: key X-Timestamp: 1366829142.94203 Content-Type: text/plain; charset=utf- 8 Accept-Ranges: bytes

Other command examples: $ swift stat --lh $ swift stat martin_container $ swift stat martin_container myfiles/test.zip

Advanced Object Information

Similar to the previous section you can be specific about container and filename(s)

$ swift stat container filename

Upload Segmented Files (Large File Support)

Using segmented files can improve performance in some circumstances. Required for files over 5GB, please read the Openstack documentation in order to understand how this works. There are two standards for large objects, DLO and SLO.

$ swift upload --use-slo -S 1073741824 martin_container testdir/test.file

Above example to upload a Static Large Object (SLO). The -S specifies use of segmentation and the segmentation size (1GB in this example).

12/11/2013 14:10 Confidential – Internal Document Page 6 of 9

Sohonet Media Storage Swift CLI User Guide media storage

Download Segmented File

NOTE: The server handles stitching all of the segments together, so this command is exactly the same as it would be for downloading a non-segmented file.

$ swift download martin_container testdir/test.file

Getting Further Help

To see all switches for a particular command see:

$ swift --help

A great deal can be learned by enabling debug output, the API doc's have more detail. Please see the --debug switch.

12/11/2013 14:10 Confidential – Internal Document Page 7 of 9

Sohonet Media Storage Swift CLI User Guide media storage

API Quick Links Language-Specific API Bindings https://wiki.openstack.org/wiki/SwiftAPI

The API docs are here :- http://docs.openstack.org/developer/swift/

Useful info on large object support and how it works http://docs.openstack.org/developer/swift/overview_large_objects.html

You should also read Pseudo-Hierarchical Folders/Directories http://docs.openstack.org/api/openstack-object-storage/1.0/content/pseudo-hierarchical-folders-directories.html

Read the constraints section here http://docs.openstack.org/developer/swift/misc.html

Listing objects is paged with a max size of 10K entries http://docs.openstack.org/api/openstack-object-storage/1.0/content/list-objects.html

Have a look at this as good way to build a client and use the API https://github.com/openstack/python-swiftclient

This will flat line 1Gb/s no problem.

Sohonet Media Storage API Access

API access to the Sohonet Storage is available to all users and allows for developers and system administrators to directly interface with the object store. The interface to Sohonet Storage is based on common Internet standards - using a ReST interface over HTTPS.

Using the API to access Sohonet Store is split into two areas, Identity and Object Storage.

Identity

Passing your details to the identity service in order to obtain a authentication token. The API for getting an authentication token is here

• The London, UK identity service is available at https://lon.identity.sohonet.com/v2.0 • The Los Angeles, US identity service is available at https://lax.identity.sohonet.com/v2.0

12/11/2013 14:10 Confidential – Internal Document Page 8 of 9

Sohonet Media Storage Swift CLI User Guide media storage

API Language bindings for Sohonet Media Storage

Making object requests to Sohonet Storage - Using your authentication token to make direct requests to query, upload or download content.

The API for object storage is described here

Language bindings are available for various languages to assist in getting you up and running:

• C#/.NET: https://github.com/rackerlabs/csharp-cloudfiles

• Java: https://github.com/jclouds/jclouds

• PHP: https://github.com/rackerlabs/php-cloudfiles

• Python: https://github.com/rackspace/pyrax

• Ruby: https://github.com/ruby-openstack

• Node.js: https://github.com/tobowers/javascript-openstack-object

If you have further questions please contact :- [email protected]

12/11/2013 14:10 Confidential – Internal Document Page 9 of 9