Elastic Cloud Storage (ECS) Version 3.1 Data Access Guide 302-003-865 01 Copyright © 2013-2017 Dell Inc. or its subsidiaries. All rights reserved. Published August 2017 Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS-IS.“ DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners. Published in the USA. Dell EMC Hopkinton, Massachusetts 01748-9103 1-508-435-1000 In North America 1-866-464-7381 www.DellEMC.com 2 Elastic Cloud Storage (ECS) 3.1 Data Access Guide CONTENTS Figures 7 Tables 9 Part 1 S3 11 Chapter 1 S3 13 Amazon S3 API support in ECS................................................................... 14 S3 API supported and unsupported features...............................................14 Behavior where bucket already exists.............................................17 Bucket policy support................................................................................. 18 Creating, assigning and managing bucket policies......................... 20 Bucket policy scenarios................................................................. 20 Supported bucket policy operations.............................................. 22 Supported bucket policy conditions...............................................23 S3 Extensions.............................................................................................24 Byte range extensions................................................................... 25 Retention.......................................................................................29 File system enabled....................................................................... 30 Metadata Search........................................................................................30 Assign metadata index values to a bucket...................................... 31 Using encryption with metadata search.........................................34 Assign metadata to objects using the S3 protocol......................... 34 Use metadata search queries.........................................................35 Using Metadata Search from the ECS Java SDK .......................... 40 ECS system metadata and optional attributes............................... 40 S3 and Swift Interoperability.......................................................................41 Create and manage secret keys..................................................................43 Create a key for an object user......................................................43 Create an S3 secret key: self-service............................................ 44 Authenticating with the S3 service............................................................. 47 Using s3curl with ECS................................................................................ 48 Use SDKs to access the S3 service............................................................ 49 Using the Java Amazon SDK..........................................................49 Java SDK client for ECS.................................................................51 Part 2 OpenStack Swift 53 Chapter 2 OpenStack Swift 55 OpenStack Swift support in ECS................................................................56 OpenStack Swift supported operations......................................................56 Swift extensions.........................................................................................58 Swift byte range extensions....................................................................... 58 Updating a byte range within an object..........................................58 Overwriting part of an object.........................................................60 Appending data to an object...........................................................61 Reading multiple byte ranges within an object............................... 62 Elastic Cloud Storage (ECS) 3.1 Data Access Guide 3 CONTENTS Retention................................................................................................... 62 File systemvenabled................................................................................... 63 S3 and Swift interoperability...................................................................... 63 OpenStack Swift authentication.................................................................64 Create Swift users at the ECS Portal............................................ 64 OpenStack Version 1 authentication ............................................. 65 OpenStack Version 2 authentication..............................................67 Authentication using ECS Keystone V3 integration....................... 69 Authorization on Container......................................................................... 72 Part 3 EMC Atmos 75 Chapter 3 EMC Atmos 77 EMC Atmos API support in ECS................................................................. 78 Supported EMC Atmos REST API Calls...................................................... 78 Unsupported EMC Atmos REST API Calls.................................................. 80 Subtenant Support in EMC Atmos REST API Calls..................................... 81 API Extensions............................................................................................ 81 Appending data to an object...........................................................81 ECS support for retention and retention expiration periods for Atmos objects................................................................................82 Part 4 CAS 89 Chapter 4 CAS 91 Setting up CAS support in ECS.................................................................. 92 Cold Storage.............................................................................................. 92 Compliance.................................................................................................93 Platform hardening and Compliance.............................................. 93 Compliance and retention policy....................................................94 Compliance agent..........................................................................95 CAS retention in ECS................................................................................. 96 Advanced retention for CAS applications: event-based retention, litigation hold, and the min/max governor.................................................................98 Set up namespace retention policies......................................................... 104 Create and set up a bucket for a CAS user............................................... 105 Set up a CAS object user.......................................................................... 106 Set up bucket ACLs for CAS..................................................................... 107 ECS Management APIs that support CAS users....................................... 109 Content Addressable Storage (CAS) SDK API support..............................110 Part 5 ECS Management REST API 111 Chapter 5 ECS Management REST API 113 ECS Management REST API introduction..................................................114 Authenticate with the ECS Management REST API...................................114 Authenticate without cookies .......................................................114 Logout.......................................................................................... 116 ECS Management REST API whoami command............................117 ECS Management REST API summary..........................................117 4 Elastic Cloud Storage (ECS) 3.1 Data Access Guide CONTENTS Part 6 HDFS 121 Chapter 6 ECS HDFS Introduction 123 ECS HDFS Introduction............................................................................ 124 Configuring Hadoop to use ECS HDFS .....................................................125 Hadoop authentication modes...................................................................125 Accessing the bucket as a file system.......................................... 126 Bucket Custom Group ACLs and Default Group........................... 127 Hadoop superuser and supergroup............................................... 127 Multi-protocol (cross-head) access............................................. 128 Proxy user....................................................................................128 Equivalence user.......................................................................... 128 Migration from a simple to a Kerberos Hadoop cluster..............................129 Hadoop Kerberos authentication mode........................................ 129 File system interaction.............................................................................. 130 Supported Hadoop applications................................................................ 130 Chapter 7 Configure a simple Hadoop cluster with ECS HDFS 131 Integrate a simple Hadoop cluster with ECS HDFS................................... 132 Install Hortonworks