Open Object Storage as the Foundation for Open Sync and Share in Science
Simon Traill – [email protected] https://swiftstack.com
© SwiftStack All Rights Reserved Characteristics: File Sync and Share
• Deals with unstructured data • Highly distributed over multiple geographic regions • Highly concurrent access • Requires infrastructure resilience • Requires high durability • Ever Growing!
2 © SwiftStack All Rights Reserved Characteristics: Object Storage
• Designed for unstructured data • Consistency Model enables multi datacenter deployments • Scales Linearly • Routes around hardware failure • Data is highly durable • Start small and expand later
3 © SwiftStack All Rights Reserved Anatomy of a SwiftStack Cluster
Applications SwiftStack Controller File Services (and Agents) SwiftStack NFSSwift API SMB S3 API Controller Proxy Services • Deployment and Provisioning over Account, Container and Object Services ZeroMQ based fabric Consistency Processes • Monitoring & Utilization SwiftStack Agents • Metrics and Reports Standard servers, drives, & networking • Services exposed via
Site 1 Site 2 Site 3 REST API
© SwiftStack All Rights Reserved 4 Anatomy of a SwiftStack Cluster
Applications Access via Swift and File Services S3 REST APIs SwiftStack NFSSwift API SMB S3 API Controller Proxy Services • PUT to upload • GET to download Account, Container and Object Services • POST to modify Consistency Processes • DELETE to remove
SwiftStack Agents
Standard servers, drives, & networking
Site 1 Site 2 Site 3
© SwiftStack All Rights Reserved 5 Anatomy of a SwiftStack Cluster
Applications Proxy Services File Services SwiftStack NFSSwift API SMB S3 API • Serve client requests via Controller Swift / S3 Proxy Services • Each object stored as many replicas: Account, Container and Object Services OBJECT NODES Consistency Processes
SwiftStack Agents HTTP Standard servers, drives, & networking PROX Site 1 Site 2 Site 3 Y
© SwiftStack All Rights Reserved 6 Anatomy of a SwiftStack Cluster
Applications
File Services SwiftStack NFSSwift API SMB S3 API ACO Services Controller Proxy Services • Store data on disk. • All data in Swift is Account, Container and Object Services namespaced into an account, container and http://example.com/v1/account/container/object Consistency Processes object.
SwiftStack Agents
Standard servers, drives, & networking
Site 1 Site 2 Site 3
© SwiftStack All Rights Reserved 7 Anatomy of a SwiftStack Cluster
Applications
File Services SwiftStack NFSSwift API SMB S3 API Consistency Controller Processes Proxy Services
Account, Container and Object Services • Constantly audit and replicate data to Consistency Processes where it is needed
SwiftStack Agents
Standard servers, drives, & networking
Site 1 Site 2 Site 3
© SwiftStack All Rights Reserved 8 Anatomy of a SwiftStack Cluster
Applications Commodity Hardware File Services SwiftStack NFSSwift API SMB S3 API Controller • Standard x86 servers Proxy Services run Swift processes • Data is initially placed Account, Container and Object Services as uniquely as
Consistency Processes possible across disks, servers, racks and SwiftStack Agents sites Standard servers, drives, & networking • Consistency processes
Site 1 Site 2 Site 3 replicate in the background where needed © SwiftStack All Rights Reserved 9 File Sync and Share
10 © SwiftStack All Rights Reserved FS&S Integration Points
• Scale capacity as adoption rises (and preferably not before…) • Provision space for users easily Ideally: • Have dual access to data – via both FS&S and Swift / S3 aware applications
11 © SwiftStack All Rights Reserved Scaling
12 © SwiftStack All Rights Reserved Scaling Hardware with Swift
- Scale by the box, or scale by the rack. File Services NFS SMB
Proxy Services
Account, Container and Object Services
- Use SSDs for account and container storage. - Think carefully about your load balancing strategy
SwiftStack Reference Architectures: http://learn.swiftstack.com/rs/034-CBF-009/images/DataSheet-SwiftStack-Hardware-Reference-Architecture-2016-06-03.pdf
13 © SwiftStack All Rights Reserved Scale a box at a time: PACO
14 © SwiftStack All Rights Reserved Real World Example: Scale By the Box
Load Balancer SwiftStack HA (10G) Controller
LEGEND • Outward-facing (10G) • Cluster-facing (10G) Core Switch Core Switch
Aggregation Aggregation Switch Switch
Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 3
ToR Switch (10G) ToR Switch (10G) ToR Switch (10G)
Proxy | Account | Proxy | Account | Proxy | Account | Container | Object Container | Object Container | Object
15 © SwiftStack All Rights Reserved Real World Example: Scale By the Box
Load Balancer SwiftStack HA (10G) Controller
LEGEND • Outward-facing (10G) • Cluster-facing (10G) Core Switch Core Switch
Aggregation Aggregation Switch Switch
Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 3
ToR Switch (10G) ToR Switch (10G) ToR Switch (10G)
Proxy | Account | Proxy | Account | Proxy | Account | Container | Object Container | Object Container | Object
Proxy | Account | Container | Object
16 © SwiftStack All Rights Reserved Real World Example: Scale By the Box
Load Balancer SwiftStack HA (10G) Controller
LEGEND • Outward-facing (10G) • Cluster-facing (10G) Core Switch Core Switch
Aggregation Aggregation Switch Switch
Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 3
ToR Switch (10G) ToR Switch (10G) ToR Switch (10G)
Proxy | Account | Proxy | Account | Proxy | Account | Container | Object Container | Object Container | Object
Proxy | Account | Proxy | Account | Container | Object Container | Object
17 © SwiftStack All Rights Reserved Real World Example: Scale By the Box
Load Balancer SwiftStack HA (10G) Controller
LEGEND • Outward-facing (10G) • Cluster-facing (10G) Core Switch Core Switch
Aggregation Aggregation Switch Switch
Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 3
ToR Switch (10G) ToR Switch (10G) ToR Switch (10G)
Proxy | Account | Proxy | Account | Proxy | Account | Container | Object Container | Object Container | Object
Proxy | Account | Proxy | Account | Proxy | Account | Container | Object Container | Object Container | Object
18 © SwiftStack All Rights Reserved Scale by a rack at a time: P/ACO*
* Or PACO.
19 © SwiftStack All Rights Reserved Real World Example: Scale By the Rack
Load Balancer SwiftStack Firewall HA (10G) Controller LEGEND • Outward-facing (10G) • Cluster-facing (10G) • Controller network (1G) Core Switch Core Switch • Out-of-band (1G)
Aggregation Aggregation Switch Switch
Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 3
ToR Switch (10G) ToR Switch (10G) ToR Switch (10G)
Proxy Proxy Proxy
Account | Container | Account | Container | Account | Container | Object Object Object
Account | Container | Account | Container | Account | Container | Object Object Object
Account | Container | Account | Container | Account | Container | Object Object Object
Mgmt Switch (1G) Mgmt Switch (1G) Mgmt Switch (1G)
20 © SwiftStack All Rights Reserved Real World Example: Scale By the Rack
Load Balancer SwiftStack Firewall HA (10G) Controller LEGEND • Outward-facing (10G) • Cluster-facing (10G) • Controller network (1G) Core Switch Core Switch • Out-of-band (1G)
Aggregation Aggregation Switch Switch
Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 Rack 4 Zone 3 4 ToR Switch (10G) ToR Switch (10G) ToR Switch (10G) ToR Switch (10G)
Proxy Proxy Proxy Proxy
Account | Container | Account | Container | Account | Container | Account | Container | Object Object Object Object
Account | Container | Account | Container | Account | Container | Account | Container | Object Object Object Object
Account | Container | Account | Container | Account | Container | Account | Container | Object Object Object Object
Mgmt Switch (1G) Mgmt Switch (1G) Mgmt Switch (1G) Mgmt Switch (1G)
21 © SwiftStack All Rights Reserved Provisioning
22 © SwiftStack All Rights Reserved Provisioning Strategy - SwiftStack, and every FS&S solution spoken about earlier, can authenticate users using LDAP, including Active Directory - Combine Swift account auto provisioning with FS&S quotas for the path of least resistance - Not all FS&S products optimise their storage strategy for Swift. - Swift performs best when objects are reasonably well dispersed between accounts and containers: http://example.com/v1/account/container/object
- Dual access – where FS&S file share the same names as objects in Swift - is easier to accommodate, without added need for ACLS - Examine your chosen FS&S provider and choose provisioning strategies that enable the above where possible.
23 © SwiftStack All Rights Reserved Example: Provisioning With
- OwnCloudOwnCloud utilises two types of logical storage. - One Primary storage area: - Uses single container in swift - Renames Objects >> Will not scale; no dual access model - Many External Storage areas: - Can be tied to many accounts / containers - Do not rename objects - Quotas operate on user level. - Best practices then: - Provision user folders using External Storage - Apply quotas on OwnCloud user level SwiftStack and OwnCloud integration guide: https://goo.gl/fkXoQU
24 © SwiftStack All Rights Reserved Example: Provisioning With OwnCloud 1. Add user to “owncloud_users” LDAP group in your directory.2. Swift: Create a container for straill_home: # curl –X PUT –H ‘x-auth-token: XXXX’ http://swift.evaluation.lan/v1/AUTH_straill/home_straill 3. OwnCloud: provision user’s external storage with OCC CLI:
# cat << ‘EOF’ > /tmp/straill.json [ { "mount_point": “\//home/straill", "storage": "\\OC\\Files\\Storage\\Swift", "authentication_type": "openstack::openstack", "configuration": { "service_name": "Swift Object Storage (http:\/\/swift.evaluation.lan)", "region": "default", "bucket": "straill_home", "timeout": "", "user": "owncloud", "password": "owncloud", "tenant": "AUTH_straill", "url": "http:\/\/swift.evaluation.lan\/auth\/v2.0" }, "options": { "encrypt": true, "previews": true, "enable_sharing": false, "filesystem_check_changes": 1 }, "applicable_users": [ "straill" ], "applicable_groups": [] } ] EOF # occ files_external:import /tmp/straill.json
3. OwnCloud: Set user quota via user provisioning REST API:
# curl -X PUT http://administrator:[email protected]/ocs/v1.php/cloud/users/straill -d key="quota" -d value="100MB" 25 © SwiftStack All Rights Reserved Example: Provisioning With Seafile
- Seafile professional supports Swift and S3 backends. - Uses three containers in one account at most to store objects, named with UUIDS. - Won’t scale out of the box, and no dual access, but… - ...is suitable for use with SwiftStack autosharder middleware to remediate scaling concerns. - Quotas operate on user level, as with OwnCloud. - Best practices (currently): - Enable Swift autosharding for Seafile account - Allow users to handle their own libraries or provision via Seafile API
26 © SwiftStack All Rights Reserved Example: Provisioning With Seafile
1. Add user to “seafile_users” LDAP group in your 2.directory. Enable autosharding for user account via SwiftStack 3. ProvisionAPI. user home directory via Seafile API, as user in question:
# curl -v -d "name=home_straill&desc=home folder" \ -H 'Authorization: Token xxxxxxxx' -H 'Accept: application/json; indent=4' \ http://my.seafile.org:8000/api2/repos/ 4. Set user quota, as admin, via Seafile API:
# curl -v -X PUT -d "storage=1073741824”\ -H "Authorization: Token yyyyyyyy” \ -H 'Accept: application/json; indent=4' \ http://my.seafile.org.:8000/api2/accounts/[email protected]/
27 © SwiftStack All Rights Reserved Summary
- OpenStack Swift and FS&S are largely complimentary - Open source or not, look for deployment and
SwiftStack andprovisioning OwnCloud Integration: strategies that maximize Swift’s https://goo.glperformance/fkXoQU Seafile with Openstack Swift backend: https://manual.seafile.com/deploy_pro/setup_with_swift.html Deploying SwiftStack Object Storage for Storage Made Easy: http://learn.swiftstack.com/rs/034-CBF-009/images/How-To-Guide-Deploying-SwiftStack-Object-Storage-for-Storage-Made-Easy.pdf
28 © SwiftStack All Rights Reserved Thank you.
Simon Traill – [email protected] https://swiftstack.com
© SwiftStack All Rights Reserved