Open Object Storage as the Foundation for Open Sync and Share in Science Simon Traill – [email protected] https://swiftstack.com © SwiftStack All Rights Reserved Characteristics: File Sync and Share • Deals with unstructured data • Highly distributed over multiple geographic regions • Highly concurrent access • Requires infrastructure resilience • Requires high durability • Ever Growing! 2 © SwiftStack All Rights Reserved Characteristics: Object Storage • Designed for unstructured data • Consistency Model enables multi datacenter deployments • Scales Linearly • Routes around hardware failure • Data is highly durable • Start small and expand later 3 © SwiftStack All Rights Reserved Anatomy of a SwiftStack Cluster Applications SwiftStack Controller File Services (and Agents) SwiftStack NFSSwift API SMB S3 API Controller Proxy Services • Deployment and Provisioning over Account, Container and Object Services ZeroMQ based fabric Consistency Processes • Monitoring & Utilization SwiftStack Agents • Metrics and Reports Standard servers, drives, & networking • Services exposed via Site 1 Site 2 Site 3 REST API © SwiftStack All Rights Reserved 4 Anatomy of a SwiftStack Cluster Applications Access via Swift and File Services S3 REST APIs SwiftStack NFSSwift API SMB S3 API Controller Proxy Services • PUT to upload • GET to download Account, Container and Object Services • POST to modify Consistency Processes • DELETE to remove SwiftStack Agents Standard servers, drives, & networking Site 1 Site 2 Site 3 © SwiftStack All Rights Reserved 5 Anatomy of a SwiftStack Cluster Applications Proxy Services File Services SwiftStack NFSSwift API SMB S3 API • Serve client requests via Controller Swift / S3 Proxy Services • Each object stored as many replicas: Account, Container and Object Services OBJECT NODES Consistency Processes SwiftStack Agents HTTP Standard servers, drives, & networking PROX Site 1 Site 2 Site 3 Y © SwiftStack All Rights Reserved 6 Anatomy of a SwiftStack Cluster Applications File Services SwiftStack NFSSwift API SMB S3 API ACO Services Controller Proxy Services • Store data on disk. • All data in Swift is Account, Container and Object Services namespaced into an account, container and http://example.com/v1/account/container/object Consistency Processes object. SwiftStack Agents Standard servers, drives, & networking Site 1 Site 2 Site 3 © SwiftStack All Rights Reserved 7 Anatomy of a SwiftStack Cluster Applications File Services SwiftStack NFSSwift API SMB S3 API Consistency Controller Processes Proxy Services Account, Container and Object Services • Constantly audit and replicate data to Consistency Processes where it is needed SwiftStack Agents Standard servers, drives, & networking Site 1 Site 2 Site 3 © SwiftStack All Rights Reserved 8 Anatomy of a SwiftStack Cluster Applications Commodity Hardware File Services SwiftStack NFSSwift API SMB S3 API Controller • Standard x86 servers Proxy Services run Swift processes • Data is initially placed Account, Container and Object Services as uniquely as Consistency Processes possible across disks, servers, racks and SwiftStack Agents sites Standard servers, drives, & networking • Consistency processes Site 1 Site 2 Site 3 replicate in the background where needed © SwiftStack All Rights Reserved 9 File Sync and Share 10 © SwiftStack All Rights Reserved FS&S Integration Points • Scale capacity as adoption rises (and preferably not before…) • Provision space for users easily Ideally: • Have dual access to data – via both FS&S and Swift / S3 aware applications 11 © SwiftStack All Rights Reserved Scaling 12 © SwiftStack All Rights Reserved Scaling Hardware with Swift - Scale by the box, or scale by the rack. File Services NFS SMB Proxy Services Account, Container and Object Services - Use SSDs for account and container storage. - Think carefully about your load balancing strategy SwiftStack Reference Architectures: http://learn.swiftstack.com/rs/034-CBF-009/images/DataSheet-SwiftStack-Hardware-Reference-Architecture-2016-06-03.pdf 13 © SwiftStack All Rights Reserved Scale a box at a time: PACO 14 © SwiftStack All Rights Reserved Real World Example: Scale By the Box Load Balancer SwiftStack HA (10G) Controller LEGEND • Outward-facing (10G) • Cluster-facing (10G) Core Switch Core Switch Aggregation Aggregation Switch Switch Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 3 ToR Switch (10G) ToR Switch (10G) ToR Switch (10G) Proxy | Account | Proxy | Account | Proxy | Account | Container | Object Container | Object Container | Object 15 © SwiftStack All Rights Reserved Real World Example: Scale By the Box Load Balancer SwiftStack HA (10G) Controller LEGEND • Outward-facing (10G) • Cluster-facing (10G) Core Switch Core Switch Aggregation Aggregation Switch Switch Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 3 ToR Switch (10G) ToR Switch (10G) ToR Switch (10G) Proxy | Account | Proxy | Account | Proxy | Account | Container | Object Container | Object Container | Object Proxy | Account | Container | Object 16 © SwiftStack All Rights Reserved Real World Example: Scale By the Box Load Balancer SwiftStack HA (10G) Controller LEGEND • Outward-facing (10G) • Cluster-facing (10G) Core Switch Core Switch Aggregation Aggregation Switch Switch Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 3 ToR Switch (10G) ToR Switch (10G) ToR Switch (10G) Proxy | Account | Proxy | Account | Proxy | Account | Container | Object Container | Object Container | Object Proxy | Account | Proxy | Account | Container | Object Container | Object 17 © SwiftStack All Rights Reserved Real World Example: Scale By the Box Load Balancer SwiftStack HA (10G) Controller LEGEND • Outward-facing (10G) • Cluster-facing (10G) Core Switch Core Switch Aggregation Aggregation Switch Switch Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 3 ToR Switch (10G) ToR Switch (10G) ToR Switch (10G) Proxy | Account | Proxy | Account | Proxy | Account | Container | Object Container | Object Container | Object Proxy | Account | Proxy | Account | Proxy | Account | Container | Object Container | Object Container | Object 18 © SwiftStack All Rights Reserved Scale by a rack at a time: P/ACO* * Or PACO. 19 © SwiftStack All Rights Reserved Real World Example: Scale By the Rack Load Balancer SwiftStack Firewall HA (10G) Controller LEGEND • Outward-facing (10G) • Cluster-facing (10G) • Controller network (1G) Core Switch Core Switch • Out-of-band (1G) Aggregation Aggregation Switch Switch Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 3 ToR Switch (10G) ToR Switch (10G) ToR Switch (10G) Proxy Proxy Proxy Account | Container | Account | Container | Account | Container | Object Object Object Account | Container | Account | Container | Account | Container | Object Object Object Account | Container | Account | Container | Account | Container | Object Object Object Mgmt Switch (1G) Mgmt Switch (1G) Mgmt Switch (1G) 20 © SwiftStack All Rights Reserved Real World Example: Scale By the Rack Load Balancer SwiftStack Firewall HA (10G) Controller LEGEND • Outward-facing (10G) • Cluster-facing (10G) • Controller network (1G) Core Switch Core Switch • Out-of-band (1G) Aggregation Aggregation Switch Switch Rack 3 Zone Rack 1 | Zone 1 Rack 2 | Zone 2 Rack 4 Zone 3 4 ToR Switch (10G) ToR Switch (10G) ToR Switch (10G) ToR Switch (10G) Proxy Proxy Proxy Proxy Account | Container | Account | Container | Account | Container | Account | Container | Object Object Object Object Account | Container | Account | Container | Account | Container | Account | Container | Object Object Object Object Account | Container | Account | Container | Account | Container | Account | Container | Object Object Object Object Mgmt Switch (1G) Mgmt Switch (1G) Mgmt Switch (1G) Mgmt Switch (1G) 21 © SwiftStack All Rights Reserved Provisioning 22 © SwiftStack All Rights Reserved Provisioning Strategy - SwiftStack, and every FS&S solution spoken about earlier, can authenticate users using LDAP, including Active Directory - Combine Swift account auto provisioning with FS&S quotas for the path of least resistance - Not all FS&S products optimise their storage strategy for Swift. - Swift performs best when objects are reasonably well dispersed between accounts and containers: http://example.com/v1/account/container/object - Dual access – where FS&S file share the same names as objects in Swift - is easier to accommodate, without added need for ACLS - Examine your chosen FS&S provider and choose provisioning strategies that enable the above where possible. 23 © SwiftStack All Rights Reserved Example: Provisioning With - OwnCloudOwnCloud utilises two types of logical storage. - One Primary storage area: - Uses single container in swift - Renames Objects >> Will not scale; no dual access model - Many External Storage areas: - Can be tied to many accounts / containers - Do not rename objects - Quotas operate on user level. - Best practices then: - Provision user folders using External Storage - Apply quotas on OwnCloud user level SwiftStack and OwnCloud integration guide: https://goo.gl/fkXoQU 24 © SwiftStack All Rights Reserved Example: Provisioning With OwnCloud 1. Add user to “owncloud_users” LDAP group in your directory.2. Swift: Create a container for straill_home: # curl ±X PUT ±H `x-auth-token: XXXX' http://swift.evaluation.lan/v1/AUTH_straill/home_straill 3. OwnCloud: provision user’s external storage with OCC CLI: # cat << `EOF' > /tmp/straill.json [ { "mount_point": ª\//home/straill", "storage": "\\OC\\Files\\Storage\\Swift", "authentication_type": "openstack::openstack", "configuration": { "service_name":