Reshaping Storage Accelerate for the Next OpenStack* Generation Data Together Center * OpenStack is a registered trademark of the OpenStack Foundation Agenda • Data Explosion & Storage Pain Points • Software Defined Storage (SDS) vision • OpenStack SDS Controller Prototype • Ceph and Virtual Storage Manager • Fujitsu Hyperscale Storage for OpenStack • OpenStack Swift • Summary OpenStack Summit, Paris, Nov. 3-7, 2014 2 Agenda • Data Explosion & Storage Pain Points • Software Defined Storage (SDS) vision • OpenStack SDS Controller Prototype • Ceph and Virtual Storage Manager • Fujitsu Hyperscale Storage for OpenStack • OpenStack Swift • Summary OpenStack Summit, Paris, Nov. 3-7, 2014 3 Data Explosion: From 2013 to 2020, the digital universe will grow by a factor of 10, from 4.4 ZB to 44 ZB It more than doubles every two years. … cost challenges continue to grow………. ….data complexity is increasing …. And DC’s spend 40% of their budget on storage Data needs are growing at a rate unsustainable with today’s infrastructure and labor costs Source: IDC – The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things - April 2014 OpenStack Summit, Paris, Nov. 3-7, 2014 4 Storage Pain Points Storage Silos (Traditional) Challenges • Application mapped to specific • Cost: silo’d storage solutions appliance - Data Growth • Storage resources optimized to run - Maintenance, Operations & Support specific workload - Infrastructure • Isolated storage resources • Choice: lock-in - Vertically integrated solutions - Flexibility to innovate • Cloud deployment - Limited Scalability - Need for massively shared data Traditional storage management is too complex and inefficient OpenStack Summit, Paris, Nov. 3-7, 2014 5 Agenda • Data Explosion & Storage Pain Points • Software Defined Storage (SDS) vision • OpenStack SDS Controller Prototype • Ceph and Virtual Storage Manager • Fujitsu Hyperscale Storage for OpenStack • OpenStack Swift • Summary OpenStack Summit, Paris, Nov. 3-7, 2014 6 Software Defined Infrastructure (SDI) Services Delivery SERVICE ASSURANCE Policy based dynamic provisioning App/SLA mapping to underlying Infrastructure Application Application Application Application A B C D Orchestration Software Orchestration Software PROVISIONING MANAGEMENT Orchestration provisions, manages and optimally allocates resources based on the unique Infrastructure Attributes requirements of an application Power Performance Security Thermals Utilization Location POOLED RESOURCES Resource Pool Network, Storage and Compute elements are abstracted into resource pools Storage Network Compute Efficient SDI requires Application Infrastructure interaction Servers play a critical role for Compute, Network, Storage, Security, Energy and Manageability OpenStack Summit, Paris, Nov. 3-7, 2014 7 SDS – A Key Component of SDI Dynamic, policy-driven storage resource management Services Delivery Abstracting Software from Hardware, App #1 App #2 App #3 App #4 providing flexibility & scalability Aggregating diverse provider solutions, Orchestration Software increasing flexibility and drive down costs Infrastructure Attributes Provisioning resources dynamically (pay-as- Power Performance Thermals Utilization Location Latency Durability you-grow) increasing efficiency Software Defined Storage Controller Orchestrating application access to diverse Storage Systems storage systems through Service Level Agreements (SLAs), increasing flexibility and handle data complexity SDS is a framework that delivers a scalable, cost-effective solution to serve the needs of tomorrow’s Data Center OpenStack Summit, Paris, Nov. 3-7, 2014 8 SDS Architecture Data Services SDS Controller • Visibility and Control of • Application that runs in ALL storage resources DP to optimize storage Applications • Communication between • Ex: Predictive Analytics Apps, Orchestrator and • Ex: De-Duplication Storage Systems • Ex: Tiering Orchestrator • Allocates storage resources to meet SLA’s Data Services SDS Controller SDN Controller Compute Controller Storage System Storage System Storage System ……….. Storage System [SAN] ……….. [Capacity] [Performance] [NAS] SAN NAS Node JBOD Node Node Node Node SDS : Consolidated Management of Scale-Out and Scale-Up Storage and plugs into SDI OpenStack Summit, Paris, Nov. 3-7, 2014 9 Agenda • Data Explosion & Storage Pain Points • Software Defined Storage (SDS) vision • OpenStack SDS Controller Prototype • Ceph and Virtual Storage Manager • Fujitsu Hyperscale Storage for OpenStack • OpenStack Swift • Summary OpenStack Summit, Paris, Nov. 3-7, 2014 10 SDS Functional Partitioning Free Pool Outside world (UI, CLI, Compute, Network Provision Controllers) Discover Compose Maintain Data Mgmt SDS Controller Block File Object Provision Storage Volume Operations Share Operations Object Operations System (Control Plane) (Control Plane) (Control Plane) OpenStack Summit, Paris, Nov. 3-7, 2014 11 SDS Controller – One Control Plane Heat Horizon CLI Provision Storage System Manage Pools App App Manage Shares App Manage Volumes App (e.g. Nova) attach/ (e.g. VM) (e.g. Nova) (e.g. Webapp) detatch attach/ Provision Volumes volume detatch Template PUT/GET/DELETE share mgmt account, container, Get Storage URL object Cinder Manila Manage volumes, Glance Shares, images Provision Object volume Provisioning, Storage SDS Controller Maintenance Discovery, etc. Cinder+ Manila+ QoS etc. (Cinder+) Glance+ Drivers Drivers Drivers Provisioning, Object Maintenance etc. Metrics, metering Servers Storage Storage Storage E.g., Object Storage – System System System Swift, Ceph, Atmos, Amplidata Ceilometer Work with dev community to identify path to enable SDS Controller natively in OpenStack OpenStack Summit, Paris, Nov. 3-7, 2014 12 SDS Controller Prototype SDS Controller Provision Discover Maintenance Compose Data Type Life (Storage (Storage (Storage (Virtual Pools) Cycle Mgmt System) Capabilities) Systems, Pools) • Expand Cluster (Ceph • Discover storage • CUDL virtual pools • CUDL volumes & shares • Deassgin resource VSM, SwiftStack system, tiers, capabilities • Attach/Detach tiers to • Attach & Detach • Unprovision resource Controller) - driver framework virtual pools volumes & shares • Retire storage system (Swift, Ceph) • Admin UI for pool • Attach object store & virtual pools composition • Clone, Snapshot, • Support for Prototype Scope • Object support Backup, Extend, Shrink maintenance mode • Get volume stats • QoS monitor, alert & Legend enforce Not in Openstack • Policies, data life-cycle Exists in Openstack mgmt. automation Note: Operations may be within storage system and • CUDL = Create, SLO based resource among storage systems Update, Delete, List requests, scheduling • Virtual pools for all operations Publish prototype by end of Dec’14 in external git repo • Data service chaining, app assignment OpenStack Summit, Paris, Nov. 3-7, 2014 13 Agenda • Data Explosion & Storage Pain Points • Software Defined Storage (SDS) vision • OpenStack SDS Controller Prototype • Ceph and Virtual Storage Manager • Fujitsu Hyperscale Storage for OpenStack • OpenStack Swift • Summary OpenStack Summit, Paris, Nov. 3-7, 2014 14 Ceph Introduction An “Open source” distributed storage software that provides object, block and file storage access KEYSTONE SWIFT API GLANCE CINDER NOVA Intel Focus • Optimize Ceph block storage for Qemu/KVM Open stack deployments CEPH Object Gateway CEPH Block Device (RBD) • Deliver Open stack and Ceph S3 and Swift compatible object Virtual block device for virtual machines Reference Architectures gateway and apps CEPH RADOS Storage Cluster • Lead SDS Controller integration with Ceph (prototype and A Reliable, Autonomous, Distributed beyond) Object Store for unstructured data • Ceph is the “only” Open source virtual block storage option. Also provides object, file (experimental) • Strong customer interest - several production implementations already OpenStack Summit, Paris, Nov. 3-7, 2014 15 Intel Virtual Storage Manager - Design Goals Consistent Configuration Segregated Storage Support • VSM controls cluster configuration through pre- • Separate storage assets by function or defined manifests performance – Storage Groups • Operator-selectable configuration options are • Drives with similar characteristics explicitly defined and validated by OEM or System identified – no ambiguous discovery Integrator • Capacity utilization and performance tracked • Reduces maintenance and support cost due to by Storage Group customer mis-configuration, “exotic” configurations Efficient Asset Management Familiar Look and Feel • OpenStack UI schema • Manage server lifetime – add and remove • “Flat” navigation storage at unit of server • Tabular data grouping – minimize drill- • Manage device lifetime - repair storage at down unit of drive • Comprehensive one-page summary • Error and Status Reporting Intel Virtual Storage Manager for Ceph OpenStack Summit, Paris, Nov. 3-7, 2014 16 Virtual Storage Manager Web-based UI . Administrator-friendly interface for management and monitoring Configuration management . Storage Groups aggregate drives with similar performance . Zones aggregate drives within failure domains Capacity Management . Pools segregate allocation by application . Capacity utilization by storage group and pool Cluster Management . Manage capacity growth . Manage Server & disk maintenance Cluster Monitoring . Capacity and performance . OSD, PG, and monitor state VSM RESTful API . Software interface supports automation Management framework = Consistent configuration