OpenSDS- An Industry Wide Collaboration For SDS Management Lawrence Lai – Huawei Rakesh Jain – IBM 1 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. State of Storage Management - Fragmented Framework CLOUD ENTERPRISE • How to connect storage to a framework? • How to provision storage for multiple frameworks? IT Integration Plug-in Plug-in Plug-in Plug-in Plug-in Plug-in Plug-in Controller • Which controller to use for the framework? Flocker Cinder Manila OceanStor • Is my storage supported by the controller? RexRay libStorage Ceph Cinder Manila Swift CoprHD 10+ drivers RexRay ViPR Ceph 60+Cinder drivers 10+Manila drivers Swift DJ OceanStorDJothers • Does it allow 3rd-party /IT integration? RexRay libStorage Ceph 60+ drivers 10+ drivers Swift CoprHD 60+ drivers 10+ drivers DJ Storage • Is there a storage driver for the controller? • Will I be able to leverage enterprise storage features? Local Storage SAN NAS AFA Scale-Out Cloud • Can I use commodity hardware? 2 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Storage Pains Enterprise Customers Want a common solution to manage storage needs for both internal and external customers How about a single SDS controller to connect to all frameworks and storage backends? Developers Want a consistent method of consuming storage easily How about a standard API that abstracts storage complexity? Vendors Want to enable their storage everywhere How about standardization of storage controls that works with all frameworks? 3 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Overview OpenSDS is an open source community working to address storage integration challenges, particularly in scale-out cloud native environments with heterogeneous storage platforms. Cross-Community Collaboration 4 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Mission and Goals The OpenSDS project promotes the use of simplified storage interfaces using a scalable storage controller architecture with open standard APIs, with the objective of providing application-oriented storage services. To develop the open source core Code components and API’s needed for building an SDS solution Community To grow a community building value for end users and extends to other open-source communities Goals To collaborate among vendors, users, Collaborate and standards bodies to solve real- world storage management problems To get broad industry acceptance amongst Acceptance users and vendors with the implementation and adoption of OpenSDS 5 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Timeline Project Proposal Dell-EMC Joins New Members! Sep 2016 Dec 2016 Jun 2017 •Dell-EMC, •Hitachi, Huawei, •CloudNativeCon Huawei, Hitachi, Fujitsu, Western Berlin Intel •Linux Digital, OSU, •IBM Foundation Vodafone •NTT Project Proposal •Yahoo Japan First Contact Project Launched Prototype Demo Mar 2016 Nov 2016 Apr 2017 6 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Key Value Propositions-For End-users Cloud Transformation Optimized Utilization No Vendor Lock-in End-users can transform their Virtualization of on-demand storage Common integration and existing storage into a platform for resources, improving storage standardized API’s keep storage cloud native workloads resource utilization ecosystem open 7 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Key Value Propositions – For Developers • Standard API offers a consistent method of consuming storage • Abstraction of storage complexity enables developers to focus on their application design • Develop once, run everywhere becomes a reality with a common storage control platform • Accelerate application development without worrying about storage infrastructure 8 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Key Value Propositions – For Vendors • Unified SDS standards and development of open SDS solutions • Efficient industry-wide collaboration and development • Leverage OpenSDS ecosystem to build new solutions and businesses quickly • Win-Win-Win for customers, partners, and company engineering and business • Accelerate cloud transition leveraging the OpenSDS community • Interoperability with OpenSDS solutions with standardization • Reduce development cycle and improve efficiency so vendors can focus on real innovation instead of doing plumbing work 9 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. OpenSDS Project Framework OpenSDS Plug-ins Dashboard Cloud Native Computing Foundation - CSI Monitoring Microsoft Azure OpenStack Others GCE AWS Azure HEC Cloud Stack Metering Docker Kubernetes Mesos Foundry TOOLS & EXTENSIONS Performance COMPUTE INTEGRATION CLOUD INTEGRATION Logging OpenSDS Controller Backup OPENSDS API DR Block File Object Services Services Services Migration SOUTHBOUND DRIVERS AND PROTOCOLS Tiering Native Drivers Cinder Manila libStorage Swordfish Others Forecast DATA PLANE COMPONENTS Analytics Local Storage SAN NAS NVMeF Others Others 10 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Technical Introduction 11 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Controller Project - The Universal Storage Controller A universal storage controller with scalable architecture and open standard APIs OpenSDS Controller Open Standard API Discovery, Pooling Provisioning, Lifecycle Open API for block, file, and object services OPENSDS API Block File Object Services Services Services Policy-Based Orchestration SOUTHBOUND DRIVERS AND PROTOCOLS Policy-based controls for provisioning, Native Cinder Manila libStorage Swordfish Others lifecycle management and more Drivers Wide Storage Support Comes with support for wide range of Cinder, and Manila storage drivers Local HDD/SSD AFA SAN NAS Cloud Enterprise Class Enabler OpenSDS API’s and policies enable the use of enterprise-class storage features 12 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Plug-ins Project – Compute Framework Plug-Ins A group of plug-ins enabling seamless integration with common compute and application frameworks Common Standard Plug-in Use a single common plug-in for a framework to access OpenSDS storage services CLOUD ENTERPRISE Seamless Integration Plug-in Plug-in Plug-in Plug-in Plug-in Plug-in Plug-in Plug-in Plug-ins enable each framework to leverage OpenSDS storage services seamlessly OpenSDS Controller Discovery, Pooling Provisioning, Lifecycle Simplify Storage Abstract the underlying storage complexities away from the compute and application frameworks 13 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Design Goals Cloud Universal Storage Control Orchestrator Single control plane for storage management Abstract Storage Management Host Discover – Available storage resources and their capabilities Aggregate – Physical resources gathered into pools Virtualize – Applications remain ignorant of storage physicality Anonymize – Vendor neutral Runtime OpenSDS OpenSDS OpenSDS Cloud Native Deployment and Scaling Plugin Host Vendor-specific code confined to a single process Host Rules Support distributed and/or centralized server deployment model Implicitly HA and scalable Pool Pool Pool Rules-based Centralized Controls Simple rules map logical storage requests into physical storage requests 14 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Design Requirements Cluster-level View of Storage Federate/aggregate multiple disparate storage systems in to a seamless whole Client and Target (Vendor) agnostic Whole Storage Life-cycle Management Full life-cycle management of storage (creation, access, destruction) Universal Client Code No vendor code in client Zero Client Configuration Common API For all compute and orchestration clients Expose and abstract enterprise storage functionalities 15 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. OpenSDS Design CLOUD ENTERPRISE Common Interface Universal OpenSDS plug-ins for different frameworks. Open API for integration Plug-in Plug-in Plug-in Plug-in Plug-in Plug-in Plug-in IT Integration Policy-Based Controls API Pooling of storage resources with policy-based CONTROLLER controls for provisioning, data protection, lifecycle management and more Discovery, Pooling, Lifecycle, Data Protection OpenStack OpenSDS Other Wide Storage Support drivers native drivers Comes with support for range of Cinder and Manila storage drivers. Extensible for new storage Local Storage SAN NAS NVMf Scale-Out Cloud Enterprise Class OpenSDS API’s and policies support x86 HW and enable the use of enterprise-class storage features 16 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Advance Storage Controls Applications Advance API’s OpenSDS provides API’s to drive policies for lifecycle, tiering, QoS, data protection etc. OpenSDS Controller Block File Object Tiering Caching QoS Lifecycle Snapshot Mirror Backup Clone Replication Migration Sharing Groups Dedup Compress Encrypt Quota High-end storage Converged storage All-Flash storage Distributed storage 17 2017 Storage Developer Conference. © OpenSDS All Rights Reserved. Tenant Concept • Storage Profile: A storage service profile created Storage Profile by the Admin – maps to one or more Virtual Pools - Name {profile name} - Tier0 • Service Catalog: A list of Storage Profiles available Service Catalog for storage provisioning requests Storage Profile - Name {profile name} request storage • Storage Request: A storage provisioning request - Tier0 by a Tenant based on a Storage