Wicked Easy Ceph Storage & OpenStack Deployment with Crowbar

Kamesh Pemmaraju, Dell Neil Levine, Inktank

1 Outline

• Introduction – Dell OpenStack-Powered Cloud Solution – Ceph – Crowbar • Why Crowbar + Ceph in OpenStack? – OpenStack Block Storage gap – Automation, scale • What we have done to enable things? • Customer benefits, drivers – an example • What’s coming next?

2 Clouds Require an Operational Focus

• Clouds demand significant operational and process controls

SW OPS • Operational decisions drive hardware and software decisions HW • We are finding ways to productize operations into best practices

3 Dell OpenStack-Powered Cloud Solution

“Crowbar”

SW CloudOps

Software

SW

OPS Reference OPS Architecture HW Services & Consulting

4 Dell OpenStack-Powered Cloud Solution HW + SW + Services • C6220, C6105,R720, R720XD servers HW reference • Storage and compute architecture Hardware • Force 10 S60 and PowerConnect Configuration • Min 6 nodes. Max 60 nodes • OpenStack Installer and continuous integration (Crowbar) Software • OpenStack cloud SW • Ceph for distributed storage Software • Ubuntu 12.04 (host) • Windows (guests) • (guests) Hypervisor • KVM • Hardware integration – onsite or merge center • Software installation – onsite Deployment and consulting • OpenStack consulting services Services − Assessment, design and installation − Delivered via partnerships with and • HW/Crowbar: Dell ProSupport Support • OpenStack support via Canonical (coming soon) 5 Ceph – Unified Storage Software

OBJECTS VIRTUAL DISKS FILES & DIRECTORIES

CEPH CEPH CEPH BLOCK STORAGE

A powerful S3- and Swift- A distributed virtual block A distributed, scale-out compatible gateway that device that delivers high- filesystem with POSIX brings the power of the performance, cost-effective semantics that provides Ceph Object Store to storage for virtual machines storage for a legacy and modern applications and legacy applications modern applications

6 Ceph – Key Differentiators

• Unified storage platform (Object + Block + File) • Ceph Intelligent Placement (CRUSH) • Ceph Intelligent Devices (Self-healing, P2P) • Ceph Intelligent Objects (Embedded Software Classes) • Ceph Integration (Linux Kernel, OpenStack, Cloudstack)

7 Ceph Architecture APP APP HOST/VM CLIENT

Ceph Object Ceph Block Ceph File LIBRADOS Gateway Device System A allowing (RGW) (RBD) (CephFS) apps to directly A bucket-based REST A reliable and fully- A POSIX-compliant access RADOS, gateway, compatible distributed block distributed file with support for with S3 and Swift device, with a Linux system, with a Linux , C++, Java, kernel client and a kernel client and Python, Ruby, QEMU/KVM driver support for FUSE and PHP

Ceph Storage Cluster (RADOS) A reliable, autonomous, distributed object store comprised of self-healing, self- managing, intelligent storage nodes 8 Ceph RADOS Components M Ceph Monitor Nodes: Ceph OSD Nodes: – Maintain cluster map • One Ceph OSD daemon per disk – Provide consensus for (recommended) distributed decision-making • At least three nodes in a cluster – Must have an odd number • Serve stored objects to clients – These do not serve stored • Intelligently peer to perform objects to clients tasks

• Supports object classes

9

OSD OSD OSD OSD OSD

Ceph OSD FS FS FS FS FS Node

DISK DISK DISK DISK DISK

Ceph Storage M M M Cluster (RADOS) 10

10 What is Crowbar?

Mission: “A Zero Touch Cloud Installer” Servers in boxes to full function cloud in under 2 hours

• Fast & Flexible – Bare metal install including BIOS & RAID config – Users can choose how their system is configured (“barclamps”) • DevOps Embracing – Ongoing Operations Model (DevOps for Clouds) – Leverages & Wraps Opscode Chef • Open – Not specific to OpenStack – Dell using for other Apps – Not restricted to Dell hardware – Apache 2 licensed

11 Images vs. Layers: Overview

Images: Single Unit Layers: Stacked Pieces

Integrations Configuration Application Foo Integrations + Application Bar Applications + Utilities + Operating Utilities System Configuration Operating System

12 Images vs. Layers: Lifecycle

Images: Replacement Layers: Upgrade

Config

Config Config I I

I+A+U+O/ Foo Foo I+A+U+O/ I+A+U+O/ Bar v1 Bar v2

S S S U U

Config Config OS OS

Config Bar v2 I+A+U+O/ S

13 Crowbar Uses the Layer Model for Deployment

APIs, User Access, • Includes all the components required to implement an

& Ecosystem entire cloud infrastructure including ecosystems partners

Partners

Cloud • Pluggable components deploy cloud infrastructure Infrastructure & • Allow for addition of Dell IP and expansion by the Dell IP Extensions community services and customers

Core Components & • Can integrate with Dell existing products Operating Systems • Delivers basic data center services and required cloud

infrastructure. Dell “Crowbar” Dell

Ops Management Ops • Provision bare-metal servers from box to cloud WITHOUT Physical Resources user intervention (other than racking/cabling and some minimal configuration questions) NOT EXCLUSIVE TO DELL HARDWARE

* Crowbar has potential to service other programs beyond OpenStack

14

How Does Crowbar Work?

Orchestration State Machine Apps Apps

Run List Network Network Chef Config Config Server Operating System

Happens! Config. Crowbar Magic Magic Crowbar

Dell “Crowbar” Dell Hardware Config Ops Management Ops DHCP (uses “Sledgehammer”) Server

15 What is a Barclamp?

Barclamp Crowbar API & O/S App Components & Chef Recipes

Partial UI Packages Scripts

• Crowbar API & Partial UI • Adds states and transitions to Orchestration • Customizes UI specific to barclamp • Visual extensions to nodes/networks/utils • Chef Recipes • DevOps description of application

• Enforced repeatable configuration Dell “Crowbar” Dell

Ops Management Ops • O/A App Packages • Operating system dependencies (Debs & RPMs) • Components & Scripts • Ad hoc configuration not covered by Chef

16 Modular Design: Barclamps The Configuration You Want to Deploy

Nagios Ganglia Dashboard APIs, User Access, & Ecosystem Partners

enStratus RightScale Horizon

Nova Swift Glance Cloud Infrastructure & Dell IP Extensions Ceph Force10 Keystone

Crowbar DNS Logging Core Components &

Operating Systems Deployer NTP Dell“Crowbar”

OpsManagement Provisioner BIOS IPMI Physical Resources Network RAID

17 The 4 Stages of Cloud Storage Thinking

“We should do “No SPOFs “How do the Web things the new “App can’t be 2.0 folks do it?” please!” way!” re-architected!”

Use distributed Use Object Use local storage! Use this expensive storage on utility Storage SAN! hardware!

18 19 Ceph Block Storage for Dell Openstack

Integrated: with Cinder and Nova (via qemu-kvm)

Compatible: with Glance

Foundations: no SPOFs, self-healing, CRUSH

Features: Copy-on-Write, Snapshots, Cloning

Integrated: with Crowbar for automation

20 Scale-Out Operations With OpenStack

If you can’t automate it, you can’t scale it

I I I I I I I

Crowbar Dash Nova Nova Nova Cinder API & Nagios

Chef Board Cntrl DB API API

Config

Config Config Config Config Config Services Config U U U U U U U OS OS OS OS OS OS OS

I I I I I I

Compute Nova Nova Nova Nova Nova Nova

Nodes Node Node Node Node Node Node

Config Config Config Config Config U U U U U Config U OS OS OS OS OS OS

I I I I I I

Storage Ceph Mon Ceph Mon Ceph Mon Ceph OSD Ceph OSD Ceph OSD

Nodes Node Node Node Node Node Node

Config

Config

Config

Config Config U U U Config U U U OS OS OS OS OS OS

21 Ceph Related Barclamps for Crowbar

• https://github.com/ceph/barclamp-ceph – ceph.com packages into the Crowbar Admin Node – Ceph OSD roles (ceph-store) – Ceph Mon roles (ceph-mon) • https://github.com/crowbar/barclamp-nova – Hypervisor with RBD (ceph-client)

22 What Have We Done to Enable?

• Dell and Inktank have partnered to bring a complete solution for OpenStack + Ceph + Automated deployment with Crowbar • The joint solution provides: – Crowbar barclamp to – Deploy Ceph clusters automatically in a very short time – Connect those clusters to OpenStack – Professional Services, Support, and Training › Collaborative Support for Dell hardware customers – Joint Solution › Validated against Dell Reference Architectures via the Technology Partner program

23 Customer: University to Deploy OpenStack + Ceph Using Crowbar Situation • University employs close to 900 researchers and receives • In the end, University hundreds of million dollars in Grants as a Top 10 Research chose Dell/Inktank to Institution primarily working on Cancer and Genomic Projects. The University has a need to provide a centralized architect a platform data repository for Researchers in order to insure that would be very compliances concerning retention of data. scalable and provide • The intent of the Data Repository is to provide 2TB of free storage space to each Researcher, with opportunity to lost costs per TB and purchase from the university more capacity at a very was the best of all reasonable cost. worlds that provide Decision Drivers • University investigated using an traditional SAN storage compute and storage solution which was very expenses on a per TB basis. They on the same also investigated using public cloud storage options which hardware. also proved to be too expensive. Finally, they looked at Hadoop for the project but found that it was not a good fit for this use case. 24

1. Dell case study: 7-Eleven; 2, Salesforce.com What’s Next?

• May – Ceph Cuttlefish • Summer 2013 – Dell Grizzly support – Ceph Dumpling (August) • https://github.com/crowbar/crowbar/wiki/Crowbar-2.0 – Crowbar v2.0 work is happening in the open. Check it out and participate!

25

Don’t Miss the OpenStack Summit Sessions

Havana Sessions: – Planning the Ceph Roadmap for Openstack Wednesday April 17, 2013 1:50pm - 2:30pm in Room - B113

– Features for Ceph with Cinder and Beyond Thursday, April 18, 2013 9:50 – 10:30am in Rooms – C120+121+122

26

Contact Information

Reach Kamesh and Neil for additional information: Dell.com/OpenStack Dell.com/Crowbar Inktank.com/Dell [email protected] @kpemmaraju [email protected] @neilwlevine Visit the Dell and Inktank booths in the OpenStack Summit Expo Hall

27