Compass – a Streamlined Openstack Deployment System

2013年11月7日星期四

Compass – A Streamlined OpenStack Deployment System

Shuo Yang Principal Architect of Cloud Computing, US R&D Center

Outline of This Talk

1 Scope of Problem for Compass

2 Compass Explained

3 DRY, Truly Open Deployment

Compass at a Glimpse

Think Big, Start Small

 A General System to Deploy Distributed Systems, Extensibility as a Primary Design Goal

 Not Limited to OpenStack, but Streamlined Our OpenStack Deployment Like a Charm

To Be Open Sourced – Apache 2.0 Soon

 100% Python, 5000 Line of Python Code

 Successfully Deployed Several Dogfood Clusters

 Compass Wiki Page: https://wiki.openstack.org/wiki/Compass

Data Center as a Computer

Open Cloud OS (OpenStack) Open Deployment (Compass)

OpenStackLinux Quantumeth0, lo Nova/proc Cinder/dev Live Auto CD / Deploy GRUB NIC CPU Disk SwitchSwitchSwitch CPUCPUServer DiskStorage NIC CPU Disk (5020) (2285/1285) (N8000/N900)

OpenStack Control Channel LILO/GRUB/LiveCD for OpenStack HW/SW Configuration Deployment Channel

Why We Are Doing This?

Full HW Portfolio in Data Center

 No.1 as Storage Revenue Growth

 No. 2 as `x86 Server Revenue Growth

 Needless to Say, Networking Gears..

OpenStack Makes the Above a Full Global Excellent Telecom Cloud Solution Cloud Solution Provider of year 2012

OpenStack Deployment System Overview

Crowbar TripleO (“under the cloud” mode)

 Pioneer effort, a Ruby web app, Chef based  Attractive concept to OpenStack folks: configuration management deploy OpenStack from OpenStack

Fuel DevStack

 A great web apps, Puppet based  A great tool for simple OpenStack setup: configuration management developers use it

Streamlining the Deployment of a System

Blueprint AI

Rack and Stack Robot

Zero Touch Installation Compass

Real Demo

 Video Clips Recording Our First Deployment

Outline of This Talk

1 Scope of Problem for Compass

2 Compass Explained

3 DRY, Truly Open Deployment

A Life of Deployment – How to Streamline All Controls

Plus a lot of scripts:  Boilerplate  Target system description

Chef / Puppet / …

Host OS Host OS Host OS Cobbler / Razor / …

IPMI Server 1 Server 2 Server n

SNMP

Switch 1 Switch 2 Switch k

Compass Philosophies – Programmability and Extensibility

Programmability – To Be a Programmable Framework

 A Restful API service instead of a web app

Extensibility – NOT To Be A Conglomerate

 Functional modules with plug-in architecture (e.g., 200 Loc to support HP Switch)

 Carefully designed boundary to mature tools like Cobbler, Chef and etc.

 Work WITH the above tools, not to replace any of them.

Compass Internals

Rest rd Huawei UI Client 3 Party ISV UI

Compass -- Open Source Restful API Server

Message bus (RabbitMQ) DB

Hardware Discover Package Deployment OS Provisioning

Huawei Plugin microkernel Chef plugin Ansible plugin Cobbler plugin Razor plugin

Compass – A Programmable Framework

Mapping Fundamental Concepts during Software Deployment into Restful Resources

 Machine – physical compute resource

 Switch – management network for machines

 Cluster – pool of machines managed together

 Host – machine in a cluster

 Adapter – plug-in module to discover/deploy/configure hosts

Resource Example: Machines

GET /machines Lists all discovered machines GET /machines/id Lists a machine’s hardware details and status

POST /machines/id/action Modify the hardware configuration for a machine

Query parameters for filtering

 switch=ID  marker=markerID  address=string  limit=int  cluster=ID  changes-size=dateTime  status=string  details=ID,switch,address,cluster,devices,status,modified

Actions

 modify(hwConfig)

13 13

A Life of OpenStack Deployment -- as Restful Calls

(1) Find all available machines: /machines?switch=0e8756dc (2) Select machines for my system to be deployed on to: {1, 2, 3, 4} (3) Select an adapter, which describes the result system: /?adapter=OpenStack (4) Find components for result systems {“controller”, “compute”} (5) Tell Compass a plan {1 -> “controller”; [2,3,4] -> “compute”} (or specify ‘?auto=true’)

(6) Poll the deployment progress Time

NOT to Be Conglomerate – Plug-in Architecture

Plug-in architecture for configuration management system

 Currently we use Chef as deployment management Engine

 Puppet or Ansible devopers, we want to work with you

Plug-in architecture for Hardware vendors IPMI / SNMP

 Networking gear based bare metal discovery (topolology awareness)

 Currently we are working on RAID Controller for Huawei

 OCP community, what’s your thought

NOT to Be Conglomerate – Roles in the System

Policy Databag (e.g., exclusive; default) OpenStack Cookbook

Lib writer Chef-Server Restful Reqs

Compass snippets kickstart files

App writer Cobbler-server

Outline of This Talk

1 Scope of Problem for Compass

2 Compass Explained

3 DRY, Truly Open Deployment

Extensible in Use Cases OS

ESXi ?

Windows Server?

CentOS

Ubuntu

Huawei OpenStack Hadoop Ceph? Distributed System HP OCP?

Hardware

Extensible in Tool Chain

OS Provisioning Tools

Razor

Cobbler

Chef Puppet Ansible Salt Configuration Management Tools

To Summarize

Open Sourced under Apache 2.0

 General Purpose Deployment System (Ultimate Goal)

 Deploys OpenStack Cloud Infrastructure Smoothly (Dogfoods Done)

 Restful API: Allow Third-party UI Integration

Open to Collaboration Contact us: [email protected] Santa Clara R&D Center, US

Copyright©2013 Huawei Technologies Co., Ltd. All Rights Reserved. The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice.

Compass – Web Enabled Deployment

Simple Web zzz… From baremetal actions (40mins) to OpenStack DC

Internet Intranet Compass Administrators

Administrators Server Farm

Compass Internal: Current Implementation

 Flask for Web Server Layer  RabbitMQ as Message Broker

 Celery for Task Management  Chef as Package Management Executor

 Relational DB as Transaction Records  Cobbler as Host System Management

Compass API Design Convention

Follow OpenStack API conventions

 Authentication  Changes-since query parameter  JSON or XML request/response data formats  Version in URI or header  Paginated collections  Common representation for faults

Allow multi-tenancy as an option ( tenantID not required)

Use POST to create a new resource and return its ID

Use PUT to replace, PATCH/POST for a partial update