FORMATION OPENSTACK
1 ABOUT THESE TRAINING MATERIALS
2 . 1 TRAINING MATERIALS WRITTEN BY PARTICULE
ex Osones/Alterway Cloud Native experts Devops experts Our training offers: https://particule.io/en/trainings/ Sources: https://github.com/particuleio/formations/ HTML/PDF: https://particule.io/formations/
2 . 2 COPYRIGHT License: Creative Commons BY-SA 4.0 Copyright © 2014-2019 alter way Cloud Consulting Copyright © 2020 particule. Since 2020, all commits are the property of their owners
2 . 3 INTRODUCTION
3 . 1 GOALS OF THE TRAINING: CLOUD Understand concepts and benefits of cloud Know the vocabulary related to cloud Overview of cloud market players and focus on AWS and OpenStack Know how to take advantage of IaaS Be able to decide what is cloud compatible or not Adapt its system administration and development methods to a cloud environment
3 . 2 GOALS OF THE TRAINING: OPENSTACK Discover OpenStack and use its different services Know how the project works and its capabilities Understand the internals of each OpenStack component Be able to make the right configuration choices Be capable of manually deploying an OpenStack cloud providing IaaS Know the best practices for deploying OpenStack Be able to track down the cause of an error in OpenStack Be able how to react in front of a bug and know the fix process
3 . 3 REQUIREMENTS Advanced sys admin skills for Linux such as Ubuntu, Red Hat or Debian, including: Package management Configuration files and services handling LVM (Logical Volume Management) and filesystems Notions: Virtualization: KVM (Kernel-based Virtual Machine), libvirt Network: iptables, namespaces SQL Optional: Comfortable in a Python environment
3 . 4 CLOUD, OVERVIEW
4 . 1 FORMAL DEFINITION
4 . 2 SPECIFICATIONS Provide one or more service(s)... Self service Through the network Sharing resources Fast elasticity Metering Inspired by the NIST definition https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800- 145.pdf
4 . 3 SELF SERVICE User goes directly to the service No humain intermediary Immediate responses Services catalog for their discovery
4 . 4 THROUGH THE NETWORK User uses the service through the network The service provider is remote to the consumer Network = internet or not Usage of standard network protocols (typically: HTTP)
4 . 5 SHARING RESSOURCES A cloud provided services to multiple users/organizations (multi-tenant) Tenant or project: logical isolation of resources Resources are available in large quantities (considered unlimited) Resources usage is not visible Accurate location of resources is not visible
4 . 6 FAST ELASTICITY Provisionning and deletion of resources almost instantaneous Enables scaling Ability to automate these scaling actions Virtually no limit to this elasticity
4 . 7 METERING Usage of cloud resources is monitored by the provider The provider can do capacity planning and billing from these informations User is therefore billed depending on accurate usage of resources User can take advantage of these informations
4 . 8 MODELS service models: IaaS, PaaS, SaaS deployment models: public, private, hybrid
4 . 9 IAAS Infrastructure as a Service Infrastructure: Compute Storage Network Target users: administrators (system, storage, network)
4 . 10 PAAS Platform as a Service Two concepts: Environment to develop/deploy an application (language/framework specific - example: Python/Django) Higher level resources than infrastructure, example: DBMS Target users: application developers
4 . 11 SAAS Software as a Service Target users: end users Not to be confused with the economic definition of SaaS
4 . 12 SOMETHING AS A SERVICE? Load balancing as a Service (Infra) Database as a Service (Platform) MyApplication as a Service (Software) etc.
4 . 13 SERVICE MODELS IN ONE DIAGRAM
IaaS - PaaS - SaaS (source: Wikipedia) 4 . 14 PUBLIC OR PRIVATE CLOUD? Who is it for? Public: everyone, available on internet Private: to an organization, available on its network
4 . 15 HYBRID CLOUD Usage of multiple public and/or private clouds Attractive concept but implementation is hard a priori Some use cases fit perfectly Continuous integration (CI) Incentives: Avoid lock-in Cloud bursting
4 . 16 VIRTUALIZATION INSTANT Let's make it clear. Virtualization is a technology that can implement the compute function A cloud providing compute resources can use virtualization But it can also use: Bare-metal Containers
4 . 17 APIS ARE KEY Reminder: API stands for Application Programming Interface In the software sense: Interface for a program to use a library In the cloud sense: Interface for a program to use a service (XaaS) Programming interface (through the network, often HTTP) Explicit boundary between the provider and the user Defines how the user interacts with the cloud to manage their resources Manages: CRUD (Create, Read, Update, Delete)
4 . 18 API REST One ressource == one URI (Uniform Resource Identifier) Usage of HTTP verbs to define operations (CRUD) GET POST PUT DELETE Usage of HTTP return codes Resources are represented in the HTTP responses' body
4 . 19 REST - EXAMPLES
GET http://endpoint/volumes/ GET http://endpoint/volumes/?size=10 POST http://endpoint/volumes/ DELETE http://endpoint/volumes/xyz
4 . 20 REAL EXAMPLE
GET /v2.0/networks/d32019d3-bc6e-4319-9c1d-6722fc136a22 { "network":{ "status":"ACTIVE", "subnets":[ "54d6f61d-db07-451c-9ab3-b9609b6b6f0b" ], "name":"private-network", "provider:physical_network":null, "admin_state_up":true, "tenant_id":"4fd44f30292945e481c7b8a0c8908869", "provider:network_type":"local", "router:external":true, "shared":true, "id":"d32019d3-bc6e-4319-9c1d-6722fc136a22", "provider:segmentation_id":null } }
4 . 21 WHY CLOUD? ECONOMICAL POINT OF VIEW Consider IT resources as service provider resources Shift the "investment" budget (Capex) to the "operation" budget (Opex) Cut costs by sharing resources, and maybe with economies of scale Reduce delivery times Match costs to the real usage of resources
4 . 22 WHY CLOUD? TECHNICAL POINT OF VIEW Abstract from the lower layers (server, network, OS, storage) Get rid of the technical administration of resources and services (DB, firewalls, load-balancing, etc.) Design infrastructures which can scale on the fly Act on resources through lines of code and manage infrastructures "as code"
4 . 23 MARKET
4 . 24 AMAZON WEB SERVICES (AWS), LEADER
AWS logo
Started in 2006 At first: "e-commerce" web services for developers Then: other services for developers And finally: infrastructure resources Recently, SaaS
4 . 25 PUBLIC IAAS ALTERNATIVES TO AWS Google Cloud Platform Microsoft Azure Rackspace DreamHost DigitalOcean In France: Cloudwatt (Orange Business Services) Numergy (SFR) OVH Ikoula Scaleway Outscale
4 . 26 PRIVATE IAAS OpenStack CloudStack Eucalyptus OpenNebula
4 . 27 OPENSTACK IN A FEW WORDS
OpenStack logo
Was born in 2010 OpenStack Foundation since 2012 Written in Python and distributed under Apache 2.0 license Large support from the industry and various contributions
4 . 28 PUBLIC PAAS EXAMPLES Amazon Elastic Beanstalk (https://aws.amazon.com/fr/elasticbeanstalk) Google App Engine (https://cloud.google.com/appengine) Heroku (https://www.heroku.com)
4 . 29 PRIVATE PAAS SOLUTIONS Cloud Foundry, Foundation (https://www.cloudfoundry.org) OpenShift, Red Hat (https://www.openshift.org) Solum, OpenStack (https://wiki.openstack.org/wiki/Solum)
4 . 30 INFRASTRUCTURE AS A SERVICE CONCEPTS
4 . 31 BASICS Infrastructure: Compute Storage Network
4 . 32 COMPUTE RESOURCES Instance Image Flavor Keypair (SSH)
4 . 33 INSTANCE Dedicated to compute Short typical lifetime, to be considered ephemeral Should not store persistent data Non persistent root disk Based on an image
4 . 34 CLOUD IMAGE Disk image containing an already installed OS Infinitely instanciable Can talk to the metadata API
4 . 35 METADATA API http://169.254.169.254 Available from the instance Provides informations about the instance Exposes userdata The cloud-init tool helps take advantage of this API
4 . 36 FLAVOR Instance type in AWS Defines an instance model regarding CPU, RAM, disk (root), ephemeral disk The ephemeral disk has, like the root disk, the advantage of often being local and thus fast
4 . 37 KEYPAIR SSH public key + private key Cloud manages and stores the public key This public key is used to give SSH access to the instances
4 . 38 NETWORK RESOURCES 1/2 L2 network Network port L3 network Router Floating IP Security group
4 . 39 NETWORK RESOURCES 2/2 Load Balancing as a Service VPN as a Service Firewall as a Service
4 . 40 STORAGE RESOURCES Cloud provides two kinds of storage Block Object
4 . 41 BLOCK STORAGE Volumes that can be attached to an instance Access to raw devices such as /dev/vdb Ability to use any kind of file system Ability to use LVM, encryption, etc. Compatible with all existing applications Requires provisioning space by defining volume size
4 . 42 SHARED STORAGE? Block storage is not a shared storage solution like NFS NFS is at a higher layer: file system A volume is a priori connected to a single host
4 . 43 "BOOT FROM VOLUME" Starting an instance with its root disk on a volume Root disk data persistence Gets similar to classical server
4 . 44 OBJECT STORAGE API: CRUD on data Push and retrieve objects in/from a container/bucket No data hierachy, no directories, no file system Read/write access through APIs only No provisioning necessary Application must be designed to take advantage of object storage
4 . 45 ORCHESTRATION Orchestrate creation and management of resources in the cloud Architecture definition in a template Resources created from a template make a stack There are also orchestration tools (rather than services)
4 . 46 USAGE BEST PRACTICES
4 . 47 WHY BEST PRACTICES? Two differents possible views: Don't change anything Risk not meeting expectations Limit usage to test & dev use case Adapt to new cloud compliant practices to take advantage of it
4 . 48 HIGH AVAILABILITY (HA) Cloud control plane (APIs) is HA Managed resources might not be
4 . 49 PET VS CATTLE How to consider instances? Pet Cattle
4 . 50 INFRASTRUCTURE AS CODE With code Provision infrastructure resources Configure said resources, instances in particular The job is changing: Infrastructure Developer
4 . 51 SCALING Scale out rather than Scale up Scale out: horizontal scaling Scale up: vertical scaling Auto-scaling Managed by the cloud Managed by an external component
4 . 52 CLOUD READY APPLICATIONS Store their data in an appropriate place Are architected to be fault tolerant Etc. Cf. https://12factor.net/
4 . 53 BEHIND CLOUD
4 . 54 HOW TO IMPLEMENT A COMPUTE SERVICE Virtualization (system) Containers Bare metal
4 . 55 STORAGE IMPLEMENTATION: (SOFTWARE DEFINED STORAGE) SDS
Warning: not to be confused with the block vs object topic Use of commodity hardware No hardware RAID Software is responsible for the data Hardware failures are taken into account and managed The Ceph project and the OpenStack Swift component implement SDS See also Scality
4 . 56 SDS - CAP THEOREM Consistency - Availability - Partition tolerance
4 . 57 OPENSTACK: THE PROJECT
5 . 1 OVERVIEW
5 . 2 HIGH LEVEL
Simple version
5 . 3 HISTORY Started in 2010 Goal: the Free Open Source Cloud Operating System Merge of two projects from Rackspace (Storage) and NASA (Compute) Free software distributed under Apache 2.0 license Birth of the Foundation in 2012
5 . 4 MISSION STATEMENT To produce a ubiquitous Open Source Cloud Computing platform that is easy to use, simple to implement, interoperable between deployments, works well at all scales, and meets the needs of users and operators of both public and private clouds.
5 . 5 RELEASES Austin (2010.1) Bexar (2011.1), Cactus (2011.2), Diablo (2011.3) Essex (2012.1), Folsom (2012.2) Grizzly (2013.1), Havana (2013.2) Icehouse (2014.1), Juno (2014.2) Kilo (2015.1), Liberty (2015.2) Mitaka (2016.1), Newton (2016.2) Ocata (2017.1), Pike (2017.2) Queens (2018.1), Rocky (2018.2) Stein (2019.1), Train (2019.2) Early 2020: Ussuri
5 . 6 SOME OF THE SUPPORTERS/CONTRIBUTORS ... Editors: Red Hat, Suse, Canonical, Vmware, ... Hardware makers: IBM, HP, Dell, ... Hardware makers/network: Juniper, Cisco, ... Hardware makers/storage: NetApp, Hitachi, ... Also: NASA, Rackspace, Yahoo, OVH, Citrix, SAP, ... Google! (since July 2015) https://www.openstack.org/foundation/companies/
5 . 7 ... AND USERS All the previously mentioned contributors In France: Cloudwatt and Numergy Wikimedia CERN Paypal Comcast BMW Etc. Not counting confidential deployments https://www.openstack.org/user-stories/
5 . 8 THE DIFFERENT SUB-PROJECTS https://www.openstack.org/software/project-navigator/ OpenStack Compute - Nova OpenStack (Object) Storage - Swift OpenStack Block Storage - Cinder OpenStack Networking - Neutron OpenStack Image Service - Glance OpenStack Identity Service - Keystone OpenStack Dashboard - Horizon OpenStack Telemetry - Ceilometer OpenStack Orchestration - Heat
5 . 9 THE DIFFERENT SUB-PROJECTS (2) But also: Bare metal (Ironic) Queue service (Zaqar) Database service (Trove) Data processing (Sahara) DNS service (Designate) Shared File Systems (Manila) Key management (Barbican) Container (Magnum) Others Client CLI and libraries OpenStack deployment tools Libraries used by OpenStack Tools used to develop OpenStack 5 . 10 APIS Each project supports its OpenStack API Some projects support the corresponding AWS API (Nova/EC2, Swift/S3)
5 . 11 THE 4 OPENS Open Source Open Design Open Development Open Community https://governance.openstack.org/tc/reference/opens.html https://www.openstack.org/four-opens/
5 . 12 THE OPENSTACK FOUNDATION Main governance entity and legal representation of the project Board members are part of the sponsoring companies and elected by individual members Everyone can (freely) become an individual member Human resources: marketing, event managemement, release management, a few developers (mainly on infrastructure) 600 organizations across the world 80000 individual members in 170 countries
5 . 13 THE OPENSTACK FOUNDATION
Main entities of the Foundation
5 . 14 OPEN INFRASTRUCTURE Lately, the OpenStack Foundation expands to Open Infrastructure Beyond OpenStack, new projects: Kata Containers Zuul Airship StarlingX
5 . 15 RESOURCES Announcements (new versions, security advisories): [email protected] Documentation portal: https://docs.openstack.org/ API/SDK: https://developer.openstack.org/ Project governance: https://governance.openstack.org/ Releases: https://releases.openstack.org/ Support: https://ask.openstack.org/ [email protected] #openstack@Freenode
5 . 16 RESOURCES News: Official blog: https://www.openstack.org/blog/ Planet: http://planet.openstack.org/ Superuser: http://superuser.openstack.org/ Commercial resources: https://www.openstack.org/marketplace/ among others Job board: https://www.openstack.org/community/jobs/
5 . 17 USER SURVEY Regular survey done by the Foundation (every 6 months) Targets deployers and users Usable data: https://www.openstack.org/analytics
5 . 18 CERTIFIED OPENSTACK ADMINISTRATOR (COA) The only certification: Approved by the OpenStack Foundation Not linked to a specific company Content: Mainly OpenStack cloud user oriented https://www.openstack.org/coa/requirements/ Practical aspects: Practical exam, remote, duration: 2.5 hours Cost: $300 (one re-take possible) Ressources https://www.openstack.org/coa/ Tips: https://www.openstack.org/coa/tips/ Handbook: http://www.openstack.org/coa/handbook (unofficial) Exercises: https://github.com/AJNOURI/COA 5 . 19 RESOURCES - FRENCH COMMUNITY AND ASSOCIATION
Logo OpenStack-fr https://openstack.fr/ - https://asso.openstack.fr/ Meetups: Paris, Lyon, Toulouse, Montréal, etc. OpenStack Days France (Paris): https://openstackdayfrance.fr Attending events such as Paris Open Source Summit Communication channels: [email protected] #openstack-fr@Freenode
5 . 20 INTERNALS
5 . 21 ARCHITECTURE
Detailed view of the services 5 . 22 IMPLEMENTATION Everything is written in Python (Django for the web part) Each project is split in multiple services (example: API, scheduler, etc.) Re-use of existing components and existing libraries Usage of oslo.* libraries (developed by and for OpenStack): logs, config, etc. Usage of rootwrap to call underlying programs as root
5 . 23 IMPLEMENTATION - DEPENDENCIES Database: relational SQL (MySQL/MariaDB) Communication between services: AMQP (RabbitMQ) Caching: Memcached Distributed storage of configuration (to come): etcd
5 . 24 DEVELOPMENT MODEL
5 . 25 STATS (2017) 2344 developers 65823 changes (commits) https://www.openstack.org/assets/reports/OpenStack- AnnualReport2017.pdf
5 . 26 DEVELOPMENT: IN DETAILS Open to all (individuals and companies) 6 months release cycle Each cycle starts with a Project Team Gathering (PTG) During each cycle, an OpenStack Summit takes place
5 . 27 TOOLS AND COMMUNICATION Code: Git (GitHub is used as a mirror) Peer review for code: Gerrit Continous Integration (CI): Zuul Blueprints/specifications and bugs: Launchpad StoryBoard Communication: IRC and mailing-lists Translation: Zanata
5 . 28 DEVELOPMENT: IN DETAILS
Change workflow in OpenStack
5 . 29 RELEASE CYLE: 6 MONTHS The schedule is published, example: https://releases.openstack.org/stein/schedule.html Milestone releases Freezes: Feature, Requirements, String RC releases Stable releases Special case for some projects: https://releases.openstack.org/reference/release_models.html
5 . 30 PROJECTS Project Teams: https://governance.openstack.org/reference/projects/index.html Each deliverable has its own versioning - Semantic versioning https://releases.openstack.org/
5 . 31 WHO CONTRIBUTES? Active Technical Contributor (ATC) Person with at least one recent contribution in a recognized OpenStack project Voting rights (TC and PTL) Core reviewer: ATC with permissions to approve patches in a project Project Team Lead (PTL): elected by the ATC of each project Stackalytics provides stats on contributions http://stackalytics.com/
5 . 32 WHERE TO FIND INFORMATIONS ABOUT THE OPENSTACK DEVELOPMENT How to contribute https://docs.openstack.org/project-team-guide/ https://docs.openstack.org/infra/manual/ Various informations, on the wiki https://wiki.openstack.org/ Blueprints and bugs on Launchpad/StoryBoard https://launchpad.net/openstack/ https://storyboard.openstack.org/ https://specs.openstack.org/
5 . 33 WHERE TO FIND INFORMATIONS ABOUT THE OPENSTACK DEVELOPMENT Proposed patches and their reviews on Gerrit https://review.openstack.org/ CI state (among others) http://status.openstack.org/ Code (Git) and tarballs are available https://git.openstack.org/ https://tarballs.openstack.org/ IRC Freenode network Logs and meetings informations: http://eavesdrop.openstack.org/ Mailing-lists http://lists.openstack.org/
5 . 34 UPSTREAM TRAINING 2 days training Learn how to become an OpenStack contributor Tools Processes Work and collaborate in an open way
5 . 35 OPENSTACK INFRA Team in charge of the OpenStack development infrastructure Works like the OpenStack developement teams and uses the same tools Result: Infrastructure as code open source https://opensourceinfra.org/ Uses (hybrid) cloud Develops some tools: Zuul yaml2ical
5 . 36 OPENSTACK SUMMIT Every 6 months at the middle of the development cycle In the USA until 2013, now between North America and Asia/Europe A few dozens at the beginning to thousands attendees today At the same time: conference (users, decision makers)and Forum (developers/operators, replaces part of the previous Design Summit) Defines the name of the next release: place/city near the Summit
5 . 37 EXAMPLE OF THE APRIL 2013 SUMMIT IN PORTLAND Photo: Adrien Cunin
5 . 38 EXAMPLE OF THE OCTOBER 2015 SUMMIT IN TOKYO
Photo: Elizabeth K. Joseph, CC BY 2.0, Flickr/pleia2 5 . 39 EXAMPLE OF THE OCTOBER 2015 SUMMIT IN TOKYO
Photo: Elizabeth K. Joseph, CC BY 2.0, Flickr/pleia2 5 . 40 EXAMPLE OF THE OCTOBER 2015 SUMMIT IN TOKYO
Photo: Elizabeth K. Joseph, CC BY 2.0, Flickr/pleia2 5 . 41 EXAMPLE OF THE OCTOBER 2015 SUMMIT IN TOKYO Photo: Elizabeth K. Joseph, CC BY 2.0, Flickr/pleia2
5 . 42 PROJECT TEAM GATHERING (PTG) Since 2017 At the beginning of each cycle Replaces part of the previous Design Summit Dedicated to developers
5 . 43 TRANSLATION Official i18n team Only some parts are translated, like Horizon The French translation is one of the most complete today Use of a web platform based on Zanata: https://translate.openstack.org/
5 . 44 DEVSTACK: QUICKLY RUN OPENSTACK
5 . 45 DEVSTACK USE CASES Quickly deploy OpenStack Used by developers to test their changes Used for demos Used to the the APIs without bothering about a deployment Must NOT be used for production
5 . 46 DEVSTACK INTERNALS Support for Ubuntu 16.04/17.04, Fedora 24/25, CentOS/RHEL 7, Debian, OpenSUSE A shell script is responsible for everything: stack.sh A config file: local.conf Installs all the required dependencies (packages) Clones all the git repositories (master branch by défaut) Launches all the components
5 . 47 CONFIGURATION: LOCAL.CONF Example
[[local|localrc]] ADMIN_PASSWORD=secrete DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD SERVICE_TOKEN=a682f596-76f3-11e3-b3b2-e716f9080d50 #FIXED_RANGE=172.31.1.0/24 #FLOATING_RANGE=192.168.20.0/25 #HOST_IP=10.3.4.5
5 . 48 USAGE TIPS DevStack installs a lot on the machine It is recommended to work inside a VM To test all the OpenStack components in good conditions, multiple Go of RAM are necessary Use of Vagrant is recommended
5 . 49 USING OPENSTACK
6 . 1 PRINCIPLE All the features are available through the API Clients (including Horizon) go through the API Credentials are required, with the OpenStack API: user password project (tenant) domain
6 . 2 OPENSTACK APIS One API per OpenStack service Versioned, backwards compatiblity is guaranteed Body of requests and responses is formatted with JSON REST architecture Managed resources are specific to a project https://developer.openstack.org/#api
6 . 3 API ACCESS Direct, using HTTP, with tools like curl With a library Official implementations in Python OpenStackSDK Other implementations, including for other languages (example: jclouds) Shade (Python library which includes business logic) With official command line tools With Horizon Through higher-level third-party tools (example: Terraform)
6 . 4 OFFICIAL CLIENTS OpenStack provides official clients Historically: python-PROJECTclient (Python library and CLI) Today: openstackclient (CLI) CLI tools Authentication is done by passing credentials as parameters, environment variables or configuration file The --debug parameter shows the HTTP connection
6 . 5 OPENSTACK CLIENT Unified CLI client openstack
6 . 6 KEYSTONE: AUTHENTICATION, AUTHORIZATION AND SERVICE CATALOG
6 . 7 PRINCIPLES Keystone is responsible for authentication and authorization and service catalog. Standard user authenticates against Keystone Admin user often interacts with Keystone
6 . 8 API API v3: port 5000 Manages: Users, groups Projects (tenants) Roles (link between user and project) Domains Services and endpoints (service catalog) Provides: Tokens (authentication tokens)
6 . 9 SERVICE CATALOG For each service, multiple endpoints are possible depending on: region interface type (public, internal, admin)
6 . 10 TYPICAL USAGE SCENARIO
Interactions with Keystone
6 . 11 NOVA: COMPUTE
6 . 12 PRINCIPLES Mainly manages instances Instances are created from images provided by Glance Instances' network interfaces are associated with Neutron ports Block storage can be provided to instances by Cinder
6 . 13 INSTANCE PROPERTIES Ephemeral, a priori not HA Defined by a flavor Built from an image Optional: volume attachments Optional: boot from volume Optional: public SSH key Optional: network ports
6 . 14 API Managed resources: Instances Flavors (instance types) Keypairs: resources dedicated to each user (not part of a project)
6 . 15 ACTIONS ON INSTANCES Reboot / shutdown Snapshot Logs VNC access Resize Migration (admin)
6 . 16 GLANCE: IMAGE REGISTRY
6 . 17 PRINCIPLES Image (and snapshot) registry Image properties
6 . 18 API API v2: current version, manages images and snapshots API artifacts: future version, more common
6 . 19 IMAGE TYPES Glance supports a wide range of image types, limited by Nova's underlying technology support raw qcow2 ami vmdk iso
6 . 20 IMAGE PROPERTIES IN GLANCE The user can define a number of properties among which some will be used at instance creation Image type Architecture Distribution Distribution version Minimum disk space Minimum RAM
6 . 21 IMAGE SHARING Public image: available to all projects By default, only the admin can make an image public Shared image: available to one or multiple other project(s)
6 . 22 DOWNLOADING IMAGES Most OS provide regularly updated images: Ubuntu : https://cloud-images.ubuntu.com/ Debian : https://cdimage.debian.org/cdimage/openstack/ CentOS : https://cloud.centos.org/centos/
6 . 23 NEUTRON: NETWORK
6 . 24 API The API exposes these main resources: Network: layer 2 Subnet: layer 3 Port: can be attached to an instance interface, a load- balancer, etc. Router Floating IP, security group
6 . 25 FLOATING IPS In addition to fixed IPs which are set on instances Allocation (reservation for the project) of an IP from a pool Association of an allocated IP to a port (of an instance, for example) Not directly set on instances
6 . 26 SECURITY GROUPS Similar to a firewall in front of each instane An instance can be associated to one or multiple security groups Ingress and egress access rules Rules per protocol (TCP/UDP/ICMP) and port Targets an IP address, a network or another security group
6 . 27 ADDITIONAL FEATURES Beyond the basic L2 and L3 networking features, Neutron may provide other services: Load Balancing Firewall: different from security groups VPN: to reach a private network without floating IPs QoS
6 . 28 CINDER: BLOCK STORAGE
6 . 29 PRINCIPLES Provides volumes (block storage) that can be attached to instances Manages different volume types Manages volume snapshots and volume backups
6 . 30 USAGE Additional volume (and persistent storage) on an instance Boot from volume: OS is on the volume Backup to object store (Swift ou Ceph) feature
6 . 31 HEAT: ORCHESTRATION
6 . 32 PRINCIPLES Heat is the native OpenStack solution, orchestration service Heat provides an API to manage stacks from templates A Heat template follows the HOT (Heat Orchestration Template) format, based on YAML
6 . 33 A HEAT ORCHESTRATION TEMPLATE (HOT) TEMPLATE parameters - resources - outputs heat_template_version: 2013-05-23 description: Simple template to deploy a single compute instance resources: my_instance: type: OS::Nova::Server properties: key_name: my_key image: F18-x86_64-cfntools flavor: m1.small
6 . 34 BUILD A TEMPLATE FROM EXISTING RESOURCES Multiple projects are being developed Flame (Cloudwatt) HOT builder Merlin
6 . 35 HORIZON : WEB DASHBOARD
6 . 36 PRINCIPLES Provides a web interface Uses existing APIs to provide a user interface Ability to log in without specifiying a project: Horizon determines the list of available projects
6 . 37 USAGE One interface per project (ability to switch) Availability of the service catalog Download of a clouds.yaml config file Restricted “admin” area
6 . 38 DEPLOY OPENSTACK
7 . 1 WHAT WE ARE GOING TO SEE Install OpenStack manually https://docs.openstack.org/install-guide/ Understand its internals Go through each component in more details Overview of deployment solutions
7 . 2 DETAILED ARCHITECTURE
Vue détaillée des services
7 . 3 SERVICES ARCHITECTURE Machines "physiques" et services
7 . 4 A FEW GLOBAL CONFIGURATION ITEMS All the components must be configured to talk with Keystone Most must be configured to talk with MySQL/MariaDB and RabbitMQ Components split in multiple services sometimes have one configuration file per service The policy.json configuraton file specify the required permissions for each API call
7 . 5 OPERATING SYSTEM Linux OS with Python Ubuntu Red Hat SUSE Debian, Fedora, CentOS, etc.
7 . 6 PYTHON
Python logo
OpenStack is Python 2.7 compatible Python 3 comptability almost complete So as not to reinvent the wheel, a lot of dependencies are necessary
7 . 7 MYSQL/MARIADB DATABASE Stores most of the data managed by OpenStack Each component has it own database OpenStack uses the SQLAlchemy Python ORM Theoretical support of what SQLAlchemy supports (and migrations support) MySQL/MariaDB is the most tested and used implementation SQLite is mainly used for tests and demos Some deployments work with PostgreSQL
7 . 8 MESSAGE BUS AMQP: Advanced Message Queuing Protocol Messages, queues, routing OpenStack processes interact through AMQP Multiple possible implementations: Qpid, 0MQ, etc. RabbitMQ by default
7 . 9 RABBITMQ
RabbitMQ logo
RabbitMQ is written in Erlang An Erlang virtual machine is therefore required
7 . 10 “HELLO WORLD” RABBITMQ
Simple example of RabbitMQ operation
7 . 11 MEMCACHED CACHE Multiple services make use of a caching mechanism Memcached is the default implementation
7 . 12 KEYSTONE: AUTHENTICATION, AUTHORIZATION AND SERVICE CATALOG
7 . 13 INSTALL AND CONFIGURATION APT package: keystone WSGI web server integration (Apache by default) Configuration file: /etc/keystone/keystone.conf Users/groups backends: SQL, LDAP (or Active Directory) Projects/roles/services/endpoints backends: SQL Tokens backends: SQL, Memcache, none (depending on the token type)
7 . 14 TOKENS DRIVERS Uuid PKI Fernet
7 . 15 BOOTSTRAP Services and endpoints creation (starting with Keystone) Users, groups and domains creation Bootstrap feature
7 . 16 NOVA: COMPUTE
7 . 17 NOVA API Two jobs API to manage instances for the utilisateur API for the instances: metadata API The metadata API must be available at http://169.254.169.254/ The metadata API provides personalized configuration informations to each instance
7 . 18 NOVA COMPUTE Manages instances (virtual or bare metal machines) Takes advantage of libvirt or other APIs such as XenAPI Drivers: libvirt (KVM, LXC, etc.), XenAPI, VMWare vSphere, Ironic Ability to retrieve console logs and VNC console
7 . 19 NOVA SCHEDULER Service in charge of scheduling instance requests to compute nodes Filter, Chance, Multi Scheduler Filters, by default: AvailabilityZoneFilter,RamFilter,ComputeFilter Sort by weigh, by default: RamWeigher
7 . 20 NOVA SCHEDULER IN ACTION
nova-scheduler operation
7 . 21 NOVA CONDUCTOR Optional service which improves security Act as a proxy between compute nodes and DB Compute nodes, at risk, therefore don't have DB access anymore
7 . 22 GLANCE: IMAGE REGISTRY
7 . 23 BACKENDS Swift or S3 Ceph HTTP Local directory
7 . 24 INSTALL APT package: glance-api
7 . 25 NEUTRON: NETWORK AS A SERVICE
7 . 26 PRINCIPLES Software Defined Networking (SDN) Was Quantum and nova-network neutron-server: provides the API DHCP agent: provides DHCP service to instances L3 agent: manages network layer 3, routing Plugin: LinuxBridge by default, other open source / proprietary, software/hardware implementations exist
7 . 27 ADDITIONAL FEATURES Beyonce basic layer 2 and 3 networking features, Neutron can provide other services: Load Balancing (HAProxy, ...) Firewall (vArmour, ...): different from security groups VPN (Openswan, ...): to reach a private network without floating IPs These features are also based on plugins
7 . 28 PLUGINS ML2 Modular Layer 2 LinuxBridge OpenVSwitch OpenDaylight Contrail, OpenContrail Nuage Networks VMWare NSX cf. OpenFlow
7 . 29 IMPLEMENTATION Each network is a bridge Bridges are expanded across hosts using tunnels (VXLAN type) if necessary Neutron takes advantage of Linux kernel network namespaces to allow IP overlapping Metadata proxy is a component that allows instances isolated in their network to reach the metadata API provided by Nova
7 . 30 DIAGRAM User view of the network
7 . 31 DIAGRAM 7 . 32 CINDER: BLOCK STORAGE
7 . 33 PRINCIPLES Was nova-volume Provides volumes Volume attachement through iSCSI by default
7 . 34 INSTALL Package cinder-api: provides the API Package cinder-volume: creation and management of volumes Package cinder-scheduler: scheduling of volume creation requests Package cinder-backup (optional): backup to an object store
7 . 35 BACKENDS Use of multiple backends in parallel possible LVM (by default) GlusterFS Ceph Proprietary storage systems such as NetApp DRBD
7 . 36 HORIZON: WEB DASHBOARD
7 . 37 PRINCIPLES Horizon is a Django module OpenStack Dashboard is the official implementation of this module
Python Django web framework logo
7 . 38 CONFIGURATION local_settings.py Services appear in Horizon if they exist in Keystone service catalog
7 . 39 SWIFT: OBJECT STORAGE
7 . 40 PRINCIPLES SDS: Software Defined Storage Use of commodity hardware CAP theorem: choosing two Completly decentralized architecture No central database
7 . 41 IMPLEMENTATION Proxy: API server for all the requests Object server: storage server Container server: maintains list of objects in containers Account server: maintains list of containers in accounts Each object is replicated n times (3 by default)
7 . 42 THE RING Problem: how to decide which data is going onto which object server The ring is split in partitions Each data is located in the ring to find out the partition A partition is associated to multiple servers
7 . 43 DIAGRAM
Swift architecture
7 . 44 CEILOMETER: METRICS COLLECTION
7 . 45 MONITOR USAGE OF INFRASTRUCTURE WITH CEILOMETER Stores different metrics regarding usage of cloud services Provides APIs to retrieve these data Base to build billing tools (example: CloudKitty)
7 . 46 CEILOMETER Retrieves data and stores them Used to be stored in MongoDB Now stored in Gnocchi
7 . 47 GNOCCHI: TIME-SERIES DATABASE Why Gnocchi? To solve Ceilometer + MongoDB scaling issues Initiated by Ceilometer developers and integrated in the Ceilometer project team Provides an API to read and write data Uses a relational DB and an object storage system
7 . 48 HEAT: RESOURCES ORCHESTRATION
7 . 49 ARCHITECTURE heat-api heat-engine
7 . 50 SOME OTHER INTERESTING COMPONENTS
7 . 51 TROVE: DATABASE AS A SERVICE trove-api: API trove-taskmanager: manages DB instances trove-guestagent: internal agent in instances
7 . 52 DESIGNATE: DNS AS A SERVICE Manages different backends: BIND, PowerDNS, etc.
7 . 53 BARBICAN: KEY MANAGEMENT AS A SERVICE Possible backends: Encrypted files PKCS#11 KMIP Dogtag
7 . 54 MAGNUM: CONTAINER INFRASTRUCTURE AS A SERVICE Backends: Swarm, Kubernetes
7 . 55 OPENSTACK IN PRODUCTION AND OPERATIONS
8 . 1 BEST PRACTICES FOR A PRODUCTION DEPLOYMENT
8 . 2 WHICH COMPONENTS SHOUD I INSTALL? Keystone is mandatory Use of Nova goes with Glance and Neutron Cinder and Swift usefulness depends on storage needs Swift can be used separately from other components Heat doesn't cost much Higher level services need to be evaluated case by case https://docs.openstack.org/arch-design/
8 . 3 THINK ABOUT FUNDAMENTAL CHOICES AT THE BEGINNING Distribution and deployment method Update and upgrade policy Drivers/backends: hypervisor, block storage, etc. Network: what architecture and what drivers
8 . 4 DIFFERENT INSTALLATION METHODS Forget about DevStack for production Manual deployment as seen previously is not recommended by unmaintainable Packaged and ready to use OpenStack distributions Classical distributions and configuration management Continuous deployment
8 . 5 MAJOR UPGRADES OpenStack supports N → N+1 upgrades Swift: very good rolling upgrade support Other components: test with you data first Read release notes Cf. CERN blog posts https://techblog.web.cern.ch/techblog/
8 . 6 STABLE UPDATES Major bug and security fixes are provided OpenStack includes these fixes as patches in the stable branch Point releases are published and includes fixes from the stable branch Stable version support timeframe is variable, depending on integrators' interest
8 . 7 ASSIGN ROLES TO MACHINES Lots of documentations mention these roles: Controller node: APIs, DB, AMQP Network node: DHCP, router, floating IPs Compute node: Hypervisor/instances management This simplified model is not HA.
8 . 8 HIGH AVAILABILITY IaaS High Availability MySQL/MariaDB, RabbitMQ: classical HA (Galera, Clustering) API services Are stateless and HTTP: scale out and load balancers Most other OpenStack services are able to scale out as well HA guide: https://docs.openstack.org/ha-guide/ Talks by Florian Haas, Hastexo: https://www.openstack.org/community/speakers/profile/398/florian- haas
8 . 9 HIGH AVAILABILITY OF THE NEUTRON L3 AGENT Distributed Virtual Router (DVR) L3 agent HA (VRRP)
8 . 10 APIS CONCERNS Uniform URLs for all the APIs: Use a reverse proxy Don't forget to update the service catalog Apache/mod_wsgi to serve APIs when possible (Keystone, etc.) Operations guide: https://docs.openstack.org/openstack- ops/content/
8 . 11 NETWORKS Management network: administrative network Data/instances network: network for inter instances traffic External network: external network, in the existing network infrastructure Storage network: network for Cinder/Swift storage API network: network containing API endpoints
8 . 12 SECURITY RELATED CONCERNS Must-have: HTTPS for external API access Securing MySQL/MariaDB and RabbitMQ traffic One MySQL/MariaDB access per database and per service One Keystone user per service Limit read permissions to configuration files (passwords, token) Security vulnerabilities: OSSA (OpenStack Security Advisory), OSSN (... Notes) Security guide: https://docs.openstack.org/security-guide/
8 . 13 SEGMENT A CLOUD Host aggregates: physical hosts with similar features Availability zones: hosts depending on the same electrical supply, same switch, same DC, etc. Regions: each region has its own API Cells: gather multiple clouds within a unique API https://docs.openstack.org/openstack- ops/content/scaling.html#segregate_cloud
8 . 14 HOST AGGREGATES Nova specific Admin defines host aggregates through the API Admin associates flavors and aggregates through common key/values 1 aggregate ≡ 1 similarity, ex: GPU User chooses an aggregate through their flavor choice when creating an instance
8 . 15 AVAILABILITY ZONES Nova and Cinder specific Hosts groups Split in terms of availability: Rack, Datacenter, etc. User chooses an availability zone when creating an instance Use can request instances to be started in the same zone, or on the contrary in different zones
8 . 16 REGIONS Generic in OpenStack AWS regions counterpart A service can have different endpoints in different regions Each region is autonomous Use case: very large cloud (such as some public clouds)
8 . 17 CELLS Nova specific Only one nova-api in front of mutiple cells Each cell has its own DB and message bus Adds a scheduling layer (cell choice)
8 . 18 OPENSTACK PACKAGING - UBUNTU Packaging is done in multiples distributions, RPM, DEB and others Ubuntu historically is the reference platform for OpenStack developement Packaging in Ubuntu closely follows OpenStack development, and automated tests are performed Canonical provides the Ubuntu Cloud Archive, which includes the latest OpenStack version for the latest Ubuntu LTS
8 . 19 UBUNTU CLOUD ARCHIVE (UCA)
OpenStack support in Ubuntu through UCA
8 . 20 OPENSTACK PACKAGING IN OTHER DISTRIBUTIONS OpenStack is integrated in Debian's official repository Red Hat provides RHOS/RDO (deployment based on TripleO) Like Ubuntu, Fedora's release cycle is synchronized with OpenStack's
8 . 21 OPENSTACK DISTRIBUTIONS StackOps: history Mirantis: Fuel HP Helion: Ansible custom etc.
8 . 22 TRIPLEO OpenStack On OpenStack Goal: ability to deploy an OpenStack cloud (overcloud) from an OpenStack cloud (undercloud) Autoscaling of the cloud itself: deployment of new compute nodes when necessary Works jointly with Ironic for bare metal deployment
8 . 23 BARE METAL DEPLOYMENT OpenStack bare metal hosts deployment can be managed with the help of dedicated tools MaaS (Metal as a Service), by Ubuntu/Canonical: works with Juju Crowbar / OpenCrowbar (initially Dell): uses Chef eDeploy (eNovance): image based deployment Ironic through TripleO
8 . 24 TEMPEST Test suite of an OpenStack cloud Makes API calls and checks the result Used a lot by developers through continuous integration Deployers can use Tempest to check their cloud's compliance See also Rally
8 . 25 CONFIGURATION MANAGEMENT Puppet, Chef, CFEngine, Saltstack, Ansible, etc. These tools can help deploying an OpenStack cloud ... but also to manage instances (next section)
8 . 26 MODULES PUPPET, PLAYBOOKS ANSIBLE Puppet OpenStack and OpenStack Ansible: Puppet modules and Ansible playbooks Developed as part of the OpenStack project https://wiki.openstack.org/wiki/Puppet https://docs.openstack.org/developer/openstack- ansible/install-guide/
8 . 27 CONTINUOUS DEPLOYMENT OpenStack maintains an always stable master (trunk) Possible to deploy master on a daily basis (CD: Continous Delivery) Requires setting up an important infrastructure Eases upgrades between major versions
8 . 28 FACING ISSUES
8 . 29 TIPS IN CASE OF ERROR OR FAULTY BEHAVIOR Are we working on the appropriate project? Is the API responding with an error? (the dashboard may hide some informations) If going further is required: Look into logs on thje cloud controller (/var/log/
8 . 30 IS IT A BUG? If the CLI client crashes, it's a bug If the web dashboard or the API responds with an error 500, it might be a bug If the logs show a Python stacktrace, it's a bug Otherwise, you decide
8 . 31 OPERATIONS
8 . 32 LOGS MANAGEMENT Centralize logs API logs Other OpenStack components logs DB, AMQP, etc. logs
8 . 33 BACKUP Databases Deployment mechanism, rather than configuration files
8 . 34 MONITORING API response Checking OpenStack services and dependencies
8 . 35 QUOTAS USAGE Limit the number of allocable resources Per user or per tenant Support in Nova Support in Cinder Support in Neutron
8 . 36 OPENSTACK : DÉPLOYER AVEC ANSIBLE
9 . 1 RAPPELS : ANSIBLE Déploiement de configuration Masterless, agentless Tâches, playbooks, rôles, variables (yaml) Inventaire Collections de modules Écrit en Python
9 . 2 OPENSTACK-ANSIBLE (OSA) Projet officiel OpenStack : https://opendev.org/openstack/openstack-ansible Déployer et maintenir un cloud OpenStack en production, choix de la release Ansible : ensemble de playbooks, de rôles et de scripts (python, bash) Support de Debian/Ubuntu, CentOS, openSUSE Déploiement à partir des sources (git clone) Services déployés dans des conteneurs LXC Nécessité de créer des bridges sur les hosts : support LinuxBridge et Open vSwitch Déploiement de Ceph via ceph-ansible : https://github.com/ceph/ceph-ansible
9 . 3 PRINCIPAUX COMPOSANTS SUPPORTÉS Keystone Nova Cinder Glance Neutron Octavia Swift Heat Horizon Ceilometer, Aodh Tempest
9 . 4 MISE EN OEUVRE DE LA HAUTE DISPONIBILITÉ HAProxy -> Keepalived MySQL -> Galera cluster RabbitMQ -> Clustering Memcached -> Farming
9 . 5 RÉSEAU L2 : LinuxBridge ou Open vSwitch Routeur virtuel : centralisé ou distribué (DVR)
9 . 6 RSYSLOG CENTRALISÉ (OPTIONNEL) Un conteneur rsyslog Tous les services lui transmettent les logs Ce rsyslog peut lui même transférer les logs ailleurs
9 . 7 OSA EN MULTI-RÉGIONS Un déploiement OSA par région Un déploiement pour le Keystone central
9 . 8 OSA WORKFLOW
9 . 9 MACHINE DE DÉPLOIEMENT Composant principal : openstack-ansible Scripts Playbooks Inventaire dynamique
$ git clone https://opendev.org/openstack/openstack-ansible /opt/openstack-ansible $ cd /opt/openstack-ansible $ git checkout stable/
9 . 10 RÉCUPÉRER LES RÔLES Un rôle par service URLs et versions définies dans openstack-ansible/ansible-role-requirements.yml openstack-ansible/scripts/bootstrap-ansible.sh installe ansible et télécharge les rôles dans /etc/ansible/roles
9 . 11 CONFIGURER /etc/openstack_deploy/ ./openstack_user_config.yml -> description de l'infra et des réseaux ./user_variables.yml -> configuration des services ./user_secrets.yml
9 . 12 DÉPLOYER Dans /opt/openstack-ansible/playbooks, avec le wrapper openstack-ansible : 1. Création des conteneurs LXC : openstack-ansible setup-hosts.yml 2. Déploiement des services infra : openstack-ansible setup-infrastructure.yml 3. Déploiement des services OpenStack : openstack-ansible setup-openstack.yml Enchaînement automatique : openstack-ansible setup-everything.yml
9 . 13 METTRE À JOUR Dans /opt/openstack-ansible : git pull git checkout de la release cible scripts/bootstrap-ansible.sh cd playbooks; openstack-ansible setup-everything.yml
9 . 14 SÉCURITÉ Rôle ansible-hardening Lancée par setup-hosts.yml Durcissement des hosts Implémente les exigences exprimées dans le Security Technical Implementation Guide http://docs.openstack.org/developer/ansible-hardening/
9 . 15 OPENSTACK-ANSIBLE-OPS
Outils pour OSA Exemples : Supprimer les anciens venvs Restaurer RabbitMQ Simuler des pannes https://opendev.org/openstack/openstack-ansible-ops
9 . 16 CLOUD-READY ARCHITECTURES
10 . 1 DESIGN AN APPLICATION FOR THE CLOUD
10 . 2 ADAPT OR THINK “CLOUD READY” APPLICATIONS 1/3 See OpenStack project design tenets and Twelve-Factor https://12factor.net/ Distributed rather than monolithic architecture Eases scaling Limits failure domain Loose coupling between components
10 . 3 ADAPT OR THINK “CLOUD READY” APPLICATIONS 2/3 Message bus for inter-component communication Stateless: allows multiple access routes to the application Dynamic: application must adapt to its environnement and reconfigure itself when necessary Allow deployment and operation using automation tools
10 . 4 ADAPT OR THINK “CLOUD READY” APPLICATIONS 3/3 Limit as much as possible hardware or software specific dependencies that may not work in a cloud Integrated fault tolerance Do not store data locally, but rather: Database Object storage Use standard logging tools
10 . 5 DESIGN AN INFRASTRUCTURE FOR THE CLOUD
10 . 6 AUTOMATION Automate infrastructure management: mandatory Resources creation Resources configuration
10 . 7 INFRASTRUCTURE AS CODE Work like a developer Describe your infrastructure as code (Heat/Terraform, Ansible) Track changes in a VCS (git) Set up code review Use testing mechanisms Take advantage of continuous integration and deployment systems
10 . 8 ORCHESTRATION NEED Manage all kind of resources through a unique entrypoint Infrastructure description in a file (template) Heat (included in OpenStack), Terraform
10 . 9 TESTS AND CONTINUOUS INTEGRATION Code style Syntax validation Unit tests Integration tests Full deployment tests
10 . 10 FAULT TOLERANCE Take advantage of application abilities Don't try to make the compute infrastructure HA
10 . 11 AUTOSCALING GROUP Group of similar instances Variable number of instances Automated scaling depending on metrics Enables horizontal scaling
10 . 12 MONITORING Take into account instances' lifecycle: DOWN != ALERT Monitor the service rather than the server
10 . 13 BACKUP Be able to recreate your instances (and the rest of the infrastructure) Data (application, logs): block, object
10 . 14 HOW TO MANAGE IMAGES? Use of generic images and personalization at launch Creation of intermediary and/or completely personalized images: Cold modification: libguestfs, virt-builder, virt- sysprep Modification through an instance: automation possible with Packer Build from scratch: diskimage-builder (TripleO) Build from scratch with distribution-specific tools (openstack-debian-images for Debian)
10 . 15 CONCLUSION
11 . 1 TO CONCLUDE Cloud is a revolution for IT OpenStack is the open source flagship project for the IaaS part Deploying OpenStack is not an easy task Using an IaaS cloud implies changes of practice Software and infrastructure architecture jobs are changing
11 . 2