What’s in a cloud? An Overview of Virtualization and Openstack
2014-03-24
1/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 1/5555 • Software-as-a-Service? • Platform-as-a-Service? • Infrastructure-as-a-Service? • Magic?
Demistifying “The Cloud” There’s a lot of talk about Cloud Computing - but what do we even mean when we say “Cloud?”
2/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/5555 • Platform-as-a-Service? • Infrastructure-as-a-Service? • Magic?
Demistifying “The Cloud” There’s a lot of talk about Cloud Computing - but what do we even mean when we say “Cloud?” • Software-as-a-Service?
2/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/5555 • Infrastructure-as-a-Service? • Magic?
Demistifying “The Cloud” There’s a lot of talk about Cloud Computing - but what do we even mean when we say “Cloud?” • Software-as-a-Service? • Platform-as-a-Service?
2/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/5555 • Magic?
Demistifying “The Cloud” There’s a lot of talk about Cloud Computing - but what do we even mean when we say “Cloud?” • Software-as-a-Service? • Platform-as-a-Service? • Infrastructure-as-a-Service?
2/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/5555 Demistifying “The Cloud” There’s a lot of talk about Cloud Computing - but what do we even mean when we say “Cloud?” • Software-as-a-Service? • Platform-as-a-Service? • Infrastructure-as-a-Service? • Magic?
2/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/5555 Demistifying “The Cloud” There’s a lot of talk about Cloud Computing - but what do we even mean when we say “Cloud?” • Software-as-a-Service? • Platform-as-a-Service? • Infrastructure-as-a-Service? • Magic?
2/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 2/5555 Cloud means: • On-demand (self service) • Elasticity (easily scale up/down) • Multi-tenancy
What exactly is a cloud?
NIST Definition Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.
3/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 3/5555 What exactly is a cloud?
NIST Definition Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Cloud means: • On-demand (self service) • Elasticity (easily scale up/down) • Multi-tenancy
3/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 3/5555 Other resources can be: • Network (virtual networks, Load Balancing, etc…) • Storage (Object and Block)
Infrastructure-as-a-Service (IaaS)
Wikipedia In the most basic cloud-service model, providers of IaaS offer computers - physical or (more often) virtual machines - and other resources.
4/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 4/5555 Infrastructure-as-a-Service (IaaS)
Wikipedia In the most basic cloud-service model, providers of IaaS offer computers - physical or (more often) virtual machines - and other resources. Other resources can be: • Network (virtual networks, Load Balancing, etc…) • Storage (Object and Block)
4/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 4/5555 Infrastructure-as-a-Service (IaaS)
Wikipedia In the most basic cloud-service model, providers of IaaS offer computers - physical or (more often) virtual machines - and other resources. Other resources can be: • Network (virtual networks, Load Balancing, etc…) • Storage (Object and Block)
4/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 4/5555 + Can take full advantage of virtualization + Own the hardware − Own the hardware − Have to manage cloud stack on top of infrastructure
+ Scaling & cost better (can go beyond your HW) + Don’t own the hardware or the cloud stack − Don’t own the hardware or the cloud stack − Vendor lock-in
With a private IaaS cloud:
With a public IaaS cloud:
IaaS types
IaaS clouds come in two flavors: public and private
5/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/5555 + Scaling & cost better (can go beyond your HW) + Don’t own the hardware or the cloud stack − Don’t own the hardware or the cloud stack − Vendor lock-in
+ Own the hardware − Own the hardware − Have to manage cloud stack on top of infrastructure With a public IaaS cloud:
IaaS types
IaaS clouds come in two flavors: public and private With a private IaaS cloud: + Can take full advantage of virtualization
5/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/5555 + Scaling & cost better (can go beyond your HW) + Don’t own the hardware or the cloud stack − Don’t own the hardware or the cloud stack − Vendor lock-in
− Own the hardware − Have to manage cloud stack on top of infrastructure With a public IaaS cloud:
IaaS types
IaaS clouds come in two flavors: public and private With a private IaaS cloud: + Can take full advantage of virtualization + Own the hardware
5/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/5555 + Scaling & cost better (can go beyond your HW) + Don’t own the hardware or the cloud stack − Don’t own the hardware or the cloud stack − Vendor lock-in
− Have to manage cloud stack on top of infrastructure With a public IaaS cloud:
IaaS types
IaaS clouds come in two flavors: public and private With a private IaaS cloud: + Can take full advantage of virtualization + Own the hardware − Own the hardware
5/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/5555 + Scaling & cost better (can go beyond your HW) + Don’t own the hardware or the cloud stack − Don’t own the hardware or the cloud stack − Vendor lock-in
With a public IaaS cloud:
IaaS types
IaaS clouds come in two flavors: public and private With a private IaaS cloud: + Can take full advantage of virtualization + Own the hardware − Own the hardware − Have to manage cloud stack on top of infrastructure
5/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/5555 + Scaling & cost better (can go beyond your HW) + Don’t own the hardware or the cloud stack − Don’t own the hardware or the cloud stack − Vendor lock-in
With a public IaaS cloud:
IaaS types
IaaS clouds come in two flavors: public and private With a private IaaS cloud: + Can take full advantage of virtualization + Own the hardware − Own the hardware − Have to manage cloud stack on top of infrastructure
5/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/5555 + Don’t own the hardware or the cloud stack − Don’t own the hardware or the cloud stack − Vendor lock-in
IaaS types
IaaS clouds come in two flavors: public and private With a private IaaS cloud: + Can take full advantage of virtualization + Own the hardware − Own the hardware − Have to manage cloud stack on top of infrastructure With a public IaaS cloud: + Scaling & cost better (can go beyond your HW)
5/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/5555 − Don’t own the hardware or the cloud stack − Vendor lock-in
IaaS types
IaaS clouds come in two flavors: public and private With a private IaaS cloud: + Can take full advantage of virtualization + Own the hardware − Own the hardware − Have to manage cloud stack on top of infrastructure With a public IaaS cloud: + Scaling & cost better (can go beyond your HW) + Don’t own the hardware or the cloud stack
5/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/5555 − Vendor lock-in
IaaS types
IaaS clouds come in two flavors: public and private With a private IaaS cloud: + Can take full advantage of virtualization + Own the hardware − Own the hardware − Have to manage cloud stack on top of infrastructure With a public IaaS cloud: + Scaling & cost better (can go beyond your HW) + Don’t own the hardware or the cloud stack − Don’t own the hardware or the cloud stack
5/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/5555 IaaS types
IaaS clouds come in two flavors: public and private With a private IaaS cloud: + Can take full advantage of virtualization + Own the hardware − Own the hardware − Have to manage cloud stack on top of infrastructure With a public IaaS cloud: + Scaling & cost better (can go beyond your HW) + Don’t own the hardware or the cloud stack − Don’t own the hardware or the cloud stack − Vendor lock-in
5/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 5/5555 Virtualization
6/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 6/5555 – Robert Goldberg, 1974
• Main idea: transform a single machine into many • Don’t emulate every instruction, emulate only privileged ones • Virtual Machine Manager → VMs
Virtual Machines
“Virtual machines have finally arrived. Dismissed for a number of years as merely academic curiosities, they are now seen as cost-effective techniques for organizing computer systems resources to provide extraordinary system flexibility and support for certain unique applications.”
7/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 7/5555 • Main idea: transform a single machine into many • Don’t emulate every instruction, emulate only privileged ones • Virtual Machine Manager → VMs
Virtual Machines
“Virtual machines have finally arrived. Dismissed for a number of years as merely academic curiosities, they are now seen as cost-effective techniques for organizing computer systems resources to provide extraordinary system flexibility and support for certain unique applications.” – Robert Goldberg, 1974
7/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 7/5555 Virtual Machines
“Virtual machines have finally arrived. Dismissed for a number of years as merely academic curiosities, they are now seen as cost-effective techniques for organizing computer systems resources to provide extraordinary system flexibility and support for certain unique applications.” – Robert Goldberg, 1974
• Main idea: transform a single machine into many • Don’t emulate every instruction, emulate only privileged ones • Virtual Machine Manager → VMs
7/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 7/5555 ◦ Behavior in VM = Behavior in bare metal
◦ VMM has complete control
◦ Most instructions should be executed natively • Equivalence
• Resource control
Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit: • Efficiency
1 Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974 8/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/5555 ◦ Behavior in VM = Behavior in bare metal
◦ VMM has complete control
• Equivalence
• Resource control
Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit: • Efficiency ◦ Most instructions should be executed natively
1 Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974 8/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/5555 ◦ VMM has complete control
◦ Behavior in VM = Behavior in bare metal • Resource control
Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit: • Efficiency ◦ Most instructions should be executed natively • Equivalence
1 Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974 8/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/5555 ◦ VMM has complete control
• Resource control
Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit: • Efficiency ◦ Most instructions should be executed natively • Equivalence ◦ Behavior in VM = Behavior in bare metal
1 Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974 8/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/5555 ◦ VMM has complete control
Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit: • Efficiency ◦ Most instructions should be executed natively • Equivalence ◦ Behavior in VM = Behavior in bare metal • Resource control
1 Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974 8/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/5555 Popek and Goldberg Virtualization Requirements 1
A Virtual Machine Manager should exhibit: • Efficiency ◦ Most instructions should be executed natively • Equivalence ◦ Behavior in VM = Behavior in bare metal • Resource control ◦ VMM has complete control
1 Popek and Goldberg, “Formal Requirements for Virtualizable Third Generation Architectures,” 1974 8/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 8/5555 Examples: Virtualbox, QEMU
Types of hypervisors
Examples: Xen, VMWare ESX
9/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 9/5555 Types of hypervisors
Examples: Xen, VMWare ESX
Examples: Virtualbox, QEMU
9/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 9/5555 x86 and Privilege Rings
Ring 3
Ring 2 Least privileged
Ring 1
Ring 0
Kernel
Most privileged Device drivers
Device drivers
Applications
• Only ring 0 can execute privileged instructions • Linux/Windows: ring 0 (supervisor/kernel) and 3 (user) 10/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 10/5555 • No MMU virtualization • 1998 VMware: binary translation
Question What are the advantages/disadvantages of this method?
Virtualization on x86 x86: reputation for being unfriendly to virtualization • Not all privileged operations generate traps (e.g. popf)
11/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5555 • 1998 VMware: binary translation
Question What are the advantages/disadvantages of this method?
Virtualization on x86 x86: reputation for being unfriendly to virtualization • Not all privileged operations generate traps (e.g. popf) • No MMU virtualization
11/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5555 Question What are the advantages/disadvantages of this method?
Virtualization on x86 x86: reputation for being unfriendly to virtualization • Not all privileged operations generate traps (e.g. popf) • No MMU virtualization • 1998 VMware: binary translation
11/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5555 Question What are the advantages/disadvantages of this method?
Virtualization on x86 x86: reputation for being unfriendly to virtualization • Not all privileged operations generate traps (e.g. popf) • No MMU virtualization • 1998 VMware: binary translation
11/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5555 Question What are the advantages/disadvantages of this method?
Virtualization on x86 x86: reputation for being unfriendly to virtualization • Not all privileged operations generate traps (e.g. popf) • No MMU virtualization • 1998 VMware: binary translation
11/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5555 Virtualization on x86 x86: reputation for being unfriendly to virtualization • Not all privileged operations generate traps (e.g. popf) • No MMU virtualization • 1998 VMware: binary translation
Question What are the advantages/disadvantages of this method? 11/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 11/5555 Question What are the advantages/disadvantages of this method?
Para-virtualization
12/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 12/5555 Para-virtualization
Question What are the advantages/disadvantages of this method? 12/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 12/5555 Hardware Assisted Virtualization (HAV)
In 2006, Intel VT-x and AMD AMD-V ⇒ VMM much easier to implement • Allowed for an unmodified guest OS • Introduced guest-mode execution • When a guest performs a privileged instruction, trap to VMM via a VM Exit
13/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 13/5555 VM Entry/Exit Example
Guest mode Host mode
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Guest mode Host mode Launch VM
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Guest mode Host mode Launch VM VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Guest mode Host mode Launch VM VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Allocate new memory
Guest mode Host mode Launch VM VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Allocate new memory VM Exit: PAGE_FAULT Guest mode Host mode Launch VM VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Allocate new memory VM Exit: PAGE_FAULT Guest mode Host mode Launch VM VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Allocate new memory VM Exit: PAGE_FAULT Guest mode Host mode Launch VM VM Entry VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Allocate new memory VM Exit: PAGE_FAULT Guest mode Host mode Launch VM VM Entry VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Allocate new memory Context Switch VM Exit: PAGE_FAULT Guest mode Host mode Launch VM VM Entry VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Allocate new memory Context Switch VM Exit: PAGE_FAULT VM Exit: CR_ACCESS Guest mode Host mode Launch VM VM Entry VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Allocate new memory Context Switch VM Exit: PAGE_FAULT VM Exit: CR_ACCESS Guest mode Host mode Launch VM VM Entry VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Allocate new memory Context Switch VM Exit: PAGE_FAULT VM Exit: CR_ACCESS … ….... Guest mode Host mode … Launch VM VM Entry VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 VM Entry/Exit Example
Allocate new memory Context Switch Turn off machine VM Exit: PAGE_FAULT VM Exit: CR_ACCESS VM Exit: VM_OFF … ….... Guest mode Host mode … Launch VM Deallocate VM VM Entry VM Entry
14/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 14/5555 Intel VT-x
A new datastructure: VMCS • On VM Exit, contains the CPU & control state of a guest • Relevant guest state is passed to the VMM (I/O port, etc...) In 2008, VT-d features: more than CPU (EPT/IOMMU) Question What are the advantages/disadvantages of HAV?
15/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 15/5555 Guest execution Host execution
Virtual VirtualVirtual VM_exit_handler(vcpu, vmcs) { Event Buffers Machines Machines Event_forwarder(vcpu, vmcs) vcpu1 vcpu2 vcpu3 vcpu4 Machines // existing code Event_processing(…) VM exit VM_resume() events } Monitor 1.1 (e.g. HRKD) Monitor 2.1 Monitor h.1 In-host View (e.g. HRKD) (e.g. HHD) (ps/top, Tasks Manager) Kernel data structure Exported helper functions: Monitor 1.2 set_gpage_protect(…); (e.g. GOSHD) … …
Process 1 1 2 3 read_gpage(…); … VM 1 VM 2 (vcpu1, vcpu2) (vcpu3, vcpu4) … Hypervisor Process 3 Hypervisor
VMIAI tools Hypervisor Process 1 Existing VMI View
Process 2 VM Guest Process 1 (DKOM rootkit) Process 3 Guest execution Host execution Process 3
Virtual VirtualVirtual VM_exit_handler(vcpu, vmcs) { Machines Machines Event_forwarder(vcpu, vmcs) Event Multiplexer VMIAI View Machines MMU Privilege // existing code CR3 TSS Event_processing(…) Process 1 VM_resume() VM exit Monitor 1.1 Process 2 } events (e.g. HRKD) Monitor n.1 Monitor h.1 VM Events (e.g. HRKD) (e.g. HHD) Process 3 HVA Hardware 2 Monitor 1.2 Exported helper functions: (e.g. GOSHD) … … HyperTap set_gpage_protect(…); VM 1 VM n read_gpage(…); … (vcpu1, vcpu2) … (vcpu3, vcpu4) Hypervisor
Hypervisor VMIAI tools
Architecture View Hardware • Can use HAV for robust security+reliability monitoring
VM exit events VM_Exit_handler() { Event Mul plexer Non-blocking
Virtual Event Forwarder Blocking vcpu1, vcpu2 events Machine 1 vcpu3, vcpu4 events ... (vcpu 1, vcpu 2) } Monitor 1.1 API call (e.g. HRKD) Monitor n.1 … (e.g. HRKD) Helper APIs Monitor 1.2 VM Monitors (e.g. PED) … Virtual set_gpage_protect(…);
Machine n read_gpage(…); … VM 1 VM n … (vcpu 3, vcpu 4) monitors monitors HyperTap Core Hypervisor HyperTap tools Guest execution Host execution
Monitors run Virtual Machines Monitors in host machine DomU Virtual Machines
Virtual Virtual PED Dom0 HRKD HRKD
PED Machine 1 Machine n 2 HRKD HRKD Virtual Virtual C. Pham, Z. Estrada, P. Cao, Z. Kalbarczyk, R. Iyer “Reliability and Security Monitoring of Virtual Machines Machine 1 Machine n Using Hardware Architectural Invariants” 2014 16 Event Mul plexer Event /55 Zak Estrada What’s in a Cloud? An OverviewEvent Forwarder of Virtualization and Openstack 16/55 Mul plexer Helper APIs (ioctl interfaces) Helper APIs Kernel module (hypercalls) KVM Hypervisor Event Forwarder Linux kernel (host OS) XEN hypervisor Open-Source Virtual Machine Monitors (VMMs)
17/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 17/5555 Xen
• Xen para-virt upstream in Linux since 3.0 • also supports HVM (HAV) • Amazon EC2 18/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 18/5555 KVM
• vCPUs scheduled as processes in Linux • Paravirtualized drivers: virtio • Default for Openstack
19/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 19/5555 Thoughts on Virtualization
• Virtualization has overhead • Many VMs run only one application • Many people running Linux on Linux (e.g. KVM w/Linux guest)
Question Can we do better?
20/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 20/5555 Linux Containers (LXC)
• All “guests” share the same kernel - no “trap and emulate” • Isolation via cgroups/namespaces Question What are the advantages/disadvantages of containers? 21/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 21/5555 Virtualization Summary
Paravirtualization • Xen HAV • KVM Overhead/double work • Linux Containers
22/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 22/5555 Openstack
23/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 23/5555 virsh virt-manager virt-factory oVirt
VIRTUALIZATION API
KVM LXC OpenVZ UML Xen ESX other...
• Standardized method for managing multiple hypervisors
How to manage VMs? • Xen provides mgmt, KVM doesn’t • need for common interface
24/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 24/5555 How to manage VMs? • Xen provides mgmt, KVM doesn’t • need for common interface
virsh virt-manager virt-factory oVirt
VIRTUALIZATION API
KVM LXC OpenVZ UML Xen ESX other...
• Standardized method for managing multiple hypervisors 24/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 24/5555 • Can setup network/storage • Utilizes local user/group services Question What are the limitations of something like libvirt?
• Network across entire environment? • No rapid VM creation • Weak multi-tenancy • Not a cloud (no utility abstraction)
libvirt
• Provides APIs/tools for managing VMs (mainly on Linux)
25/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5555 • Utilizes local user/group services Question What are the limitations of something like libvirt?
• Network across entire environment? • No rapid VM creation • Weak multi-tenancy • Not a cloud (no utility abstraction)
libvirt
• Provides APIs/tools for managing VMs (mainly on Linux) • Can setup network/storage
25/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5555 Question What are the limitations of something like libvirt?
• Network across entire environment? • No rapid VM creation • Weak multi-tenancy • Not a cloud (no utility abstraction)
libvirt
• Provides APIs/tools for managing VMs (mainly on Linux) • Can setup network/storage • Utilizes local user/group services
25/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5555 Question What are the limitations of something like libvirt?
• Network across entire environment? • No rapid VM creation • Weak multi-tenancy • Not a cloud (no utility abstraction)
libvirt
• Provides APIs/tools for managing VMs (mainly on Linux) • Can setup network/storage • Utilizes local user/group services
25/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5555 • Network across entire environment? • No rapid VM creation • Weak multi-tenancy • Not a cloud (no utility abstraction)
libvirt
• Provides APIs/tools for managing VMs (mainly on Linux) • Can setup network/storage • Utilizes local user/group services Question What are the limitations of something like libvirt?
25/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5555 libvirt
• Provides APIs/tools for managing VMs (mainly on Linux) • Can setup network/storage • Utilizes local user/group services Question What are the limitations of something like libvirt?
• Network across entire environment? • No rapid VM creation • Weak multi-tenancy • Not a cloud (no utility abstraction) 25/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 25/5555 Openstack Overview • Originated in 2010, by Rackspace and NASA • Rapid development - new (stable?) release every 6 months • Modular architecture made up of multiple projects that provide a separate piece of an IaaS service, each with their own REST API • Command-line and GUI interfaces • Seeks to be to IaaS what Linux is to OSes
26/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 26/5555 Interest in Openstack Google Trends for Openstack 100
80
60
Interest 40
20
0
Jun 2010Dec 2010Jun 2011Dec 2011Jun 2012Dec 2012Jun 2013Dec 2013 Date 27/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 27/5555 Openstack Architecture
28/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 28/5555 Openstack Architecture
Okay, that wasn’t so bad, why do you complain about the complexity of dealing with this?
29/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 29/5555 Openstack Architecture - expanded
30/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 30/5555 Openstack Architecture
Question That looks more complex than libvirt, why do I want to do this?
• Life is easier AFTER setting it up (mostly) • Lots of automation available - not always if you want to do something special
31/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 31/5555 ◦ e.g. You can connect to the compute service via http://cloud1:8774
• Contains a list of all the components in the system • Multiple backends (LDAP, MySQL, etc..) • Also acts as the index for API endpoints
Example command to list known services: root@logging:~# keystone service-list +------+------+------+------+ | id | name | type | description | +------+------+------+------+ | 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service | | 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service | | 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service | | b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service | | f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service | +------+------+------+------+
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications
32/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5555 ◦ e.g. You can connect to the compute service via http://cloud1:8774
• Multiple backends (LDAP, MySQL, etc..) • Also acts as the index for API endpoints
Example command to list known services: root@logging:~# keystone service-list +------+------+------+------+ | id | name | type | description | +------+------+------+------+ | 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service | | 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service | | 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service | | b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service | | f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service | +------+------+------+------+
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications • Contains a list of all the components in the system
32/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5555 ◦ e.g. You can connect to the compute service via http://cloud1:8774
• Also acts as the index for API endpoints
Example command to list known services: root@logging:~# keystone service-list +------+------+------+------+ | id | name | type | description | +------+------+------+------+ | 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service | | 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service | | 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service | | b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service | | f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service | +------+------+------+------+
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications • Contains a list of all the components in the system • Multiple backends (LDAP, MySQL, etc..)
32/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5555 ◦ e.g. You can connect to the compute service via http://cloud1:8774
Example command to list known services: root@logging:~# keystone service-list +------+------+------+------+ | id | name | type | description | +------+------+------+------+ | 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service | | 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service | | 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service | | b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service | | f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service | +------+------+------+------+
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications • Contains a list of all the components in the system • Multiple backends (LDAP, MySQL, etc..) • Also acts as the index for API endpoints
32/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5555 Example command to list known services: root@logging:~# keystone service-list +------+------+------+------+ | id | name | type | description | +------+------+------+------+ | 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service | | 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service | | 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service | | b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service | | f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service | +------+------+------+------+
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications • Contains a list of all the components in the system • Multiple backends (LDAP, MySQL, etc..) • Also acts as the index for API endpoints ◦ e.g. You can connect to the compute service via http://cloud1:8774
32/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5555 Example command to list known services: root@logging:~# keystone service-list +------+------+------+------+ | id | name | type | description | +------+------+------+------+ | 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service | | 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service | | 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service | | b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service | | f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service | +------+------+------+------+
Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications • Contains a list of all the components in the system • Multiple backends (LDAP, MySQL, etc..) • Also acts as the index for API endpoints ◦ e.g. You can connect to the compute service via http://cloud1:8774
32/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5555 Piece-by-piece: Identity (keystone)
• Provides authentication for users and applications • Contains a list of all the components in the system • Multiple backends (LDAP, MySQL, etc..) • Also acts as the index for API endpoints ◦ e.g. You can connect to the compute service via http://cloud1:8774
Example command to list known services: root@logging:~# keystone service-list +------+------+------+------+ | id | name | type | description | +------+------+------+------+ | 6dc206b8e4dd4b378123dcf354aebb8f | nova | compute | Openstack Compute Service | | 785fb6f0251742afa43a06b2c6f8d730 | cinder | volume | Cinder Service | | 8fccfcdac18a4efc98b7f1e149c752ec | keystone | identity | OpenStack Identity Service | | b1f1c9dd25864eca89aee1b52dc6e66d | glance | image | Openstack Image Service | | f484b860fc8542928bb9d0db2a1b83b3 | nova_ec2 | ec2 | EC2 Service | +------+------+------+------+ 32/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 32/5555 ◦ Support for libvirt, ESX, Hyper-V, etc...
• Communicates with the hypervisor to create/destroy virtual machines • Similar to job scheduler in HPC
• Each compute node connects to the others via a message bus (AMQP)
Piece-by-piece: Compute (nova)
• Originally contributed by NASA
COMPUTE33/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 33/5555 ◦ Support for libvirt, ESX, Hyper-V, etc...
• Similar to job scheduler in HPC
• Each compute node connects to the others via a message bus (AMQP)
Piece-by-piece: Compute (nova)
• Originally contributed by NASA • Communicates with the hypervisor to create/destroy virtual machines
COMPUTE33/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 33/5555 ◦ Support for libvirt, ESX, Hyper-V, etc... • Each compute node connects to the others via a message bus (AMQP)
Piece-by-piece: Compute (nova)
• Originally contributed by NASA • Communicates with the hypervisor to create/destroy virtual machines • Similar to job scheduler in HPC
COMPUTE33/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 33/5555 • Each compute node connects to the others via a message bus (AMQP)
Piece-by-piece: Compute (nova)
• Originally contributed by NASA • Communicates with the hypervisor to create/destroy virtual machines • Similar to job scheduler in HPC ◦ Support for libvirt, ESX, Hyper-V, etc...
COMPUTE33/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 33/5555 Piece-by-piece: Compute (nova)
• Originally contributed by NASA • Communicates with the hypervisor to create/destroy virtual machines • Similar to job scheduler in HPC ◦ Support for libvirt, ESX, Hyper-V, etc... • Each compute node connects to the others via a message bus (AMQP)
COMPUTE33/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 33/5555 • Copies image from storage to the compute node, with a copy-on-write file for each instance • Can use local storage or an object store
Piece-by-piece: Image (glance)
• Provides VM images to the compute service for booting
34/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 34/5555 • Can use local storage or an object store
Piece-by-piece: Image (glance)
• Provides VM images to the compute service for booting • Copies image from storage to the compute node, with a copy-on-write file for each instance
34/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 34/5555 Piece-by-piece: Image (glance)
• Provides VM images to the compute service for booting • Copies image from storage to the compute node, with a copy-on-write file for each instance • Can use local storage or an object store
34/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 34/5555 glance and nova with libvirt
openstack-glance openstack-nova VM libvirt libvirt base instance disks
VM
• Compute server pulls the image from glance at instance start • Instance disks are copy-on-write images (e.g. QCOW) off of this base • QCOW images are deleted upon instance termination
35/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 35/5555 • Can be used as boot disks or additional storage - dynamically • Built on top of iSCSI, etc • Integration for enterprise block storage (NetApp, IBM, etc...) • NFS/Ceph/gluster as well
Piece-by-piece: Block Storage (cinder)
• Block storage: persistent disk for instances
36/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 36/5555 • Built on top of iSCSI, etc • Integration for enterprise block storage (NetApp, IBM, etc...) • NFS/Ceph/gluster as well
Piece-by-piece: Block Storage (cinder)
• Block storage: persistent disk for instances • Can be used as boot disks or additional storage - dynamically
36/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 36/5555 • Integration for enterprise block storage (NetApp, IBM, etc...) • NFS/Ceph/gluster as well
Piece-by-piece: Block Storage (cinder)
• Block storage: persistent disk for instances • Can be used as boot disks or additional storage - dynamically • Built on top of iSCSI, etc
36/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 36/5555 • NFS/Ceph/gluster as well
Piece-by-piece: Block Storage (cinder)
• Block storage: persistent disk for instances • Can be used as boot disks or additional storage - dynamically • Built on top of iSCSI, etc • Integration for enterprise block storage (NetApp, IBM, etc...)
36/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 36/5555 Piece-by-piece: Block Storage (cinder)
• Block storage: persistent disk for instances • Can be used as boot disks or additional storage - dynamically • Built on top of iSCSI, etc • Integration for enterprise block storage (NetApp, IBM, etc...) • NFS/Ceph/gluster as well
36/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 36/5555 • Object storage: distributed object/file store • Similar to Amazon S3 • Eventually consistent • Online capacity adjustment • Versioned writes • Quota management • rsync based replication
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace
37/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5555 • Similar to Amazon S3 • Eventually consistent • Online capacity adjustment • Versioned writes • Quota management • rsync based replication
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace • Object storage: distributed object/file store
37/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5555 • Eventually consistent • Online capacity adjustment • Versioned writes • Quota management • rsync based replication
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace • Object storage: distributed object/file store • Similar to Amazon S3
37/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5555 • Online capacity adjustment • Versioned writes • Quota management • rsync based replication
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace • Object storage: distributed object/file store • Similar to Amazon S3 • Eventually consistent
37/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5555 • Versioned writes • Quota management • rsync based replication
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace • Object storage: distributed object/file store • Similar to Amazon S3 • Eventually consistent • Online capacity adjustment
37/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5555 • Quota management • rsync based replication
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace • Object storage: distributed object/file store • Similar to Amazon S3 • Eventually consistent • Online capacity adjustment • Versioned writes
37/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5555 • rsync based replication
Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace • Object storage: distributed object/file store • Similar to Amazon S3 • Eventually consistent • Online capacity adjustment • Versioned writes • Quota management
37/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5555 Piece-by-piece: Object Storage (swift)
• Originally contributed by Rackspace • Object storage: distributed object/file store • Similar to Amazon S3 • Eventually consistent • Online capacity adjustment • Versioned writes • Quota management • rsync based replication
37/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 37/5555 ◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc...
• Now its own project (formerly quantum) • Provides abstraction layer for creating virtual networks
• Also provides services like load balancing • Part of the official release since 2012.2
Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute
38/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5555 ◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc...
• Provides abstraction layer for creating virtual networks
• Also provides services like load balancing • Part of the official release since 2012.2
Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute • Now its own project (formerly quantum)
38/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5555 ◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc... • Also provides services like load balancing • Part of the official release since 2012.2
Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute • Now its own project (formerly quantum) • Provides abstraction layer for creating virtual networks
38/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5555 • Also provides services like load balancing • Part of the official release since 2012.2
Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute • Now its own project (formerly quantum) • Provides abstraction layer for creating virtual networks ◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc...
38/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5555 • Part of the official release since 2012.2
Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute • Now its own project (formerly quantum) • Provides abstraction layer for creating virtual networks ◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc... • Also provides services like load balancing
38/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5555 Piece-by-piece: Network (neutron)
• In earlier openstack releases, networking was a part of compute • Now its own project (formerly quantum) • Provides abstraction layer for creating virtual networks ◦ OpenVSwitch, Linux Bridge, OpenFlow, Cisco offerings, etc... • Also provides services like load balancing • Part of the official release since 2012.2
38/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 38/5555 Openstack Architecture (Reprise)
39/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 39/5555 Tying it all together (simplified)
API identity
...
network compute image
libvirt
OS KVM Hardware 40/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 40/5555 But wait, there’s more!
I only focused on the “core components” for a working cloud, but there are plenty more • Ceilometer - Metering/Monitoring • Heat - Automation (Similar to AWS CloudFormation) • Savannah - Tighter Hadoop Integration
41/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 41/5555 An actual production use case
42/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 42/5555 • Not worth runnning tightly-coupled HPC workloads • Need OS bypasss for infiniband and GPUs • “Developer productivity went through the roof” ◦ Easy, low-cost experimentation • IaaS good for data-centric workloads (bioinformatics) ◦ Many folks that run hadoop do so on top of AWS
ANL Magellan experiences
• CPU Performance was good • I/O bandwidth was decent • I/O latency was terrible
43/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 43/5555 • “Developer productivity went through the roof” ◦ Easy, low-cost experimentation • IaaS good for data-centric workloads (bioinformatics) ◦ Many folks that run hadoop do so on top of AWS
ANL Magellan experiences
• CPU Performance was good • I/O bandwidth was decent • I/O latency was terrible • Not worth runnning tightly-coupled HPC workloads • Need OS bypasss for infiniband and GPUs
43/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 43/5555 ANL Magellan experiences
• CPU Performance was good • I/O bandwidth was decent • I/O latency was terrible • Not worth runnning tightly-coupled HPC workloads • Need OS bypasss for infiniband and GPUs • “Developer productivity went through the roof” ◦ Easy, low-cost experimentation • IaaS good for data-centric workloads (bioinformatics) ◦ Many folks that run hadoop do so on top of AWS
43/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 43/5555 ◦ Can give them root
− Things are always changing + Noticeable improvement with each release
◦ Always run into some problem ◦ Never the same problem twice • Automated tools are your friend (packstack, puppet) • Really convenient for kernel hacking • Developers like it a lot
• Rapid Release
• VMs are just as reliable as with libvirt, mgmt not so much
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana)
44/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5555 ◦ Can give them root
− Things are always changing + Noticeable improvement with each release
◦ Never the same problem twice • Automated tools are your friend (packstack, puppet) • Really convenient for kernel hacking • Developers like it a lot
• Rapid Release
• VMs are just as reliable as with libvirt, mgmt not so much
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana) ◦ Always run into some problem
44/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5555 ◦ Can give them root
− Things are always changing + Noticeable improvement with each release
• Automated tools are your friend (packstack, puppet) • Really convenient for kernel hacking • Developers like it a lot
• Rapid Release
• VMs are just as reliable as with libvirt, mgmt not so much
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana) ◦ Always run into some problem ◦ Never the same problem twice
44/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5555 ◦ Can give them root
− Things are always changing + Noticeable improvement with each release
• Really convenient for kernel hacking • Developers like it a lot
• Rapid Release
• VMs are just as reliable as with libvirt, mgmt not so much
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana) ◦ Always run into some problem ◦ Never the same problem twice • Automated tools are your friend (packstack, puppet)
44/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5555 ◦ Can give them root
− Things are always changing + Noticeable improvement with each release
• Developers like it a lot
• Rapid Release
• VMs are just as reliable as with libvirt, mgmt not so much
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana) ◦ Always run into some problem ◦ Never the same problem twice • Automated tools are your friend (packstack, puppet) • Really convenient for kernel hacking
44/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5555 − Things are always changing + Noticeable improvement with each release
◦ Can give them root • Rapid Release
• VMs are just as reliable as with libvirt, mgmt not so much
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana) ◦ Always run into some problem ◦ Never the same problem twice • Automated tools are your friend (packstack, puppet) • Really convenient for kernel hacking • Developers like it a lot
44/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5555 − Things are always changing + Noticeable improvement with each release
• Rapid Release
• VMs are just as reliable as with libvirt, mgmt not so much
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana) ◦ Always run into some problem ◦ Never the same problem twice • Automated tools are your friend (packstack, puppet) • Really convenient for kernel hacking • Developers like it a lot ◦ Can give them root
44/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5555 − Things are always changing + Noticeable improvement with each release • VMs are just as reliable as with libvirt, mgmt not so much
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana) ◦ Always run into some problem ◦ Never the same problem twice • Automated tools are your friend (packstack, puppet) • Really convenient for kernel hacking • Developers like it a lot ◦ Can give them root • Rapid Release
44/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5555 + Noticeable improvement with each release • VMs are just as reliable as with libvirt, mgmt not so much
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana) ◦ Always run into some problem ◦ Never the same problem twice • Automated tools are your friend (packstack, puppet) • Really convenient for kernel hacking • Developers like it a lot ◦ Can give them root • Rapid Release − Things are always changing
44/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5555 • VMs are just as reliable as with libvirt, mgmt not so much
My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana) ◦ Always run into some problem ◦ Never the same problem twice • Automated tools are your friend (packstack, puppet) • Really convenient for kernel hacking • Developers like it a lot ◦ Can give them root • Rapid Release − Things are always changing + Noticeable improvement with each release
44/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5555 My Experiences
• Set up 5 environments (Essex, Folsom x2, Grizzly, Havana) ◦ Always run into some problem ◦ Never the same problem twice • Automated tools are your friend (packstack, puppet) • Really convenient for kernel hacking • Developers like it a lot ◦ Can give them root • Rapid Release − Things are always changing + Noticeable improvement with each release • VMs are just as reliable as with libvirt, mgmt not so much
44/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 44/5555 Hypervisor Performance Comparison
45/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 45/5555 Experimental Setup Bioinformatics Applications: Systems: • Paired-end short read • Dell R720 (2013) (Machine 1) alignment • Homebuilt (2011) (Machine 2) • Burrows-Wheeler Aligner • Ubuntu 12.04 LTS • Novoalign
46/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 46/5555 Memory Performance
8000 STREAM Results PHY 7000 KVM XEN-pv 6000 LXC
5000
4000 MiB/s 3000
2000
1000
0 Copy Scale Add Triad 47/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 47/5555 Disk Performance
600 Bonnie++ Results 1400 PHY KVM 1200 500 XEN-pv LXC 1000 400 800 300
MiB/s 600 Seeks/s 200 400
100 200
0 0 Seq. Write Seq. Read Seek
48/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 48/5555 1.25 BWA Runtime (Machine 2)
1.20
1.15 1.11 1.10
1.05 1.00 1.00 1.00 1.00
Mean Runtime0 (normalized to. physical) 95 PHY KVM XEN-pv LXC
Novoalign Runtime 1.10 Novoalign Runtime (Machine 2) 1.04 1.03 1.08 1.06 1.06 1.02 1.00 1.00 1.04 1.00 0.99 1.02 1.00 1.00 0.99 0.98 0.98 0.96 0.96
Mean Runtime (normalized to physical) PHY KVM XEN-pv LXC Mean Runtime (normalized to physical) PHY KVM XEN-pv LXC
0.00
1.25 BWA Runtime
1.20 1.19
1.15
1.10 1.09
1.05 1.00 1.00 1.00
Mean Runtime0 (normalized to. physical) 95 PHY KVM XEN-pv LXC
49/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 49/5555 1.25 BWA Runtime (Machine 2)
1.20
1.15 1.11 1.10
1.05 1.00 1.00 1.00 1.00
Mean Runtime0 (normalized to. physical) 95 PHY KVM XEN-pv LXC
1.10 Novoalign Runtime (Machine 2)
1.08 1.06 1.06
1.04
1.02 1.00 1.00 0.99
0.98
0.96
Mean Runtime (normalized to physical) PHY KVM XEN-pv LXC
0.00
1.25 BWA Runtime
1.20 1.19
1.15
1.10 1.09
1.05 1.00 1.00 1.00
Mean Runtime0 (normalized to. physical) 95 PHY KVM XEN-pv LXC
Novoalign Runtime
1.04 1.03
1.02 1.00 1.00
1.00 0.99
0.98
0.96
Mean Runtime (normalized to physical) PHY KVM XEN-pv LXC 49/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 49/5555 1.10 Novoalign Runtime (Machine 2)
1.08 1.06 1.06
1.04
1.02 1.00 1.00 0.99
0.98
0.96
Mean Runtime (normalized to physical) PHY KVM XEN-pv LXC
0.00
1.25 BWA Runtime 1.25 BWA Runtime (Machine 2)
1.20 1.19 1.20
1.15 1.15 1.11 1.10 1.09 1.10
1.05 1.05 1.00 1.00 1.00 1.00 1.00 1.00 1.00
Mean Runtime0 (normalized to. physical) 95 Mean Runtime0 (normalized to. physical) 95 PHY KVM XEN-pv LXC PHY KVM XEN-pv LXC
Novoalign Runtime
1.04 1.03
1.02 1.00 1.00
1.00 0.99
0.98
0.96
Mean Runtime (normalized to physical) PHY KVM XEN-pv LXC 49/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 49/5555 1.25 BWA Runtime 1.25 BWA Runtime (Machine 2)
1.20 1.19 1.20
1.15 1.15 1.11 1.10 1.09 1.10
1.05 1.05 1.00 1.00 1.00 1.00 1.00 1.00 1.00
Mean Runtime0 (normalized to. physical) 95 Mean Runtime0 (normalized to. physical) 95 PHY KVM XEN-pv LXC PHY KVM XEN-pv LXC
Novoalign Runtime 1.10 Novoalign Runtime (Machine 2) 1.04 1.03 1.08 1.06 1.06 1.02 1.00 1.00 1.04 1.00 0.99 1.02 1.00 1.00 0.99 0.98 0.98 0.96 0.96
Mean Runtime (normalized to physical) PHY KVM XEN-pv LXC Mean Runtime (normalized to physical) PHY KVM XEN-pv LXC 49/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 49/5555
0.00 Multiple VMs
1.35BWA Runtime (8 Concurrent Executions) Novoalign Runtime (8 Concurrent Executions)
1.30 1.28 1.29 1.16 1.15 1.25 1.22 1.12 1.20 1.19 1.17 1.18 1.10 1.15 1.08 1.08 1.08 1.07 1.07 1.15 1.12 1.06 1.10 1.05
1.05 1.00 1.00 Mean Runtime (normalized to single) Mean Runtime0 (normalized to. single) 95 0.95 PHY KVM LXC PHY KVM LXC PHY-pin XEN-pv LXC-pin PHY-pin XEN-pv LXC-pin KVM-pin KVM-pin XEN-pv-pin XEN-pv-pin • Only on the Dell R720 • Still not utilizing this machine fully
50/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 50/5555 Application Run Time Statistics
BWA System Statistics Novoalign System Statistics 1.5 1.5 1.0 User 1.0 User 0.5 Sys 0.5 Sys 0.0 0.0 Active CPUs Active CPUs 0 20 40 60 80 100 0 20 40 60 80 100
3 3 2 Used 2 Used Mapped Mapped 1 1 Memory Cache Memory Cache % 0 % 0 0 20 40 60 80 100 0 20 40 60 80 100
600 400 300 400 Read Read 200 200 Write Write MiB/s MiB/s 100 0 0 0 20 40 60 80 100 0 20 40 60 80 100 Run completion (%) Run completion (%) • Gathered using collectl • Only for a single run
51/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 51/5555 • Performance for a given VMM is dependent on the application being executed • CPU-bound applications run at near-baremetal speeds in virtualized environments • VMM scheduling overheard is low when VMs are not overprovisioned • Linux Containers have low performance overhead
Some Conclusions
• CPU pinning can be effective (but be careful)
52/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 52/5555 • CPU-bound applications run at near-baremetal speeds in virtualized environments • VMM scheduling overheard is low when VMs are not overprovisioned • Linux Containers have low performance overhead
Some Conclusions
• CPU pinning can be effective (but be careful) • Performance for a given VMM is dependent on the application being executed
52/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 52/5555 • VMM scheduling overheard is low when VMs are not overprovisioned • Linux Containers have low performance overhead
Some Conclusions
• CPU pinning can be effective (but be careful) • Performance for a given VMM is dependent on the application being executed • CPU-bound applications run at near-baremetal speeds in virtualized environments
52/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 52/5555 • Linux Containers have low performance overhead
Some Conclusions
• CPU pinning can be effective (but be careful) • Performance for a given VMM is dependent on the application being executed • CPU-bound applications run at near-baremetal speeds in virtualized environments • VMM scheduling overheard is low when VMs are not overprovisioned
52/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 52/5555 Some Conclusions
• CPU pinning can be effective (but be careful) • Performance for a given VMM is dependent on the application being executed • CPU-bound applications run at near-baremetal speeds in virtualized environments • VMM scheduling overheard is low when VMs are not overprovisioned • Linux Containers have low performance overhead
52/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 52/5555 For more info...
https://www.usenix.org/legacy/events/sec2000/robin.html http://www.vmware.com/pdf/asplos235_adams.pdf http://www.vmware.com/files/pdf/VMware_ paravirtualization.pdf http://software.intel.com/en-us/blogs/2009/06/25/ virtualization-and-performance-understanding-vm-exits http://wiki.xen.org/wiki/Event_Channel_Internals http: //www.docstoc.com/docs/45758154/Understanding-Intel-% C2%AE-Virtualization-Technology-(VT) http://wiki.openstack.org http://devstack.org
53/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 53/5555 Slide theme courtesy Flip Tanedo, Cornell Images are rights of their respective owners
54/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 54/5555 Machine Statistics
Machine 1: • dual-socket 8 core Intel Xeon E5-2660 2.2GHz CPUs (3.0GHz Turbo boost), 20MiB cache • 128GiB of DDR3-1333MHz • 8 SAS 10K RAID 10, 1024MiB cache Machine 2: • dual-socket 6 core Intel Xeon E5645 2.40GHz (2.67 GHz Turbo boost), 12MiB cache • 32GiB of DDR3-1333Mhz • 1 TiB SATA Disk
55/ Zak Estrada What’s in a Cloud? An Overview of Virtualization and Openstack 55/5555