Open Source Virtualization - Software Freedom Day 2008
Michael Clark
Sep 20, 2008 1 Presentation Overview
A bit on Software Freedom i.e. why we are here What is Virtualization & why do we need it Available Virtualization Technologies How it all works underneath How Xen and KVM do it Examples on deploying Xen Also EVMS Logical Volume Management KVM Virtualization Demo
Sep 20, 2008 2 What is Open Source / Free Software?
Sep 20, 2008 3 Open Source Software
You rely on it every day Two thirds of web sites are served using the Apache Open Source Software The biggest sites on the Internet such as Google, Ebay, Yahoo and Amazon are all using Open Source Software Approximately 20% of all new servers are shipped or installed with the Linux OS
Sep 20, 2008 4 Open Source Software
You rely on it every day The vast majority of the Domain Name System (DNS) is implemented using Open Source Software Linux, an Open Source Operating System is being used by Corporations, Governments, Schools, Businesses and Individuals. The Fabric of the Internet is built on Open Source Software!
Sep 20, 2008 5 What is Open Source?
Open Source can be seen as 3 things:
Open Source Software
Open Source Development Model
Open Source Community
Sep 20, 2008 6 What is Open Source?
Open Source Software Software that is unencumbered – Source code for developers – Binary code for users Allows free redistribution – No costly licensing fees, change to a support model Allows creation of derived works – You have the source to change it to suit your needs See http://www.opensource.org/ for a full definition
Sep 20, 2008 7 What is Open Source?
Open Source Development Model A superior development model that takes advantage of this access to source code Build on existing code, lower cost of development Shared common benefit – Amortise development cost with others Open collaboration using the Internet End users can contribute directly to the development process – it is not being done behind closed doors
Sep 20, 2008 8 What is Open Source?
Open Source Community The individuals, IT vendors, end user corporations and business that are producers, consumers or supporters of Open Source Software Communication is rapid and distributed Many choices for end user support – Go straight to the developers for support – Get commercial support from a OSS solution vendor such as Red Hat, Novell, Dell, HP, Sun, IBM, etc, ... – Use the source Luke, do it yourself!
Sep 20, 2008 9 What is Virtualization?
Sep 20, 2008 10 What is Virtualization?
Virtual Machines create the appearance of a real machine “virtual” defines something that is not “real” but appears “real” ie. virtually a machine Provides a simulation of some real machine A program running in a Virtual machine should behave as it would running under on a real machine One physical machine can run many virtual machines
Sep 20, 2008 11 What is Virtualization? Normal non-virtualized architecture
Application Application A B
Operating system
PC Hardware
Sep 20, 2008 12 What is Virtualization? Fully Virtualized architecture
App A App B
OS OS (Each virtual machine Virtual Virtual hardware hardware has separate RAM, disk, and share of CPU plus their own Virtulization software network address).
OS / Hypervisor
PC Hardware
Sep 20, 2008 13 Why do we need Virtualization?
Flexibility Run multiple OS on one machine. Solaris x86, FreeBSD, Linux, Windows, etc... Run an application not available on your OS Easy testing of software on different operating systems without a desk full of machines. – e.g. testing web applications with different browser/OS versions. Simulate a network of machines. Desktop / Server Virtualization
Sep 20, 2008 14 Why do we need Virtualization?
Server consolidation Reduce management of many physical machines – Host many virtual machines on a smaller number of larger and more powerful physical machines. Utilize resources (CPU/Disk/RAM) more efficiently. – Left over resources can't traditionally be used. Easily and dynamically control resource distribution – RAM, CPU, Disk Deploy new servers on-demand – Quicker time-to-market
Sep 20, 2008 15 Why do we need Virtualization?
Increased Security Partition untrusted applications from each other. – Minimize impact when an application is compromised. Partition applications across machines for separate management and administrative domains. – e.g. person with Administrator access to Customer Service dept. machine does not have Administrator access to Accounts dept.
Sep 20, 2008 16 Why do we need Virtualization?
Example server utilisation 6 x Dual Xeon servers
wasted capacity
server1 server2 server3
not enough capacity
server4 server5 server6
Sep 20, 2008 17 Why do we need Virtualization? Virtual machines consolidated 2 x Eight Core Xeon servers
vm4 vm3 vm5 vm2
vm6 vm1
(SAN is optional server1 server2 but popular Fibre Channel SAN architecture)
Sep 20, 2008 18 Types of virtualization
Operating system level virtualization Time-sharing / Virtual Memory Virtual Private Servers / Resource Containers Full-virtualization Emulated – slow using CPU emulation Hybrid – combination of emulation and hardware Hardware assisted – Intel VT, AMD Pacifica Para-virtualization Modified operating system kernel
Sep 20, 2008 19 Virtualization Technologies
VMWare http://www.vmware.com/ Provides “Full Virtualization” of X86 architecture. Has performance penalties for Ring 0 emulation. Proprietary commercial software. Runs on Windows, Linux and Mac OS X Multiple flavours: – Server (Freeware) – Workstation – ESX – ESX Lite - BIOS hypervisor
Sep 20, 2008 20 Virtualization Technologies
UML – User mode Linux http://http://user-mode-linux.sourceforge.net/ Provides virtualization of X86, X86-64, and PowerPC Linux userspace ABI (Application Binary interface). Linux kernel is effectively ported to run as a user program instead of on a hardware interface. One kernel image per virtual machine. Open Source software
Sep 20, 2008 21 Virtualization Technologies
QEMU http://www.qemu.org/ Provides full virtualization of x86, PowerPC, MIPS, Sparc through CPU simulation. Slow! Runs on many Operating Systems – Linux, FreeBSD, Solaris, Mac OS X, Windows, ... QEMU is open source – although there exists a proprietary module for x86 which provides accelerated simulation similar to VMWare
Sep 20, 2008 22 Virtualization Technologies
Linux Virtual Server http://linux-vserver.org/ Operating system level virtualization for Linux Provides enhanced security chroot (change root) type virtualization. ie. system runs only one kernel. Single kernel image shared by all virtual machines. Open Source software
Sep 20, 2008 23 Virtualization Technologies
OpenVZ http://openvz.org/ Operating system level virtualization for Linux Support all architectures that Linux supports Runs a single Linux kernel with security containerization / isolation – Lighter-weight than full virtualisation although offers less isolation between virtual machines Open Source software Technology behind proprietary Virtuozzo product
Sep 20, 2008 24 Virtualization Technologies
Xen http://xensource.org/ Initially a para-virtualization only solution – para-virtualization requires a modified operating system kernel Now supports hardware-assisted virtualization – Intel VT, AMD Pacifica Hypervisor-based Open Source software
Sep 20, 2008 25 Virtualization Technologies
KVM – Kernel Virtual Machine http://kvm.qumranet.com/ A full-virtualization solution for Linux – Intel VT, AMD Pacifica Does not require modification to operating system Kernel-based (as against hypervisor based) Open Source software
Sep 20, 2008 26 Virtualization Technologies
Many more ... Parallels Desktop – Proprietary software – Runs on Linux, Windows and Mac OS X Microsoft Virtual Server – Proprietary software – Runs on Windows only VirtualBox - http://virtualbox.org/ – Open Source software ...
Sep 20, 2008 27 How it all works...
Xen and KVM
Sep 20, 2008 28 Why is virtualization hard?
Design“flaws” of intel x86 x86 has 4 privilege levels named “rings” (0-3) and each ring is protected from each other. Executing privileged instructions causes a trap (or fault). – lgdt, lldt, hlt, updating page tables OSes such as Linux runs its kernel code in Ring 0 and user code in unprivileged Ring 3 x86 is not fully virtualizable by design. Certain Ring 0 instructions cannot be run in Ring 1 and trapped so Ring 0 can't be virtualized without special tricks. Full virtualization techniques such as those in VMWare need to emulate or rewrite code running in Ring 0 (big slow down!)
Sep 20, 2008 29 How Xen does it?
Xen Paravirtualization Xen implements a hypervisor that runs at Ring 0 and the Guest OS kernel runs at Ring 1 Xen requires small poritions of the Linux kernel source code to be modified. Instead of the Linux kernel calling the privileged instructions directly, a call is made to the Xen hypervisor to virtual memory update page tables, halt the processor, etc. This has the advantage of much higher performance than full virtualization techniques like VMWare as not code inspection/ emulation is required of code executing in Ring 0.
Sep 20, 2008 30 How Xen does it?
Xen hypervisor The xen hypervisor is actually loaded instead of the linux kernel and arbitrates access to system resources and virtual memory. Domain0 The Domain0 kernel is the first virtual machine kernel that is loaded and it has access to the physical hardware on the underlying pyshical machine. Only one Domain0 runs in the first memory partition. DomainU's The DomainU kernels are the virtual machines kernels. Many domainU kernels run in a Xen system. Each one has a partition of memory allocated to it. DomainU kernels access network and storage through the Domain0 kernel as they do not have access to hardware.
Sep 20, 2008 31 Xen 3.0
Xen requires source modification on standard x86 Paravirtualization improves performance due to lack of full virtualization support in x86 (by avoiding Ring 0 code inspection techniques such as those used in VMWare). Intel VT and AMD Pacifica instructions Intel adding to the Pentium D and other CPUs - allows Xen to offer full virtualization with no source modification. Ring 0 will be able to be fully emulated with these extensions Xen can run unmodified Windows XP VT Will also speed up software such as VMWare AMD has its own similar but incompatible virtualization extensions called 'Pacifica'
Sep 20, 2008 32 How KVM does it?
KVM is a hardware-assisted full virtualization solution Only works on new hardware with Intel VT or AMD Pacifica instruction sets Does not require modification to linux kernel – Runs as a loadable module – Each VM runs as a Linux process Kernel based instead of hypervisor on bare-metal – Does not require you to boot a hypervisor to run – Any new linux desktop on modern hardware can run KVM without modifications.
Sep 20, 2008 33 Kernel-based vs Hypervisor based
Hypervisor-based Kernel-based
dom1 dom2 dom3 vm1 vm2 vm3 kernel kernel kernel kernel kernel kernel
kvm dom0 VT kernel normal Hypervisor kernel IO MMU IO MMU PC Hardware PC Hardware
Sep 20, 2008 34 Building and Installing Xen
Building with EVMS support on Debian Sarge Xen source and binaries download here: http://www.xensource.com/downloads/ Xen-3.0 supports x86, x86-64, x86 PAE, Intel VT PAE allows you to use more than 4GBs RAM on 32bit x86 hardware that supports this (Xeon)
Sep 20, 2008 35 Building and Installing Xen
Install EVMS, python twisted and libcurl
# aptget install evms evmscli evmsncurses evmsgui kernelpatchevms \ python2.3dev python2.3twisted libcurl3dev
Download, configure, compile, patch & install Xen
# wget http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads/xen3.0.2src.tgz # tar xzf xen3.0.2src.tgz # cd xen3.0.2 # make kernels # cd linux2.6.16xen # zcat /usr/src/kernelpatches/diffs/evmsbdclaim/2.6bdclaim.patch.gz | patch p1 # cd .. # make all # make install
Sep 20, 2008 36 Building and Installing Xen
Customising kernel config (Xen 3.0.2) Domain0 and DomainU share same kernel
# cd linux2.6.16xen # make menuconfig # cd .. # make install
Sep 20, 2008 37 Building and Installing Xen
Configure bridged network (xen-br0) Virtual machines used bridged networking by default and need their own IP allocated to them Debian example /etc/network/interfaces – aptget install bridgeutils
#auto eth0 auto xenbr0 iface xenbr0 inet static bridge_ports eth0 address 192.168.?.? netmask 255.255.255.0 broadcast 192.168.?.255 gateway 192.168.?.?
Sep 20, 2008 38 Building and Installing Xen
Example GRUB configuration /boot/grub/menu.lst timeout 5 default 0 root (hd0,0) title 2.6.16xen kernel /boot/xen3.gz dom0_mem=262400 module /boot/vmlinuz2.6.16xen root=/dev/hda1
Sep 20, 2008 39 Building and Installing Xen
Create symlinks in init.d
# ln s ../init.d/xend /etc/init.d/rc0.d/K99xend # ln s ../init.d/xendomains /etc/init.d/rc0.d/K95xendomains # ln s ../init.d/xend /etc/init.d/rc2.d/S99xend # ln s ../init.d/xendomains /etc/init.d/rc2.d/S99xendomains # ln s ../init.d/xend /etc/init.d/rc6.d/K99xend # ln s ../init.d/xendomains /etc/init.d/rc6.d/K95xendomains Disable TLS if using a TLS enabled libc
# mv /lib/tls /lib/tls.disabled
Or install libc6-xen May need to add 'hwcap 0 nosegneg' to ld.so.conf Reboot the system!
Sep 20, 2008 40 Creating block device storage with EVMS
What is EVMS? EVMS (Enterprise Volume Management System) Is a form of a Logical Volume Manager and can actually work with LVM formats although has a much easier GUI management tool Works together well with Xen as allocating storage for Virtual machines needs something more powerful than limited DOS partitions
Sep 20, 2008 41 Creating block device storage with EVMS
EVMS Concepts Segments (Partitions) Container – aka Volume Group in some other LVMs. Region – an allocation section of storage from a container Volume – this is the final block device ie. /dev/evms/volumnename
Sep 20, 2008 42 Creating block device storage with EVMS
Create a segment on a spare disk
Sep 20, 2008 43 Creating block device storage with EVMS
Create a container with the new segment Select segment, Choose Create -> Container – Use LVM2 as type of container Create regions for root and swap Select container, Choose Create -> Region – Use LVM2 region manager Create volumes for root and swap Right click on region and choose “Create EVMS Volume ...” Create filesystems for root and swap Right click on volume and choose “Make filesystem ...”
Sep 20, 2008 44 Creating Xen virtual machines
Create configuration /etc/xen/xentest1
kernel = "/boot/vmlinuz2.6.16xen" memory = 256 name = "xentest1" disk = [ 'phy:/dev/evms/xentest1_root,hda1,w', 'phy:/dev/evms/xentest1_swap,hda2,w' ] hostname= "xentest1" root = "/dev/hda1 ro" vif = [ 'mac=aa:00:00:00:fe:10, bridge=xenbr0' ] restart = 'onreboot'
Sep 20, 2008 45 Creating Xen virtual machines
Populate filesystem
# mkswap /dev/evms/xentest1_swap # mkdir /mnt/xentest1 # mount /dev/evms/xentest1_root /mnt/xentest1 # debootstrap etch /mnt/xentest1 # umount /mnt/xentest1 Make required changes to fstab, network configuration, etc
# vi /mnt/xentest1/etc/fstab # vi /mnt/xentest1/etc/network/interfaces # umount /mnt/xentest1 ...
Sep 20, 2008 46 Creating Xen virtual machines
Start the domain (virtual machine)
# xm create xentest1 Using config file "/etc/xen/xentest1". Started domain xentest1, console on port 9602
Connecting to its console
# xm console xentest1 Using config file "/etc/xen/xentest1". Started domain xentest1, console on port 9602# xm console xentest1 ************ REMOTE CONSOLE: CTRL] TO QUIT ******** Linux version 2.6.11.12xenU (root@monty) (gcc version 3.4.5 20050821 (prerelease) (Debian 3.4.48)) #2 Fri Oct 21 17:53:45 SGT 2005 BIOSprovided physical RAM map: Xen: 0000000000000000 0000000010000000 (usable) 256MB LOWMEM available. DMI not present. IRQ lockup detection disabled Allocating PCI resources starting at 10000000 (gap: 10000000:f0000000)
Sep 20, 2008 47 Creating Xen virtual machines
Listing running domains
# xm list Name Id Mem(MB) CPU State Time(s) Console Domain0 0 745 0 r 131.8 xentest1 2 256 0 b 0.7 9602
Other useful commands xm help – list all xm commands xm dmesg – list hypervisor boot messages xm info – general info such as free and allocated memory xm shutdown – cleanly shutdown a domain xm destroy – forcefully kill a domain
Sep 20, 2008 48 Creating Xen virtual machines
Domains can be autostarted on boot symlink the file into /etc/xen/auto – ln s ../xentest1 /etc/xen/auto/xentest1 Memory for a domain can be shrunk The max memory is set in the config file and it can be lowered within this limit – xm balloon xentest1 128 Domains can even be 'teleported' Requires shared storage ie. Fibre Channel SAN – Memory is Suspended to disk from one physcial machine and resumed on another physical machine.
Sep 20, 2008 49 Questions? - Thank You
Michael Clark
Sep 20, 2008 50