:FM

8[d`e`jk\i`e^m`iklXcdXZ_`e\jn`k_DCE N8K:?@E> K?<=CF:B

This simple command-line tool helps you keep ahead of your virtual infrastructure. BY KYRRE BEGNUM AND ÆLEEN FRISCH

he simplest virtualization scenar- tion, the open source MLN VM adminis- ฀ ฀฀฀฀฀฀฀฀ ios are very easy to manage. In tration tool might be just what you need. implement complex VM deployments. Kthe real world, though, virtual- MLN, which stands for “Manage Large ฀ ฀฀฀฀฀฀ ization does not always simplify system Networks,” lets you build sophisticated, ployment should be as efficient as pos- administration. In fact, many of the most dynamic, virtual infrastructures with the sible. powerful uses of virtualization result in use of freely available virtualization plat- The configuration language should be additional administrative tasks and chal- forms. The versatile MLN currently sup- extensible, so users can customize MLN lenges. For example, large data centers ports the widely used VMware Server [1] easily for their unique needs. use virtualization to consolidate several and Xen [2] packages, as well as User- MLN is designed to work in concert servers onto the same physical hard- Mode [3]. with other VM management tools. For ware, thereby minimizing space, cost, For Xen, MLN also provides live mi- example, if you have a VMware Server, and power requirements. gration of entire network topologies, you can use MLN for efficient building These virtual machines (VMs) can mi- which is otherwise available in applica- and redeploying of complex virtual net- grate between servers to achieve higher tions only at the highest end. MLN’s works and also use VMware Server Con- uptimes and flexible load balancing. simple command-line and config file in- sole for starting and stopping individual However, managing such a complex vir- terfaces serve the needs of complex vir- virtual machines. tual infrastructure is at least as challeng- tualization scenarios, but it is easy Download MLN from SourceForge [4]. ing as administering the equivalent enough for administrators who are new The installation process is simple and number of separate server systems, and to virtualization. similar to the procedure for other Linux it requires very different skills. programs: Many high-end commercial virtualiza- 8YflkDCE tion products come with sophisticated The MLN environment helps you man- # wget 5 VM management features. If you are age large numbers of virtual machines http://mln.sourceforge.net/5 wondering whether the through a simple command-line inter- files/mln-latest.tar.gz community offers an equivalent applica- face. MLN has three main design goals: # tar xzfv mln-latest.tar.gz

22 ISSUE 98 JANUARY 2009

022-030_mln.indd 22 13.11.2008 17:57:08 Uhr MLN :FM

# cd mln-latest Listing 1: ldaptest.mln # ./mln setup 01 global { 17

(Settings affecting all VMs and 18 host ldapserver { ;\]`e`e^Gifa\Zkj switches in the file.) 19 xen Another Xen VM. MLN organizes virtual machines into 02 project ldaptest Project name. units called projects. Projects are groups 20 memory 256M 03 } of virtual machines and their associated 21 lvm 04 virtual networks. The simplest project 22 size 2GB 05 host ldapclient { consists of a single virtual machine. A 23 template ubuntu-ldap.ext3 more powerful approach is to group vir- 06 xen This is a Xen VM. (Initial OS image.) tual machines that belong together: all 07 memory 128M Memory size for VM. 24 nameserver 10.0.0.15

virtual machines that are web servers, 08 lvm Create logical volume 25 network eth0 { This VM uses a an entire replica of a small business net- for storing VM. static address. work, all the test systems for a software 09 size 2GB Size of the VM. 26 address 10.0.0.2

test suite. 10 template ubuntu-client.ext3 27 netmask 255.255.255.0

Projects are defined in MLN configura- 11 nameserver 10.0.0.15 Set the 28 gateway 10.0.0.1 tion files. The configuration file shown name server. 29 switch LDAPlan Connect to in Listing 1 (ldaptest.mln) defines two 12 network eth0 { Config. virt. virtual LAN. net.interface. virtual machines connected by a virtual 30 } LAN. Annotations in the file highlight 13 address dhcp Use DHCP for 31 } the most important configuration set- IP address. 32 tings. 14 switch LDAPlan Connect to Listing 1 starts with the global stanza, virt. switch. 33 switch LDAPlan { } (Create virtual switch connecting 15 } wherein the name of the project is de- the VMs.) fined. Next follows a declaration of a vir- 16 users { kyrre l47/Y.NtB9p7w } tual machine called ldapclient – a Xen (Create user and set encoded host with 128MB of memory built from password.)

022-030_mln.indd 23 13.11.2008 17:57:11 Uhr :FM

Table 1: Working with the mln Command Clearly, writing host stanzas for large numbers of virtual machines would get Action Command tedious. The MLN configuration lan- Build VM mln build -f project-config-file [-r] guage allows you to define superclasses: Reconfigure VM (including live migration) mln upgrade -f modified-config-file named groups of settings that can be Start VM mln start -p project [-h host] applied to multiple hosts. Stop VM mln stop -p project [-h host] Query status of all/ specified projects mln status [-p project] Jlg\iZcXjj List defined projects mln list Listing 2 illustrates the use of a super- Remove a project mln remove -p project class. The superclass, which is called Register a template mln rt -t file-system-image-file basic in Listing 2, provides a bundle of Backup an entire project mln export -p project -d location [-z] settings that are then available for host Restore a saved project mln import -p project -d location definitions. Listing 2 defines three hosts based on the superclass. The third host the template ubuntu-client.ext3. The command to build the virtual machines illustrates the technique of overriding a host has one network interface config- and virtual network inside it. For exam- setting from the superclass. Of course, ured to use DHCP for its IP address. The ple, the following commands create the many more settings could be placed into third stanza defines another Xen host, virtual hosts and network defined in the superclass or individual host stanzas ldapserver, with similar settings, al- Listing 1 and then start the hosts and as required. One project file can include though it is given more memory. Note virtual network: multiple superclasses, and the super- that the second host uses a different classes can even be nested. template as its initial # mln build -f ldaptest.mln Substanzas, such as network interface image and has a static IP address. The # mln start -p ldaptest definitions, which appear both within final stanza defines a virtual switch to superclass and host definitions, are which both virtual machines are con- The build subcommand instantiates the merged. Directives that appear in the nected; this switch allows them to com- entities in the specified project file, and host definition have precedence over municate with one another. the start subcommand boots the virtual directives within the superclass. Most of the settings are straightfor- machines and activates the virtual net- ward, but the template specifications work. ;`jki`Ylk\[Gifa\ZkjXe[ deserve special mention. Templates are Typically, the build subcommand is `m\D`^iXk`fe pre-fabricated VM filesystems. Each time used only once – initially to create the So far, all the examples have been lo- you build a new virtual machine, you project – whereas the start subcommand cated on a single server. However, MLN     specify a template to use as its base. The and its sibling stop are used to activate supports distributed projects in which virtual machine will be individualized and deactivate the project hosts and net- virtual machines are located on one or by MLN during the build process. Some work. more remote hosts. The host and switch people like to use very generic templates By including the -h option, you can definitions in a distributed project re-  $$ ' $$ for all their virtual machines and cus- apply the command to a single host quire an additional service_host direc- % & % & tomize them later, whereas others create within a project: tive, which specifies the server location: very specialized templates, like a fully functional LAMP server or a copy of a # mln stop -p ldaptest -h 5 host rem-vm { typical employee desktop. ldapserver vmware The definition for a VMware Server template rhel5.vmdk virtual machine differs only slightly from Table 1 summarizes the most important ...  the host definition in Listing 1. The vm- subcommands related to project man- service_host bigvmbox ware keyword is used in the VM host agement. }    &   definition stanzas instead of xen, and a   !  " #  VMware format template is used instead Listing 2: Working with a Superclass   & . / of a filesystem image. The switch defini- 01 global { project italy } 12 }    ()  !    tion stanza also requires the vmware 02 13  !             keyword. 03 superclass basic { 14 host uno { superclass basic }  % *+   ,   ! " # Valid host and switch names are case 04 vmware 15 host due { superclass basic }   0 ,-   $%&&'   sensitive and can contain lowercase and 05 template rhel5.vmdk 16 host tre { uppercase letters, numbers, and hy-    ""# 06 memory 128M 17 superclass basic    ()  !  phens.  !   07 network eth0 { 18 memory 256MB  % *+   , DXeX^`e^MDjn`k_k_\dce 08 address dhcp 19 } :fddXe[ 09 switch lan 20    ,- 10 } 21 switch lan { vmware }    ! Once you have defined a project in a configuration file, you can use the mln 11 service_host milano                 24 ISSUE 98 JANUARY 2009

022-030_mln.indd 24 13.11.2008 17:57:11 Uhr MLN :FM

Superclasses can include the service_host To move virtual machines from one noted by bigvmbox to the one denoted directive as well (as shown in Listing 2). server to another, all you need do is newvmbox: For servers that will host virtual ma- modify the service_host settings within chines, the MLN daemon must be run- the project file to reflect the desired new # vi italy.mln ning. In addition, the MLN daemon’s destination and then rebuild the project Change bigvmbox to newvmbox configuration file, /etc/mln/mln.conf, with the mln upgrade command. For ex- # mln upgrade -f italy.mnl must define the server to be referenced ample, the following commands modify in project files and must grant access to and rebuild the italy project shown in Note that this command can run even remote systems from which projects will Listing 2, resulting in the virtual when the virtual machines within the be managed. For example, the MLN machines migrating from the server de- project are currently running. daemon configuration file on the target for the rem-vm virtual ma- Listing 3: MLN with a SAN chine would contain directives like the 01 host odysseus { 01 host odysseus {

following: 02 xen 02 xen

03 lvm 03 lvm

service_host bigvmbox 04 lvm_vg volume-group-name 04 lvm_vg volume-group-name daemon_allow 192.168.10.* 05 service_host scylla 05 service_host scylla

06 ... 06 ... In this case, the daemon_allow setting 07 } 07 } permits access only from systems on the 08 08 local subnet. 09 host ulysses { 09 host ulysses { The argument to service_host can be either a resolvable hostname or an IP 10 xen 10 xen address. 11 filepath /nfs-mount-point 11 filepath /nfs-mount-point With MLN, it is a very small step from 12 service_host charybdis 12 service_host charybdis distributed projects to live VM migration 13 ... 13 ... in Xen environments. 14 } 14 }    

 $$ ' $$ % & % &

    &     !  " #    & . /    ()  !     !              % *+   ,   ! " #   0 ,-   $%&&'      ""#    ()  !   !    % *+   ,    ,-    !

               

022-030_mln.indd 25 13.11.2008 17:57:11 Uhr :FM

Listing 4: Use of the for lab work, with each group using a Listing 5 shows the main file, oscourse. network of one Linux and one Windows mln. (Note that this file has no global #include Statement virtual machine to solve the course exer- stanza.) 01 global { cises. The Linux virtual machine will The hvm directive in the definition of 02 $gnum = 03 have two network cards and share its the virtual machine running Windows 03 $userpasswd = unique-value-1 connection to the LAN with the Win- XP indicates that it uses Xen full (hard- 04 $rpasswd = unique-value-2 dows virtual machine. ware) virtualization. The Linux virtual 05 $vncpasswd = unique-value-3 The challenge for the instructor is to machine uses Xen paravirtualization. As 06 project = os$[gnum] create 60 of those mini-networks such, networking configuration is inter- 07 } quickly, each with an individual public nal to this virtual machine (as IP address 08 IP address and password. 10.0.0.2), so it is not configured by MLN 09 #include oscourse.mln The main philosophy for this task is and there is no network substanza. This “design once, deploy often.” All 60 mini- configuration file also introduces the Live migration also requires a few networks must be as consistent as possi- root_passwd keyword, as well as two preparatory steps: ble, and the system also must be easy to keywords that set up the VNC display. ฀ ฀฀฀฀฀฀฀฀ reconfigure, so if you later decide that With a shell or script, you can daemon. Access control for the dae- the Windows virtual machines need loop over the various project files with mon must grant access to the other more memory, you can modify all of the mln create, the mln update, or both servers involved in the migration. them easily. Putting all of the virtual ma- commands if necessary, as in when a ฀ ฀฀฀฀฀฀ chines into a single project might seem change occurs to a customized template storage to hold images that will be ac- like an easy way to accomplish this, but file used by the Linux virtual machine. cessible to all relevant servers. The a single project would make it more dif- Also, you can run mln commands manu- best solution is usually a storage area ficult to manage one group’s virtual ma- ally. network (SAN) controlled by the logi- chines separately. cal volume manager (LVM), but you A better solution is to use one project ;\]`e`e^M`iklXc:cljk\ij can also use a shared NFS directory per student group. However, it will not Virtualization is also an effective way for testing, especially when perfor- be necessary to write 60 complete proj- of using a cluster of physical machines. mance is not a consideration. The lo- ect definitions. The #include statement A cluster can essentially act as a server cation is specified in the host defini- lets you compress as much information farm for virtualization; in fact, the stu- tions in the project (see Listing 3). as possible, so each project file only con- dent networks described previously In addition, with a SAN, the /etc/mln/ tains the information unique to that could run on the various nodes in a clus- mln.conf file on all relevant servers must project. For example, you can create 60 ter. The physical cluster can serve as contain the san_path directive. The argu- small project files and let them all point physical hardware for one or more vir- ment for san_path specifies the location to the same main configuration file (see tual clusters. Listing 6 shows a project of the SAN, which is either the volume Listing 4). file that automatically creates a virtual group name (like lvm_vg within the host The items beginning with a dollar sign cluster of 36 nodes. definition) or the local mount point for are variable definitions, which will be This project uses the autoenum plugin the SAN. used within the main configuration file. (included with MLN) to automatically Xen must be configured to allow mi- gration on all relevant servers via the fol- Listing 5: oscourse.min lowing entries in the Xen daemon’s con- 01 superclass common { 19 template os_linux_1.ext3

figuration file /etc/xen/xend-config.sxp: 02 xen 20 memory 128M

03 lvm 21 network eth0 {

(xend-address '') 04 memory 256M 22 address 10.0.0.1

(xend-relocation-hosts-allow 5 05 network eth0 { 23 }

'^localhost$ ^.5 06 switch lan 24 network eth1 {

*\\.ahania\\.com$') 07 netmask 255.255.255.0 25 address dhcp

08 } 26 }

The argument in the second directive is 09 users { 27 }

a single-quoted list of regular expres- 10 osuser$[gnum] $userpasswd 28 sions specifying allowed hosts. In this 11 } 29 host win { case, we specify the local host and hosts 12 root_passwd $rpasswd 30 superclass common anywhere within the ahania.com do- 13 } 31 hvm main. 14 32 template winXP.template Fe\@j

26 ISSUE 98 JANUARY 2009

022-030_mln.indd 26 13.11.2008 17:57:12 Uhr Anzeige kommt separat

022-030_mln.indd 27 13.11.2008 17:57:12 Uhr :FM

Listing 6: Creating a Virtual Cluster Listing 7: Plugins 01 global { 11 } 01 global {

02 project bioinfo 12 } 02 project webserv

03 autoenum { Plugin to auto assign 13 $gateway_ip = 192.168.33.211 03 apache {

IP and service host. 14 } 04 doc_root /var/www

04 superclass cluster-node 15 05 }

05 numhosts 36 Number of VMs to 16 superclass cluster-node { 06 sshkey { create. 17 template ubuntu_mpi.ext3 07 /home/nagios/.ssh/ 06 address auto Assign IP pubkey.outgoing 18 memory 312M addresses ... 08 } 19 free_space 1G 07 addresses_begin 150 09 } (Required free space in the (starting with this host number ...) storage filesystem.) 10 08 net 128.39.73.0 ...on this 20 network eth0 { 11 host web14 { subnet. 21 gateway $gateway_ip 12 ... 09 service_hosts { Service hosts (Default gateway setting.) 13 apache { to use for VMs ... 22 } 14 max_connections 300

10 #include /bio/servers.txt 23 } 15 } (...stored in an external file.) 16 }

generate virtual machines with succes- configuration task: in this case, assign- and the maximum number of simultane- sive IP addresses and service hosts taken ing sequential IP addresses and service ous connections (once again, the direc- from the list specified in the plugin’s hosts. tives in the global stanza and the host service_hosts substanza. The MLN configuration language has stanza are merged). Listing 6 creates 36 hosts on the basis an open architecture that simplifies the The sshkey plugin will copy the public of the settings in the cluster-node super- task of writing plugins. MLN plugins are key from the local nagios user account to class, and the list of service hosts comes Perl routines that are invoked within the the .ssh/authorized_keys file in the home from an external file referenced with the global stanza of a project file. directory of the user nagios in the virtual #include directive. Listing 7 shows two more examples of machine web14 (creating the file and di- The superclass definition also intro- plugins. One (apache) sets various con- rectory as needed). duces two more configuration file entries figuration parameters for the Apache that specify the required amount of free web server within a virtual machine. An- :fe]`^li`e^DCE;\]Xlckj space within the filesystem in which the other (sshkey) installs the specified SSH Within the main configuration file, /etc/ VM image will be stored (another way of key file(s) into the designated user ac- mln/mln.conf, you can set many MLN specifying its size) and the default gate- count. This technique is very practical defaults. The installed version of this file way (router) IP for a network interface. because it allows you to build virtual contains extensive comments document- machines that can be automatically ac- ing the available entries. Some of the Gcl^`ej]fiJg\Z`Xc$Gligfj\ cessed from other systems, such as by a most useful and important settings are :fe]`^liXk`feFgk`fej monitoring tool like Nagios. shown in Table 2. The preceding example illustrated the The apache plugin will cause MLN to Note that the -P, -T, and -F options to use of a plugin designed for a specific set the Apache document root location the mln command, which override de-

Table 2: MLN Configuration File Entries Directive Use Default projects /path Project files directory (override with mln’s -P option) /opt/mln/projects/$USER templates /path Template directory (override with mln’s -T option) /opt/mln/templates default_template file Default VM template None default_memory nnnM Default VM memory size 64MB default_size Default VM filesystem size 250MB service_host name-or-IP Define server as a location for VMs None daemon_allow ip-address-pattern Specify remote access control None daemon_listen_address ip Network interfaces on which to listen for remote requests Listen on all interfaces daemon_status_query host Include host in mln daemon_status command output None lvm_vg vg-name Volume group for VM volumes None san_path vg-name SAN volume group None san_path /mount-point SAN local mount point None files /path Location for files to be copied to VM with files directive /opt/mln/files/$USER (override with mln’s -F option)

28 ISSUE 98 JANUARY 2009

022-030_mln.indd 28 13.11.2008 17:57:12 Uhr Anzeige kommt separat

022-030_mln.indd 29 13.11.2008 17:57:12 Uhr :FM

fault directory locations, appear before # mln daemon -D /var/run/mln.pid Apply the usual system hardening tech- the desired subcommand (for example, niques to virtual machines and, espe- mln -P /mln/projects start ldaptest). The following command will display the cially, to the physical servers that host status of the MLN daemon on all hosts them. :fekifcc`e^k_\DCE specified in the daemon_status_query ;X\dfe lines in /etc/mln/mln.conf: 9XZblgj The MLN daemon, mlnd, is started at Also, think about backups. Either you boot time via the file /etc/init.d/mlnd # mln daemon_status can choose to back up virtual machines (linked to the appropriate rcn.d direc- in the usual manner, within your enter- tory). When you set up MLN to manage vir- prise backup scheme, or back up virtual Also, you can run this script manually tual machines and networks, it is a good machines at the virtual level. with the usual start, stop, and restart ar- idea to use LVM for flexible VM storage, guments. including expansion capabilities. :feZclj`fej To start the daemon manually, use the Anticipate resource use before deploy- Virtualization products are everywhere. following command: ing virtual machines, and monitor it on What makes MLN so different is its abil- an ongoing basis with software like ity to work in a very wide range of de- INFO Munin or Cacti. ployments. MLN works well for virtual- [1] VMware Server: To limit remote VM management ization beginners because it removes the vmware. com/ products/ server and live migration, use access control gritty details of VM configuration files, and don’t forget security. Virtual ma- and, at the same time, you can use MLN [2] Xen: www. xen. org chines are not inherently more secure to deploy far more complex scenarios [3] User-Mode Linux: than physical systems, contrary to many than most vendors offer. p user-mode-linux. . net vendor claims. In fact, in the absence of [4] MLN at SourceForge: precautions, they can even be less secure mln. sourceforge. net because they offer new forms of attack.

Creating Templates for Virtual Machines Under both Xen and various free and com- template); eliminating specific network UÊ Ê7ˆÌ ÊÌ iÊdd command, you can harvest mercial flavors of VMware, creating a vir- configurations; and removing device an image from an installed operating tual machine starts with making an empty names, /etc/fstab entries, and so on. system, copying the entire partition into virtual machine. On the first boot, an oper- UÊ Ê œÜ˜œ>`Ê8i˜ÊÌi“«>ÌiÃÊvÀœ“ʘÌiÀ˜iÌÊ an image file. ating system is installed just as it would be sites (e.g., jailtime. org). Note that such Once you have a template, you can modify on physical hardware, often from the same images typically correspond to paravirtu- it easily by mounting it in loopback mode, installation media, or, more recently, from alized virtual machines, meaning that the as in the following examples. the corresponding CD/ DVD image files. included operating system knows that it For Xen images: Once you have a virtual machine with an is running in a Xen virtual environment # mount -o loop guest.img 5 installed virtual system, its image could be and contains special features for efficient copied to create new virtual machines, al- execution. This also means these sys- /somewhere though the copies might require customi- tems do not contain bootable kernels but and VMware images: zation. rather rely on the kernel and initial ram- # mount -o loop,offset=5

MLN is designed for complex virtualization disk on the virtualization server for boot- 32256 guest-flat.vmdk 5 tasks. As such, it does not install operating ing. /somewhere systems from standard media or ISO im- UÊ Ê œÜ˜œ>`Ê6 ʈ“>}iÃÊvœÀÊ6 Ü>ÀiÊ If the image is a Linux operating system, ages; rather, it relies on installed operating Server from Internet sites (e.g., virtualap- you can chroot /somewhere to access the system image files – what it calls templates pliances. net, jumpbox. com). These im- image. This allows you to use the VM oper- – as the basis for instantiating virtual ages are typically special-purpose virtual ating system’s own tools to make modifica- machines (relying on the ability of VMware machines ready to run a specific applica- tions, something that is especially helpful and Xen to create fully installed virtual tion or fulfill a specific purpose. They are for ensuring proper functioning when you machines as well as empty ones). normal VMware VM image files (usually >``ÊÜvÌÜ>Ài°ÊvÊÌ iʈ“>}iʈÃÊ>Ê7ˆ˜`œÜÃÊ A few of the options for creating MLN -flat.vmdk preallocated disk image files). operating system, you will have to use ex- templates are as follows: Note that you can convert VMware im- ternal tools to modify items within it. ages for use with Xen with the qemu-img UÊ Ê œ«ÞÊi݈Ã̈˜}ÊۈÀÌÕ>Ê“>V ˆ˜iÃÊ>˜`ÊÕÃiÊ Once prepared, templates must be regis- convert command. them as templates. Before copying, how- tered with the MLN daemon before you ever, it is a good idea to boot the virtual UÊ Ê/ iÊ8i˜Ê/œœÃÊ«>VŽ>}iÊ«ÀœÛˆ`iÃÊ>˜Êi>ÃÞÊ can use them to build virtual machines: machine and make it as generic as possi- way to create templates for Xen para- # mln register_template -t 5 ble to allow for different deployment virtualized virtual machines from Debian, contexts and scenarios. This includes re- Ubuntu, Fedora, and other Linux distri- file-system-image-file moving local users and groups (because butions. See the xen-create-image Also, you can use rt as an abbreviation for MLN can configure these as required for command for easy creation and customi- the register_template subcommand. each virtual machine created from the zation.

30 ISSUE 98 JANUARY 2009

022-030_mln.indd 30 13.11.2008 17:57:12 Uhr