A Comparison and Critique of Eucalyptus, OpenNebula and Nimbus Peter Sempolinski and Douglas Thain University of Notre Dame Abstract as-a-service, accomplished by providing virtual ma- chines to users. Amazon’s EC2 cloud is arguably one Eucalyptus, OpenNebula and Nimbus are three ma- of the best examples of this paradigm. [2] Of course, jor open-source cloud-computing software platforms. this is not to minimize vast varieties of system config- The overall function of these systems is to manage the urations that can be referred to as “clouds”. We also provisioning of virtual machines for a cloud providing point out that Amazon is not the only player in cloud infrastructure-as-a-service. These various open-source computing market, as Sun (Sun Cloud), IBM (Blue projects provide an important alternative for those who Cloud), Microsoft (Azure), and many others have their do not wish to use a commercially provided cloud. own systems as well. [7] [13] [21] We provide a comparison and analysis of each of In the setting we are considering, a cloud is a group these systems. We begin with a short summary com- of machines configured in such a way that an end-user paring the current raw feature set of these projects. can request any number of virtual machines (VMs) of After that, we deepen our analysis by describing how a desired configuration. The cloud will spawn these these cloud management frameworks relate to the many VMs somewhere on the physical machines that it owns. other software components required to create a func- The word “cloud” in this context is meant to convey tioning cloud computing system. We also analyse the the semi-ethereal nature of these VMs. The end-user overall structure of each of these projects and address neither knows nor cares where exactly these VMs are how the differing features and implementations reflect physically located or the configuration of the underly- the different goals of each of these projects. Lastly, we ing hardware, so long as they can access their bank of discuss some of the common challenges that emerge in properly configured VMs. This kind of setup is ideal setting up any of these frameworks and suggest avenues for applications where a specific hardware configura- of further research and development. These include the tion is needed or users only occasionally need the high problem of fair scheduling in absence of money, evic- compute capacity. tion or preemption, the difficulties of network configu- However, commercial cloud services charge, by the ration, and the frequent lack of clean abstractions. hour, for CPU time. In some settings, such as a large organization with many users, it might be more cost effective for the organization to purchase hardware to 1 Introduction create its own private cloud. This is where open-source cloud frameworks such as Eucalyptus,[3] OpenNebula In recent years, due to the high demand for com- [5] and Nimbus [4] enter the picture. These software puting resources for processing jobs that require hun- products are designed to allow an organization to set dreds or even thousands of cores, different paradigms up a private group of machines as their own cloud. In have been developed for harnessing the computational this work, we analyze these three open-source cloud power of large groups of processors. Depending on the frameworks. We selected these three because they rep- computational setting and the problem being solved, resent three different points of interest in the design different approaches are being taken, including batch space of this particular type of open-source cloud. job systems such as SGE [11], cycle scavenging with In this analysis, we will first briefly address the pre- Condor [27], or virtual machine spawning with Ama- vious comparisons which have been made of these cloud zon’s EC2 cloud [2]. architectures. These previous works largely focused on The various definitions of cloud, grid or distributed the feature set of these clouds. We will briefly summa- computing are open to some debate. [29] We will con- rize that work. In this analysis, however, we wish to go fine ourselves to considering the idea of infrastructure- beyond feature comparisons. We will discuss how these 1 software frameworks act as managers that stand in the middle of a number of other software components. Un- 4 derstanding how these pieces fit together is critical for 5 understanding any cloud framework. Next, we will an- alyze how core decisions in the basic architecture and Cloud Controller: overall structure of Eucalyptus, OpenNebula and Nim- Eucalyptus, Open Nebula or bus impact the kind of settings and applications for Nimbus DHCP which each framework is most suitable. Third, we will identify several opportunities for improving these soft- 6 DNS ware stacks by identifying some of the challenges that Compute Nodes: 2 are common to all three. VM VM 2 Previous Work 3 hypervisor In looking for comparisons between Eucalyptus, OS 1 OpenNebula and Nimbus, we found several papers which, in some manner, gave comparisons of some as- Figure 1. Abstract Cloud Architecture pects of at least some of these systems. [17], [21], [12], The layout of the different parts of a generic cloud com- [8], [23], [9] Most of this previous work focused on the puting system. Note that the role of the cloud control feature set of these software projects. In Table 1, we software is to coordinate all of these pieces and to pro- summarize the basic feature comparisons from these vide sufficient abstraction so that a user can simply prior works. request VMs with minimal concern for how these com- The open source projects developing Nimbus, Open- ponents must interact. The numbers in this diagram Nebula and Eucalyptus are all rapidly expanding. Pre- correspond to the components as we list them in this vious work as recent as 2009 does not include some of section. the new features developed by these projects. For ex- ample, the Nimbus project has recently added support for KVM hypervisors, which is not noted in some of the hardware and operating systems that are on the vari- aforementioned citations. In such cases, we update the ous physical machines in the system. While these must table to as recently as possible. However, we we fully be set up properly for any software system, we make expect any table of features, including ours, to become especial note of the physical hardware and the base op- obsolete. This is especially the case with regard to sup- erating system for two reasons. First, if the processors port for standard API (such as EC2). Moreover, the of the physical nodes do not have the needed hardware standard API themselves often change, rendering prior extensions to run pure virtualization, this limits the support obsolete. Nevertheless, for features, Table 1 system to paravirtualization only. While such a cloud is the starting point of our comparison. system can be setup, it greatly limits both the speed of the VMs and the flexibility available in choosing soft- 3 The Cloud Computing Stack ware components for the system. Second, open-source frameworks, unlike commercial frameworks, must be Before delving into the structures that make Euca- flexible enough to work with many underlying systems. lyptus, OpenNebula and Nimbus unique, we will make (Whereas, commercial clouds only need their system to a quick overview of the entire cloud computing software work with the hardware that they have.) stack. These three cloud infrastructures only make The second component is the network. This includes up part of the software stack needed to make a cloud the DNS, DHCP and the subnet organization of the computing system work. In many ways, software such physical machines. It also includes virtual bridging of as Eucalyptus, OpenNebula and Nimbus stand in the the network that is required to give each VM a unique middle a large number of other components. Further- virtual MAC address. This bridging is accomplished more, there are often many options for these compo- using programs such as bridge-utils, iptables or nents. We include an overview of the software pieces ebtables. Moreover, in addition handling the physi- in Figure 1. cal nodes, DHCP and DNS processes must be config- In a generic open-source cloud computing system, ured, in concert with the cloud framework, to handle we can identify six basic components. First, we have the MAC and IP addresses of virtual nodes as well. 2 Table 1. Cloud Architectures Compared Eucalyptus OpenNebula Nimbus Disk Image Options Options set In private cloud, Depends on by admin most libvirt options left open. configuration Disk Image Storage Walrus, which A shared file system, Cumulus (recent imitates Amazons S3 by default NFS, or SCP update from GridFTP) Hypervisors Xen, KVM (VM Ware Xen, KVM, VMware Xen, KVM in non-open source) Unique Features User management VM migration Nimbus context web interface supported broker The actual setup of these components depends heavily The fifth component is the front-end for users. on whether one is using OpenNebula, Nimbus or Euca- There must be some interface for users to request vir- lyptus, as these systems can have different expectations tual machines, specify their parameters, and obtain for both physical and virtual network configuration. needed certificates and credentials in order to log into The third component is the virtual machine hyper- the created VMs. Some front-ends perform various visor, (also known as a Virtual Machine Monitor or types of scheduling by giving users an allotment of re- VMM). Popular VMMs include Xen and KVM, which sources which they are not allowed to exceed. Other are open-source, VirtualBox, which has an open-source front-ends implement standard API such as EC2.