Criteria for Evaluation of Open Source Computing Solutions

Ivan Voras, Branko Mihaljević, and Marin Orlić Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia [email protected], [email protected], [email protected]

Abstract. promises on- IT departments to elastically deploy resources demand scalability and flexibility for the from a centrally managed pool. Open source enterprise environment and open source solutions, with lower initial investments, are a products have a large presence in this area. We viable alternative to commercial vendors, have devised an elaborate set of criteria for especially as business conditions deteriorate and evaluation and comparison of open source IaaS the economy constrains. cloud computing solutions which can be used for ranking and choosing between the available 3. Criteria products. We explain each criteria group, and give both the rationale and the estimated impact Evaluation of complex products requires an of the criteria. We also briefly introduce a elaborate and complete set of evaluation criteria number of most common open source cloud as a baseline for comparison. Even though the computing IaaS solutions to be evaluated using primary focus of our evaluation is on open the explained criteria. source IaaS products, we found the opportunity to objectively compare such products to those Keywords. Cloud Computing, Open Source from closed/commercial domain to be very Cloud, Infrastructure , Enterprise, valuable. Furthermore, some of the criteria can Evaluation be directly applied to or easily extended to apply to other cloud computing models (SaaS and 1. Introduction PaaS). This is especially true for high-level criteria such as management, security and service Cloud computing is a recent combination of levels. resource management and provisioning We have devised a set of about 100 criteria technologies build on the concepts of elasticity, grouped into six main categories: storage, ease of use and mass deployment. Cloud , network, management, security computing implementations are centered around and vendor support. These categories, targeting three service models [3]: Software-as-a-Service features interesting for enterprise deployment, (SaaS), offering its users a ready-made can further be divided into basic IaaS properties application with little or no control or commonly described in literature (storage, customizability of its innards or underlying virtualization, network and management) [3,4], infrastructure; Platform-as-a-Service (PaaS), and additional generic features (security, vendor offering developers a development and support). Assigning weights to areas of special deployment platform with a rich set of , interest enables high-level decisions. programming languages and tools used to create customized applications for end-users, with 3.1. Storage criteria limited control over the entire environment; and Infrastructure-as-a-Service (IaaS), offering full Virtual machine storage management is vital control of virtualized low-level infrastructure for cloud computing architectures as it must be used to build customized applications or higher- powerful and flexible enough to allow the level products. implementation of core cloud goals: flexibility, scaling and ease of use. Storage criteria cover the 2. Motivation main technology groups in which the storage may be implemented: direct-attached storage Most important goals for enterprise IT (DAS), storage area networks (SAN) and departments are agility and cost control. Cloud network-attached storage (NAS). Backup computing fulfills these requirements and allows technologies are covered as a related feature. These main technology groups include a list (IEEE 802.1p), and the Integration group of criteria which detailed support for specific describes the support for IPv6 and virtual private technologies, e.g. the NAS group considers the networks (VPNs) for access and management. support for NFS and CIFS network file systems, and the DAS group the supported file system and 3.4. Management criteria block-based replication. Similarly, the backup group contains criteria for backup over local Management category describes features (tape and DAS) and remote (NAS) storage. directly related to the way evaluated products Organizations can thus choose products manage their clouds. The criteria within this depending on their internal storage strategies category are grouped into Integration, (e.g. if they invested in SAN storage) or even Accounting, Reporting and Recovery. plan future strategies based on selected products. The Integration group focuses on software integration between the host and the guest 3.2. Virtualization criteria operating systems, support for specific and mass management of both, and available options for Virtualization technologies are at the core of integration with third party software via exposed cloud computing idea of flexible and scalable and consumed APIs. The Accounting group computing resources. Open source products are deals with data collecting and billing resource ahead of the commercial/closed products in the usage to cloud users. Along with billing, we number of supported virtualization technologies, included criteria for reporting on current system often supporting several different open source, status and historical logs in the Reporting and to some extent even commercial . category, while the Recovery category contains Virtualization criteria are grouped as follows: criteria describing the product's automatic VM Type, VM Technology, Quotas, recovery abilities, high availability features and Prioritization, Migration, Cloning, Hot administrative alerting about exceptional configuration and Provisioning. These groups operational conditions. address the virtualization type (full, Easy and comprehensive management is paravirtualized or containers), specific crucial for successfully implementing clouds, virtualization technology (, KVM, OpenVZ and we believe that the proposed criteria can be etc.), quotas and prioritizing of CPU, network, used to asses and compare these features across memory and disk, live and offline migration and varied products. cloning, hot reconfiguration and provisioning, respectively. 3.5. Security criteria As the support for certain feature-sets differs among supported virtualization types, we Security features are important for all decided that the evaluation process should computer system deployments and clouds are no consider the feature superset, with additional exception. Security criteria groups are commentary describing which features are Integration, Permissions, Reporting, Support, available in which virtualization products. Encryption, Certification and Auditing. The Integration group describes the product's 3.3. Network criteria ability to integrate with Active Directory and generic LDAP servers for user authentication, Network is important both as means of using while the Permission group deals with the services deployed inside the virtual machines authorization levels for specific resource (e.g. and for managing the entire cloud environment. VM server or storage servers) or virtual machine We divided low-level network support into four access. Reporting covers the security report types groups: VLAN, Firewall, Performance and available to cloud administrators, and Auditing Integration. covers the types of events which can be audited. The VLAN group describes the support for Certification details third-party product VLAN packet tagging (IEEE 802.1Q) for evaluations for compliance with security network management and isolation, the Firewall standards. Finally, Encryption contains the group describes the level of support for network criteria describing support for encrypted virtual filtering in the form of stateful packet inspection, disk storage and secure management access. the Performance group covers Ethernet QoS Security criteria are more important in public classical grades 1-5 used in education systems. and hybrid clouds than in private clouds, but a Alongside these grades, each criteria in the comprehensive list of criteria allows comparison evaluation sheet can contain a note with detailed and evaluation to be used in making decision information about the relation of specific criteria about overall security architecture. to the product, if required.

3.6. Vendor support criteria 5. Open source cloud computing solutions

Large open source projects are often This chapter presents (alphabetically) a supported or even created by commercial selected set of open source cloud computing entities. Support for such projects is often solutions. divided between a community, which is almost exclusively oriented toward open source 5.1. Abiquo Community Edition development, and a vendor in a strict sense, which may offer commercial support or even Abiquo1 presents cloud management solutions commercial versions of the products. Our criteria in the form of commercial Abiquo Enterprise cover both potential sides of a project and Edition and as open source Abiquo Community contain the following groups: Community, Edition, differing mostly in resource limits and Certification, Documentation, Vendor, Support management features. It provides network, and Overall. storage and workload management, multiple- The Community group deals with support image public, shared and private libraries, and channels freely available to everyone via the multi-tenancy hierarchical user management. open source community gathered around the Open source version supports Appliances project (e.g. mailing lists, forums), as well as the repository storage for virtual images in NFS quality of these channels. Certification covers the shared folders, while commercial version possibility of third-party auditing for compliance additionally supports virtual block devices. This with laws, regulations and business rules. The solution consists of several components: Abiquo documentation is evaluated for its completeness. Core, which contains the business logic and The Vendor group concerns itself with direct resides on Abiquo server, BPM that executes vendor support for the products (if any) and complex asynchronous tasks, and Appliance includes the possibility of signing a SLA contract Manager for image library management. Remote with the vendor. The Support group covers services provide virtual and physical resources general methods of customer relations and management with overall system monitoring, and includes criteria for public issue tracking (public integration through standardized REST API. disclosure of bugs and security vulnerabilities), Support for Amazon EC2 enables combining proactive updates and CRM-like approaches to public and private services on various servers customer relations. Finally, the Overall group types e.g. Java EE, database, cloud node, storage contains criteria describing the completeness of etc. It can be deployed on numerous OSs open source product versions and estimates about including most of distributions (Ubuntu, the project's openness, past track record and Debian, CentOS, RHEL, OpenSUSE, and future viability. Fedora), Windows, OpenSolaris, and OS X. Vendor and community support are of Furthermore, it supports various virtualization critical importance in enterprise deployments and standards including Xen, KVM, ESX and ESXi, our goal in criteria design was to cover important Hyper-V, XenServer, and VirtualBox. Besides aspects of these support types. these benefits, it implements a powerful web management with powerful features such as 4. Usage and grading with the criteria drag-and-drop service deployment.

Our method for grading products based on 5.2. Community Cloud (ECC) the presented criteria includes a numeric aspect and a descriptive aspect. The numeric grade is in One of the most adopted cloud computing the range of 0 to 5 where 0 designates absolutely architectures is the scalable IaaS framework no support for the criteria within the product and Eucalyptus with academic roots at the University the other grades have meaning similar to the 1 Abiquo, http://www.abiquo.com of California, Santa Barbara. Besides generic agent skeleton should be extended. Main commercial Eucalyptus Enterprise Edition, there features include support for cloud ontology, user- is an open source Eucalyptus Community Cloud centric service level agreements, and dynamic (ECC)2 [1], with hierarchical architecture of negotiation of resources. The work on the project flexible and highly modular system components is ongoing on usage patterns, cloud ontology, with well-defined interfaces. Components are and description of API, but currently software exposed in the form of stand-alone, standardized deliverables are not available. and easily replaceable Web services. Additionally, ECC utilizes standardized 5.4. language-independent communication and virtual network overlay that isolates user network Nimbus4, “cloud computing for science”, traffic and utilizes cluster transparency. represents a set of open source software cloud Interoperability concerns are addressed through computing components with emphasis on the implementation of AWS API and emulation of needs of scientific community. Nimbus consists SOAP and REST interfaces, allowing seamless of several components. Standalone virtual integration with existing Amazon EC2 and S3 machine and cloud management Workspace public cloud services, thus enabling hybrid service supports WSRF frontend and Amazon clouds. Operations and data structures of services EC2 via SOAP and REST interfaces. Cumulus is are described in the form of WSDL documents, an open source implementation of S3 REST API and optionally secured using WS-Security which presents scalable quota-based storage for policies. Cloud Controller component provides multiple cloud configurations. Context Broker main user interface, which interacts with a set of service enables coordination of large-scale resource, data and interface services used for virtual clusters and management of common managing resources. Storage Controller (Walrus) cloud configuration in secure context across is a stream and storage service, which accesses resources provisioned from potentially multiple and stores user data and virtual machine images. clouds or distributed providers with hybrid Cluster Controller collects information from host clouds. It supports Xen and KVM virtualization, nodes, schedules run requests, and manages and virtual machine schedulers OGE and PBS. virtual network overlay. Node Controller queries Besides, it can be combined with OpenNebula virtualization and OS on the host node, thus . Currently Nimbus is controlling resources availability, authorization, mostly used for scientific purposes, but there is a and execution. ECC currently supports Xen and possible pathway for business utilization. KVM virtualizations and can be deployed on all major Linux distributions (Ubuntu, CentOS, 5.5. OpenNebula Debian, RHEL, openSUSE, SLES etc.), but the preferred installation is Ubuntu. Open source software toolkit for cloud computing OpenNebula5 is used for management 5.3. mOSAIC of private and public clouds. It orchestrates existing systems and services, but does not Consortium of several European academic contain virtualization, network, storage or and industry partners recently created the security technologies. Some of the main design mOSAIC project3, a joint initiative for creation principles of OpenNebula are fully opened of open source portable platform for cloud architecture with standardized interfaces, services based on platform- and language- integration, interoperability, portability, and independent API for resources and usage scalability. It can be used for virtual patterns. This promising platform is intended for infrastructure management of private clouds in developing multi-cloud oriented applications, data centers or clusters, or hybrid scalable cloud which interfaces extend the existing platform- environments. Cloud-bursting is supported with dependent APIs with composite features based Amazon EC2, providing simultaneous access to on usage patterns. For representation of various multiple clouds and cloud federation. Some of user’s resources and functionalities an adaptable the features include secure user management of virtual images, virtual machines, virtual 2 Eucalyptus Community Edition, http://open.eucalyptus.com 4 Nimbus, http://www.nimbusproject.org 3 mOSAIC, http:// www.mosaic-cloud.eu 5 OpenNebula, http:// www.opennebula.org networks, and storage, using secure OpenStack7, with a mission to produce the authentication, multi-tenancy and quota ubiquitous cloud for public management. Abstraction from an infrastructure and private clouds, massively scalable regardless and modular approach supports standardization of size, and simple to implement. There are and interoperability with most common interrelated projects named Compute and virtualization standards (Xen, VMware, and Storage, which deliver three products. First KVM), interfaces (Amazon AWS, VMware product, OpenStack Compute is used for vCloud, and OGF OCCI), and APIs (Java, Ruby deployment, management and provisioning on and XMLRPC). A welcome addition to the large-scale virtual private servers. This IaaS toolkit is a set of tools, extensions and plug-ins platform orchestrator is used as a cloud fabric named OpenNebula EcoSystem, which enhances controller for management of various resources, integration with existing virtualization and networking, and security options. It does not clouds products, services and management tools. contain any virtualization, but interacts with OpenNebula is currently highly adopted in underlying virtualization mechanisms on hosts scientific institutions (e.g. CERN), telecom and over web-based API. It extends standard cloud hosting providers. features (e.g. of Amazon EC2) with group settings under virtual “projects”, thus supporting 5.6. openQRM multiple users, keys, volumes, instances, images, and VLANs. openQRM6 is advertised as a data-center OpenStack Compute supports KVM, Xen management platform. It follows the modular UML, MS Hyper-V and QEMU virtualizations. design and has no functionality itself, but instead OpenStack Compute can be deployed and runs focuses on abstracting storage and resources on Ubuntu, but tests are performed on CentOS (computing resources, running virtual machines, and RHEL. Second product named OpenStack VM images and other objects), moving features Object Storage is used as a redundant and to plugins which use the exposed services. This scalable storage system on clusters of commodity aims to make the whole system more stable and servers, providing data replication, failover and easier to manage. openQRM can be installed on cluster integrity. Third component is named a variety of Linux operating systems: Debian, OpenStack Imaging Service and enables retrieval Ubuntu, SuSE, CentOS, and Fedora. To achieve of virtual machine images. OpenStack relies on its goal of managed virtualized data-center, some external tools e.g. Eucalyptus’s euca2ools openQRM provides server and storage for image management and OpenStack management, high-availability, real-time Dashboard, a reference implementation of a web- monitoring and virtual machine deployment and based management console for Compute. Images provisioning services, among others. can be stored in local file system, OpenStack openQRM plugins provide a wide range of Object Storage or . OpenStack’s open services, from integrated storage management design process already delivered afore- (supporting direct-attached storage, and various mentioned products in a Bexar release which will SAN and NAS variants), abstraction of be soon replaced with next release called Cactus. virtualization (Xen, KVM, Linux-VServer, VMware Server and ESX VMs), migration from 5.8. Red Hat Cloud Foundations physical to virtual machines (P2V, V2P and V2V of different VM type), high-availability (with Red Hat Cloud Foundations (RHCF) is failover from physical to virtual machines, and primarily presented as a set of products for cloud virtual to virtual failover between machines of computing, virtualization, scheduling and same, or different type), and VM image application management, but it also includes templates or appliances. other Red Hat’s8 products e.g. Red Hat Enterprise Linux (RHEL) OS, middleware, and 5.7. OpenStack reference architectures, combined with documentation, training and consulting services. One of the upcoming and promising open RHCF suite of open source software provides source players is collaborative software project infrastructure for public and private cloud

7 OpenStack, http:// www.openstack.org 6 openQRM, http://www.openqrm.com 8 Red Hat, http://www.redhat.com solutions, and in current Edition One consists of EBS in conjunction with CLC provides usage of Red Hat Enterprise Virtualization (RHEV) used persistent block devices and point-in-time for end-to-end virtualizations, system volume snapshots. Furthermore, UEC provides management product Red Hat Network (RHN) security layers for authentication and Satellite, clustering solution Red Hat Cluster authorization, network isolation in system, static, Suite (RHCS), and Red Hat Enterprise MRG managed and managed-noVLAN modes, and which is a distributed computing platform machine instance isolation on based providing messaging, real-time and grid machine, networking, and OS levels. It is functionalities. RHEV consists of server obvious that UEC uses all of the advantages of virtualization system RHEV Manager (RHEV- underlying ECC, and ECC is at current time M) with storage management, high availability, mostly adopted in the form of UEC system. live migration, scheduler etc., and RHEV Hypervisor (RHEV-H), based on KVM 5.10. Other hypervisor and deployed standalone or on top of RHEL. RHN Satellite enables configuration Enomaly ECP Community Edition10 and its management, updates, provisioning and fork OpenECP11 was to be included in this monitoring, while RHCS supports load evaluation, but was omitted due to discontinuous balancing, service failover and high availability. work and legal issues. RHCF supports multiple hosts of RHEV-H hypervisors and/or KVM hypervisors on RHEL 6. Conclusion and future work with Windows and RHEL tenant VMs, and various MRG, RHEL, and JBoss applications. We believe the proposed set of criteria to be RHCF is intended for private clouds [2], but also very applicable to capturing the most important supports hybrid clouds where application features offered by open source products, and instances could be deployed in Amazon EC2, also that it can be easily adapted or directly used with dynamic balancing and scheduling using for evaluation of closed/commercial products. MRG Grid. Besides, Red Hat is intensely Having a common evaluation framework for a participating and investing in various open wide range of products creates a necessary source projects related to cloud computing, comparison baseline and allows IT professionals which serve as add-on features on its products. and management to make educated decisions.

5.9. Ubuntu Enterprise Cloud (UEC) 7. References

Canonical’s cloud computing services present [1] Nurmi D. et al. The Eucalyptus Open-source two open source cloud IaaS solutions: Ubuntu Cloud Computing System. In Proc. of 9th Server Edition (USE) with support for Amazon IEEE/ACM International Symposium on EC2 public cloud services, and Ubuntu Cluster Computing and the Grid, 2009. Enterprise Cloud (UEC)9, which comprises of [2] Red Hat, Inc. Red Hat Cloud Foundations USE with ECC architecture integrated with Reference Architecture: Private IaaS KVM hypervisor. UEC [5] infrastructure is Clouds. Red Hat Reference Architecture compatible with AWS API and can be used in Series, Version 1.1, April 2010. creation of private clouds. Additionally, UEC [3] Sriram, I., Khajeh-Hosseini, A. Research delivers images runnable on Xen hypervisor or Agenda in Cloud Technologies. LSCITS UEC/ECC/KVM. UEC exposes the same high- Technical Report, 2010. level components with the same functionalities [4] Staten, J. Is Cloud Computing Ready for the as ECC: Cloud Controller, Walrus Storage Enterprise?. Forrester Research report, Controller (WS3), Cluster Controller, and Node 2008. Controller, with an addition of Elastic Block [5] Wardley, S., Goyer E., Barcet N. Ubuntu Storage Controller (EBS). CLC comprises three Enterprise Cloud Architecture. Technical interfaces: standard SOAP-base API compatible White Paper, Canonical, 2009. with Amazons EC2, REST interface which uses euca2ools and Elastic Fox, and Web interface. 10 Enomaly ECP Community Edition, 9 Ubuntu Enterprise Cloud, http://src.enomaly.com http://www.ubuntu.cloud/private 11 OpenECP, http://www.openecp.org