A Multi-Site Virtual Cluster System for Wide Area Networks Takahiro Hirofuchi∗ Takeshi Yokoi∗ Tadashi Ebara∗† Yusuke Tanimura∗ Hirotaka Ogawa∗ Hidetomo Nakada∗ Yoshio Tanaka∗ Satoshi Sekiguchi∗ ∗National Institute of Advanced Industrial Science and Technology (AIST) †Mathematical Science Advanced Technology Laboratory Co., Ltd. Abstract cent virtualization technologies contribute to great im- provements in efficient resource usage, scalability and A virtual cluster is a promising technology for reduc- flexibility are limited by the total amount of physical ing management costs and improving capacity utiliza- resources at one site. For virtualization of high perfor- tion in datacenters and computer centers. However, re- mance computing environments, emerging virtual clus- cent cluster virtualization systems do not have the max- ter systems allow rapid deployment of customized clus- imum scalability and flexibility required, due to lim- ter systems. They basically reduce deployment costs and ited hardware resources at one site. Therefore, we are improve resource sharing efficiencies. However, exist- now developing an advanced cluster management sys- ing virtual cluster systems focus only on single-site com- tem for multi-site virtual clusters; which provides a vir- puter resources; users cannot create large-scale clusters tual cluster composed of distributed computer resources beyond single-site hardware resources and dynamically over wide area networks. This system has great advan- extend running clusters in a flexible manner, on demand. tages over other cluster management systems designed only for single-site resources; users can create a clus- We designed and implemented a multi-site virtual ter of virtual machines from local and remote physical cluster management system, which enables administra- clusters in a scalable manner, and dynamically change tors to create a virtual cluster composed of virtual ma- the number of cluster nodes on demand, seamlessly. In chines from distributed computer centers easily, and dy- our system, a multi-site cluster achieves a monolithic namically change its components for ever-changing de- system view of cluster nodes to enable existing appli- mands from users, making efficient use of surplus com- cations to be deployed quickly and managed flexibly, putational resources. just as in physical clusters. In this paper, we propose an advanced cluster virtualization mechanism composed In this paper, we propose two key mechanisms for of two key techniques. An L2 network extension of multi-site support of cluster virtualization: An L2 ex- virtual machine networks allows transparent deployment tension of virtual machine networks allows transparent over networks for distributed virtual cluster nodes, and virtual cluster deployment over WANs, and contributes a transparent package caching mechanism greatly opti- to the same usability as a single physical cluster, through mizes data transfers in virtual cluster deployment over unmodified application programs. In addition, a package network latencies. Experimental results show multi-site caching mechanism is an essential component for scal- virtual clusters have sufficient feasibility in WAN envi- able deployment of multi-site virtual clusters. It dramat- ronments and promise great scalability for a large-scale ically reduces backbone network traffic among sites and number of virtual nodes. alleviates network resource requirements to the lowest feasible levels. 1 Introduction Section 2 clarifies the concept of multi-site virtual clusters and Section 3 presents key mechanisms for Large-scale datacenters and computer centers need to multi-site support. Section 4 gives design details and maintain enough computer system resources for their Section 5 evaluates prototype implementation through customers in a cost-effective manner. However, it is experiments. Section 6 notes additional discussions, and difficult to make appropriate investments against unpre- Section 7 summarizes related work. Finally, we conclude dictable demands on processing powers. Although re- this paper in Section 8. 1 propriately hidden from the inside of the virtual cluster. Remote computer resources are integrated into a uniform computing environment over its underlying heterogene- ity, so that existing cluster applications can be applied to distributed environments with smaller efforts. A user ba- sically operates a virtual cluster in the same manner as a physical one, using unmodified programs designed for physical clusters. 3 Virtual Cluster Management System This section describes our virtual cluster management system. We recently decided to extend our single-site vir- tual cluster management system [10] for multi-site clus- ter virtualization. Its system architecture was carefully Figure 1: Multi-site virtual cluster concept designed to be composed of standard technologies, and its components were well-modularized for further exten- sion and customization. Therefore, multi-site extensions 2 Multi-Site Virtual Cluster can be seamlessly integrated with it without large modi- fications. The basic mechanism of cluster virtualization A multi-site virtual cluster is an advanced cluster virtu- is the same as that of multi-site cluster virtualization. alization concept for greater scalability and flexibility, as In our system, virtual clusters are based on three well as sufficient usability. Beyond physical hardware types of hardware virtualization; virtual machine mon- limitations at one site, it aims to integrate distributed itors (VMMs) for processor virtualization, logical vol- computational resources into a single cluster form by ex- ume manager (LVM) and iSCSI [16] for storage access, ploiting recent broad bandwidth backbone advances in and VLAN [6] for network isolation. A VMM works on WAN environments. A conceptual overview is illustrated each node of a physical cluster and hosts multiple vir- in Figure 1, and its advantages are summarized as fol- tual machines on it. Storage devices of a virtual machine lows. are attached via iSCSI from dedicated storage servers, in Scalability: A virtual cluster overlaying multiple sites which an LVM allocates storage space from large hard can be composed of a greater number of computational disks. A physical node hosts several virtual machines, nodes than can single-site hardware resources. This each of which is a member of different virtual clusters. breaks hardware resource limitations at a single site, and The private network of a virtual cluster is logically sep- integrates distributed computer resources as virtual clus- arated from the physical cluster’s networks by VLAN. ters. In recent years, broadband networks over 1Gbps Virtual clusters are completely isolated from each other, have been widely deployed between large-scale data cen- so that users can never use sniffer and tamper with net- ters and computer centers, and end-to-end reservations work packets of other clusters. of network resources are becoming possible in academic A user interface is provided through a web browser. and commercial backbone networks. These networks en- An administrator can specify the upper limit of physical able virtual clusters to be extended to other remote sites resources allocated for a virtual cluster, and users make in a scalable manner. reservations of virtual clusters with the number of virtual Flexibility: A virtual cluster can be dynamically ex- nodes, the amount of the memory and storage required, panded according to user requests, by adding more and and the start / end time. Users can also specify an operat- more virtual machines from remote sites. It is also pos- ing system and applications which are then automatically sible to shrink it by releasing virtual machines. The vir- installed in a reserved virtual cluster. tual nodes belonging to a virtual cluster can be substi- Our system is fully compatible with a widely-used tutable with other virtual machines of remote sites in cluster management framework, NPACI Rocks [14], in a fairly transparent manner; for instance, some virtual order to allow users to utilize its pre-packaged soft- nodes can be moved to other sites for hardware mainte- ware repository of various cluster applications. A self- nance or power saving of physical clusters. contained ISO image, called a Roll, includes a set of Usability: Even though a virtual cluster consists of RPM files for a cluster application, as well as its cluster- distributed virtual machines located at remote sites, it can aware configuration scripts. It should be noted that our be seen as a single system image ”cluster” for users and system is also distributed as a Roll, which means Rocks applications. Its underlying networking topology is ap- users can easily install virtual cluster management sys- 2 scalable, flexible, and usable as much as possible for available hardware resources. We consider this basic de- sign architecture is also applicable to multi-site virtual clusters. Its multi-site support should be achieved by a straightforward extension of the existing design; scala- bility, flexibility, and usability are improved in a transpar- ent manner for users, applications, and cluster manage- ment systems. Users should be able to utilize multi-site virtual clusters seamlessly by using existing programs for physical clusters. Figure 2: Virtual cluster management system for a single Second, multi-site virtual clusters should be allocated site rapidly, and their internal operating systems and appli- cations should be also installed and configured quickly, with the minimum