Easing Scientific Computing and Federated Management in The
Total Page:16
File Type:pdf, Size:1020Kb
Easing Scientific Computing and Federated Management in the Cloud with OCCI Zdenekˇ Sustrˇ 1, Diego Scardaci2,3, Jirˇ´ı Sitera1, Boris Parak´ 1 and V´ıctor Mendez´ Munoz˜ 4 1Department of Distributed Computing, CESNET, Zikova 4, 160 00, Praha 6, Czech Republic 2European Grid Initiative, Science Park 140, 1098 XG Amsterdam, Netherlands 3INFN Sezione di Catania, Via S. Sofia 64, I-95123 Catania, Italy 4Computer Architecture & Operating Systems Department (CAOS), Universitat Autonoma` de Barcelona (UAB), Bellaterra, Spain Keywords: Federated Cloud, Cloud Standards, Cloud Interoperability, Cloud Solution Design Patterns, Cloud Application Architectures, Cloud Middleware Frameworks, Open Cloud Computing Interface. Abstract: One of the benefits of OCCI stems from simplifying the life of developers aiming to integrate multiple cloud managers. It provides them with a single protocol to abstract the differences between cloud service implemen- tations used on sites run by different providers. This comes particularly handy in federated clouds, such as the EGI Federated Cloud Platform, which bring together providers who run different cloud management plat- forms on their sites: most notably OpenNebula, OpenStack, or Synnefo. Thanks to the wealth of approaches and tools now available to developers of virtual resource management solutions, different paths may be cho- sen, ranging from a small-scale use of an existing command line client or single-user graphical interface, to libraries ready for integration with large workload management frameworks and job submission portals relied on by large science communities across Europe. From lone wolves in the long-tail of science to virtual or- ganizations counting thousands of users, OCCI simplifies their life through standardization, unification, and simplification. Hence cloud applications based on OCCI can focus on user specifications, saving cost and reaching a robust development life-cycle. To demonstrate this, the paper shows several EGI Federated Cloud experiences, demonstrating the possible approaches and design principles. 1 INTRODUCTION cloud – for instance tools only available for oper- ating systems not compatible with available grid OCCI, the Open Cloud Computing Interface (Nyren´ infrastructures, tools distributed by vendors in the et al., 2011; Nyren´ et al., 2011; Metsch and Ed- form of virtual machine images, etc. monds, 2011b; Metsch and Edmonds, 2011a), is a Lower the acceptance threshold for users trained standard developed by the Open Grid Forum to stan- • in using their existing environment without hav- dardize virtual resource management in a cloud site ing to change it or re-qualify for a different one. (OGF, 2016). It was released in 2011 and several im- plementations and real-world deployments followed. On reflection, this suits the need of users in the “long Among others, OCCI also became the standard of tail of science” who typically lack the resources to choice for the management of virtualized resources have their solutions tailored or adjusted to the grid in EGI’s Federated Cloud Platform (Wallom et al., and cannot invest in re-training for different tools or 2015). environments. Even relatively small-scale resources, The EGI Federated Cloud Platform has been con- available in the EGI Federated Cloud Platform at the ceived as an alternative way to access EGI’s consid- time of writing, can fit the needs of these “long-tail” erable computing resources, which are primarily ac- communities. cessed through grid middleware. Cloud-flavoured ser- Firstly, this article will introduce relevant imple- vices are meant to: mentations of OCCI, both on the server and client side Attract user communities relying on tools that are (Section 2). Secondly, Section 3 will introduce a scale • not easily ported to grid but can scale well in the of cloud usage patterns relying on OCCI for interop- 347 Šustr, Z., Scardaci, D., Sitera, J., Parák, B. and Muñoz, V. Easing Scientific Computing and Federated Management in the Cloud with OCCI. In Proceedings of the 6th International Conference on Cloud Computing and Services Science (CLOSER 2016) - Volume 2, pages 347-354 ISBN: 978-989-758-182-3 Copyright c 2016 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved OCCI 2016 - Special Session on Experiences with OCCI erability and abstraction, and also briefly expand on point (Fig. 1). There are several of such server-side other mechanisms users must ensure beyond the ba- frameworks whose resources can be managed through sic OCCI functionality for their solutions to be truly OCCI: productive. Finally, Section 4 will briefly outline new OpenStack, which comes with an OCCI interface developments, and Section 5 will sum up the topic of • (occi-os) as part of the stack and is soon due to experience with OCCI in the area of scientific com- receive an all-new implementation in the form puting. of OOI – the OpenStack OCCI Interface (Garc´ıa et al., 2016). Synnefo with its own OCCI implementation (GR- 2 OCCI IMPLEMENTATIONS • NET, 2016). OpenNebula, which is accessed through an OCCI OCCI was gradually gaining support throughout its • translation service – the rOCCI-server (Parak´ lifetime and there are currently multiple implementa- et al., 2014). tions available on the server side as well as the client side. At the time of writing, the version of OCCI Although there is currently no such site participat- supported by most implementations mentioned below ing in the Federated Cloud Platform, it is also techni- is 1.1. They will all follow their own, independent cally possible to use OCCI to manage resources in a schedules to adopt OCCI 1.2, which is due to come fourth type of cloud management framework: into effect shortly. Amazon Web Services (AWS), which can be ac- • cessed with OCCI through the rOCCI-server as 2.1 Server-side OCCI Support well (CESNET, 2016). Aside from the aforementioned, there are other Server-side OCCI implementations originate mostly cloud services which support OCCI or its subsets from service providers who wish to make their re- (Fogbow, 2016), or which are going to be made sources available in a standardized way. Where appli- OCCI-capable in the foreseeable future. For instance, cable, the OCCI interfaces are being pushed upstream the development work is underway to implement an to Cloud Management Framework developers. OCCI translation layer for Microsoft Azure. In the EGI Federated Cloud Platform, multiple For the sake of completeness, it is also worth sites make their cloud resources available to users. mentioning that there are OCCI implementations cur- Although different contributing sites employ differ- rently being used in existing solutions, but apparently ent Cloud Management Frameworks (CMFs), OCCI no longer maintained. They are: (plus an X.509-based authentication mechanism) is pyssf – the Service Sharing Facility (PySSF, the unifying factor, allowing users to access any site • 2016), which strives to provide building blocks in a uniform way, indeed, without actually know- for grid, cloud, cluster or HPC services. ing what flavor of CMF is installed on the end- occi-os – the original OpenStack OCCI interface • (occi os, 2016), which is gradually being replaced Clients Servers with OOI, already mentioned above. Custom client app. http NativeNative RubyJava occi application rjOCCI-api While the latter is scheduled to phase out in fa- rjOCCI-core vor of OOI, the former – it must be acknowledged – # Shell script has never been declared as discontinued, and devel- Application or script using rOCCI-cli http opment activity may resume; especially with the in- occi Synnefo command-line rOCCI-api executables troduction of OCCI 1.2 specification. rOCCI-core Custom client app. http OpenNebula Native Ruby occi 2.2 Client-side Tools and Libraries application rOCCI-api r application e v r rOCCI-core e Available s - MS Azure I C C Independent http O r OCCI is a text-based protocol. Although there are un- OCCI-compliant occi client official extensions to support OCCI transport through message queues (Limmer et al., 2014), the only of- Figure 1: Ecosystem of OCCI implementations envisioned ficially standardized transport method for OCCI is for products relevant to EGI Federated Cloud and its user HTTP. Therefore it is technically possible to interact communities. with an OCCI-capable server through a generic HTTP 348 Easing Scientific Computing and Federated Management in the Cloud with OCCI client, and implement at least a subset of OCCI func- around the EGI Federated Cloud. tionality with a few pre-formatted OCCI messages. This approach is often used in interoperability test- 3.1 Science Gateways and Workload ing, where a fixed set of actions is tested over and Management Frameworks over, but is highly unsuitable for real world applica- tions wherein a general-purpose client is required. The great potential offered by OCCI is evident when At present, there are at least two independent high-level tools are built on or connected to its inter- client-side stacks available: the rOCCI framework face. End users of PaaS and SaaS services that ex- and the jOCCI library (Kimle et al., 2015). ploit cloud resources through OCCI can benefit from The rOCCI framework offers a set of Ruby li- a large amount of resources belonging to heteroge- braries (rOCCI-core and rOCCI-api), wherein the neous cloud sites adopting different cloud manage- former implements the OCCI class structure, pars- ment frameworks. ing and rendering, while the latter implements HTTP In the context of the EGI Federated Cloud (Wal- transport. Calls to these libraries may be used by na- lom et al., 2015; del Castillo et al., 2015), this oppor- tive Ruby clients to interact with any OCCI-enabled tunity has been exploited by many technical providers cloud site. who have extended their platforms to support the The rOCCI framework also comes with a general- OCCI standard providing an alternative and auto- purpose command line client – the rOCCI-cli – which mated way to access the Federated Cloud, hiding the can be used by end users to complete simple tasks, or IaaS layer from their users.