EGI-InSPIRE
Deploying Helix Nebula flagship applications in a hybrid scenario
Salvatore Pinto Cloud Technologist EGI.eu
9/18/2013 1 EGI-InSPIRE RI-261323 www.egi.eu Agenda
• Helix Nebula and the EGI Federated Cloud
• The ESA flagship on Helix Nebula
• ESA Proof of Concept on the EGI Federated Cloud
• Next Steps
9/18/2013 2 EGI-InSPIRE RI-261323 www.egi.eu Helix Nebula and EGI FedCloud
The EGI Federated Cloud is federation of institutional private Clouds, offering Cloud Infrastructure as a Service to scientists in Europe and worldwide.
• Standards and validation: federation is based on common open-standards – OCCI, CDMI, OVF.
• Heterogeneous implementation: no mandate on the cloud technology, the only condition is to expose the chosen interfaces and services.
EGI-InSPIRE RI-261323 www.egi.eu Interoperability between Helix Nebula and EGI FedCloud
Considering the Helix Nebula, Blue Box approach, the obvious interoperability solution for VM management is to add a OCCI connector to the Blue Box
OCCI FedCloud BLUE BOX
We concentrated our effort on one of the Blue Box solutions under test, SlipStream, which is an Open Source solution and has an EGI-InSPIRE mini-project to support OCCI cloud connectivity.
NOTE: enStratius proposed Blue Box does not support OCCI at the time
9/18/2013 5
EGI-InSPIRE RI-261323 www.egi.eu The ESA flagship
9/18/2013 7 EGI-InSPIRE RI-261323 www.egi.eu The ESA flagship (SSEP)
ESA flagship is integrated into the ESA’s SuperSite Exploitation Platform project, which exploit cloud resources to
1. Provision IaaS resources from different cloud providers as well as private cloud resources (hybrid multi-sourced cloud) 2. Allow users (scientists) to search and access/download data at high performance and data providers (ESA and other agencies) to upload data. 3. Allow users (scientists) to process data on dedicated sandboxes; for algorithm development and small data processing 4. Provider users (scientists) access to large data processing where to deploy and exploit their applications via a portal. 5. All nicely integrated with a common security framework
More info at: http://ssep.eo.esa.int
9/18/2013 8 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC on EGI Federated Cloud
The ESA Proof of Concepts on the EGI Federated Cloud focuses on demonstrating the possibility to provide Processing Services to ESA scientists using the EGI Federated Cloud resources
• Participants: – User community • ESA Research and Service Support section: Configuration and execution of the tests – Technology providers • Sixsq: Provided Open Source SlipStream software and OCCI connector – Resource providers • CESNET (OpenNebula): Performance tests and multi-site tests • GRNET (synnefo): Multi-site tests • CESGA (OpenNebula): Hosting of the SlipStream server
9/18/2013 9 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC Scenario 1: Single site Processing Cluster
Resource Providers ESA Broker
Computing Cluster Deployment
Hosted by EGI
OCCI
OCCI Interface CESNET (OpenNebula)
9/18/2013 10 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC Scenario 1: Single site Processing Cluster
Resource Providers ESA Broker
Computing Computing Cluster Cluster Deployment Cluster Working Head Nodes Hosted by EGI
SlipStream Orchestrator OCCI
OCCI Interface CESNET (OpenNebula)
9/18/2013 11 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC Scenario 1: Single site Processing Cluster
Resource Providers ESA ESA Grid
Computing Cluster Globus GRAM Job Submission Cluster Working Head Nodes
CESNET (OpenNebula)
9/18/2013 12 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC Scenario 1: Single site Processing Cluster
Resource Providers ESA Broker
Terminate Computing Cluster Cluster Cluster Working Head Nodes Hosted by EGI
OCCI
OCCI Interface CESNET (OpenNebula)
9/18/2013 13 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC Scenario 1: Single site Processing Cluster (with different back-ends technologies)
Resource Providers ESA Broker
Computing Cluster Computing Cluster Computing Cluster Deployment Cluster Working Cluster Working Head Nodes Head Nodes
Hosted by EGI
SlipStream SlipStream OCCI Orchestrator Orchestrator
OCCI OCCI Interface Interface GRNET CESGA (synnefo) (OpenNebula)
9/18/2013 14 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC Scenario 2: Multi site Processing Cluster
Resource Providers ESA Broker
OCCI GRNET Interface (synnefo)
Multi-Site Computing Cluster OCCI Cluster Deployment Working Nodes
Hosted by EGI
Cluster Working Head Nodes
OCCI CESNET Interface (OpenNebula)
9/18/2013 15 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC Tests and Results (1/4)
• Automatic Deployment of Processing Cluster – Time required for complete startup: 22 minutes
Time is inline with the expectation, considering that: • All the software is installed and configured on startup, the image is a basic OS installation • Cluster is started and then operated for at least one week • The same procedure, performed manually, usually takes about 1 day of work
9/18/2013 16 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC Tests and Results (2/4)
• VM creation/deletion using direct OCCI API
• Job submission to the Computing Cluster – Average job overhead: 5 seconds
• Computing Cluster end-to-end availability – 99.56% during 5 days (31/7021 failures)
• Stress test (10WN Cluster under CPU/RAM/Disk stress) – Average job overhead under stress: 5 seconds
9/18/2013 17 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC Tests and Results (3/4)
• Internal network bandwidth – Average 9.2 Gbit/sec
• Local storage performances – Average: Write 102M/sec, Read 103M/sec
• Network latency (time to download an empty file using SCP) – Averaged for internal network: 0.12 s – Averaged ESA to FedCloud: 0.6 s
EGI sample site performances are inline with Helix Nebula commercial providers
9/18/2013 18 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC Tests and Results (4/4)
• Multi-Site Deployment of Processing Cluster
Cluster Deployment was correctly completed, but network issues in the sites connections caused the job submission tests to fail
Still to be investigated
9/18/2013 20 EGI-InSPIRE RI-261323 www.egi.eu ESA PoC Test Results Summary
OCCI Automatic FedCloud Cluster Deployment Computing Cluster
Globus Cluster Working Head Nodes Job submission
• Proved interoperability between academic and commercial clouds.
• Globus-based computing cluster deployment in one-click.
• Cloud technology agnostic. SlipStream worked with the same connector and configuration on OpenNebula and synnefo sites.
• Test of site availability, performances, etc… (results shows provided resources are inline with commercial providers).
9/18/2013 21 EGI-InSPIRE RI-261323 www.egi.eu Next Steps: CERN PoC (October 2013)
CERN
Computing Cluster Deployment
Broker
BLUE BOX
Resource Providers OCCI
OCCI OCCI OCCI OCCI CESNET GRNET FZJ INFN Commercial Cloud (OpenNebula) (sysnefo) (OpenStack) (WNoDeS) Providers
FedCloud
9/18/2013 22
EGI-InSPIRE RI-261323 www.egi.eu Next Steps: SlipStream FULL connector (January 2014)
SlipSteream FULL connector, providing – Support for “Image Creation” feature – Support for EGI FedCloud Contextualization system (OCCI User Data extension) – Support for attached storage (block or object storage) – Better error catching (to debug issues in the sites) – Available by default in SlipStream 2.0
9/18/2013 23
EGI-InSPIRE RI-261323 www.egi.eu EGI-InSPIRE
Thank you
Questions?
EGI-InSPIRE RI-261323 www.egi.eu