Declarative Task Delegation in Openmole
Total Page:16
File Type:pdf, Size:1020Kb
Declarative Task Delegation in OpenMOLE Romain Florent Mathieu Thierry Nicolas David Reuillon Chuffart Leclaire Faure Dumoulin Hill ISC-PIF LIP ISC-PIF LISC LISC LIMOS CNRS INRIA, ENS CNRS Cemagref Cemagref CNRS, UBP [email protected] ABSTRACT cessor architectures and available applications. Even if grid infrastructures are able to take advantage of stor- In this paper we present OpenMOLE, a scientific frame- age and computational power through the grid mid- work providing a virtualized runtime environment for dis- dleware, a fine-grain analysis is still necessary at the tributed computing. Current distributed execution systems user level to achieve a successful execution on such an do not hide the hardware and software heterogeneity of environment. computing and data resources whereas OpenMOLE pro- vides generic services to develop distributed scientific al- The main object of this paper, is to describe gorithms independently from the execution environment ar- distributed computing facilities within OpenMOLE. chitecture. OpenMOLE uses abstraction layers to dele- OpenMOLE is a scientific framework, which aims at gate computing tasks with the same high level interface providing a virtualized runtime environment. In this for the major underlying architectures: local processors, approach the user can be confident that an experi- batch systems, computational grids, Internet computing ment successfully executed on a local computer will be and cloud computing. The file access abstraction layer is successfully executed on other execution environments another key feature helping a generic usage of the compu- without preliminary tests or software installation. tation power provided by grids and clusters. The Open- MOLE framework has been tested with the exploration This paper is organized as follows. Section one de- of a bacterial biofilm simulation with an individual-based scribes existing tools to execute jobs on large scale model. grid computing and clusters. The lack of existing tools compared to our main goal are pointed out. Sec- tion two presents novel methods implemented in Open- KEYWORDS: Task Delegation, Grid Computing, MOLE providing a transparent access to various dis- High Perfomance Computing, Distributed Simulation tributed computational environments. Exploration, Design of Computer Experiments. 2. RELATED WORK IN DISTRIBUTED EX- 1. INTRODUCTION ECUTION MANAGEMENT TOOLS The computational grid concept was originally pre- 1 2 sented as a metaphor of the US power grid [1]: a global Batch systems such as openPBS , Condor or Sun Grid 3 infrastructure where end-users can access computing Engine (SGE) and grid middleware like globus [2], power on demand. But this concept clashes with re- gLite [3] or DIET [4] provide integrated commands ality. Indeed, end-users are faced with heterogeneity and/or specific Application Programming Interfaces of the execution environments. Power grid has to deal (API) to submit and manage jobs as well as transfer with potential difference and electricity flow, whereas data. computational grid has to deal with not only network 1http://www.pbsgridworks.com/ and computing power availability considerations, but 2http://www.cs.wisc.edu/condor/ also higher level concepts like operating systems, pro- 3http://gridengine.sunsource.net/ 978-1-4244-6830-0/10/$26.00 ©2010 IEEE 55 Even though these distributed execution systems share tributed execution environments. Currently, the g- many common concepts, the set of commands provided Eclipse project supports implementations for two grid by each of them is specific. In this context, using ab- middleware systems: gLite7 [3] and gria [9] and one straction layers is a key issue to easily benefit from cloud computing service: Amazon EC2 (Elastic Com- the distributed computing power independently of the pute Cloud)8. These three environments are based on underlying architecture. A well designed generic ab- similar concepts but on radically dissimilar architec- straction layer for computational resources should al- tures, illustrating the genericity of the g-Eclipse mid- low the use of local processors, batch systems, compu- dleware abstraction layer. tational grids, Internet computing and cloud comput- ing through the same high level interface. Besides the generic access to several grid systems, g- Eclipse provides an implementation of the Eclipse file- 2.1. Ganga system (efs) to access grid storages. efs is an abstrac- tion layer in which files are referenced by Uniform Ganga [5] is a Python application for job management. Resource Identifier (uri)s 9. Actual file accesses are Ganga allows local and distributed executions on many achieved using dynamically registered modules con- grid middleware and batch systems based on: openpbs, taining implementations of the file system access pro- lsf (distributed by Platform4), dirac [6] and gLite [3]. tocols. Various implementations of efs are available, among them: SFTP, HTTP, CVS. Implementations Even though Ganga provides a generic job manage- for gLite [3] file access protocols have been developed ment on many cluster and grid environments, it does by the g-Eclipse project, including the following proto- not provide a full support for accessing grid storage cols: GridFTP (the grid file transfer protocol) and srm elements and file servers. Ganga enables the user to (Storage Resource Management). An implementation define input and output files for jobs. This function- for the on-line storage Amazon S3 is also available. ality fits the \bag-of-task" [7] paradigm, in which a job is transfered jointly with data it refers to in an G-Eclipse provides very useful classes to interact with object called input sandbox. However in many real various execution environments in a generic way. How- distributed simulation cases, jobs need to access large ever its development has been more focused on build- files. In this case, this voluminous file should not be ing a nice graphical interface interacting with the grid mentioned in the input sandbox. Otherwise the file than a nice application programming interface (API). would be uploaded whenever the job is launched in- A consequence for developers is that further develop- stead of being stored once and for all on a storage ment based on it, might be tricky. element. 2.3. JavaGAT and JSAGA In our experience, this missing functionality prevents designing generic jobs. Indeed, the lack of generic Java Grid Application Toolkit (JavaGAT) [10] is a file access interface necessitates writing specific (for Java API designed to abstract various distributed ex- a given execution environment) file access instructions ecution environments. It is designed as an extensible in the job executable code. Therefore, the file access environment in which modules for accessing computing abstraction layer is another key feature for generic us- platforms can be plugged. Modules, also called \adap- age of the computation power provided by grids and tors", for execution systems (such as gLite, Globus, clusters. SGE, SSH) and for remote file system access (such as HTTP, FTP, SFTP or GridFTP) are currently avail- 2.2. G-Eclipse able. On this work basis, the Simple API for Grid Ap- plications (SAGA) [11] generalizes the JavaGAT archi- G-Eclipse5 [8] is an integrated workspace environment tecture by providing a language-independent specifica- for accessing grid computing power. It is based on tion for accessing distributed execution environments. Eclipse6. G-Eclipse provides high-level abstractions Several implementations of SAGA have been achieved for grid resources such as job management services, in C++, Python and Java. This approach, endorsed authentication services and information services. This by the Open Grid Forum (OGF), is the most promis- abstraction layer is adaptable for various kinds of dis- 7http://glite.web.cern.ch/glite/ 4http://www.platform.com/ 8http://aws.amazon.com/ec2/ 5http://www.g-eclipse.eu/ 9URIs are normalized in the RFC3986, they allow identifying resources 6http://www.eclipse.org/ on a network such as servers, web services or files 56 ing we have seen for a transparent access to remote 3. OpenMOLE should be autonomous in accessing computing and storage resources. various remote execution environments, without requiring third party servers, 3. OPENMOLE 4. the end-user's desktop computer can be con- nected to the Internet without being necessarily Tools such as G-Eclipse, Ganga or JSAGA simplify the the owner of a public IP address (Internet access access to distributed execution environments, however via a Network Address Translation (NAT) system they do not hide the hardware and software hetero- is enough). geneity of computing resources. From our previous work on grid computing [12, 13], this drawback pre- vents developing distributed scientific algorithms in- Constraints 1 and 2 have been established so that the dependently from the execution environment architec- user can work on his own algorithms, without knowing ture. In order to tackle this problem, OpenMOLE how to deploy or execute them on various execution takes advantage from generic interface of JSAGA and environments. OpenMOLE should provide means to provides, on top of that, a virtualized runtime environ- integrate external programs, which can then be exe- ment as shown on Figure 1. Transparent delegation of cuted directly and with confidence on targeted execu- scientific algorithms to remote environment becomes tion environments despite the hardware and software possible within OpenMOLE. Indeed, virtualizing guar- heterogeneity. antees a successful