PoS(EGICF12-EMITC2)081 http://pos.sissa.it/ ∗ [email protected] [email protected] Speaker. The Advanced Resource Connector (ARC) Grid middlewareREX including was its designed Computing almost Service 10 A- years ago,solution. and has Historically proved to ARC be was anon attractive using remote distributed clusters. proprietary computing interfaces With time fority as managing coverage became computational of important. various jobs Grid Few middlewaresadoption expanded attempts of interoperabil- have OGSA been Basic made Executioncovering Service to enough (BES) functionality address for standard. this production use. issue. Unfortunately VariousGrid it implementations provided middlewares One proved by - to of different including be them ARC not was extensions - hence solve breaking that idea of problem compatibility. byinterface Recent implementing (EMI-ES) development specifications different of produced proprietary EMI Execution much Service richerwhich takes and into almost account production-ready requirements definition of 3To prove participating its middlewares usability - and to gLite, test Unicore its andprovide completeness ARC. and few interoperability implementations. capabilities This it is paper important represents to made the in implementation the of A-REX the service. EMI-ES interface ∗ Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licence. c

EGI Community Forum 2012 / EMI26-30 Second March, Technical 2012 Conference Munich, Germany Martin Skou Andersen Niels Bohr Institute, Blegdamsvej 17, 2100E-mail: København Ø, Denmark EMI Execution Service implementation in ARC Aleksandr Konstantinov University of Oslo, P.b.1048 Blindern, N-0316 Oslo,E-mail: Norway PoS(EGICF12-EMITC2)081 ], 6 ] can be 7 ], gLite[ ]. This stan- 5 3 ], ARC[ 4 Aleksandr Konstantinov ] and defines non-strict 12 ] service of gLite and UNICORE 9 ] CREAM[ 8 2 ] and corresponding job description language JSDL[ 2 ] to develop common interface with rich enough capabilities, flexible and with ]. 10 14 ]. The EMI-ES is one of its core activities. Support for delegation of client credentialsality. to service It closely allows linked for to different data delegateddata. staging credentials function- to be assigned to different input and output To accommodate for various implementationsonly of basic job job processing state cycle. life-cycle And EMI-ES states defines related to supported functionalities - data staging, fail- Integrated support for data pre-staging andtions. post-staging Staging with of flexible specific control data forputational is staging job. conditional op- depending on job request and outcome of com- Quick development cycle tied to developmentdiate of response EMI to components needs allows of for participating almost middlewares. imme- 11 This new interface was called EMI Execution Service (EMI-ES) [ The plethora of non-compatible interfaces is driving development of client applications and This complex situation raised effort backed by 3 middlewares within European Middleware Historically not so many efforts for standardizing computational job control interfaces can be The most notable effort to provide true standard among Grid computing services is OGSA • • • • 2. Description of interface dard was adopted by multipleetc. Grid middlewares. But Among interoperability them testsplementations. UNICORE[ show only The very OGSA basic BES functionalitydeveloped provides by achieved various only between groups different very simultaneously im- basicdards are functionality development tuned and procedure for its is different extensions aims usuallyproduction and being slow middleware. overlap. and not Also consistent stan- with much quicker life-cycleframeworks of capable to communicate using multiplementioned interfaces. as one Among of those oldest. CondorG2 ARC [ own also proprietary provides interfaces, set OGSA of client BES-enabled[ atomic side services[ plug-ins for communication with Initiative project[ super-set of functionalities from production versionstive of features all of participating new middlewares. interface The - distinc- especially if compared to the BES - are: fast development cycle. The EMI isdistributed a computing. project developing The a EMI software platform Griddistributed for middleware computing high is infrastructures performance used all by over the scientificGrid[ world research including communities the and Worldwide LHC Computing EMI Execution Service implementation in ARC 1. Introduction identified in Grid community.are Both mostly Local adopting Resource proprietary Management solution.Service Systems And technology and although that their the still only tendency Grid provides in layers communication Grid layer world leaving is semantics to incompatible. useBasic Execution Web Service (BES)[

PoS(EGICF12-EMITC2)081 EMI ADL EMI

] standard JDL 1

]. All three are JSDL 14 ] was also imple- Job

16

Aleksandr Konstantinov xRSL

description

EMI ES EMI CREAM BES CREAM

] and UNICORE/X[ Unicore BES Unicore 13

Job control BES ARC

- implemented in C++ with a set of dynamically

3 ARC GridFTP ARC

Architecture of ARC client EMI ES EMI BES

ARC Client Library ARC Client

Figure 1: ARC client library WSRF+GLUE2

Target ARC (LDAP) ARC

information EMI Registry EMI shows currently available set of modules implemented for the ARC client library.

1 Top/Site-BDII ] model is a base for representing computational services internally. Significantly ex-

15 ARC EGIIS ARC ure reason, etc. - are represented bystates. state This modifiers allows which for are greater only loosely flexibilityimplemented. coupled in to how specific and at which state specific functionalities are Target The figure The EMI provides three implementations of computing services offering different set of fea- The ARC middleware implements EMI-ES interface both at client and service side. The client discovery EMI Execution Service implementation in ARC GLUE2[ 3 additional modules were designed asguage part (EMI of ADL) EMI-ES handling implementation module, - EMI-ESvice EMI computational information job query job description module. control lan- Also module for andservice discovering EMI-ES discovery EMI-ES ser- module service as which part can of talk Grid to infrastructure new EMI Registry service EMIR[ tures and suited for different purposes - A-REX, CREAM[ tended from their prototypes, those structurescoincides are well able with to express EMI-ES widecapabilities. interface range of because applications. it This also chooses GLUE2 for representing service 3. Implementation implementing EMI-ES interfaces as part of their EMI participation. part of ARC is a library -loadable called plug-ins each representing specificindexing module service for or communicating processing withaddition job computing of request service, new language. interfaces anduses This functionalities. structures modular initially For design describing based allows computational on for job JSDL seamless internally and ARC later significantly extended. The OGF[ PoS(EGICF12-EMITC2)081 ]. 18 Batch system Aleksandr Konstantinov Batch interface 4 . Communication part is handled by WS framework of 2 Job Data Data staging Module Architecture of ARC computing service collector Management (Grid Manager) (Grid Information ] also has modular design. Different parts of functionality are split 17 Figure 2: BES EMI ES Currently neither the client nor the service parts provide full EMI-ES implementation. Miss- On service side ARC implements EMI-ES interface as part of A-REX service. The ARC Delegation Delegation Infosysinterface WS interfaces: GridFTP GridFTP interface And credentials delegation is implementedbridge as between a communication dedicated and job libraryof control of new modules. C++ interface classes Such relatively which separation simpleof provides makes task. interfaces implementation active simultaneously. It That also makeson allows it production to systems possible have hence even providing multiple to possibility deploy samement. to experimental and test interfaces new different interface kinds in real production environ- ing functionality is mostly related toof a it way - A-REX most processes notable jobs are internally. multiplemulti-node There prologues request is and support. not epilogues However for so some executable much of applicationfeatures and the in compatible EMI-ES A-REX specs hence triggered implementation enhancingES of its specification new usefulness. was During identified the as implementation covering process most the of EMI- the use cases of ARC middleware. Whether over multiple modules as seen on figure mented as part of thecomputational activities library. within Grid In infrastructure this based way only on ARC common client agreed library EMI provides interfaces. Computing full Service solution A-REX[ for handling ARC with job processing implemented as separatedone module in known another as reusable Grid module Manager. which Data provides staging balanced is data download/upload functionality[ EMI Execution Service implementation in ARC PoS(EGICF12-EMITC2)081 ] , , ] ] ] Aleksandr Konstantinov , (2008) Berichte des Forschungszentrums , ] GFD-R.108 , http://glite.cern.ch/ ,[ doi:10.1016/j.future.2006.05.008 5 http://hdl.handle.net/2128/3695 (2007) [ 23 Job Submission Description Language (JSDL) Specification, Version 1.0 http://www.ogf.org/documents/GFD.136.pdf (2010), ISSN 0944-2952 [ http://www.ogf.org , Advanced Resource Connector middleware for lightweight computational Grids UNICORE 6 - Recent and Future Advancements OGSA Basic Execution Service Version 1.0 (2008), [ JUEL-4319 http://www.ogf.org/documents/GFD.108.pdf [ gLite - Lightweight Middleware for Grid Computing GFD-R.136 A. Streit et al., Jülich M. Ellert et al., Future Generation Computer Systems Open Grid Forum I. Foster at al., A. Anjomshoaa at al., The first interoperability tests showed promising results although not full interoperability was The implementation of new EMI-ES interface in ARCPreliminary shows interoperability promising tests run results between different as implementations in of EMI-ES some client and That should drive further development of EMI-ES specifications to cover issues discovered This work was done as part of the EMI project and partially funded by the European Commis- [6] [4] [5] [1] [2] [3] EMI Execution Service implementation in ARC the non-covered functionality is reallyplemented used in in EMI-ES production functionality environmentdevelopment is of and still EMI-ES how specifications. to it For be purpose couldcomings determined of of be better and EMI-ES understanding im- may capabilities in and production become short- environment,interface subject ARC to for community be is enabled future planning on to sites promote using EMI-ES ARC middleware. 4. Interoperability immediately achieved. Besides few technicalmore mistakes serious shortcomings in were interpreting revealed. EMI-ES The specifications mostrepresentation important also in of them GLUE2 is model. weak definition That ofand makes service use impossible EMI-ES for services automated and tools made to interoperability tests properly a discover manual work. 5. Status and Conclusion fields it covers even broader functionality than was available before. service show that basicbetween functionality interfaces is properly constituting covered servicestill by and are specifications. way undefined. service But describes InEMI-ES interconnection itself is case capable through of to GLUE2 communicate ARC to model that other means implementations the that whole although setduring failed each interoperability to plugin tests. perform. implementing part of 6. Acknowledgment sion under Grant Agreement INFSO-RI-261611. References PoS(EGICF12-EMITC2)081 , ] Aleksandr Konstantinov Journal of Physics: , , ] , , (2009) ] , 6 GFD-R-P.147 , http://wlcg.web.cern.ch/ , http://www.eu-emi.eu , (2011) , (2012) , (2006), ISBN 10:0230-63017-0, ISBN 13:978-0230-63017-8. Proc. XV International Conference on Computing in High Energy and , (2001) , http://grid.pd.infn.it/omii/cream-bes 062006 https://twiki.cern.ch/twiki/bin/view/EMI/EMIRegistry , , GLUE Specification v. 2.0 CREAM: A simple, Grid-accessible, Job Management System for local Adaptive data management in the ARC Grid middleware ARC Computing Element. System Administrator Guide EMI Execution Service Specification Condor-G: A Computation Management Agent for Multi-Institutional Grids , http://research.cs.wisc.edu/condor/doc/condorg-hpdc10.pdf http://www.ogf.org/documents/GFD.147.pdf http://dx.doi.org/10.1088/1742-6596/331/6/062006 http://www.nordugrid.org/documents/arc-ce-sysadm-guide.pdf D. Cameron et al., http://grid.pd.infn.it/cream/ S. Andreozzi at al., EMI Registry (EMIR) F. Paganelli at al., NORDUGRID-MANUAL-20 B. Schuller at al., CREAM (Computing Resource Execution And Management) Service UNICORE/X http://www.unicore.eu/unicore/architecture/service-layer.php#anchor_xnjs Nuclear Physics (CHEP’06) European Middleware Initiative WLCG - Worldwide LHC Computing Grid BES-Enabled CREAM P. Andreetto at al., Computational Resources J. Frey et al., Proceedings of the Tenth IEEE SymposiumSan on Francisco, High California Performance Distributed Computing (HPDC10) [ https://twiki.cern.ch/twiki/pub/EMI/EmiExecutionService/EMI-ES-Specification_v1.07.odt [ Conference Series 331 [ [8] [9] [7] [18] [15] [16] [17] [12] [13] [14] [10] [11] EMI Execution Service implementation in ARC