EGI USER FORUM 2011 Book of Abstracts
held in conjunction with the emi technical conference EGI BOOK OF USER FORUM ABSTRACTS 2011
held in conjunction with the emi technical conference
11–14 April 2011, Vilnius, Lithuania First published in 2011 in the Netherlands by EGI.eu Science Park 105 FORUM HOSTS 1098 XG Amsterdam The Netherlands www.egi.eu
ISBN 978 90 816927 1 7
Copyright © 2011 by Members of the EGI-InSPIRE Collaboration
This work is licenced under the Creative Commons Attribution-Noncommercial 3.0 Licence. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/3.0 or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.
The work must be attributed by attaching the following reference to the copied elements: “Copyright © Members of the EGI-InSPIRE Collaboration, 2011. See www.egi.eu for details of the EGI-InSPIRE project and the collaboration.” Using this document in a way and/or for purposes not foreseen in the licence requires the prior written permission of the copyright holders. PROGRAMME COMMITTEE EGI-InSPIRE (“European Grid Initiative: Integrated Sustainable Pan-European Infrastructure for Researchers in Europe”) is a project co-funded by the European Alberto Aimar CERN Yannick Legre HealthGrid Commission as an Integrated Infrastructure Initiative within the Sergio Andreozzi EGI.eu Rob van der Meer EGI.eu 7th Framework Programme. EGI-InSPIRE began in May 2010 and will run for 4 years. Steve Brewer EGI.eu Alberto Di Meglio CERN
Daniele Cesini INFN Johan Montagnat CNRS The Organisers and the Publisher have made every effort to provide accurate and complete information in this Book. However, changes or corrections may Diana Cresti INFN Steven Newhouse EGI.eu occasionally be necessary after publication. Michel Drescher EGI.eu Morris Riedel CHJUELI
Florida Estrella ERNC Horst Schwichtenberg RF AUNHOFER
Tiziana Ferrari EGI.eu Jamie Shiers CERN
Laurence Field CERN Gergely Sipos EGI.eu
Patrick Fuhrmann DESY Giuliano Taffoni INAF
Catherine Gater EGI.eu Claudio Vuerli INAF
Francesco Giacomini INFN John Walsh TCD
Emidio Giorgio INFN Cecile Germain CNRS
Editors: Sara Coelho, Neasan O’Neill Maria Girone CERN Ioannis Liabotis GRNET Design: Viviane Li Forum poster: Ori Ginale Balázs Kónya LUND Contents
Overview...... xii A suite for distributed execution of general purpose applications on the grid for small- and mid-size VOs...... 30
Worker node software management: the VO perspective...... 32 USER SUPPORT SERVICES 2 A new integrated monitoring tool for grid site administrator...... 34 Benefits and vision for the VRC community model ...... 4 Accounting and SLA support in KnowledgeGrid Malaysia by P-GRADE portal...... 36 EGI VO Services activities ...... 6 UNICORE accounting service for Belarusian Grid Infrastructure...... 38 Applications Database...... 8
NGI-based user support in the European Grid Infrastructure collaboration...... 10 EMI 40 CTACG – The Cherenkov Telescope Array Computing Grid...... 12 What's new in EMI 1...... 42 DC-NET – Digital Cultural Heritage Network ...... 14 EMI-ES: a common interface to ARC, gLite and UNICORE computing elements...... 44 FP7-EU project ‘DECIDE’ (Diagnostic Enhancement of Confidence by an International Distributed Environment): a national prototype of e-services A uniform MPI user experience with MPI-Start in EMI...... 46 for the assisted diagnosis of neurodegenerative functional studies...... 16 EMI Data, the unified European Data Management Middleware ...... 48 GÉANT and EGI: enabling seamless distributed computing infrastructures through high-speed, dedicated networking...... 18 Supporting SRM over SSL...... 50
Logging and Bookkeeping – what can it do for you?...... 20 The ARC data client in EMI 1 ...... 52
Supporting Distributed Computing Infrastructures...... 22 Parallel NFS: EMI data in competition with high-end industry storage solutions . . . . . 54
HammerCloud: an automated service for stress and functional testing of grid sites. . . .24 gLite File Transfer Service in the EMI data area...... 56
Catering for different user profiles in the HELIO project...... 26 Storage catalogue consistency...... 58
WeNMR: the tale of virtual research community in NMR and structural biology...... 28 Software quality assurance in EMI...... 60
vi | EGI User Forum 2011 contents | vii Software quality assurance tools in EMI...... 62 U ser Environment 92
Definition, reporting and analysis of EMI software metrics...... 64 A science gateway for molecular simulations...... 94 EMI Integration and large scale testing infrastructure...... 66 Proteomics portal: an e-science gateway for Swiss bioinformatics...... 96 The gLite Data Management continuous integration and testing process...... 68 SOMA2 – gateway to grid-enabled molecular modelling workflows in www-browser...... 98 ARC tools for revision and nightly functional tests...... 70 Data-driven portal for hadron therapy ...... 100 EMI and dCache: the vanguard of standards adoption ...... 72 Job Submission Tool, web interface and WebDAV data management...... 102 Proposing a Storage Accounting Record standard...... 74 New developments of gUSE and WS-PGRADE to support e-science gateways. . . . .104 Implementation of GLUE 2.0 support in the EMI Data Area...... 76 Grid computing for biomechanical markers extraction...... 106
Virtualisation and Cloud Computing 78 Socially impacting grid services from the chemistry Heavy User Community ...... 108
e-Social Science development in Taiwan: scalable social simulation Operating grid services on the StratusLab cloud...... 80 using commodity computing...... 110
Data management at VENUS-C...... 82 Access earth science data from the EGI infrastructure ...... 112
Mantychore – dynamic networks for e-Infrastructure...... 84 Grid computing for electrophysiology ...... 114
Federated cloud computing environment for malaria fighting...... 86 Grid-empowered molecular simulators of crossed beam signals...... 116
Searching Eclipse behind the cloud...... 88 Earth science oriented data processing tools and applications for the Black Sea catchment basin...... 118 GPU computing in EGI environment using a cloud approach ...... 90 The grid rocks...... 120
e-science on earthquake hazard mitigation...... 122
viii | EGI User Forum 2011 contents | ix An overview of CMS workload management for data analysis...... 124 InSilicoLab – grid environment for supporting numerical experiment in chemistry. . . . 156
Getting a grip on the grid: a knowledge base to trace grid experiments ...... 126 The GRelC Project: main service, functionalities and relevant use cases...... 158
Shared services and tools based on the Ganga job definition and management framework...... 128 DATA MANAGEMENT 160 Cheminformatics platform for drug discovery application on grid ...... 130 An insight into the ATLAS Distributed Data Management ...... 162 VisIVO, visualisation in different grid environments, from gLite to desktop grid. . . . . 132 Improving CMS data transfers among its distributed computing facilities...... 164 e-Infrastructures integration with gCube...... 134 Evolution of the ATLAS data placement model ...... 166 Distributed multi-scale computing: the MAPPER project...... 136
Theompi: a large MPI cluster on the grid for theoretical physics ...... 138 TECHNOLOGIES FOR DISTRIBUTED COMPUTING 168 CORAL – a relational abstraction layer for C++ or Python applications...... 140 A new ‘lightweight’ crypto library for supporting an advanced grid Challenges in the adoption of the EGI paradigm for authentication process with smart cards...... 170 an e-Science / Tier 2 centre (ES-ATLAS-T2)...... 142 Federated access to grids...... 172 Monitoring of the LHC computing activities during the first year of data taking. . . . . 144 Argus, the EMI Authorisation Service...... 174 Optimisation of the job submission and data access in a LHC Tier 2...... 146 Site integral management with Puppet ...... 176 The EDGI infrastructure and its usage for the European Grid user communities. . . . .148 A new scheme for efficient distribution of information in grid infrastructures...... 178 Cross-grids simulation scenarios with the GridSFEA framework...... 150 Coarse-grained interoperability of heterogeneous grid workflow systems...... 180 Integration, sharing and exploitation of national and international e-Infrastructures. . . .152
Multidisciplinary approach for computing in Emilia Romagna (Italy) ...... 154
x | EGI User Forum 2011 contents | xi Dear Delegate, domains specific support services provided by various communities to help their users to make the best use of On behalf of the European Grid Infrastructure community, the infrastructure. specifically the EGI-InSPIRE and European Middleware • Tailored User Environments are frequently provided to Initiative (EMI) projects, and the Vilnius University we a community to simplify their use of the infrastructure. A would like to welcome you to the EGI User Forum and variety of scientific gateways and portal technologies are EMI Technical Conference in Vilnius, Lithuania. described, in addition to applications, tools and program April 2011 is the last month in the first year in both the EGI- ming libraries that can simplify access to distributed InSPIRE and EMI projects – and what a year it has been! April resources. The use of these environments across a range will see the release of EMI 1 – the first public release of an of scientific disciplines is also reported upon. integrated software offering from the gLite, ARC, UNICORE • The Virtualisation and Cloud Computing sessions provide and dCache consortia – after which planning starts on the details of the technologies and user experiences coming next release in a year’s time. from European and national activities in this area. For EGI-InSPIRE, the first year of its four-year project plan has • The Technologies for Distributed Computing sessions seen the transition to a sustainable European Grid Infrastructure highlight some of the challenges in providing federated continue, with the establishment of EGI.eu in Amsterdam and access to e-Infrastructures to achieve interoperability. the transition from the regional operational model established • Some of the high-performance Data Management within EGEE to the national model within EGI. solutions coming from the WLCG community are reviewed. The combined EGI User Forum and the EMI Technical In addition there is a rich selection of demonstrations being Conference, provides a perfect opportunity to catch up shown each day in the exhibition area, training sessions with a range of activities taking place within the community: for resource administrators and end-users on a variety of different technologies, and workshops from European • The EMI sessions provide an overview of some of the technologies being released in EMI 1 and in the projects aligned with the issues facing EGI. developments planned for file transfer and access, the Our thanks to the local organisers (LitGrid, BAIP and the software quality assurance procedures that are employed Vilnius University) and the members of the Programme in the project, and the plans for standards adoption. Committee for their suggestions and support during the • The User Support Services sessions give an overview of preparation of this meeting. the services provided by EGI.eu and its partners, and the We hope you have an enjoyable week.
Steven Newhouse Alberto Di Meglio Overview Project Director, EGI-InSPIRE Project Director, EMI
• • • • • • This track presents the User Support services provided by EGI.eu and the wider EGI-community. EGI provides user support through the GGUS ticketing system which is aimed at capturing issues relating to the operational e-Infrastructure. The Requirements Tracking system offers a mechanism for all members of the community to submit details of their future needs for the e-Infrastructure. In addition to capturing new requirements, the User Community Support team at EGI offers a range of services aimed at meeting the broader needs of research communities. A key focus in EGI is to provide communities of like- minded researchers to work together as Virtual Research Communities in order to better communicate their common needs and also to benefit from common solutions. Further- more, the EGI support team collaborates closely with the U ser Support various NGIs and other partners to assist them in their support of researchers in their own countries. This is done by making the EGI shared services available, by sharing the knowledge of user needs captured from the Virtual Research Communities, as well as by learning Services directly from the NGIs what their own direct needs are. Of course, many support services are highly specialised and relate specifically to particular projects and com- munities. A number of the presentations within this track describe such initiatives. The specialist subject areas covered include: neurodegenerative diagnostics enhancement, digital cultural heritage, gamma-ray astronomy, heliophysics and structural biology. Other technical areas covered include various accounting and virtual organisation monitoring services, gLite’s Logging and Bookkeeping service and the HammerCloud auto mated testing service. The GÉANT Project’s dedicated task to Liaison and Support is also described. Steve Brewer Chief Community Officer, EGI.eu
• • • • • • Benefits and vision for the VRC community model IMPACT CONCLUSION The VRC is defined as an organisational The presentation gives an overview grouping that brings together transient of the benefits of EGI VRCs, the VRC Virtual Organisations within a persistent accreditation process and the profile of and sustainable structure. A VRC must already established VRCs. VRCs benefit be a self-organising group that collects from the services provided by members of and represents the interests of a focussed the EGI collaboration, as well as from the collection of researchers across a clear services provided by other VRCs. VRCs and well-defined field. Named contacts can influence the evolution of EGI software OVERVIEW made available by EGI stakeholders are agreed upon by the VRC to perform and services through various mechanisms, through open source middleware specific roles and these then form the primarily through the User Community Scientific research is no longer being software solutions. VRC members communication channel between the VRC Board. The EGI VRC model contributes to conducted within a single research group can store, process and index large and EGI. EGI welcomes any scientific the sustainability of European Distributed in a single institution in a single country but datasets and can interact with partners community to submit a proposal for the Computing Infrastructures and to the in Virtual Research Communities (VRCs) using the secured services of the EGI accreditation of an EGI VRC. VRC proposal sustainability of European scientific that span national borders encompassing infrastructure. must demonstrate that it represents a communities to a large extent. many different organisations but with a 2. The user support units of the NGIs community of researchers that has an need to share ICT resources. EGI provides and EGI.eu help VRC members during established existence outside of the Become an EGI VRC: the e-Infrastructure that enables the the routine usage of the systems and VRC, i.e. that it has structure (such as an www.egi.eu/user-support/becoming_a_ researchers within a VRC to collaborate, provide assistance to access and ESFRI project, EIROFORUM laboratory, community communicate, share resources, access utilise the largest multi-disciplinary national research structure. professional remote computers or equipment and grid in the world. organisation or affiliation, etc.) and that this produce results as effectively as if they, 3. VRCs will have the ability to establish body represents this particular community. and the resources they require, were their own Virtual Organisations (VOs) This community must also show that it physically co-located. Becoming an as collections of hardware, software has an established governance model and identified VRC within EGI offers several and human resources configured in open mechanisms for new participants to benefits for researchers, scientists and order to share capacities, to collaborate enter (or leave) the organisation, and that the developers of distributed scientific with partners and to run data intensive all members of this organisation will have applications. The talk overviews the EGI simulations. The VOs can benefit from access to all the services offered by the VRC model, the benefits of establishing the resources provided by NGIs and VRC, i.e. beyond those who are just part EGI VRCs for scientific communities and other VRCs. of the proposal. These conditions enable the landscape of already established VRCs. 4. NGIs provide trainers and technology EGI.eu to recognise the VRC as being specialists for VRCs to support them the ‘voice’ of a particular community of DESCRIPTION during the integration and adaptation users within the infrastructure. EGI can of their legacy applications and provide help and advice on suggested best EGI is a partnership between National datasets to the EGI infrastructure. practices for such organisational models Grid Initiatives (NGIs), EIROs and a The combination of VRC’s own if needed. coordinating body – named EGI.eu – to training resources together with EGI’s operate a sustainable, pan-European grid infrastructure-related modules can infrastructure for international scientific provide comprehensive packages communities. EGI.eu is seen as the glue for VRC members in an efficient and enabling coherence between the NGIs for timely manner. the benefit of their users and members 5. VRCs can influence the evolution of of Virtual Research Communities or EGI’s services through representation VRCs. VRC membership in EGI offers the in the User Community Board and the following key benefits for researchers, User Services Advisory Group. Based scientists and the developers of distributed on requirements collected from its scientific applications: members, VRCs can advise EGI on its 1. VRCs can access computing, data planning and operational priorities. storage and other types of resource S teve Brewer [email protected] EGI.eu
4 | EGI User Forum 2011 User Support Services • • • • • • services capacities and costs from the CONCLUSION EGI VO Services activities perspectives of the major players involved: service provider, VO operations and User The technical integration of any research Support. The impact of such documents community as a Virtual Organisation in a intends to raise the awareness of VO production grid environment is a demand Administrators regarding the range of ing task from the operations point of view. services needed to support standard In EGI, that activity has further constraints VO requirements, and understand their due to the decentralised political organisa associated costs. VOs should understand tion, where NGIs play a leading role in the that providing services to their VO project governance model. VO Services OVERVIEW ment Frequently Asked Questions wiki and members requires an effort that should activity intends to address many of the cost/effort evaluations (from the operations be at least partially assumed by the VO. difficulties which new emerging VO The EGI-InSPIRE project aims at support point of view) of several services and This information is an important tool to managers face in the operation and tech ing both Heavy User Communities (HUC) tools that were found to be useful for new assist VO Managers in deciding and nical instantiation of their VO. During this and new emerging communities. While emerging communities. planning the range of services needed, first period of the EGI project, this was HUCs are directly supported through and whether these services can be achieved through the evaluation of services specific tasks in the EGI SA3 activity, new Several monitoring frameworks for the operated by the VO or provided through and tools which were found to be useful emerging communities lack this organisation VOs are already available under the EGI EGI catch-all instances. for VOs and VRCs. Simultaneously, some and resources and take-off can fail if framework, and information is compiled steps of the VO Management process have assistance is not provided. The VO Services in the Wiki pages. Specific effort has Another major achievement was the official been revised, and improvement require subtask, developed in the context of the been put on testing a Nagios monitoring establishment of a communication tool (via ments have been communicated to the EGI-InSPIRE TNA3.4 Technical Services instance which could be used by one or a dedicated GGUS support unit) where VO developers of the services and tools used activity, aims at providing documentation more VOs in monitoring their infrastructure. managers can ask for help and clarification. during that process. The enhancement and services to these emerging com The impact is that dedicated staff are ready of VO activities management is expected munities, allowing them to start up their Finally, an official communication channel to provide guidance through the VO life cycle; to continue through future activities as own Virtual Organisations (VO), Virtual has been established via a support unit assist VO managers through the process well as the testing of new VO monitoring Research Communities (VRC) and boost in GGUS where VO Managers can ask for of deploying technical services; and framework offered from Nagios tool. their integration and usage of the produc clarifications on using some special tool, receive VO managers’ requests (in terms tion infrastructure. handling some special operation or just of services/tools). https://wiki.egi.eu/wiki/VO_Services expressing a requirement that they would DESCRIPTION like to see fulfilled. Finally, the VO management documentation was revised, and a Frequently Asked The VO Services offer new emerging Future work will continue with the Questions wiki-page was created to communities the documentation to guide investigation of the possibility to extend properly address new comers’ VO issues VO managers through the various steps the use of dashboards for new emerging (https://wiki.egi.eu/wiki/VO_Services). of the VO life-cycle process; support VO communities. Presently, the available managers in deploying, using and adapting dashboards used in the HUC framework All these achievements are contributing technical services which may simplify are based on commercial software data so that VO operations become more their user community’s access to the bases, which prevents the possibility to transparent from all points of view: VO infrastructure and promote collaboration extend its use to a broader range of users. management, operation and usage. The within the VO; and collect feedback Other alternatives are being explored. future actions are also pointing in that from VOs and VRCs regarding the used direction with the testing of VO tools services, mapping them into requirements IMPACT for monitoring. for improvement. One of the main achievements during the Documentation is presently offered through first activity period was the production a wiki-page that gathers in a central of an inventory of basic services and and unique point information regarding monitoring tools needed by every VO, and protocols and procedures relevant for emphasising characteristics considered VO management activities, such as the VO important like purpose of the service; Gonçalo Borges [email protected] LIP • Jorge Gomes [email protected] LIP • Ignacio Blanquer registrations process in EGI, VO Manage hardware requirements; known issues; [email protected] UPV
6 | EGI User Forum 2011 User Support Services • • • • • • Applications Database IMPACT CONCLUSION At the time of this writing, the AppDB The AppDB subtask has used the first counts 272 application and tool entries, six months of the project to migrate and and 505 people profiles. Since the public transform existing data from the EGEE release of the latest major version in era, to initially provide a new read-only mid-November 2010, there has been portal to the community, with subsequent interest from various countries such as write-enabled authenticated access. For Switzerland, Ireland, and Norway for the the next six months, there exist plans for testing and addition of entries. The site's additions to the data and object model, OVERVIEW and techniques as MVC and AJAX have traffic has been monitored through the additions to the portal's functionality, been employed. Finally, another important use of Google Analytics, and during this mainly in order to integrate with other The EGI Applications Database (AppDB) feature of AppDB, around which much of one-month period, there have been 182 services such as the GOCDB. The CIC stores information about tailor-made the development effort was concentrated, visits and 1151 page views in total, from portal, and localised versions of the computing tools for scientists to use, and was providing write-enabled access to 25 countries. Additionally, the GISELA portal itself, and integration with the EGI about the programmers and scientists who registered users, by linking AppDB to the project has declared its intention to share Community Software Repository. have developed them. The applications and EGI SSO system, thus giving them control 52 of its applications with the EGI AppDB, tools filed in AppDB are finished products, and responsibility over the data. an intention which may bring further Further requirements have been laid out for ready to be used on the European Grid collaboration between initiatives in later development, such as integration with Infrastructure. Its main mission is to provide For the next version of the AppDB, there the future. monitoring and information services of the all the necessary information about the are several enhancements planned related various supported middlewares, in order applications running on EGI, thus enabling to the information provided, such as adding It should be noted that the AppDB to provide information about application people to search for applications matching an ‘experience’ section to people's profiles, is unique as a service in the EGI availability and resources in sites across a pattern (such as a scientific domain) and keywords and a tagging mechanism infrastructure, and through its web API, NGIs. However, due to the lack of a and also to contact the corresponding to applications. Another important planned which is under development, new localised uniform manner for information retrieval, authors for guidance on application usage feature is integration with GOCDB/ instances are expected to spring out. such requirements have been put on hold, or further development. Storing pre-made CIC through their web APIs, in order to This is in turn expected to help promote until there arises a way to overcome the applications means that scientists don't provide detailed NGI and VO information, scientific work more efficiently by targeting technical difficulties. have to spend research time developing which will require architectural changes specific niches, while the AppDB remains their own software. Hence, by storing pre- and careful planning. Integration with the the master instance. http://appdb.egi.eu made applications and tools, AppDB aims EGI Community Software Repository is to avoid duplication of effort across the EGI also a feature planned which will enable user community, and to inspire scientists developers to register and submit software less familiar with programming into using releases through the AppDB interface. the European Grid Infrastructure. Apart from enhancements, another key DESCRIPTION aspect planned for the next release is the further development of an XML-based web Activities since the beginning of the API. This requirement has stemmed from project have been primarily focused on the need for NGIs and VRCs to be able migrating existing data from the EGEE era. to provide their own localised, custom Support for all EGI-endorsed middleware interfaces to the service, which could, was added and the concept of tools for example, display entries relevant to was introduced, by porting the entries of their county or discipline only, or in their the existing RESPECT programme into own native language. Upon successful the database. The storing of personal deployment of such a web API, the profiles for each application developer possibility of developing an alternate was introduced, aiming at simplifying the REST API on a dedicated FQDN will be search for experts. In order to provide the considered, depending on feedback user with a consistent and responsive end and impact. result, such architectural design patterns William Vassilis Karageorgos [email protected] IASA
8 | EGI User Forum 2011 User Support Services • • • • • • the previously described user support CONCLUSION NGI-based user support in the European Grid fields. The EGI.eu UCST has defined Infrastructure collaboration and runs mechanisms to understand the The presentation gives an overview of strengths and weaknesses of each of user support processes and activities the NGI USTs and to map their offerings from NGIs’ perspective: how the NGIs are and capabilities to the requirements of involved in the process and how they can multi-national user communities. These improve their commitment even further. mechanisms, namely the Wiki pages, the The key focus of the integrated activities knowledge bases, the email lists, the Web is to support sustainable international pages, surveys, web forms, Request Tracker research groups through the Virtual OVERVIEW DESCRIPTION system and various user-facing technical Research Community (VRC) model. services will be reviewed in the presentation. An EGI VRC represents a community of The stated goal of the European Grid The primary stakeholders of EGI user researchers with an established presence Infrastructure is to facilitate the work support services are the User Support IMPACT in a scientific or other research field and of established and emerging multi- Teams of National Grid Initiatives/ with the shared goal to develop themselves national user communities. The growing Infrastructures (NGIs) and EIROs. While Efficient support for users is the primary into self-sustainable grid communities. The user demands have provided, and will the primary responsibility of these NGI interest of all the EGI stakeholders. The long term sustainability of EGI depends continue to provide, the necessary push User Support Teams is to serve users operation of efficient support activities is a on the success of organising existing EGI for development and extension of the grid and user communities inside their non-trivial task, given the highly distributed users and emerging scientific communities infrastructure. Therefore, the active support countries, through their involvement in nature and heterogeneity of infrastructure into VRCs. for these communities is a primary concern international collaborations – under the components, user support teams and user for the members of the EGI collaboration. EGI umbrella – they share and expand communities. EGI has managed to define www.egi.eu/user-support The presentation introduces the goals of their services for the benefit of large, and setup processes and tools to deal with https://wiki.egi.eu/wiki/TNA3.3_NGI_User_ EGI user support activities, the structure multi-national user communities. The User this complexity. The presentation provides Support_Teams and linkage of user support teams and Community Support Team (UCST) of the an excellent opportunity for all the EGI gives an overview of the services and tools Amsterdam-based EGI.eu organisation stakeholders to learn about user support that the different stakeholders provide for provides coordination for this activity. The mechanisms and to see how their partners multi-national user communities. UCST ensures that the national efforts fit and the various NGIs are involved in together at the European level, they satisfy this process. the demands of multi-national scientific collaborations and that the evolution of support tools and mechanisms is aligned with the long term goals of communities.
The user support services are implemented as a mixture of human services, software services and feedback mechanisms. While users are mostly served by human services (e.g. by trainers who provide courses, by consultants and technicians who help porting of applications) software services allow remote interactions with experts (e.g. a helpdesk system) or facilitate the provision of human oriented services (e.g. a training event registry).
NGIs are heterogeneous in terms of the size and composition of their user communities, the size and expertise of their user support teams. Consequently, NGIs are not equally active and strong in all Gergely Sipos [email protected] EGI.eu
10 | EGI User Forum 2011 User Support Services • • • • • • CTACG – The Cherenkov Telescope Array IMPACT CONCLUSION Computing Grid In 2010, 60,000 files containing 109 The combined power of computing centres simtelarray showers have been produced, at different CTA institutes, grouped in the outnumbering by far any production on grid infrastructure, allows massive Monte dedicated computer clusters. Carlo simulations. Our aim is to keep the development for CTA at a minimum by The current production jobs are very staying grid middleware compliant and demanding in memory size (up to 4GB relying on tools already used in the grid RAM) and disk space (5-10GB scratch). community. Current simulation production OVERVIEW DESCRIPTION Not all available computing centres meet is much more efficient than anything these requirements. Therefore only a being done at single computing centres Gamma-ray astronomy is one of the most CTA is currently in the preparatory phase subset of all sites is currently used. Future and allows the estimation of the future active topics in astroparticle physics and no dedicated computing infrastructure plans for upgrading computer centres CTA computing requirements. The grid involving both particle physicists and is available. Using grid infrastructure will take into account the CTA memory infrastructure can be recommended for astrophysicists leading to the design of the computing centres of participating requirements. The necessary scratch disk large scale computing in CTA. new types of research infrastructures. institutes can provide computing resources space and disk operations can be reduced The Cherenkov Telescope Array – CTA – for CTA. Currently, the CTA virtual by running processes parallel within one http://lappwiki01.in2p3.fr/CTA-FR/doku. is a proposed new project for ground- organisation (vo.cta.in2p3.fr) comprises job and piping the output directly. It will php?id=cta-computing-grid-public based gamma-ray astronomy. This 8 computing centres with several 1,000 be necessary to require several cores on project is driven by the CTA consortium, CPUs and about 500TB of storage. The multi-core machines on job submission. comprising 132 institutes in 25 countries. participating computing centres are Tier The CTA Computing Grid (CTACG) uses 1 and Tier 2 centres and provide very The grid provides the necessary infra grid technology to perform heavy Monte heterogeneous computing resources. structure to help participating institutes to Carlo simulations and to investigate the easily supply their computing power to the potential of grid computing for the future The CTA computing grid is currently used CTA community. By relying on official grid data reduction and analysis pipeline. This for massive Monte Carlo simulations. standards the work load on the technical talk presents the tools developed in this For job submission and monitoring staff at the computing sites is reduced and framework, the performance achieved and we use EasiJob (EASy Integrated JOB the development of the grid can be easily the lessons learned. submission), a tool developed within the followed. In this way CTA can concentrate MUST framework (Mesocentre de calcul nearly all its efforts on scientific study. et de stockage ouvert sur la grille EGEE/ LCG), the computing infrastructure from LAPP (Laboratoire d'Annecy-le-Vieux de Physique des Particules). This tool includes a web interface to define and configure a grid production.
The configuration and submission is currently being done manually by the production manager. In the future we plan to set up an automated data pipeline where each newly produced simulation file will be automatically processed. This automated data processing will be based on a data management system. We are investigating the potential of AMI (ATLAS Metadata Interface) for the needs of CTA. The outcome of this investigation will be N ukri Komin [email protected] LAPP, CNRS/IN2P3 • Giovanni Lamanna [email protected] presented in the talk. LAPP, CNRS/IN2P3 • Sabine Elles [email protected] LAPP, CNRS/IN2P3 • Cecile Barbier [email protected] LAPP, CNRS/IN2P3
12 | EGI User Forum 2011 User Support Services • • • • • • The community of the digital cultural institutional, national and sectorial DC-NET – Digital Cultural Heritage Network heritage is naturally larger than the eight boundaries. It should also provide services Ministries participating to DC-NET. The and facilities for collection owners, so network enlargement is on-going and that they can store, protect and present many cultural institutions, governmental their collections online without any loss agencies and Ministries in Europe and of identity, ownership or intellectual beyond are in the process of formalising property. Finally, it should address the their participation to DC-NET. most common causes of fragmentation in the digital cultural heritage sphere, IMPACT including variations in policies, languages, OVERVIEW • Access to everything. Enabling online metadata models and vocabulary. The access to more than a small proportion At political level, there are two main bodies design and implementation of such a The cultural heritage domain (libraries, of holdings; that should be mentioned: the Member data infrastructure dedicated to the digital museums, archives and other cultural • Delivering the best. Providing access to States Experts Group for Digital Libraries cultural heritage is central to the vision bodies) has seen quite a revolution in the the highest quality multimedia materials established by the European Commission of DC-NET and is at the core of its Joint last twenty years. Increasing amounts of online, including audio-visual, 3D, text (MSEG); and the Joint Programming Activities Plan. cultural heritage data are being converted and images; Initiative for Cultural heritage (JPI-CH) to digital form and made available via • Searching beyond text. Enabling established by the Member States. The CONCLUSION the World Wide Web to researchers, users to use images, sounds, shapes DC-NET community is fully involved in students and the interested public. This and other aspects of items as search MSEG, where many experts of DC-NET DC-NET coordinates interaction and represents an important opportunity for criteria; participate as national representatives. liaison across borders and also across the improvement of the quality of life of the • From active to passive. Allowing inter With regard to the JPI-CH, DC-NET the interface between cultural heritage European citizen, for the delivery of cultural action and manipulation of digital items; shares with NET-HERITAGE the operative and e-Infrastructure. Across borders, education, for the promotion of cultural • Maintaining control and rights foundation of the Initiative, being the the Cultural Ministries of the EU work tourism, for the development of the creative management. Cultural heritage former target to the research for the digital together to identify and prioritise the new industry, for increasing the value which material belongs to a huge number of heritage and the latter to the research for actions which should be undertaken in society enjoys from its common patrimony. autonomous entities, each with its own the tangible heritage. order to bring digital cultural heritage to a intellectual property rights; new level of development. Across sector However, there has been no revolutionary • Long-term preservation of digital At project level, in parallel with DC-NET, the boundaries, the cultural heritage experts technological leap forward since the materials. Both digitised and community is active at international level and the e-Infrastructure experts match digitisation and online publication began. born digital. with the INDICATE project targeted to the the capabilities of the e-Infrastructure Bandwidth, storage and search power are Mediterranean region and in supporting and the ambitions of the digital heritage all major constraints. Each of these challenges can be met by Europe with many projects, among which community. The European Ministries of exploiting the results derived from R&D are the two Best Practice Networks Culture, owners and managers of the The establishment of a dialogue and a projects for ICT applied to the cultural ATHENA and LINKED HERITAGE. national programmes for the research factual relationship between the cultural heritage, including national, regional and in the field of the digital cultural heritage sector, the research sector and the EC initiatives and by redeploying these The next step in the vision of DC-NET is interact with their respective national research infrastructures in Europe is the results to e-Infrastructures. the deployment of a persistent and robust executive bodies responsible for the key factor to overcome these limits. data infrastructure for digital cultural e-Infrastructure development to develop In the framework of the DC-NET project, heritage, built on the facilities available an action plan that will strengthen their DESCRIPTION eight governments and research agencies from the e-Infrastructures sector. This cooperation at national and cross-border are working to reach an agreement on a data infrastructure should be able to level, in pursuit of a common objective: a Digital Cultural heritage has several exciting set of common priorities for digital cultural deliver services and facilities that enable European research infrastructure for digital and important challenges which need to heritage research and are discussing with researchers to treat collections of data cultural heritage. be addressed in order to increase the value e-Infrastructures to identify how best to from many heterogeneous data sources that society as a whole derives from its carry out this research using the National as a continuum, overcoming linguistic, www.dc-net.org heritage. These include: Research and Education Networks and the • Multilingual and multi-national other grid providers. collaboration. Combining national cultural heritage research initiatives to However, the scope of DC-NET goes European-wide and international scale; beyond the current ERA-NET project. antonella fresa [email protected] ICCU – Italian Ministry for Cultural Heritage and Activities
14 | EGI User Forum 2011 User Support Services • • • • • • FP7-EU project ‘DECIDE’ (Diagnostic Enhancement of 2. computationally intensive processing, CONCLUSION 3. image processing on patient images Confidence by an International Distributed Environment): residing locally, compliant with the The implementation of the DECIDE strict data-sharing hospital policies. infrastructure and service should be a national prototype of e-services for the assisted regarded not only as a step towards IMPACT streamlining and enhancing confidence diagnosis of neurodegenerative functional studies in early diagnosis of neurodegenerative The potential impact for clinical use and pathologies, but as a concept that can be research of the proposed e-Infrastructure successfully extended to other pathologies will be on a large scale by enabling and communities, not only in the field OVERVIEW will be gathered, to implement the clinicians from hospitals with no access to of neurology (e.g. brain cancer), but to e-Infrastructure in an innovative way; sophisticated computational algorithms, other organs as well (e.g. cardiology or Aim of the Diagnostic Enhancement of • Deploy on this e-Infrastructure a resources, and large sets of reference traumatology). An international approach Confidence by an International Distributed secure and user-friendly service for images to carry out analyses remotely will be especially beneficial in view of such Environment (DECIDE) project is to the early diagnosis and research on and efficiently. This will be reached by the extension in scope, as it will help bring design, implement, and validate a grid- dementia and other brain diseases, and use of a centralised user friendly web-grid together other groups who are working on based e-Infrastructure building upon exploiting large distributed reference service. The service will be accessed by the subject in different fields, sharing with neuGRID and relying on the pan-European databases of multimodal neuroimages; the medical community through a science them approaches and results, and making backbone GÉANT and the NRENs. Over • Validate the e-Infrastructure and gateway portal. the best of the state of the art in this field. this e-Infrastructure, a service will be service through application to real provided for the computer-aided extraction patient cases (Alzheimer’s disease, The project will define with national and www.eu-decide.eu of diagnostic markers for Alzheimer’s neurodegenerative dementias) and European regulatory agencies protocols disease and schizophrenia from medical validate the DECIDE model on and rules for the qualification of experts images. DECIDE will offer access to large cutting-edge diagnostic conditions enabled to use the service for clinical distributed reference databases (850 and (e.g. schizophrenia); diagnostic purposes. 2,200 datasets from normal subjects and • Propose a long-term vision for the from neurological subjects, respectively), sustainability of the e-Infrastructure The DECIDE consortium strongly believes high computation and storage resources and its extension to new communities; that a European, and even wider approach is (more than 1,000 CPU core processors ethical issues related to the needed to tackle the problem of Alzheimer’s and 70 TB of storage) and intensive management and exploitation of disease and neurodegeneration in general. image processing tools. DECIDE will be sensitive patient clinical data will be Furthermore the rapidly ageing society, that open to the implementation of future specifically addressed as well as Europe and other developed economies algorithms based on other neuroimaging the business model for the wider has to cope with, make the societal methods; in a longer-term perspective, the exploitation of the service; emergency of the phenomenon more infrastructure could be extended to other • Disseminate the results to promote dramatic and resource-consuming. In this research and diagnostic algorithms relating the adoption of the DECIDE perspective, the project consortium and to other diseases of the brain and other e-Infrastructure and service by the pilot infrastructure should be regarded as organ systems. clinical community at large. an extensible, proof-of-concept platform, whose usage should be extended to other DESCRIPTION The planned tools will extract (i) diagnostic communities in Europe. markers of established value in the medical The project objectives are: community and might be used immediately • Provide the neuroscientific and medical in clinical settings, and (ii) markers currently community with a dedicated grid- in an earlier stage of validation, that will be based e-Infrastructure building upon brought forward by DECIDE. the FP7 e-Infrastructure neuGRID (www.neuGRID.eu) and relying on the The DECIDE applications will be implemented pan-European backbone GÉANT and into a grid middleware that will allow: the NRENs; different competences and 1. authorisation and secure access communities (neurological research, to largely distributed databases for medical community, grid, networking) reference images, F ederica Tanlongo [email protected] GARR • Laura Leone [email protected] GARR
16 | EGI User Forum 2011 User Support Services • • • • • • Finally, the GÉANT Project (GN3) has a CONCLUSION GÉANT and EGI: enabling seamless distributed task dedicated to Liaison and Support computing infrastructures through high-speed, which focuses on providing assistance Computing grids and Research and and support to projects, initiatives and Education (R&E) Networks are increasingly dedicated networking organisations such as EGI, to facilitate and growing to respond to the evolving encourage a productive use of Research requirements of demanding scientific and Education Networks. and education disciplines. Their synergy is achieved by combining the power of The presentation proposed will describe geographically distributed computation the importance of the crucial relationship and the reliability of advance networks OVERVIEW DESCRIPTION between the network and the computing/ dedicated to the research community. application layers and how the GÉANT Over the last 15 years, science and Distributed computing over large project is working in collaboration with The work presented aims at describing research have become increasingly based geographical areas and dedicated EGI to constantly improve the research both the immediate and far-reaching on cross-border collaboration between Research and Education (R&E) Networks experience of scientists. effects of the interaction, discussing researchers across the world. Sharing have emerged to respond to the how advanced R&E networking provides experiences, resources and facilities is, in requirements of the most demanding IMPACT the reliable connectivity which enables many situations, the only way to ensure scientific disciplines by combining the a scalable, sustainable distributed grid sustainability. In addition, science, research power of distributed computation and the The presentation is mainly addressed at computing infrastructure in Europe and and education are making intensive use reliability of dedicated networks. EGI/NGIs’ grid computing developers, other parts of the world. Finally, the of high-capacity computing to model users and at site administrators, network submitted presentation will describe the complex systems, to process and analyse GÉANT is the pan-European data administrators and PERT (Performance role of the GÉANT Project’s dedicated experimental results, and to deliver them network dedicated to the research and Enhancement Response Team) engineers. task to Liaison and Support. This task in real time in different locations around education community. Together with is devoted to providing assistance the globe. Computing grids for e-Science Europe's National Research and Education Dealing with how the network layer is and support to projects, initiatives and and Research and Education (R&E) Networks (NRENs), GÉANT connects 40 integrated with the distributed computing organisations such as EGI in order to Networks have emerged to respond to million users at over 8,000 institutions facility, the work submitted aims at consider- facilitate and encourage productive use of the requirements of the most demanding across 40 countries. The resulting network ing the network as a stable, solid, yet the network. Some examples of support scientific disciplines to share and combine provides an ideal infrastructure for any dynamic and customisable resource to given to grid projects will be provided the power of distributed computation scientific and research application with support grid-enabled research communities. and the reliability of dedicated networks. demanding network needs in terms of The work presented aims at describing sustained bandwidth, low latency (or Finally, it will highlight how network-related this synergy, discussing how advanced one-way delay) and low jitter (latency or activities such as monitoring and performance R&E networking provides the reliable one-way delay variation). Grid computing measurements can improve the perception connectivity which is enabling a scalable, is probably one of the most interesting of the grid computing resource from the sustainable distributed grid computing and complex of these applications. The perspective of the end users (researchers infrastructure in Europe and across reliability of such a distributed computing and scientists). Checking the status of crucial the world. facility is in fact, directly related to the links between distributed computing and reliability of the connections among storage resources and regularly assessing computing elements, storage elements, the transmission cost of big data files resource brokers and grid user interfaces. across the network can effectively improve The footprint of the GÉANT and the NREN the overall computing operations within the networks, their coverage across Europe EGI community. and the international links to networks in North America, Latin America, Africa and the Middle East, the Southern Caucasus, Central Asia and the Asia-Pacific region all enable a seamless, reliable, competitive infrastructure for the highly distributed grid computation EGI/NGIs aims to provide. D omenico Vicinanza [email protected] DANTE
18 | EGI User Forum 2011 User Support Services • • • • • • Logging and Bookkeeping – what can it do for you? IMPACT CONCLUSION The main purpose of this talk is to trigger The grid environment has matured in discussion with the user community. recent years and essential functionality It gives an overview of existing usage of the services is, more or less, available. patterns involving the L&B service, The current development activities, emphasising less well-known use cases besides pursuing standardisation and and applications often well outside the harmonisation of parallel solutions, aim scope of computing job tracking. We invite at providing advanced functionality, which users and user communities to provide can be leveraged by the emerging grid OVERVIEW Starting with its recent release 2.1, L&B is their input. They are welcome to approach user communities. able to track native (i.e. non-WMS) CREAM the L&B team with their priorities and gLite's Logging and Bookkeeping service jobs. Experimental support for PBS jobs requests to support other interesting In this talk we present, besides a summary (L&B) has been with us for almost ten and Condor is also available. L&B can also use cases. of less-known usage patterns of the years as a robust monitoring service work with data transfers (e.g. sandbox service, a view on possible directions of which gathers, aggregates and archives transfers), and the job status information As a starting point for the discussion we further development of the L&B service information on infrastructure behaviour is enhanced with the state of associated present several topics L&B wishes to as we ourselves can foresee its advanced from the perspective of users' tasks, and data movements. We believe that a great address in the near future. Feedback on use. The talk is intended to trigger discus provides a single (per VO) endpoint for opportunity to provide L&B users with a priorities or possible extensions of the list is sion with the user community, which will accessing that information. Numerous more accurate and descriptive view of more than welcome. result in a more specific and eventually use cases have emerged over the years, their jobs lies in tracking the dependencies • interesting entities to be tracked, extended work plan, better aligned with some well known and widely applied in between computational and data transfer besides gLite WMS jobs (e.g. ARC the expected user needs. gLite-based grids, some not so widely jobs, either directly through job state L&B and UNICORE CE jobs, data transfers, known but possibly important for the queries or indirectly through monitoring SRM operations…), http://egee.cesnet.cz/cs/JRA1/LB user community. tools providing VO- or site-specific views. • the ability to track dependencies among such entities (e.g. a computa The current situation gives us an L&B can store application-specific infor tional job is blocked by the wait for opportunity to stop and take time to review mation (job annotations, metrics, or status) transfer of its input), L&B's potential for various usage scenarios as user tags. The tags can be queried, • the desired level of complexity of and to plan its continued development and used to build application-specific queries (all jobs of a particular user, the according to real-world needs. We would dashboards, as we have demonstrated user's jobs within a given time interval, like to involve current and potential L&B several times. We also provide support for failed/successful jobs on a given CE, users (end users, groups, VO managers, tracking generic user workflows currently up to full SQL/XQuery/etc., power on monitoring tool developers/designers) in used in medical image processing the job data), this process. To help this happen we give (a subject of a standalone contribution). • output formats to be supported an outline of current L&B features and use (Glue-conformant WS interface, simple cases (sometimes not widely known but We can also demonstrate successful key=value text format, human-readable ready to use or prototyped) and possible use of L&B to track different entities – HTML, etc), directions of further development. CA revocation lists. Individual CRLs • history of the data to be kept (a day, a are registered in L&B in place of jobs. week, a month, etc), DESCRIPTION Whenever the CRL is updated, it • types of aggregate information, e.g. is reflected in a state update of the average queue traversal time, job failure The most common user of L&B are the corresponding ‘job’ in L&B, which triggers rate etc., level of aggregation (per user, glite-wms-job-status and glite-wms-job- delivery of L&B notifications to the sites VO, grid service instance). logging-information commands, querying subscribed for receiving CRL updates. the status of jobs handled by gLite WMS. It The payload of the notification carries the is less known that monitoring applications actual CRL update then. such as RTM, Experiment Dashboard and the grid observatory heavily rely on L&B, too. Zdenek Sustr [email protected] CESNET • Jiri Sitera [email protected] CESNET • Ales Krenek [email protected] CESNET • Frantisek Dvorak [email protected] CESNET • Ludek Matyska ludek@civ. zcu.cz CESNET • Daniel Kouril [email protected] CESNET • Jiri Filipovic [email protected] CESNET
20 | EGI User Forum 2011 User Support Services • • • • • • Supporting Distributed Computing Infrastructures IMPACT CONCLUSION EGI is a major part of a larger GGUS has shown during EGEE and environment of Distributed Computing now in EGI that it has the capabilities of Infrastructures (DCIs) from within and acting as this integration layer between outside Europe. These infrastructures user, resource providers and technology rely on the same or similar technologies providers. Extending this support and tools. The workflows defined in EGI infrastructure to cover other DCIs as well for the communication between user would help to harmonise the European communities, technology providers distributed computing landscape, would OVERVIEW All NGIs need to interface their helpdesk and resource providers can therefore ease its multi- and interdisciplinary use and systems to GGUS. To provide an easy be extended and adapted to be utilised its extension beyond Europe. By establishing EGI.eu to create and main way for small NGIs to do this, the xGUS other DCIs. This would have the benefit tain a pan-European Grid Infrastructure helpdesk template has been developed. of providing the technology providers This presentation will give an overview of with the National Grid Initiatives as building Instances of this template, while being and user communities with one channel the support infrastructure currently in place blocks, Europe has taken a major step hosted and maintained by the GGUS through which they receive or submit and plans how to further improve, extend towards providing a long-term sustainable team, can be customised and updated by requests from or to the various DCIs. and consolidate it in the near future. production quality grid infrastructure for NGI staff. xGUS comes with the standard multi-disciplinary use by international, helpdesk features and the interfaces to Such a support infrastructure servicing national and regional user communities. GGUS already included. xGUS could also various DCIs would help to harmonise With the aid of the EGI-InSPIRE project, be of interest for user communities of the the landscape for scientific computing on EGI.eu coordinates between user EGI infrastructure wanting to structure their the European scale and could also ease communities, resource providers and internal user support workflows and to link the integration of extra-European DCIs. technology providers within Europe and them to the EGI support. A major benefit would be that the user with partners from outside Europe. communities would be provided with one Several workflows that are needed contact point regardless of which DCIs As all these stakeholders are structured between EGI and the technology providers they are using. in their own organisations or projects, are also realised through GGUS. Support it is necessary to enable formalised requests by users are analysed within EGI Currently, various projects exist that aim communication between them. Here, by the Deployed Middleware Support Unit at linking multinational grid infrastructures EGI.eu plays the integrating role. (DMSU) and are transferred by the DMSU from various regions around the globe to the respective technology providers’ to the European Grid Infrastructure. The In the area of user support the integration support teams, in case their problems are relationship between these initiatives and platform is the Global Grid User Support due to a bug in the middleware. EGI, once established and formalised, (GGUS). GGUS is used as the central needs to be aided by a technical infra helpdesk for EGI, as well as for linking Releases will be announced by the structure linking their regional support for support infrastructures of other projects technology providers by submitting a user, operations and technology issues into the overall EGI set-up. GGUS ticket of a special category. EGI to EGI. GGUS could also play the role of then uses the same ticket to communicate the integration platform for these initiatives, DESCRIPTION its feedback to the technology provider. by providing tools to set up the regional A similar workflow has been defined for support, as well as by linking it to EGI. As Developed during the EGEE project, middleware bugs found in production. a prototype a helpdesk for the EUMEDGrid GGUS has also become the central The bug will be reported in a GGUS ticket project has been implemented using the integration platform for user support for to the technology provider, who will then xGUS helpdesk template. EGI. Project-wide support is handled use the ticket to communicate the date through the system and it acts as a central and number of the release fixing the bug information hub for user, operations and in the same ticket. An additional benefit technology issues. Integrating existing of handling these diverse workflows in tools rather than replacing them, GGUS the same tool is the possibility of creating enables support teams from all areas to statistics and reports on the metrics of the communicate with each other. support process. T orsten Antoni [email protected] Karlsrühe Institute of Technology
22 | EGI User Forum 2011 User Support Services • • • • • • this work, HC can flexibly accept further validate release-candidates of the grid HammerCloud: an automated service for additions of plugins for new communities. middleware, and to compare data-access stress and functional testing of grid sites methods. Further, a subset of the ATLAS IMPACT HC functional tests are deemed as critical – consecutive failures of these jobs result The HammerCloud service empowers in HC taking action to blacklist the site site administrators to undertake detailed from receiving user jobs. While the site is studies of their site's capabilities without blacklisted, HC continues to send tests; requiring any VO-specific knowledge or when the jobs succeed again the site permissions. With only three clicks, HC is informed and they can reset their site OVERVIEW the site performance evaluations these users can schedule a test and shortly online at their convenience. tests enabled were critical during the thereafter performance metrics are Many research communities rely on EGI commissioning phase of many grid sites. In made available. CONCLUSION resources to process vast quantities of the end, these manual tests motivated the data rapidly. Performance-critical activities development of an automated service to The experiences of the ATLAS experiment In summary, HammerCloud is an such as this give motivation for tools which carry out the stress tests and present the in using HC demonstrate the potential of automated testing service which can be aid in the design and configuration of a grid results via a web interface. such a tool. Since late 2008, ATLAS has used for on-demand stress testing and site to ensure that its capabilities meet or invested more than 200,000 CPU-days continuous functional testing. Such a exceed the requirements of the foreseen The HC service is composed of a back-end processing HC test jobs globally. The tool has been demonstrated to be useful user applications. which submits and monitors the test jobs, primary focus of the ATLAS stress-testing during grid site commission and to evaluate and a user front-end which allows users to has been on optimising the data access changes to site configurations. HammerCloud (HC) is an automated schedule on-demand tests and to watch method at the sites. In particular, HC testing service with two main goals. First, the progress of running tests or review was used to compare strategies such as The LHC VOs have a strong interest in it is a stress-testing tool which can deliver completed tests. Jobs are submitted copying input files to a local disk (e.g. the service. ATLAS continues to use the large numbers of real jobs to objectively and monitored using GANGA; this tool's using lcg-cp, dccp, or xrdcp) against service for on-going stress tests and site aid in the commissioning of new sites and grid Programming Interface provides reading files directly from the site storage validation tests, while CMS and LHCb are to evaluate changes to site configurations. an efficient framework to develop a grid element using the local access protocol currently incorporating HC into their grid Second, it is an end-to-end functional service which needs the flexibility to submit (e.g. rfio, dcap, xrootd). During large- operations procedures. CMS has thirteen testing tool which periodically sends short arbitrary applications to any grid back-end. scale global stress tests such as STEP09, HC users who can schedule on-demand jobs to continually validate the site and HC was used to simulate the resource stress tests, and a migration to HC from related grid services. The web interface is developed using requirements of hundreds of real users a CMS-specific tool for functional testing Django and is designed to provide by delivering a constant stream of up to is planned. The LHCb instance of HC In addition to delivering the test jobs to common web views in a core HC library 15,000 concurrent jobs throughout a two has been used to validate the rollout of grid sites, HC provides a user-friendly while allowing Virtual Organisations (VO) to week period. Tests like this have led to new storage software at RAL, and further web interface to test results. The service customise their web views in VO-specific I/O optimisations in the ATLAS software integration to the grid operations is presents plots of job efficiency and many plugins. Example metrics provided by HC resulting in improvements to the overall job in progress. performance metrics to enable quick include job success rates, timings of the throughput on the grid. comparisons to past configurations and various steps of a grid job (e.g. preparing The flexible architecture of HC will allow between other sites. input files, execution, and storing output For functional testing, HC delivers future development of new plugins for files), and I/O metrics including storage around ten types of test jobs for ATLAS. research communities who have interest DESCRIPTION latency and throughput values. These tests validate not only the basic in the service. functionality of the grid sites, but are also HammerCloud was first developed for the The current users of HC include three LHC used to test remote database access, to http://hammercloud.cern.ch ATLAS experiment at CERN. Prior to the experiments. ATLAS is the heaviest user, turn on of the Large Hadron Collider (LHC), having used the service since its inception members of this research community in late 2008, while HC plugins for CMS were often participating in coordinated and LHCb were developed in 2010. The efforts to stress test the grid sites by transition of HC from a single- to multiple- simultaneously submitting their physics VO service required the generalisation of analysis jobs. Collating the results of these core components and the development multi-user tests was time-consuming, yet of a plugin architecture. As a result of D aniel van der Ster [email protected] CERN
24 | EGI User Forum 2011 User Support Services • • • • • • the HELIO services in a user-defined CONCLUSION Catering for different user profiles in the HELIO project fashion, and, 3. Direct access that allows the user to HELIO is now well into its second year of write code to directly use the HELIO development, most of the query services services. have been developed and connected to the HELIO Query Interface with success, All the different implementations of the enabling services that provide security. access layer connect to the services Processing and storage facilities are now through the HELIO API that includes the being prototyped and will be connected to stubs of the various services and offers HELIO in the next release in January 2011. OVERVIEW DESCRIPTION an optimal entry point to the services. In particular, the grid services are designed to www.helio-vo.eu HELIO, an FP7 funded project, offers a The architecture of HELIO is composed cope with the complexity stemming from set of tools to the heliophysics community by components arranged in multiple layers the negotiation between users and back- to support scientific research. HELIO that span the conceptual space from end policies. uses services specific to HELIO and the resources to the user. The services, also external services for storage and their interfaces and the back-ends (that Security is based on authentication tokens computation; the various services of which only some services rely upon) are of with a low and high level of security to cater HELIO is composed are orchestrated different kinds: for all the permutations of user profiles and through the means of workflows. 1. services that allow searches in external implementations of the access layer data sources; HELIO's architecture is complicated by 2. services that allow an easy access IMPACT a variety of factors mainly driven by the to diverse data sources: they offer a large variety of users it intends to support: layer of abstraction to the existing data The impact of HELIO covers two main while the normal users should be able to sources; aspects. The scientific aspect covers easily select pre-defined workflows, more 3. services that offer functionalities such the impact that HELIO will have on advanced users should be able to define as computation and storage; the heliophysics community. To allow their own procedures either as workflows or 4. services that offer functionalities that a profitable feedback with the users, as scripts and programmes. Also, scientists are of no direct interest to the user special events named CDAW (Complex and the services back-ends have very (Authentication, Logging, etc...) but that Data Analysis Workshop) are scheduled different approaches towards security and are important to the system; after every major release of HELIO. The privacy, all of which have to be catered for. 5. services that allow discovery and first such workshop was held in Orly in monitoring of HELIO services. November 2010 and showed a good To meet all these requirements HELIO is interest of the scientific community in the built on a SOA, multi-layered architecture To grant access to these services in features offered by HELIO. based on the following principles: service compliance of the HELIO principles, HELIO duality, workflow agnosticism, need-to- offers an ‘access layer’, an abstraction The other aspect is technological: some know policy, flexible deployment and used to describe the different ways in of the problems faced by HELIO are of a policy compliance. which the services can be accessed. wider interest as the need of negotiation between the needs of the different user There are three possible implementations profiles and the policies of the back-ends of the access layer: is common among infrastructures for the 1. The HELIO user interface, which allows scientific community. the user to invoke the various HELIO services independently or to connect them in patterns defined on the fly. The user interface also supports an instance of a workflow engine used to execute pre-defined workflows, 2. A desktop instance of a workflow engine used by the scientist to orchestrate Gabriele Pierantoni [email protected] Trinity College Dublin
26 | EGI User Forum 2011 User Support Services • • • • • • WeNMR: the tale of virtual research community IMPACT CONCLUSION in NMR and structural biology WeNMR will strengthen the European WeNMR has established itself as a lively ties with National Grid Initiatives, the new and growing Virtual Research Community EGI and PRACE initiatives toward an in the life sciences and structural biology effective sharing of the offered service. area. It already represents one of the Collaborations will be established with the largest if not the largest Virtual Research Asian, South-African, South- and North- Organisation (VO), accounting for about American grid initiatives, in close contact 20% of the CPU on the grid within the life with existing EC projects, to extend and science area. Its global character is well OVERVIEW 3. to provide support to software open the WeNMR Life-Science Gateway at illustrated by the international nature of its developers, users and other e-Infra a global level. WeNMR will further establish user community covering more than 30 Structural biology and life sciences in structure projects in an e-science and promote best practices in life sciences countries, including South Africa and Latin general, and NMR in particular, have knowledge and training centre, and offers (virtual) training services to America. To date, more than 10% of the always been associated with advanced 4. to foster the adoption and use of worldwide researchers. users come from outside Europe. WeNMR computing. The current challenges in e-Infrastructure on a global scale by has thus developed into a successful the post-genomic era call for virtual supporting a wide range of flanking Virtual Research Community in structural research platforms that provide the disciplines within the life sciences, biology for researchers from around worldwide research community with both 5. to operate and consolidate the eNMR the world. user-friendly tools, platforms for data grid infrastructure in line with NGIs and analysis and exchange, and an underlying the EGI, and to extend it to interoperate www.wenmr.eu e-Infrastructure. WeNMR, a three-year with other worldwide grid initiatives, European Commission co-funded project 6. to develop a model to ensure started in November 2010, groups different sustainability of the project. research teams into a worldwide virtual research community. It builds on the The computational grid infrastructure established eNMR e-Infrastructure and its is central to the services offered by steadily growing virtual organisation, which WeNMR; its operation, maintenance and is currently the second largest VO in the consolidation are therefore core service area of life sciences. WeNMR provides activities. The infrastructure is being an e-Infrastructure platform and Science expanded to worldwide grid initiatives, Gateway for structural biology. It involves which is an important factor to both researchers from around the world and increase the available computational will build bridges to other areas of resources and the user community. structural biology. WeNMR also operates a Virtual Research Community portal that will address the DESCRIPTION needs of the various user communities, from general public to experienced The main objective of WeNMR is to users and software developers. For establish an e-Infrastructure-based global each community, a number of services, virtual research community for structural information channels and communication biology in the life sciences. To this end six tools is being deployed. WeNMR also objectives are defined: aims at expanding the extensive human 1. to operate, maintain and further develop collaboration network successfully a user-friendly science gateway for the implemented within the e-NMR project NMR and SAXS communities, geographically, i.e. involving users and Alexandre Bonvin [email protected] WeNMR – Utrecht University • Chris Spronk [email protected] WeNMR – SpronkNMR • Marco Verlato [email protected] WeNMR – INFN • Antonio Rosato 2. to establish a virtual research platform other stakeholders from many new [email protected] WeNMR – CERM University of Florence • Harald Schwalbe [email protected] to serve as a digital knowledge countries, in size and scientifically by frankfurt.de WeNMR – Goethe-University Frankfurt • Ernest Laue [email protected] WeNMR – repository, data exchange medium, involving new scientific communities. Cambridge University • Geerten Vuister [email protected] WeNMR – Radboud University and forum for (interaction with) the The inclusion of the SAXS community Nijmegen • Gert Vriend [email protected] WeNMR – Radboud University Nijmegen • Dmitri Svergun user community, is a first step in that direction. [email protected] WeNMR – EMBL Hamburg
28 | EGI User Forum 2011 User Support Services • • • • • • repository, metadata and log files are processing and analysis. A suite for distributed execution of general purpose stored and location information registered applications on the grid for small- and mid-size VOs into the LFC service. Our suite can be seen as a light-weight general-experiment framework which IMPACT focuses on basic functionalities, designed specifically for organisations that cannot The prototype suite we have developed afford the use of the more specialised has been successfully used in summer HEP frameworks but that still require 2010 during a large Monte Carlo simulation an easy-to-use interface to the grid. production of SuperB. Customisation of the web-interface, the OVERVIEW requirements of many different Virtual bookkeeping database, job executable and Organisations (VO). The centralised access site has been site requirements are considered the key Many research activities from several fields, the CNAF computing centre, which has points to achieve the goal as well as small such as high energy, nuclear and atomic The system design includes a central EGI provided the web-interface, the submission installation and configuration footprint. physics, biology, medicine, geophysics service site providing the web interface point, the bookkeeping database and and environmental science, rely on data to job and metadata management tools. hosted the data repository. Fifteen CONCLUSION simulation and analysis tools, which are A database system is used to store the remote sites in Italy, France, UK, USA and high CPU-consuming and/or data-driven. environment-related metadata; it works Canada, which deploy three different grid The prototype suite has proven to be Sequential computation may require as a back-end for all the suite sub-services middleware, have also been involved. reliable and efficient although it still months or years of CPU time, so a loose- and communicates via RESTful protocol requires a careful initial configuration parallel distributed execution is expected with running jobs on remote sites. The More than 11 billion simulated events have and needs a deeper abstraction from to give benefits to these applications. In whole suite is based on standard grid been produced. Over an effective period the application specifics. fact, the large number of storage and services such as the Workload of four weeks, approximately 180,000 computation resources offered by a Management System as job brokering jobs were completed with an approximate Further development is needed to provide grid environment allows to consistently service and grid flavours interoperability 8% failure rate, mainly due to executable such an abstraction layer both in the web- reduce the amount of computation time by element, the Virtual Organisation errors (0.5%), site misconfigurations (2%), interface and in the bookkeeping database. splitting the whole task in several parts and Membership Service as authentication proxy expiration (4.0%), and temporary For instance, a user-defined database executing each part on a single node. and accounting system, the Large Hadron overloading of the machine used to schema should be made available by Collider Computing Grid File Catalog and receive the data transfers from the remote providing an admin web-interface that High Energy Physics (HEP) experiments Utils, GANGA as job management system sites (2%). The peak rate reached 7,000 allows the environment configuration, pioneered work on this but their results and the Storage Resource Manager as the simultaneous jobs with an average of table modifications and setting of display are hardly available to small and mid- protocol for data access. 3,500. The total wall clock time spent and logic preferences. Moreover a better size organisations that may have similar by the simulation executables is about configuration structure is needed in order computational requirements, mostly due A web-based user-interface has been 195 years. to allow site, job script, data management to the large amount of technical expertise developed for database interactions and executable customisation. At a higher needed. The goal of our work is to provide and job preparation; it also provides The distributed infrastructure and the level, the minimal set of grid services and a suite allowing an easy, quick and highly basic monitoring functionalities. The developed suite have been fundamental protocols must be redefined in terms of customisable access to the grid. web interface initialises the bookkeeping in achieving the SuperB production standard compliance, lastingness and database and provides a submission cycles goals. easiness, in order to expand the user base. DESCRIPTION interface allowing an automatic submission An authentication management layer will be to all the available sites or a fine grain The online and offline monitor included added in order to provide a direct access This work started from the need of the selection of job submission parameters. with the web-interface keeps the metadata to the grid resources from the suite. SuperB project (a HEP experiment) to information stored in the bookkeeping simulate detector systems. The submission workflow includes the database available for querying and further www.fe.infn.it/gridprod offline sites set up (job input file transfers The effort in developing a distributed to Storage Elements and VO software simulation production system capable installation), and the bulk job submission of exploiting multi-flavour grid resources via web tool suite. Jobs communicate resulted in a general purpose design status information to the central database; L uca Tomassetti [email protected] University of Ferrara and INFN • Armando Fella based on minimal and standard sets at job completion the output files are [email protected] INFN Sezione di Pisa • Eleonora Luppi [email protected] University of grid services, capable to fit the transferred to the central site data of Ferrara and INFN
30 | EGI User Forum 2011 User Support Services • • • • • • Worker node software management: IMPACT CONCLUSION the VO perspective Maintaining dependencies of software We have presented a solution that is a time-consuming task, but it is a very allows VO managers to maintain a stack generic problem at the basis. By pooling of software applications and libraries from a large available set called pkgsrc independent of what is provided by the [http://www.netbsd.org/docs/software/ WN software. packages.html] and building the tooling for installation and maintenance on grid Although we have a working prototype, sites, the problem is reduced to defining there are still some improvements OVERVIEW DESCRIPTION packaging instructions and dependency that we want to make to address the resolution for the domain-specific parts. following issues. Grid jobs often run software that is domain- The grid is a (potentially) heterogeneous Any VO that needs to maintain a local specific, VO-specific and sometimes even environment. Therefore simply distributing software stack can adopt this method. Many VOs in the EGI project might have user-specific. They depend on a software binaries is not always sufficient. We have similar software wishes, but because of stack beyond what is generally available therefore chosen to use a model where By transferring management of software communication barriers those similarities by default on grid nodes. During the Dutch binaries are compiled remotely per site. packages from the infrastructure providers might be unknown. We therefore have VL-e project an additional distribu- This assumes that all resources in one site to the VO managers, the responsibility some thoughts about a more collaborative tion of software packages was created, are homogeneous. for the maintenance of the stack now distribution model that enables reuse of the Proof of Concept (PoC), which is lies with those who have the proper porting effort. also distributed to the WNs on the BiG This very issue comes down to domain knowledge. Grid infrastructure (Dutch NGI). However dependency management. Our solution Our current implementation is only tested this was still very generic and it had very allows a VO manager to maintain a stack The toolset that we have developed within the infrastructure of one NGI. We long update cycles and needed to be of software libraries independent of what is offers centralised package management are therefore looking forward to testing our done by site administrators. For scientific provided by the WN software, so it can be complete with dependency resolution, software with a VO that uses resources experiments more specific stacks are updated asynchronously. without the need for administrator's from multiple countries. required with shorter cycle times and privileges. It uses the VO software area, with less (or no) participation of site Also, multiple versions of libraries can which makes packages available on all administrators. be maintained side-by-side without the worker nodes. interference through the use of the The toolset that we have developed Environment Modules package. Additionally, these methods are not limited offers centralised package management to VO managers. Individual grid users can complete with dependency resolution, With the help of the Environment Modules make use of the same tools to fit their grid without the need for administrator's package [http://modules.sourceforge.net] jobs with the right environment, although privileges. It uses the VO software area, we manage all the paths and environment they will have to do this on a job-by-job which makes packages available on all the variables that are needed so that the basis as they lack the privilege to make worker nodes. software can be correctly found and such environments persist between jobs. executed by the job.
Further tooling is developed to update, monitor and test installations at multiple sites, so the users can check the validity of all systems in a single view and decide which sites can run their jobs. All these tools run as normal grid jobs that are submitted to the site with the permissions of the VO manager.
M ark Santcroos [email protected] Academic Medical Center Amsterdam • Dennis van Dok [email protected] Nikhef
32 | EGI User Forum 2011 User Support Services • • • • • • The web interface that allows the site It was already evident, while using it at a A new integrated monitoring tool for admin to look at the status of the farm medium-large site made by 1000 CPU grid site administrator exploits newly available web technologies and 700TB of storage, that the system in order to give to the final user an is able to handle a huge amount of data advanced experience. without decreasing the responsiveness of the web interface. The monitoring IMPACT system also integrates several status views for each service that should be up and All information gathered in the central running in order to have normal operational database could be aggregated and conditions on the farm. This monitoring OVERVIEW agent provides information about: user DN, presented by users, host, processes system is capable (differently from other FQAN, grid-jobid, local-jobid, queue, local etc., in order to have a much clearer view systems already developed) to give a In this work we will show the development user, VO. of what is happening on the computing complete and aggregate view of the status and the work carried on in order to build nodes. Using this monitoring facility indeed of the farm together with the capability a monitoring tool that gives an aggregate Also the StoRM and the gridftp servers makes it far easier to track down misuse of to have historical information on each view of all the users’ activities on a given provide information about the file the computing facilities from a given user. observed metric. grid site. accessed both from the farm itself and/ The database was designed to speed up or from remote sites. Also in this case the the procedure of building web pages. This CONCLUSION The tool is able to show the job submitted monitoring agents provides: DN, FQAN, was obtained by means of an accurate by each user together with information name and path of the file, VO. For each file data movement between on-line and In every grid computing farm, there are about the file accessed on the storage accessed locally through lustre file-system, near-line tables and a deep study on how several services and each of those has system. Also in a farm with posix-like the local user that access the file, the node to speed up the query needed to build the different logging or monitoring system. parallel file-system, the tool is able to track from which the file is accessed, the pid interfaces. Using this monitoring tool it is It is important to have a unique point in down both SRM standard operation and of the process accessing the file, the also possible to keep under control the which the site admin could aggregate and the local ‘posix’ file access. name of the process accessing the file. disk space used by each VO or user. This analyse all the information in order to have monitoring infrastructure was already used a clear view of what is going on. We will put a particular highlight on how Thanks to sensors installed in all the nodes with success at the INFN-Bari site in order this monitoring system works in a mixed of the farm it is possible for the site admin to have statistics on CMS dataset usage: This is more important in those computing environment like a farm used both via grid to know each accessed file over a Lustre/ this is particularly useful in order to take farms that are shared among several VOs, and with local job submission. Moreover, GPFS parallel file-system. The monitoring decisions about dataset deletion. It is also in order to guarantee that activities of a VO it could easily work with different types of agent is as lightweight as possible in order easy to monitoring the usage of the Wide are not blocked or affected by other users. computing elements and batch systems, to run it every one or few minutes. Area Network bandwidth as it is possible as it is highly modular and customisable. to have a clear view of the users that are Using this tool the site administrators in By design the monitoring system allows transferring files into (or out of) the farm, the INFN-BARI farm were able to better This monitoring system will help the sys- the sys-admin to change, add, or switch off and it is also possible know which files are fulfil the user requirements in terms of admin to have a complete and detailed each plug-in used to find out the data. transferred. Looking at the job submission, performance and reliability and also detect view of what is happening with the using the web interface, it is possible to proactively problems and failures that computing centre. The database schema is built in order to find out information on how many jobs could stop or affect the users’ activities. keep track of the dependency between are submitted from a given DN, or from a DESCRIPTION several observed values: for example, it is given VOMS group, which executables are easy to match between the job running on running on the farm, and which nodes they The monitoring system was built starting a given node, and the files accessed by the are using. from different agents and monitoring user on that machine. In order to gather as services. There is a central database much information as possible and in order system that takes care of storing, to be easily adapted to new batch system, aggregating and presenting the information we use pre- and post-exec scripts. gathered from each monitored node. We have already developed sensors for In particular each computing element several services: LCG-CE, CREAM-CE, has its own agent in order to send the StoRM, Gridftp servers, Xrootd servers, information about the jobs; indeed, this Torque/Maui, Lustre. Giacinto Donvito [email protected] INFN
34 | EGI User Forum 2011 User Support Services • • • • • • Accounting and SLA support in KnowledgeGrid IMPACT CONCLUSION Malaysia by P-GRADE portal KnowledgeGrid Malaysia is used by several This talk explains the specifications universities and user communities and and implementation of the applied hence the balanced use of the grid among accounting system and the complex SLA these communities required the measure solution offered by P-GRADE portal for ment of the use of the grid sites as well as KnowledgeGrid Malaysia. This complex the provision of a guaranteed start-up time SLA allows the portal to calculate for the users. deadlines for workflow start-up before the submission of the workflow. Based on the OVERVIEW The SLA framework can offer users The solution is generic enough that it could calculated deadline, the user can choose if of P-GRADE portal a guarantee that easily be adopted for any gLite-based VOs he or she wants to run the workflow within KnowledgeGrid Malaysia is based on gLite. his/her application would start on the of the EGI community. the given deadline or not. If yes, the jobs of Since KnowledgeGrid Malaysia is used by KnowledgeGrid Malaysia before a given the workflow will be sent to high-availability several universities and user communities deadline. The features of this SLA solution Tier 1 sites. If not, the jobs will be excluded it was important to solve the problem are the following: from Tier 1 sites. The complex SLA of accounting and SLA support for the uses dynamic values for calculating users. The single entry point for scientists The application start-up deadline is the deadline formula. is based on P-GRADE portal and hence generated by the portal based on dynamic the accounting and SLA support is built values that originate from the system state into the portal as new portlets. The talk instead of being defined by the portal will explain how KnowledgeGrid Malaysia administrator. Several parameters are taken and P-GRADE portal was extended with from the environment dynamically, such as: accounting and SLA support. 1. Current number of users and application instances that run simultaneously DESCRIPTION within the portal and could delay the start-up time of the current application. The accounting portlet of KnowledgeGrid 2. The number of resources that are Malaysia portal presents accounting available in the system and can be information from the connected grid sites assumed that will operate correctly in on a per job and per user basis. The the near future. main feature of the accounting portlet is . 3 The length of the applications that are the traceability of consumed CPU time. currently running in the grid. In the accounting portal we can define two user roles: generic user and system The talk will explain the architecture of administrator user. The generic user is able the accounting and SLA support system to check CPU time consumed by his/her as well as their user interface within the jobs. The system administrator is able to P-GRADE portal. check CPU time consumed by any user belonging to the grid.
The solution is based on the APEL accounting system, an optional module used in some EGI VOs. APEL serves as a back-end service for the newly developed account portlet. APEL is capable of collecting resource consumption statistics from EGEE Computing Elements and publishing this information into a central database. APEL can be installed as an P eter Kacsuk [email protected] MTA SZTAKI • Zoltan Farkas [email protected] MTA SZTAKI • optional component for EGEE grids. Mohd Sidek Salleh [email protected] MIMOS • Nazarudin Wijee [email protected] MIMOS
36 | EGI User Forum 2011 User Support Services • • • • • • Data is uploaded via HTTPS request. CONCLUSION UNICORE accounting service for Each agent had its own certificate signed Belarusian Grid Infrastructure with root UNICORE CA. An agent's certifi The current version of accounting service cate should be saved on the accounting due to technical requirements had server. Otherwise, the service would not been implemented as a third party web recognise the agent and the connection application written in PHP, so it cannot be would be rejected. one of the UNICORE parts. The possible future of this project is to implement the IMPACT billing system as UNICORE official add-on using obtained experience with the full OVERVIEW The application server is based on Apache UNICORE accounting service is the developing guidelines compliance. httpd server with mod_ssl and mod_php solution of accounting problem in UNICORE UNICORE middleware is widely used in modules included. This solution makes grids. The service has been working in Anyway, the service is an essential part the Belarusian Grid infrastructure, but it possible to run the accounting service Belarusian grid infrastructure since 2009 of the Belarusian grid infrastructure, it it does not have any accounting and under Linux or Windows systems. Applica and has already provided much useful provides flexibility and will be especially usage data collecting functionality. At tion server is the main component with information to users and admins. important if it will be decided to provide the beginning of 2009, the development only passive behaviour. It does not make a paid access to grid resources. of a UNICORE accounting service was any network connections. It only responds With the help of the service, our users are launched to monitor the employment to two kinds of requests: able to monitor their own usage statistics http://account.grid.by of national grid resources. The service 1. user’s requests and and prediction in tables in charts, to generate provides various accounting information 2. accounting agents’ requests. reports, bills, and then print them. Admins from grid sites through a web-interface and They both communicate with the applica can do much more. They are able to helps users and administrators to know tion server using the HTTPS protocol. see the whole grid statistics and make about every job of every user on every decisions based on it. There are several site. The presentation is mainly focused MySQL 5.x is used as database server scenarios for example: on implementation and technical details of for accounting application. Data tables 1. One of the sites is under heavy load. the new service, but also describes some use InnoDB storage engine. Performance Admins will find that many users need benefits of using it in the Belarusian grid. tuning was done, but no extra patches an application installed only on that site were applied. To avoid data loss, binary and then will make a decision to install DESCRIPTION logs and everyday backups to external the application somewhere else too. storage were set up. For best performance, 2. One of the users generates a heavy load The service was developed with several a database server was set up on a and makes other users’ jobs have to requirements in mind: dedicated physical server. wait for their turn. Admins can make • Ability to record the actual time a decision to set up priorities or even of computing resources usage in An agent is used to extract accounting provide dedicated resources for active normalised units based on sites' information from different sources, match users. performance indexes it with XUUDB user record and upload • Ability to identify commercial software results to the billing server, which verifies Collected statistics help the developers usage the format of the sent data. It is possible to of the UNICORE resource broker (another • Ability to interact with different categories develop a custom agent's version for any project of United Institute of Informatics of users and provide relevant information source, by bringing information into the Problems) to tune their service to make • Securing all connections with user and recognised state. Each record of sent data load balancing in the Belarusian grid host certificates must contain the following information: more accurate. 1. task id; Basically, the accounting service consists 2. certificate id; of three main components: 3. task termination time and 1. Application server; 4. used resources. 2. Database server and 3. Accounting agents.
Andrew Lukoshko [email protected] United Institute of Informatics Problems of NASB
38 | EGI User Forum 2011 User Support Services • • • • • • EMI represents the challenge taken up by the four major European middleware providers (ARC, dCache, gLite and UNICORE) to work on the long-term sustainability of the middleware as an essential tool for European research infrastructures like EGI. Sustainability can only come if important aspects of simplification, standardisa tion and quality are addressed in a collaborative way. The first of three years of EMI has brought important achievements in the software engineering processes, the standardisation of the middleware and its functionality. The EMI sessions offer an extensive range of presenta- tions about the activities of the past twelve months. Improvements in the way the middleware is deployed and configured have been introduced. Standard guide lines from mainstream operating systems have been applied and better compatibility with off-the-shelf package repositories like EPEL has been introduced. Standardisation activities have brought consolidation in a number of important areas, like the new EMI Execu- tion Service providing for the first time common inter- EMI faces to the Compute Element systems; like the pervasive use of standard SSL technology; or advances in the definition and implementation of specifications like GLUE 2.0, Storage Accounting Records, WebDAV and NSF 4.1. There are technological improvements, like the extension and integration of MPI support across ARC, gLite and UNICORE and the introduction of solutions to long- standing problems in data catalogues synchronisation. At the core of these initial, but important results is the EMI software engineering process, based on standard ITIL methodologies and continuously monitoring the quality of software and procedures, collecting and analysing metrics, running tests and producing reports. It has been an important first year for EMI and it has laid the foundations for even more important achievements along the hard road of sustainable, reliable and easily accessible infrastructures. Alberto Di Meglio Project Director, EMI
• • • • • • What's new in EMI 1 IMPACT CONCLUSION EMI 1 represents a significant step forward This session highlights some of the main in creating a unified middleware distribution changes introduced by EMI 1. with enhanced standards-compliance, interoperability, ease of use and reliability of the EMI products.
OVERVIEW DESCRIPTION
The European Middleware Initiative (EMI) EMI 1: Kebnekaise introduces a number is a close collaboration of four major of changes which impact not only the middleware providers, ARC, dCache, functionality, but also the mechanisms gLite and UNICORE. for obtaining and installing the software, support channels, etc. The project aims to deliver a unified middleware distribution which will EMI 1 focuses on simplification and harmonise the various middleware adoption of established best practices. products and focus on standards, It introduces significant improvements interoperability, ease of use and in interoperability between the various reliability, while simultaneously improving middleware products, for example the functionality and responding to the needs support for standard authorisation of the user communities. patterns which eases interoperability of grid credentials. The first release EMI 1: Kebnekaise is due for release at the end of April. This session The process of obtaining and installing introduces some of the new key features of the middleware has also been simplified, the middleware, and explains how the new with a single repository for all of the version differs from the previous separate middleware packages, and EPEL-compliant middleware releases. component installation.
This session outlines the mechanisms for obtaining and installing the new distribution. It also introduces the new support mechanisms which the EMI project has put in place. The session will serve as an introduction to the hands-on EMI training workshops which will take place throughout the week and will cover issues relevant to all the EMI products.
Kathryn Cassidy [email protected] TCD
42 | EGI User Forum 2011 EMI • • • • • • EMI-ES: a common interface to ARC, IMPACT CONCLUSION gLite and UNICORE computing elements The EMI-ES uses the experience from The execution service agreement the existing JSDL, OGSA-BES and some represents a significant milestone for the related specifications, combining them into EMI project, which will enable transparent a single coherent interface. job submission to the different middlewares that make up EMI. Initial implementations The EMI-ES will allow transparent job are expected mid-2011. submission to the different middlewares, based on a common web service interface. https://twiki.cern.ch/twiki/bin/view/EMI/ OVERVIEW DESCRIPTION EmiExecutionService Further standardisation activities with the The EMI Execution Service (EMI-ES) The EMI-ES specification has been deve relevant OGF groups (PGI, JSDL, BES) are represents an agreement between ARC, loped by a team consisting of leading foreseen, in which this agreement between gLite and UNICORE for a common web developers from the three grid middleware the European players is expected to have a service interface to computing elements. systems ARC, gLite and UNICORE. strong impact. Conversely, we foresee This service deals with the submission and future modifications of the specification, management of single-job activities on a The specification covers roughly the both necessitated by external require computing element. same space as the existing OGSA-BES ments, and through the experience gained and JSDL specifications, i.e. creation and from implementing the EMI-ES specification, It covers important use cases from management of single activities on a and doing interoperability testing. both high throughput and high computing element. performance computing. However, there are evolutions and differences as well: • re-designed activity description XML schema, covering data staging, execution, runtime environments, parallel applications and a simple yet powerful resource description (slots, memory, etc.). It can be seen as a merger of the JSDL 1.0 schema and some critical and often-used extensions • support for bulk operations (e.g. bulk job submission) • flexible data staging model, synthesising the capabilities of the involved middlewares • simple state model • consistently uses GLUE 2 as the information model for resource and activity descriptions
The technical realisation of the interface will be done using standard SOAP web services.
Bernd Schuller [email protected] Juelich Supercomputing Centre • Balazs Konya balazs.konya@ hep.lu.se University of Lund • Aleksandr Konstantinov [email protected] Universitetet i Oslo • Massimo Sgaravatto [email protected] INFN Padova • Luigi Zangrando [email protected] INFN Padova
44 | EGI User Forum 2011 EMI • • • • • • A uniform MPI user experience with MPI-Start in EMI IMPACT CONCLUSION User communities have frequently stated The unique interface of MPI-Start that the support for MPI applications is simplifies the task of starting the jobs too coarse and unwieldy for their needs. by handling transparently the low level The processes of job submission and details of the different Local Resource management should be simple, more Management Systems, execution transparent and should support generic frameworks and file distribution methods. parallel job types. The most common batch systems and MPI implementations are supported OVERVIEW MPI-Start provides users with an The adaptation and integration MPI- by MPI-Start and its modular architecture abstraction layer that simplifies the Start in the middleware stacks of EMI allows the easy extension of the tool to Execution of parallel applications on a grid execution of MPI and other types of parallel was performed by using the particular support new implementations. environment is a key task which requires applications on grid resources. By using mechanisms of each middleware – the cooperation of several middleware a modular and pluggable architecture it Runtime Environments of ARC, Execution With the integration of MPI-Start into the tools and services. Current middleware manages for the user the details of: Environments of UNICORE and user three computing middleware stacks of EMI, stacks provide basic access to such • Local Resource Management Systems wrapper scripts in gLite. Therefore users users have a unified experience for running applications, which may be one of the • File Distribution get a uniform experience for running their their parallel jobs. reasons for its limited use. EMI faces • Application Compilation jobs with MPI-Start, while at the same time the challenge of providing extended and • Application Execution they maintain the interface for submitting The availability of multi-core architectures seamless access to the execution of MPI the jobs of their middleware. opens the possibilities of new types of jobs across the middleware it supports. Although MPI-Start is designed to be parallel jobs. MPI-Start’s ability to define This is being done building on an existing middleware agnostic, its original target In most cases the automatic environment the way the user logical processes project, MPI-Start, originally implemented was the gLite environment. In this work detection mechanisms of MPI-Start introduces the possibility of new types in the context of the int.eu.grid and which we have integrated it into ARC and provide support for the most common of applications in the tool, such as hybrid has been used in the gLite/EGEE grid over UNICORE middlewares using their specific MPI implementations and resource MPI/OpenMP applications. the last years. In this work we present the mechanisms, Runtime Environments of managers without any administrator effort EMI is putting on MPI-Start, that the ARC CE and Execution Environments intervention, thus allowing sites with little is targeted on the provision of a uniform of UNICORE. or no previous MPI experience to support abstraction for the execution of parallel these applications without major efforts. jobs and the integration of the tool with all The latest developments of MPI-Start the middleware stacks. have also introduced a new architecture The possibility of defining how user for extensions that allows sites and users processes are mapped to physical DESCRIPTION to modify the MPI-Start behaviour. resources and the modular architecture of MPI-Start, allows the execution of Middleware support for MPI applications is Moreover, MPI-Start now supports the basic new kinds of parallel jobs. OpenMP jobs usually limited to the possibility of allocating mapping of logical processes to physical are now supported using the common a set of nodes. The user still needs to deal resources by allowing the user to specify interface of MPI-Start. with low level details that make the task how many processes they want to execute non-trivial. Furthermore, the heterogeneity on each of the available hosts. of resources available in grid infrastructures aggravates the complexity that users must face to run their applications.
E nol Fernandez del Castillo [email protected] CSIC
46 | EGI User Forum 2011 EMI • • • • • • EMI Data, the unified European Data IMPACT CONCLUSION Management Middleware EMI-Data is aiming for a universal data This presentation will report on the work management system suitable for use in of EMI-Data and on activities planned for large and highly versatile infrastructures. the rest of the project. It will focus on the It combines the expertise of the teams, implications of EMI-Data work for EGI and working on well-established products, its user communities. This contribution which have already been in production will also include details on those EMI- for many years. Under the financial and Data activities which do not have their organisational umbrella of EMI, those own presentation. OVERVIEW DESCRIPTION teams now have the chance to build a complete system, covering the needs https://twiki.cern.ch/twiki/bin/view/EMI/ At the time of the Vilnius event, one third of Besides improving individual components, of infrastructures like EGI and PRACE EmiJra1T3Data the EMI project life will have elapsed. Although EMI-Data has identified about a dozen and to attract new customers for the EMI is built upon a rather complex structure, cross middleware activities aiming to EMI distribution. our customer infrastructures can expect guarantee interoperability and to make that all the administrative gears are finally EMI data components compatible with interlocked, that realistic plans exist for common standards. This includes work future software development, maintenance on the SRM specification as well as on and deployment and that significant work its security mechanism. In terms of data has been done to achieve the goals access standards NFS 4.1 and WebDav planned for the first 12 months. will be offered by EMI storage elements. In addition to the work on interoperability of This presentation will focus on the work components and middlewares, flaws in the done within the data management part design of the currently deployed systems of EMI (aka EMI-Data). This includes, will be fixed as there is the problem of the but is not limited to, the improvement of inconsistency between the name spaces of existing components, the integration of file catalogues and storage elements. Pre- middlewares, the work on cross platform EMI activities like the migrating from GLUE projects to enable interoperability and 1.3 to 2.0, as well as the integration of the to provide standard interfaces to make Argus authorisation framework, will be EMI data components compatible with finalised within EMI-Data. Other pressing corresponding industry products. topics, like the specification and application of a common accounting record for data and a common monitoring framework will be tackled. Whenever possible, EMI-Data will use existing standards or will feedback specifications to standardisation bodies, like the Open Grid Forum. EMI-Data takes great care that modifications in specifications will be communicated and agreed on by grid frameworks outside of EMI, e.g. the US Open Science Grid.
P atrick Fuhrmann [email protected] DESY
48 | EGI User Forum 2011 EMI • • • • • • Supporting SRM over SSL IMPACT CONCLUSION Allowing clients and servers to use the Using the SRM protocol over standard SSL standard SSL protocol brings several brings several advantages. In this paper, benefits. It allows storage software we discuss these advantages, present the providers to choose which security progress in supporting SRM over SSL and implementation to use, potentially detail the future plans. bringing performance gains. It also allows better tools for diagnosing problems and consolidation of software towards OVERVIEW DESCRIPTION standards: one of the aims of the EMI project. SRM v2.2 is a standard protocol that Within the EMI project, work has started allows end-users not only to manage their to first understand how SRM may be used stored data, but also to negotiate transfer over a standard SSL transport and then protocols between data exchanging to extend the existing SRM clients and parties, independent of which software is servers to achieve this. Since SSL does not deployed. Storage software that supports provide a delegation mechanism, this work SRM v2.2 is widely deployed within WLCG: also involves establishing an alternative there are currently over 400 SRM v2.2 delegation mechanism for SRM. This work endpoints registered. is conducted in collaboration with the SRM software providers outside EMI. The SRM protocol uses Globus' Grid Security Infrastructure (GSI) protocol as both a secure transport for sending messages and mechanism to allow delegation. Delegation is the process where an SRM server obtains a short-lived credential from a user, allowing the server to act on behalf of that end-user. Such delegation is required for some SRM operations.
GSI has several drawbacks: it isn't a standard, there is only one provider of GSI software libraries, and it is incompatible with SSL, the current standard secure transport. This leads to duplication of code, as some grid software uses SSL while others use GSI.
P aul Millar [email protected] DESY • Alex Sim [email protected] Lawrence Berkeley National Laboratory • Jean-Philippe Baud [email protected] CERN • Ricardo Rocha [email protected] CERN • Giuseppe Lo Presti [email protected] CERN • Patrick Fuhrmann [email protected] DESY
50 | EGI User Forum 2011 EMI • • • • • • The ARC data client in EMI 1 IMPACT CONCLUSION The libarcdata2 library and the arc* data The ARC data clients can be used as a clients replace the ng* data clients currently general-purpose data client for all grid- in production and will be available in the related needs. The architecture, status EMI 1 distribution. and plans for the ARC data clients will be presented and usage examples will be shown.
OVERVIEW DESCRIPTION
The libarcdata2 library and the The libarcdata2 library has a modular corresponding arc* data clients are structure to support various data transfer successors of the widely used ng* data protocols. The core libarcdata2 library clients and provide a uniform way to move builds on standard libraries and does data from source to sink, with support for a not introduce any additional external number of data transfer protocols. dependencies. The plug-ins (DMC’s) for specific data access protocols can however have various external dependencies. This separation of external dependencies from the core library minimises the prerequisites for an ARC installation, while allowing the support for additional access protocols requiring special dependencies to be installed by those who need it. Most of these components and the clients are also available on several platforms (Linux, Windows, Mac, Solaris), and libarcdata2 also has API’s for Python and Java.
Jon Kerr Nilsen [email protected] UIO
52 | EGI User Forum 2011 EMI • • • • • • Besides the intense development work CONCLUSION Parallel NFS: EMI data in competition for DPM and dCache in this field, dCache. with high-end industry storage solutions org has been putting significant efforts The just-in-time availability of pNFS in into testing of pNFS with available Linux industry (IBM, NetApp, Oracle) and Open clients. A small but realistic Tier2 has been Source products (Linux kernel) allow EMI built including CPU and storage resource to provide professional storage elements, at DESY. pNFS client server interactions which can easily compete with expensive have been tested with plain I/O, ATLAS and high-end storage solutions. The discomfort CMS analysis software and with specific of currently deployed systems, in terms of setups provided by the ROOT team. product dependent protocols and client OVERVIEW DESCRIPTION DPM will catch-up in terms of testing software which needs to be installed and by making the DPM/pNFS implementation maintained, will be overcome. Significant Providing storage solutions for large For the last years, the Centre for Information available to DPM-sites in the UK for testing of pNFS in EMI data products as international e-science infrastructures, Technology Integration (CITI), as part of the community feedback. well as in industry products is in progress like EGI, is an interesting challenge, as University of Michigan has been working and more results will be presented at the such infrastructures are supposed to on a specification of a successor of the IMPACT time of the Vilnius meeting. support a large variety of diverse science overwhelmingly successful Network File communities. Most of those communities Protocol (NFS), focusing on today's needs In current deployments of EMI data storage www.dcache.org/manuals/2010/ have well established but unfortunately in terms of speed, reliability, security elements, specific proprietary software 20101102-hepix-patrick-nfs41.pdf different patterns with respect to data and modern distribution pattern of data has to be installed and maintained on data management and data access. The only sources. The effort was supported by client nodes. Moreover for RFIO, dCap and way to tackle this problem, in a generic the major storage solution vendors, others, libraries have to be linked with the way, is to provide industry standard including Sun Microsystems, IBM, NetAPP, applications, which makes this approach mechanisms in these areas. Besides Pansas, Mircosoft and dCache.org, as useless for communities where the source other solutions, the data management an unfunded partner. This work resulted code of the application is not available. For part of EMI is approaching the idea of in a specification which is now being other products, data clients might even standards-in-storage by offering access to implemented by all participants of the effort. have to be purchased. (e.g. GPFS). its storage elements through the NFS 4.1 (pNFS) protocol currently being adapted Parallel NFS (pNFS) is a part of the NFS With the common availability of pNFS in by leading industry storage providers. (e.g. v 4.1 standard that allows clients to access EMI storage elements, client software will IBM, Panasas, NetAPP etc.). This approach storage devices directly and in parallel. The be provided by the OS vendors, which makes EMI data solutions competitive pNFS architecture eliminates the scalability will free resources in EMI and very likely with expensive industry solutions and as and performance issues associated with in the infrastructure support (EGI) as well. a side effect, EMI and the corresponding NFS servers in deployments today. This Furthermore, with pNFS, the repository infrastructures are no longer in charge of is achieved by the separation of data and of EMI storage elements can be mounted maintaining the data access software on metadata, and by moving the metadata into the file system as any other local their data client entities and with such, server out of the data path. storage resource. Any unmodified resources are freed. application will have access to data using The design of pNFS perfectly matches the plain POSIX I/O, an essential requirement design of the storage elements provided by for non-HEP communities. EMI. dCache, being part of the CITI group for some years, already offers a production Last but not least, pNFS enables EMI and version of pNFS with its current releases. with that EGI, to provide storage solutions DPM provides a beta version of pNFS which can easily compete with expensive available for early testers. A production industry products and can be used as a release is expected before EMI-2. StoRM drop-in replacement. will provide pNFS as soon as IBM makes pNFS available for GPFS which is expected in 2011.
P atrick Fuhrmann [email protected] DESY/dCache.org
54 | EGI User Forum 2011 EMI • • • • • • gLite File Transfer Service in the EMI data area IMPACT CONCLUSION With the new FTS and LCG_Util, EMI will We have proposed redesigning FTS and (in have a scalable set of data transferring the long run) merging its functionalities with tools with simple configuration, lower LCG_Util using standard building blocks. administration burden and better usage We believe that the new FTS would extend of storage and network resources. FTS the usability of our data transfer services, will serve the needs of T2-T3 sites as well. lower administration and maintenance In addition, ordinary users (not only VO costs and serve a much broader set of use admins) will also be able to schedule file cases according to the latest requirements OVERVIEW DESCRIPTION transfer jobs. of the experiments.
We propose a new gLite File Transfer We summarise the limitations of the current Service (FTS) that implements the latest FTS architecture which pre-supposes requirements of experiments and serves a strict tier model which the experiment the needs of smaller (Tier 2 and Tier 3) workflows have now largely moved sites as well. beyond. To address these problems, to enable user initiated workflows and to promote efficient resource usage by a robust and easily maintainable file transfer service, we propose redesigning FTS based on standard building blocks, in three steps.
Step 1: remove the channel model, taking into account the published load on an individual SE and the network, while supporting other database back-ends
Step 2: introduce a pull model of job processing based on messaging. Main advantage: a load balanced service where the client does not need knowledge of which FTS to contact. The current push model will be maintained for backward compatibility.
Step 3: Merge FTS and LCG_Util function alities. Besides the advantages in resource handling and unified UI, FTS users get access to the catalogue interface as well.
Zsolt Molnar [email protected] CERN
56 | EGI User Forum 2011 EMI • • • • • • Storage catalogue consistency IMPACT CONCLUSION Within the EMI project, finding a way The EMI data group has proposed a draft for Storage Elements and catalogues specification and a prototype system to interact in real time in order to keep built on standard building blocks with the their metadata synchronised will give a goal of making Storage Elements interact higher degree of robustness to the data for metadata synchronisation purposes. infrastructure. The deployment of such a The ideas behind these steps will be common synchronisation subsystem will presented, together with some possible help the overall data management systems enhancements and directions. OVERVIEW DESCRIPTION in giving a better service to the users.
This talk describes the status of the work The EMI Catalogue Synchronisation performed in EMI to find practical solutions Working Group has produced a prototype to the problem of maintaining consistency system built on standard messaging tools in the grid data catalogues. This activity that successfully integrated with LFC and aims to provide grid Storage Elements and DPM. This was used as a proof of concept Central Catalogues the means to interact to show that the objective can be achieved and keep their metadata synchronised. and extended to the other Storage Element implementations in EMI. Now the activity . of the working group is headed towards developing a more comprehensive specification of the messages that are needed to fulfil the objective, and towards starting to address the aspects related to security and deployment.
The initial prototype has generated new ideas that may yield a higher degree of integration with the whole distributed storage infrastructure.
Fabrizio Furano [email protected] CERN
58 | EGI User Forum 2011 EMI • • • • • • Software quality assurance in EMI IMPACT CONCLUSION This presentation describes how adopting This presentation is a good opportunity standard software engineering and quality to understand how software quality assurance processes within academic assurance has been implemented within research environments can become a EMI, understanding the procedures, roles realistic and beneficial activity. The work and responsibilities involved in the software performed in EMI is not only targeted at lifecycle and the monitoring activities improving the quality and timeliness of carried out to control the quality goals. software delivered to the infrastructures, OVERVIEW with the documentation governing the but also at establishing a more standard software lifecycle. SA2.2 also works very and efficient software production lifecycle In EMI, the definition and monitoring of closely with the quality control tasks in as the base to make the development and software quality assurance is the respon development, maintenance and support maintenance of software more sustainable. sibility of the SA2.2 team. This talk will activities. SA2.2 defines the release criteria present the procedures defined to carry for the EMI software components that . out this activity and how to monitor their will be checked by the quality control application within the EMI software lifecycle. tasks. SA2.2 produces regular periodic reports evaluation the application of the DESCRIPTION existing procedures and the availability of proper documentation for each software SA2.2 is the task responsible for defining component. This contribution will present and monitoring the software quality the main aspects of the EMI SQAP and assurance process within the EMI project. will focus on those guidelines related to One of the documents produced by development, change management and SA2.2 is the Software Quality Assurance release of software, providing information Plan (SQAP) which specifies the tasks useful for software developers and testers. that are needed to ensure quality, the responsibilities for quality monitoring, the required software component documen tation and the procedures that will be followed to manage the software process. In practice the SQAP specifies the manner in which the EMI project is going to achieve its quality goals that are defined in terms of quality factors like usability, maintainability or portability. In order to better describe the software process, a set of guidelines have been also written giving specific details to product teams in various stages of the software lifecycle: integration, configuration, packaging, testing and certification, change management and release management. The process of defining the guidelines has involved experts from the middleware distributions and the other activities within the project. SA2.2 is responsible for managing changes in these guidelines and announcing them to the project members to make sure all product teams are up to date M aria Pradillo [email protected] CERN
60 | EGI User Forum 2011 EMI • • • • • • Software quality assurance tools in EMI IMPACT CONCLUSION The EMI project started as a consolidation This presentation shows the status of this activity of the four European middleware activity by providing a description of the distributions, namely ARC, dCache, gLite tools selected, their integration and their and UNICORE. Each of these distributions use in the project. Starting from the build started several years ago from specific system, going through ticketing systems, needs dictated by their own scientific quality assurance tools, testing infra communities. Moreover the distributions structure and dashboards, ending with the developed along the years in different and production package repositories, every OVERVIEW DESCRIPTION independent ways aiming at solving different aspect of the software engineering process scientific problems and adopting different and quality assurance is illustrated from the One of the important goals of the EMI In EMI, the selection, integration and strategies on software architecture and tools point of view. project is the unification and standardi maintenance of quality assurance tools, design. As a result, their culture, processes sation of the software engineering and starting with the identification of the and tools appear today diverse and in some quality assurance process. This can be tools initially used by each middleware aspects, contrastive. The unification of made possible only by providing a single distribution and the identification of new this heterogeneity to provide a single and and unified tool chain throughout the tools which may be required by other unified quality assurance infrastructure is project which will be used as a standard activities is the responsibility of the SA2.4 the first challenge the quality assurance environment to enforce common team. The main functions of this team are: team must face. procedures and constraints. This tool chain 1. to support all the required tools and needs then to become a stable, trustable service installations for the duration and reliable foundation for the production of the project of high quality software throughout the 2. to plan the introduction, change and project lifetime. removal of services as smoothly as possible providing a stable and reliable infrastructure 3. to integrate different tool chains used by each distribution in a single tool infrastructure able to provide at least the same functionalities 4. to enable continuous integration and testing processes by selecting and maintaining tools and resources for building and testing software either within the project or in collaboration with external resource providers
L orenzo Dini [email protected] CERN
62 | EGI User Forum 2011 EMI • • • • • • Definition, reporting and analysis IMPACT CONCLUSION of EMI software metrics Attempting to collect metrics changes the Metrics are a vital part of the quality landscape of the working environment of assurance process for large scale projects a project. The inability to generate certain such as EMI. Results will be presented for required metrics has resulted in project priority, severity, density and delays for procedural changes, which then allowed the bug-trackers of each middleware. The the metric calculation to progress. The comparison of results per product team, knock-on effect therefore, is that metrics products, components inside products are automatically causing the cohesion gives an overall view of how the project is OVERVIEW was generated to map each bug-tracking of the four middlewares into one distinct progressing, whilst highlighting the areas/ system into one unique format which can middleware. The metrics will help pinpoint products that require fixing or streamlining. Within EMI, the definition, continual be understood by the metrics collection problems on a weekly basis as well as Box and whisker charts showing medians collection, reporting and analysis of soft mechanism. Additionally, many procedures a quarterly basis, by generating reports and quartile ranges as well as histograms ware quality metrics are handled by the have been implemented to ensure that for the EMI Executive Management Team showing means resulted in plots that were SA2.3 team. The team provides information software metrics are produced coherently (EMT). The definition of carefully chosen more informative and easier to analyse. to the project’s management bodies on the across all products. metrics such as: success builds per The uniformity required to generate metrics status of the software as an instrument to product team, average time to fix a top across many products and product teams taking corrective actions. The success criteria Practically speaking, the visualisation priority bug, number of untouched open gave rise to changes in procedure for each of this task are the regular production of of specific statistics in metric reports is bugs older than two weeks old, delays middleware. In summary, metrics help reports and their use to identify areas for currently used to present a very precise in release, back-log in bug handling per the middlewares to highlight problems in improvement, as shown in this contribution. overview of specific parts of the process. quarter, bug severity density, FindBugs specific products and enforce bug-tracking For example, with careful construction plots per product team are all helping uniformity and correct usage. DESCRIPTION of suitable metrics in EMI, it is not only to produce a more heavily tested, possible to say how many components stable middleware. Metrics in EMI are to be generated are building (as in more build systems), but internally per month and visibly per it is also possible to suggest the parts of quarter. The metrics are broken into two the project that are creating bottlenecks main groupings: process metrics and in the release process. For instance, a software metrics. The process metrics normalised histogram presenting the relate to each middleware bug-tracker, number of successful/failing software the GGUS bug-tracker and each software components in a product team, singling release. The software metrics relate to out the external dependencies and the building, testing and static code analysis. dependencies provided by other product Exceptions to these two groups appear teams components, will help define the when a metric is confined to a software sources of any delay. component, usually normalised according to the number of bug-tracking defects open against that component.
This work is seemingly easy at first glance, but is complicated by the diverse number of software languages in the EMI middleware. In addition, each of the four middleware involved in EMI uses its own bug-tracking system in very different ways. However, metrics must be calculated in exactly the same way across multiple product teams, product components and bug-trackers to compare like with like. For this reason, an XML schema E ammon Kenny [email protected] TCD • Gianni Pucciani [email protected] CERN
64 | EGI User Forum 2011 EMI • • • • • • (requests for specific testbeds defining new implementation challenges. On the EMI Integration and large scale testing infrastructure use case and target scale) and supply other hand, the large-scale infrastructure (collaborative effort from a community of model, though still under validation partners available to participate to specific about effectiveness and sustainability, tests campaigns) approach was adopted brings both a flexible approach, adaptive allowing to set up testbeds fitting the to variable testing conditions required specific test use case needs. and a useful communication feedback channel with enduser communities. IMPACT Finally a less visible but equally important outcome of the work was the contribution OVERVIEW Integration testing is the part of testing and The EMI project is a close collaboration of to identification and analysis of practical certification process of a software product major European grid middleware providers issues concerning the merging and Releasing software with production where the product functionalities and (ARC, gLite, UNICORE, dCache) whose harmonisation process among different level quality, i.e. satisfying end user expected behaviour are tested against mission includes the harmonisation and middlewares converging into EMI. As expectation in terms of functionality and other related grid service components. In evolution of the middleware, ensuring it an example consider cross-middleware performance, is the final goal of every EMI’s decentralised software development responds effectively to the requirements resource discovery and publication or software collaborative project. Software model, testing and certification are charged of the scientific communities relying on common authentication/authorisation continuous and effective testing is a key to different product teams, each responsible it. Therefore the impact of setting up a framework for testbed accessibility. step in the software production process for one or more software components. centralised EMI testing infrastructure in the to match quality targets. The presentation Taking place after functionalities, testing of project’s first year is two-fold: CONCLUSION will describe infrastructural resources products in insulation has been success 1. providing the testing and certification provided by the EMI quality assurance fully carried out. Integration testing then infrastructure, a central service for The main outcome of the work was the work package for the project continuous represents the first centralised point of the project, and model definition and implementation of integration and large scale acceptance contact among different products. There 2. somehow having a living laboratory EMI’s centralised testing infrastructure testing. The two testing infrastructures fore the solution focused on permanently where actual integration and together with the operational resources follow different implementation approaches deploying instances of both production interoperability problems can be necessary for its daily usage and reflecting the different testing goals. and release candidate versions of all practically experienced, identified continuous update. Also an ‘on demand’ Integration testing testbed must provide products components for every EMI and approached. model for implementing large scale a snapshot of release products to allow release. Hence a continuously evolving acceptance testbeds has been defined to for testing correct mutual interaction of snapshot of all released and upcoming Considering the first, and the more visible match specific testing use cases with end- all related products in the release. Large versions of all EMI products is provided to of the two mentioned outcomes, the user community resource providers. scale testbed must reproduce end-user product team developers. For flexible and infrastructural and operational resources production environment for specific test dynamic creation of testbed subset views, put in place consist of: Among the outcomes of the work it is use case, therefore requiring an on- configurable central information system 1. hardware resources amounting to useful to mention the identification of demand model where end users are instances publishing resources in the roughly 100 server instances deployed key open issues to be addressed for involved in specific testing scenarios. testbed were set up. across five geographically distributed assuring the testbed homogeneous sites including monitoring, resource usability by all middlewares converging DESCRIPTION A different model was adopted to imple publishing and testing security utilities; into EMI. Future work will then focus on ment Large Scale testbed infrastructure, 2. support handling, activity coordination tracking and integrating in the testbed the The set up of integration and large scale necessary to perform acceptance, inter infrastructure and documentation; solutions to these issues, among which acceptance testing infrastructures for EMI operability and scalability tests. Here the 3. model definition and collaborative effort the most relevant with respect to direct projects was a three step process: main objective is to stress the service coordination channels set up for the effort contribution is the adaptation of the 1. testing objectives and requirements under test reproducing conditions as large scale testing infrastructure testbed infrastructure implementation to collection; similar as possible to real production the planned automation of testing process. 2. implementation model definition; environment in terms of geographical Notice that also providing a testing 3. actual set up and revision in distribution or different deployment infrastructure for both intra- and cross- https://twiki.cern.ch/twiki/bin/view/EMI/ collaboration with testbed users. scenarios or scale. Therefore a demand middleware integration testing has posed TestBed
D anilo Dongiovanni [email protected] INFN
66 | EGI User Forum 2011 EMI • • • • • • The gLite Data Management continuous integration IMPACT CONCLUSION and testing process Currently, the DM product team runs A procedure to continuously verify the every night this procedure on the latest integrity and stability of the products code produced by developers. As such, being developed in the gLite DM PT has all changes committed by developers been put in place and is being exploited during the day are immediately built and on a daily basis. This procedure relies tested every night. Furthermore reports on and integrates a number of existing are automatically produced and sent to tools making more efficient and easier developers informing them on the impact the development and release of new gLite OVERVIEW ia32_gcc346, slc4_x86_64_gcc346, sl5_ of their changes. Potential problems are DM releases. ia32_gcc412, and sl5_x86_64_gcc412). discovered in advance and the code The continuous execution of builds and quality increases. tests on newly developed software is a key The automatic execution of these tasks activity for any software development team. is managed by a new tool, called SAKET The same process can also be executed To properly implement such procedures it (Swiss Army Knife for ETICS Test), which during the release of new software is fundamental to automatise all tasks not only orchestrates the entire process versions. In this case the advantage is related to software building, deployment, but provides reports to the product team not on the continuous verification of and testing. members. These reports are sent by email the code but rather on the easiness and archived in a web accessible location to verify if the new release passes all The gLite Data Management (DM) product for historical reference. SAKET also relies existing tests. This directly contributes to team has put in place a continuous on other existing tools: a decrease in the time needed to complete integration and testing procedure to • ETICS: to run builds and tests on the release process. properly support the development remote infrastructure nodes; and release of the product team • YUM: to make available the packages software products: DPM/LFC, FTS, generated by the build; and GFAL/lcg_util. • YAIM: to deploy the available packages and to configure the services; DESCRIPTION • YAIMGEN: to execute the pre-defined tests and to generate reports. A continuous integration and testing procedure must integrate the execution Besides these tools, the continuous of source code builds, the deployment of integration and testing procedure also packages, and the execution of different relies on a well-defined controlled types of tests under a common continuous environment to ensure the reproducibility execution plan. The execution plan of the build and test executions. This defined for the DM product team defines environment is composed by the ETICS the following tasks: building the source infrastructure nodes where the builds and code from SVN (taking code from three tests actually run and the EMI certification repositories: LCGDM, LCGUTIL, FTS); testbed which provides other external creating platform specific YUM repositories services, such as BDII. containing all generated packages (including meta packages which represent the services provided by the product team); verification of the correct deployment and configuration of all services; execution of a number of functional, regression, and stress tests; and finally the creation of reports for the product team members. This execution plan is executed n-times P edro Andrade [email protected] CERN • Oliver Keeble [email protected] CERN • on all supported platforms (currently slc4_ Alejandro Ayllon [email protected] CERN
68 | EGI User Forum 2011 EMI • • • • • • ARC tools for revision and nightly functional tests IMPACT CONCLUSION The functionality tests are periodically We present automatic tools which are running carefully chosen test scenarios used to detect build and functional on ARC “nightlies” or on specific code errors in the early stages of the software revisions. The goal of these test scenarios development process. is to check a basic functionality of ARC clients and services. During these tests http://arc-emi.grid.upjs.sk/revisionTests.php we additionally check integration of clients and servers. The test scenarios, build and OVERVIEW DESCRIPTION functionality test results are stored in a data- base so they can easily be visualised and Identifying errors in the early stage of soft Automated builds enable us to track errors exported in any form using simple PHP or ware development ensures a lower cost which could break the integrity of the final python scripts. The request for a new test development process and higher quality code. For each code change committed to scenario is done through a simple web form. final software release. Automated testing ARC subversion the tool checks status of procedures in EMI performed during ARC following build steps: Autogen, Configure, middleware development are done by two Make, Make Check, Make Dist, and Build. independent tool sets. First is the ETICS The check is done for several platforms build system and second is the ARC auto (including Mac OS X) and subtrees of ARC mated build and functionality test system. sources codes.
Jozef Cernak [email protected] UPJS Kosice • Marek Kocan [email protected] UPJS Kosice
70 | EGI User Forum 2011 EMI • • • • • • EMI and dCache: the vanguard of standards adoption IMPACT CONCLUSION dCache's adoption of WebDAV means that By adopting standard protocols, such as end-users can access their data on their WebDAV and NFS v 4.1, dCache is allowing choice of platform, with their choice of access end-users to be agnostic to the nature tool. There are WebDAV clients for all major of the storage. This lowers the barrier for computing platforms, which are often new communities that need access to supplied as part of the operating system. large storage, allowing them to use grid resources without modifying their software. Several sites are deploying dCache with This work is already bringing benefits OVERVIEW DESCRIPTION WebDAV specifically to allow end-users to end-users. outside the HEP community to have dCache is a hugely successful storage The EMI project includes a programme access to large data storage; for example, software that is currently deployed in eight where the storage elements are adopting dCache and WebDAV forms the basis of out of 11 Tier-1 centres in WLCG and industry standards. Within this programme the Swedish National Infrastructure for providing over 40% of the total storage dCache has taken a trailblazer role: Computing (SNIC) national storage facility: capacity available to the LHC experiments. demonstrating the feasibility of adopting SweStore. SweStore aims to support With dCache as a founding member, the these industry standard protocols and non-HEP communities such as climate EMI project has great experience demonstrating their functionality in modelling, bioinformatics and bioimaging. in storage. production environments. With the availability of native support for The High Energy Physics (HEP) com- WebDAV is an industry standard protocol NFS v 4.1 in the various operating systems, munity has a tradition of building their that allows end-users to read and write a site-administrator can simply mount own protocols for LAN and WAN access data. dCache has been adopting WebDAV, the large data stored in dCache on local to data. This is because the existing allowing end-users to read and write analysis machines. Programmes running protocols have proven unable to sustain data over the WAN. Within dCache, on those machines may access data the required throughput. access via WebDAV is no different from like any other file, without modification. other protocols: end-users experience a This permits dCache to support analysis Unfortunately, these non-standard coherent namespace and are subject to work-flows that use software that cannot protocols introduce a major barrier to new the same set of access controls. be modified to support HEP-specific communities adopting grid computing. protocols, which permits analysis facilities These communities may have custom For local access dCache has implemented and compute farms to support a much software that would need to be made the NFS v 4.1 protocol with pNFS, one wider range of analysis activity. ‘grid-aware’; the end-users may be using of the first storage solutions to do so. commercial off-the-shelf analysis software Through the use of realistic tests, dCache Adopting NFS v 4.1 also brings advantages that cannot be modified. has demonstrated deploying NFS v 4.1 for to site administrators. Adoption of standard the high-throughput analysis work typical protocols allows a mix-and-match Recent developments in the industry of HEP communities. There are sites in the approach, where storage from different mean that a large storage system, such as process of rolling out NFS v 4.1 support. vendors may be combined, based on their dCache, must adopt industry standards relative benefits, to produce a coherent to remain competitive with solutions from storage solution. The standard also HPSS, NetApp, EMC^2, Panasas, Dell- provides better tools for monitoring. DDN, etc., without sacrificing throughput.
P aul Millar [email protected] DESY • Patrick Fuhrmann [email protected] DESY • Tigran Mkrtchyan [email protected] DESY • Gerd Behrmann [email protected] NDGF • Dmitry Litvintsev [email protected] FermiLab • Tanja Baranova [email protected] DESY • Christian Bernardt [email protected] DESY • Karsten Schwank karsten.schwank@ desy.de DESY • Antje Petersen [email protected] DESY • Per Lundqvist [email protected] National Supercomputer Centre in Sweden
72 | EGI User Forum 2011 EMI • • • • • • Proposing a Storage Accounting Record standard IMPACT CONCLUSION Storage accounting records, as defined The EMI data group has proposed a in the final StAR specification, will be standard for storage accounting records. implemented in ARC, dCache, gLite The ideas behind the StAR definition and and UNICORE, thus providing means to the envisioned implementation and usage monitor and account for distributed storage will be presented. in a standardised fashion.
OVERVIEW DESCRIPTION
To enable a shared storage infrastructure, The StAR definition describes a format, it must be possible to account for and which can account for resources report on the resources consumed by consumed on a storage system. The persons and groups in a common format. format allows for granularity on a storage While usage record formats for computing system level (e.g. separating disk and jobs have been defined by OGF (GFD98) tape consumption) as well as reporting and used for several years, there is no on both a per-user and a per-group basis. corresponding definition for storage The different properties enable a variety of accounting records. The EMI data group, aggregation scenarios. with data experts from ARC, dCache, gLite and UNICORE, has seen the need Due to a general approach the StAR and defined StAR – a common storage format is not restricted to usage in a accounting record to be proposed to OGF. distributed (grid) environment but can be applied in any storage environment.
Jon Kerr Nilsen [email protected] UIO
74 | EGI User Forum 2011 EMI • • • • • • Implementation of GLUE 2.0 support IMPACT CONCLUSION in the EMI Data Area The GLUE 2.0 schema was designed EMI has committed to the implementation to allow a more accurate and natural of GLUE 2.0 support across the relevant description of the resources available on services and clients. grid infrastructures. By supporting this, the EMI data team is promoting the optimal use of grid resources.
OVERVIEW DESCRIPTION
We report on the implementation of The EMI data area team is committed to support for the GLUE 2.0 information the implementation of GLUE 2.0 support model in the EMI Data Area. across the relevant services and clients. This has involved reaching agreement on a common EMI interpretation of the schema to ensure a consistent implementation across components. We describe the various tasks required to realise the GLUE 2.0 support and describe the status of the work. In addition, detail is given on validation, testing and rollout as well as plans to further exploit the potential of the new schema.
O liver Keeble [email protected] CERN
76 | EGI User Forum 2011 EMI • • • • • • Cloud computing has emerged in the last years as the new paradigm for the provision of on-demand computing resources. More generally, virtualisation is growing, in terms of use cases and interest from both user communities and service providers. Many user communities have very specific configuration requirements for their software. Using virtualisation the applications runtime environment can be ported through different sites. Virtual networks favour communities that have particular privacy requirements, by configuring Virtual Private Networks as an additional layer of security. Virtual resources can be accessed through the grid interface, as well as the cloud paradigm; the latter should focus on different use Virtualisation cases, for instance interactive applications as well as accessing SaaS (Software as a Service) resources. Infrastructure Providers can run their services over the cloud to improve availability, scalability, and and Cloud maintainability of their grid and non-grid services. Moreover, physical resources costs can be optimised by deploying services on private clouds. Data Centers can expose their resources as an ‘Infrastructure as a Service’ (IaaS) cloud, as well as deliver ‘Software as Computing a Service’ (SaaS). Generally they can achieve a better exploitation of their resources and can offer a diverse set of services, attracting in this way different user communities. For a grid site deployed on a private cloud it is also easier to include resources from public (commercial) clouds. The enormous potential of these technologies has attracted the interest of e-science communities, and now many projects (e.g. StratusLab, Venus-C, ECEE) are working on the cloud itself and on the integration between cloud and grid. Peter Solagna Operations Officer, EGI.eu
• • • • • • Operating grid services on the StratusLab cloud IMPACT CONCLUSION Cloud technologies can significantly StratusLab is developing a grid-optimised alter the way grid services are currently cloud distribution based on OpenNebula. deployed and operated. By exploiting the This distribution is used to offer a reference inherent capabilities of IaaS clouds for public cloud service which is used as a on-demand provision of elastic computing testbed for deploying virtualised grid sites. services, grid resource providers can The operation of these grid sites is helping optimise the utilisation of their physical us evaluate the applicability of cloud computing resources. technologies, their impact on end-user OVERVIEW focuses on the Infrastructure-as-a-Service applications and to gather requirements for (IaaS) cloud paradigm, which implies the The primary user of StratusLab is the grid grid middleware. As the distribution evolves Cloud computing has emerged in recent usage of virtualisation technologies for the site administrator, yet the adoption of the new capabilities will be added enhancing years as the new paradigm for the provision provision of computing resources. cloud paradigm is expected to indirectly the current cloud solution. In parallel of on-demand distributed computing have a positive impact to the VO managers the public cloud service is attracting resources. The inherent capabilities of the The project is integrating a cloud distribution, and end-users. The project also prepares real-life applications which will help us cloud architecture for optimised resource specifically designed with the purpose to and makes available through the appliance benchmark the technologies and validate management and consolidation appear host grid services. During the design phase repository a set of VM images, with grid the applicability of StratusLab cloud very appealing for grid resource providers. the specific requirements and/or restrictions middleware pre-installed, for the basic grid distribution for the operation of production- StratusLab is a European-funded project of grid services are taken into account in components (CE, SE, WN). The usage of level grid sites. that started in 2010 with the purpose to order to provide optimised cloud environ- VMs accelerates the instantiation of a grid investigate possibilities, implications and ments for deploying virtualised grid sites. site and makes it easier to try out new www.stratuslab.eu optimal solutions for the combination of These requirements are both technical and features or validate middleware updates. cloud and grid technologies. The project is operational. In the heart of the StratusLab integrating its own cloud distribution based cloud distribution lies the popular Moreover, for VOs and end-users the ability on the OpenNebula cloud management OpenNebula (www.opennebula.org) to use VMs with pre-installed applications middleware. The first versions of the cloud management toolkit which is being and scientific software, provides additional distribution have already been released and extended with additional capabilities, flexibility for deploying grid services a reference cloud service is being offered either developed within the project or customised for specific application to the public. This public cloud currently integrated from the ecosystem of existing domains. Also, end-users are expected hosts a minimal but complete grid site. In add-ons. The first version of the StratusLab to be impacted positively by the expected this paper we present the first experiences distribution was released in October 2010. enhanced stability and availability of grid gained by operating this grid site on the Incremental or bug-fixing versions are sites running on virtualised environments. StratusLab cloud and the impact on users released every six weeks. The distribution and their applications. is used by the project itself to set up and provide a reference cloud service. DESCRIPTION Currently two capabilities are available to The StratusLab project started in June the public: a cloud computing IaaS service 2010 with the purpose to investigate the giving the ability to users to instantiate and impact of the emerging cloud computing manage VMs, and an appliance repository paradigm in the provision of grid computing where the VM images are stored. This ref services. The primary motivation was the erence cloud service is used also internally belief that these two technologies are by the project as testbed for deploying grid complementary and their combination sites and in order to investigate potential can offer new capabilities and optimise implications of their operation over the cloud. resource utilisation. Where cloud offers flexibility in resource management the grid provides high-level services that enable collaboration and resource sharing among dispersed scientific teams. StratusLab Vangelis Floros [email protected] GRNET
80 | EGI User Forum 2011 Virtu alisation and Cloud Computing • • • • • • Data management at VENUS-C IMPACT CONCLUSION With our solution we are aiming at During the advent of cloud computing there providing a way for applications to was a lot of discussion about the traditional benefit from sustainable public cloud DCI model being extended or, in some storage offerings while avoiding the cases, replaced by commercial offerings. interface-level vendor lock-in. Moreover, Our work builds on the evolving standards integrated storage management creates in the field of data management in clouds a marketplace for possible CDMI service and aims at providing an integration deployments, as back-ends could be solution for both DCIs and cloud providers. OVERVIEW both from the public cloud computing potentially brokered from other domains offerings and local infrastructures. (both commercial and academic). www.venus-c.eu VENUS-C is an FP7 project aimed at On the technical level it works by bringing together existing DCIs and separating metadata storage (data commercial cloud providers to create store) and data abstractions (blobs, a sustainable ecosystem for e-science message queues). Current list of applications. One of the key aspects of supported back-ends includes for this ecosystem is data management. blobs: POSIX, AWS S3, MS Azure Blob; for message queues – A M Q P, More and more academic groups store AWS SQS, MS Azure Queue; and for and work on their data outside the datastore: CouchDB, AWS SimpleDB university domain. While most cloud and MS Azure Table. storage systems offer convenient access 2. Client-side SDKs in several languages APIs, they are rarely fully compatible that allow interaction with the CDMI-server. with each other. This makes migration between back-ends, or integration of CDMI-compliance means supporting of data located in various places, more standard operations on blob data items complicated. A common answer to this and message queues (table storage vendor lock-in problem lies in adopting support is planned in the next version common standards. Although there are no of the standard). Apart from the core commonly adopted standards for cloud CRUD functionality for those objects, storage, yet, the potentially most promising more advanced features, specific for the direction at the moment is SNIA’s Cloud VENUS-C offering, are foreseen that take Data Management Interface (CDMI), which into account aggregated meta-information, defines the semantics of handling data for example, optimisation of storage costs objects and the required metadata. and transparent encryption of data when storing outside the domain of trust. In this talk we present our approach to managing data in a heterogeneous Implementation is based on Python and environment: a prototype implementation uses Twisted networking framework. We of CDMI that offers a transparent also rely on the services provided by integration layer on top of both cloud Twisted for implementing access controls systems and local storage infrastructures. (Twisted Guard) and logging. For connect ing to the components’ back-ends, DESCRIPTION we either rely on the existing SDKs, for example, boto for Amazon Web Services VENUS-C data management solution and py-amqplib for talking to AMQP- consists of two main parts: compliant queue engines, or implement 1. CDMI-compliant (http://cdmi.sniacloud. the missing link ourselves. com) server that is used for bridging together a number of back-ends, I lja Livenson [email protected] KTH
82 | EGI User Forum 2011 Virtu alisation and Cloud Computing • • • • • • added new and improved capabilities Further investigation will lead to creating Mantychore – dynamic networks for e-Infrastructure to the software and ran a success pilot a distributed computing cloud between project where MANTICORE II partners got institutions, with an intended use as a the chance to run some trials on their own platform for agent-based computing. This network and equipment. The Mantychore will raise issues relating to adding and FP7 project consolidates this trend. removing sites, and relating to dynamic provision of extra links for data migration IMPACT for virtual machine migration and backups.
Mantychore FP7 will carry out pre-operational In general, by working with research OVERVIEW Layer 1, optical links. Users will be able deployments of the IP network service at communities in higher-education and to get permissions over optical devices two NRENS: HEAnet and NORDUnet. research institutions Mantychore will gather Mantychore aims to provide dynamic network- like optical switches, and configure some Part of the project effort will be dedicated real-world requirements for connecting ing as a service for e-Infrastructure users. important properties of its cards and to consolidate and enhance the community dynamic and user-controlled networks to ports. Mantychore will integrate the Argia of providers (NRENs but also commercial) traditionally managed networks. The e-science community has expressed framework which provides complete and users of the IP network service. a need for flexible and application- control of optical resources. CONCLUSION driven networks. Traditional methods for Initially three communities of researchers configuring networks for research users are Layer 2, ethernet and MPLS. Users will will benefit from this service: the Nordic In addition to the central development and impractical when the scale is international be able to get permission over Ethernet Health Data Network, the British Advance deployment of dynamic networking as a and involves several networks. and MPLS (Layer 2.5) switches, and High Quality Media Services and Grid- service for e-Infrastructure, Mantychore configure different services. In this aspect, Ireland, the Irish NGI. will innovate the business model used Mantychore takes an ‘Infrastructure as a Mantychore will integrate the ETHER for services based on IaaS, establishing Service’ (IaaS) approach to networking project and its capabilities for the manage The Grid-Ireland Operations Centre a marketplace where all infrastructure to enable National Research and ment of Ethernet and MPLS resources. (http://grid.ie) in Trinity College Dublin providers can announce their available Education Networks (NRENs) and other runs Ireland's national computing grid resources and all customers can automa e-Infrastructure providers to deploy a Layer 3, IP. Mantychore FP7 suite includes infrastructure which connects grid nodes at tically negotiate SLAs getting the best configurable, automatable network that a set of features for: many higher-education institutions around resources combination for their needs. allows virtual research communities to • Configuration and creation of virtual the country and is part of the European Furthermore, Mantychore services will control routers, switches, optical devices, networks. grid infrastructure. be used to contribute to the research and IP networks as necessary to meet the • Configuration of physical interfaces. performed in the GreenStar Network pro- needs of their applications. • Support of routing protocols, both Grid and cloud computing use-cases will ject to enable carbon-neutral infrastructures. internal (RIP, OSPF) and external (BGP). be explored by the Grid-Ireland Operations Grid and cloud computing will provide • Support of QoS and firewall services. Centre. As Grid-Ireland has resources distri- The Mantychore FP7 project is committed some of the first real uses for this • Creation, modification and deletion of buted across Ireland and connected via to incorporating as many viewpoints and technology. Mantychore's facilities should resources (interfaces, routers) both HEAnet it is an ideal testbed for Mantychore. potential uses as possible in order to reach make it easier to share computing and physical and logical. a more complete and valuable software and storage resources between grid sites or • Support of IPv6. It allows the The Grid-Ireland team will explore novel expertise pool. For this reason, the project to create a distributed computing cloud configuration of IPv6 in interfaces, approaches to sharing and connecting resources are open and available for any spanning several institutions. routing protocols, networks. computing and storage resources. This will interested individual or community to join. include connecting grid worker nodes at DESCRIPTION Mantychore FP7 is the natural evolution multiple physical sites into a single virtual www.mantychore.eu of its parent projects: MANTICORE and grid site over a dynamically configured The Mantychore FP7 project aims to allow MANTICORE II. MANTICORE implemented virtual private network. the NRENs to provide a complete, flexible a proof of concept which proves the idea network service that allows research of IP network and router as a Service communities to create an IP network under (IPNaaS), all this management in Layer 3 their control, where they can configure: where MANTICORE works. MANTICORE D avid O'Callaghan [email protected] Trinity College Dublin • Peter Lavin [email protected] II continued its steps to implement stable Trinity College Dublin • Sergi Figuerola [email protected] i2cat • Pau Minoves pau.minoves@ software with the feedback, expertise and i2cat.net i2cat • Victor Reijs [email protected] HEAnet • Dave Wilson [email protected] know-how received. Also, MANTICORE II HEAnet • Per Nihlen [email protected] NORDUnet
84 | EGI User Forum 2011 Virtu alisation and Cloud Computing • • • • • • Federated cloud computing environment IMPACT CONCLUSION for malaria fighting This is one of the first international The collaboration between both super experiences in which two supercomputing computing centres in the implementation centres create a set using HPC Cloud of this cloud, in addition to being essential computing and storage equipment support in the development of international distributed between the two institutions. research groups, is a step in the research For the connection, the centres used the of technologies for cloud computing network of academic research and RedIris, services. The success of the experience which provided the necessary bandwidth. opens the door to new collaborations OVERVIEW DESCRIPTION between national and international research groups, which could deliver very The research group of Dr. Hugo Gutierrez The aim is to generate a list of potential demanding computer applications for de Teran, Galician Public Foundation for candidate molecules to pass Bioassay in research projects almost immediately. Medical Xenómica (FPGMX) and member parallel using two independent methods of Scientists Against Malaria, needed to of ligand-receptor coupling or docking. perform, using different computational In the field of molecular modelling, this methods, virtual screening of a chemical is a method that predicts the preferred library of 350,000 chemical compounds orientation of a molecule that will be linked to study a selected therapeutic target to to another to form a stable complex. further develop, based on the findings, Knowledge of the preferred orientation new drugs against the parasite can in turn be used to predict the strength Plasmodium, which causes the disease. of the association or the binding affinity between two molecules. A cloud computing collaboration with FCSCL, using the framework of the Prior to this study, researchers from the cooperation agreement signed months ago group of Dr. Gutierrez de Terán generated between the presidents of the regions of a collection of possible molecular species Galicia and Castilla-León was used as the for each of the chemicals, considering all fastest way to solve this challenge. the possible isomers (compounds with the same atomic composition but different structural formula), tautomers (two isomers that differ only in the position of a functional group) and protonation states (addition of a proton to an atom or molecule) which led to a collection of 1,238,000 processed molecules. This type of calculation requires a large amount of computational resources, which are only possible using large data centres.
CESGA technicians used the cloud developed in the framework of the NUBA project, extending it to use resources from FCSCL. The management of both sites was performed using computer software developed in Spain, OpenNebula, which manages the nodes in both locations, deploying virtual machines in either one or the other depending on the criteria of optimising computing resources. C arlos Fernandez [email protected] FCTSG
86 | EGI User Forum 2011 Virtu alisation and Cloud Computing • • • • • • Searching Eclipse behind the cloud IMPACT CONCLUSION The Eclipse community is one of the The set up of distributed OSGi-based biggest open source communities applications and the management with worldwide with strong support from an integrated user frontend is possible industrial players. In an open community today. The combination of existing Eclipse process the important trends for the next technology helps to build a full cloud year were defined and they contained service stack including IaaS, PaaS and amongst others: SaaS layer. • Cloud and browser-based delivery OVERVIEW This talk will explain the overall picture of applications is the future. www.eclipse.org/geclipse of OSGi applications on the cloud. • Cloud and browser-based developer www.eclipsecon.org/2009/sessions?id=336 Emerging cloud infrastructures offer Furthermore it will demonstrate the set tools needed to support this future new ways to develop dynamic services. up of a search application in the cloud; process. Management, provisioning and inter including the management of the cloud, operability are the main challenges among the dynamic provisioning of the application The demonstration will show what is already others. Within the Eclipse community, there itself and the provisioning of an advanced possible today with existing open source is a clear trend from the desktop usage to AJAX-based user interface on the cloud. technology in this area. An alternative a general runtime system. With the help of This will be performed with the help of the approach for dynamic service delivery will the Eclipse ecosystem (www.eclipse.org), following Eclipse projects: also be shown. a toolbox of solutions exists already today • g-Eclipse will be used to manage and to develop new services. The Equinox configure the virtual cloud resources runtime of Eclipse – based on the OSGi based on its general Cloud model. standard – can be seen as the basis for an Access to Eucalyptus and AWS interoperable cloud platform in the future. resources is supported today. • p2 will be used to deploy the An introduction to the various Eclipse search application. projects will be given and distributed • SMILA (SeMantic Information Logistics search applications will be set up on cloud Architecture) is an extensible framework resources (AWS or/and Eucalyptus) during for building search applications for the talk. data like office documents, emails, images, audio and video files, blogs, DESCRIPTION etc. One of the features of SMILA is the parallelisation of processes/workflows, Emerging cloud infrastructures offer new so the natural deployment environment ways to develop dynamic services. Manage- of SMILA is similar to the distributed ment, provisioning and interoperability environment of the cloud. are the main challenges among others. • RAP will be used to create a simple A short time-to-service is important to all search user interface for the application kinds of applications on grid and cloud based on JAVA script and AJAX infrastructures. With the help of the Eclipse technologies. ecosystem (www.eclipse.org), a toolbox of solutions exists already today to develop This talk demonstrates existing goodies new services. As the underlying technology from various Eclipse projects, which is based on standards (JAVA and OSGi) can help to build cloud applications the application is independent from the independent from underlying infra underlying infrastructure. structures. It will show the potential power of Eclipse technology in the cloud.
H arald Kornmayer [email protected] DHBW Mannheim
88 | EGI User Forum 2011 Virtu alisation and Cloud Computing • • • • • • rules. In a similar way the termination of The approach adopted in this system is GPU computing in EGI environment using jobs are notified to a daemon that releases not focused only on GPU computing and a cloud approach the execution environment. it can be easily extended to other special hardware devices and, in general, to other In order to develop and test the whole environments (as in the IaaS Model). infrastructure, a fully working test bed has been built with the adoption of the CONCLUSION Eucalyptus software system to implement a private cloud over the cluster. In this work we presented a system to provide to EGI users a specific on-demand OVERVIEW DESCRIPTION We have also addressed the need to create GPU environment to transparently execute Virtual Machine Images that match the jobs on GPU devices. Recently GPU computing, namely the In the present work, the possibility of requirements of the execution of GPU- possibility to use the vector processors enabling the convenient usage of GPU dependent jobs, such as CUDA, OpenCL The proposed system uses a cloud of graphics cards as computational devices for VO users, exploiting the libraries and gLite middleware. approach, based on EC2 compliant general purpose units of High Performance capabilities of EGI, and the emerging clouds, in order to control the specific Computing environments, has generated paradigm of cloud computing are explored. IMPACT GPU-enabled VO's environment from EGI considerable interest in the scientific middleware interfaces. community. Some communities in the A strategy to provide on-demand execution GPU computing is increasing in the EGI European Grid Infrastructure (EGI) are environments has been proposed communities, starting from computational The system is currently in its testing reshaping their applications to exploit this through the joint usage of traditional and chemistry (COMPCHEM VO) to theoretical phase at UNI-PERUGIA grid site and new programming paradigm. Each EGI widespread gLite components and the physics (THEOPHYS VO), as well as supports COMPCHEM VO. In this community, called Virtual Organisation popular standard EC2 web-service APIs. satisfying the needs of other communities. phase the LRMS is used to join the EGI (VO), often requires specific environments, infrastructure and the UNI-PERUGIA making it necessary for each grid site to An entire job flow that enables the Local The main purpose of the present work is private cloud. We are planning to decouple enable an efficient system to fulfil VO's Resource Management System (LRMS) to dynamically provide a ready-to-use GPU the on-the-cloud allocation mechanism software requirements. to discriminate the GPU resources environment for the communities using from the LRMS and place it at the requests, through GLUE Schema EGI to share GPU resources, since a single computing element level using for example Cloud computing and more generally parameters, has been defined to allocate, GPU environment does not satisfy the the CREAM Architecture capabilities. the opportunity to transparently use com in a dynamic fashion, the required different requirements of all communities putational resources, together with the resources on a cloud-like infrastructure (such as operating systems, compilers This will allow a fine grained control over consolidation of virtualisation technologies, either public, private or hybrid. and scientific libraries). For this reason, the Virtual Instances and the Accounting. provides end-users with the required the developed system provides dynamical environment for their activities. To achieve this goal, part of the work has environments with the aim to optimise GPU been devoted to the virtualisation of the resources usage. The present work aims to provide an on- physical GPU resources in order to make demand GPU environment (GPU framework, them available in an Infrastructure as a Contextually, the cloud computing Operating System and libraries) for each Service (IaaS) private cloud. opportunity allows taking into account the VO and to make it accessible via the GPUs as a Service (IaaS). From a cloud production infrastructure using the cloud. To this end a centralised mechanism, point of view, the project carries out a responsible to listen for events generated feasibility study to understand how the by the LRMS like job scheduling and next evolution of the grid computing to termination, has been implemented to keep cloud computing, or better, how the switch track of each request. from Batch Model to Service Model could be done. These events are then used to carry out the required actions as follows: once a job is received and identified as a GPU usage request, it is treated as an event that triggers the allocation of virtualised Alessandro Costantini [email protected] UNIPG • Osvaldo Gervasi [email protected] UNIPG • resources according to simple leasing Flavio Vella [email protected] UNIPG • Riccardo Cefala' [email protected] UNIPG
90 | EGI User Forum 2011 Virtu alisation and Cloud Computing • • • • • • End-users of Distributed Computing Infrastructures typically expect that these systems can be smoothly and easily integrated into their routine research workflows. DCI-based ‘e-workplaces’ should allow users to customise their data, computational and software services, and make these available within easy reach. While highly customised user environments, built on top of e-Infrastructure services can certainly satisfy these needs, the develop ment and maintenance of such solutions can rarely justify the investment. Highly customisable systems can attract more users, however these systems require involved configuration, maintenance and training from the parties. Presentations within the User Environments track introduce various tools and projects concerned with how end-users interact with the European Grid Infrastructure. The number of user environments available to EGI communities is high. The track aims to bring together the developers, User providers and users of these environments and enable them to share experiences, issues, solutions and trends with each other and with the EGI stakeholders. The integration of National Grids into a single European Environments Infrastructure triggered the need to extend existing user environments – whether these are implemented as portals or client-side applications – with support for multiple middleware technologies. Only those tools that are interfaced with a variety of workload manager systems can successfully satisfy the needs of multi- national communities. The emergence of desktop PC and cloud-based com puting within the EGI ecosystem has been recognised by the providers of user environments. To achieve the transparent integration of users’ application and data across different infrastructures, the latest generation of user tools can connect to cluster, desktop and cloud infrastructures. Gergely Sipos Senior User Community Support Officer, EGI.eu
• • • • • • on the science gateway server and handed could offer carbohydrate recognition in A science gateway for molecular simulations over to the UNICORE infrastructures. UNICORE 6 infrastructures. G-FLUXO provides Gromacs workflows based on IMPACT P-GRADE (the first generation of WS- PGRADE) and visualisation of simulation The emerging science gateway offers a results. The workflows and the visualisation single point of entry to the whole process could be integrated into the MoSGrid of analysing data for molecular simulations science gateway. The e-NMR portal offers facilitating UNICORE 6 infrastructures. It also multiple molecular simulation tools will support multiple molecular simulation with access to a number of grid resources OVERVIEW of services including a workflow engine, tools, workflows on molecular simulations but does not offer a repository of results. so-called submitters (Java-based applica and repositories which store the Nowadays, scientists in multiple research tions), a workflow storage, an application information on structures, workflows and CONCLUSION areas like material science, structural repository, monitoring and logging. simulation results. The supported tools biology, and drug design are supported Submitters implement the grid service are selected via a survey in the MoSGrid The emerging MoSGrid science gateway by invaluable molecular simulation tools. interface of gUSE and exist for classical community including about 100 working will support the users of molecular These tools allow analysis of increasingly service and desktop grids, clouds and groups in the fields of computational simulation tools in grid infrastructures. complex chemical structures on high- clusters, unique web services. MoSGrid chemistry and bioinformatics. Workflows At the current stage of the prototype, performance computing facilities. However, added a submitter for UNICORE 6 using and results can be exchanged in the the UNICORE 6 integration for WS- the lack of graphical user interfaces, the the UCC (UNICORE Commandline Client) community via a global repository. PGRADE workflows and the SAML limitation of the usability of the tools and libraries. The submitter enables to invoke Workflows can be created, re-used and extension is operational. Furthermore, the complexity of infrastructures demands jobs of WS-PGRADE workflows on changed by the community and additional there are archetypical workflows available intuitive user interfaces. UNICORE 6 infrastructures and additionally knowledge will be obtained by cross- for quantum chemical simulations the invocation of existing UNICORE referencing different result data files. with Gaussian and molecular dynamic The project MoSGrid (Molecular Simulation workflows via WS-PGRADE. simulations with Gromacs in single portlets. Grid) addresses these issues by combining Compared to similar existing activities an easy to use portal-based infrastructure Besides the extended workflow features like ProSim, G-FLUXO, the e-NMR portal Next steps include the definition of with expert knowledge on the correct of WS-PGRADE, MoSGrid has been and projects based on EnginFrame, the workflows for the MoSGrid community use of complex methods. The emerging developing intuitive portlets for predefined MoSGrid science gateway will offer a and the uptake of the Application Specific portal will support users in all stages of the basic workflows for the most popular larger scale of tools and methods and will Module (ASM) for WS-PGRADE in the simulation process with easy access to molecular simulation tools within the support the possibility to refer to molecules developed portlets. The latter will serve as data repositories storing information about MoSGrid community. Furthermore, users and results in a global repository under an templates for further portlets on additional molecular properties, and the possibility of will be supported by repositories which open source licence. tools. The portlet developers do not have creating, editing and invoking workflows. store information about initial structures to become acquainted any more to the The project integrates the UNICORE 6 for the simulation process, parameters EnginFrame is also a workflow-enabled UNICORE integration in detail and the grid middleware and the cloud file system of workflows like used molecular simula- grid portal with existing projects for users are enabled to monitor their different XtreemFS into the workflow-enabled grid tion tools and resulting structures. molecular simulations but under a workflows on one page. portal WS-PGRADE. The repositories will allow external ref commercial licence model for non- erencing of simulation results and will academic users. ProSim and MoSGrid Furthermore, the repositories based DESCRIPTION be available replicated in XtreemFS via can benefit from each other. ProSim on XtreemFS will be integrated into the the science gateway. offers workflows and workflow templates science gateway. Hence, XtreemFS will be MoSGrid is an interdisciplinary project for carbohydrate recognition in a portal extended for authentication with SAML. which aims to facilitate molecular simula In general the authentication process for on top of WS-PGRADE with access to tion tools in grid infrastructures via a grid infrastructures relies on X.509 based several grid infrastructures. Users could http://mosgrid.informatik.uni-tuebingen. science gateway. This science gateway certificates. In MoSGrid WS-PGRADE re-use workflows of ProSim in the MoSGrid de:8080 is based on the workflow-enabled grid has been extended for the use of SAML science gateway and the ProSim portal portal WS-PGRADE which is the highly (Security Assertion Markup Language). flexible graphical user interface (including The user is enabled to create an assertion a graphical workflow editor) of the grid file from his X.509 certificate via an applet. User Support Environment (gUSE). gUSE is The certificate remains on the user’s S andra Gesing [email protected] University of Tuebingen • Peter Kacsuk kacsuk@ a virtualisation environment providing a set computer, solely the assertion file is stored sztaki.hu MTA SZTAKI, Budapest • Miklos Kozlovszky [email protected] MTA SZTAKI, Budapest
94 | EGI User Forum 2011 user environmentS • • • • • • IMPACT There is a strong synergy with the medical Proteomics portal: an e-science gateway sciences, i.e. between the proteomics for Swiss bioinformatics Data analysis in the domain of proteomics gateway community and the medical has made a lot of advances in the recent gateway, to be later developed by AMC years with a very large number of new (Amsterdam Medical Centre). A strong emerging tools to identify and quantify interest exists to extend the gateway to proteins and peptides through mass be used to analyse other types of data, spectrometry and liquid chromatography i.e. imaging, microscopy or genomics experiments. The problem is that many datasets. The concepts are identical, but tools do not adhere to standard data of course the data types and algorithms OVERVIEW interface were the reasons to choose formats, although such standards do exist. would need to be adapted. By the use of P-Grade as the baseline technology for P-GRADE portal technology it is possible Bioinformatics is a science that this specialised proteomics e-science Through the Swiss Proteomics Gateway to link this effort with the efforts of AMC, routinely makes use of large computing gateway. To start with, three different types the usage of these tools is automated and giving this development a European infrastructures including grids to of workflows were developed based on very much simplified for the end-user. dimension. understand and analyse the data and to commonly used proteomics tools and Experts take care of data transformation prepare for the next wave of instruments applications set up to be executed on the into the usable formats, chaining the tools CONCLUSION producing even more data. Not only the large clusters. Then corresponding portlets to get the most relevant information and data volume but also its complexity has were developed providing easy-to-use web inclusion of the most recent methods. Due to the recent joint development increased. The combination of complex interfaces tailored to the needs of the end- With the gateway scientists will be able between the ETH Zürich, SystemsX.ch analysis tools and large-scale infrastructure users. These portlets hide the complexity to share their analysis workflows and and MTA SZTAKI, a new web-based portal makes it difficult for the scientists to of the workflows and the DCIs allowing the the best methods to look at the data. is available to researchers in proteomics. analyse the data without expert help. end-users to focus just on the parameters More importantly, they can automate The complexity of the inner component There is a strong need to establish that are important to their own research. the workflows for repeated analysis systems is hidden, therefore the end- automated processes and workflows The three initial portlets are interfaces to with changed parameters, which was a users can focus on their science, only that can be executed easily without the Transproteomic Pipeline, a workflow manual, slow and very error prone process parameterising and executing common any knowledge of the complexity of the to perform label-free quantification and a in the past. Developers of algorithms proteomics analysis workflows by the underlying infrastructure. Thanks to a quality control metric calculation. But also can easily interface with the existing push of a button. This increases their joint project between SystemsX.ch, ETH much more complex workflows have been services and test new ideas on real data. productivity very efficiently. Zurich and MTA SZTAKI, a new web-portal implemented by more advanced users Collaborations between lab scientists for easy to use automated proteomics of the platform for dedicated research and algorithm developers are very much https://www.imsbportal.ethz.ch analysis was built using the P-Grade Portal problems and several new portlets are facilitated, nationally and internationally; the technology developed by MTA SZTAKI. in the process of being finalised. scientists can focus on their core scientific The portal has been set up to be able to discoveries as opposed to spending time execute programmes on local clusters of There was also a clear need to make the in the details of data transformations. ETH and on the national distributed grid, usage of the portal’s security components Due to these advantages the whole the Swiss Multi-Science Computing Grid as easy as possible with no shortcuts Swiss research community interested in based on the ARC middleware. taken in terms of security. This was proteomics analysis can set up workflows achieved by the integration of the Swiss and run applications on the local and DESCRIPTION national AAI infrastructure based on national DCIs. Shibboleth2 into the portal and providing The P-Grade Portal provides a web- access to the grid automatically using the based user interface where the users SLCS service to generate user certificates can develop/manage applications on based on the user’s AAI login to the portal. various types of DCIs. The portal is able This way the users only have to log in once to submit jobs using gLite or ARC-based and the portal can then submit jobs to the middlewares as well as on LSF or PBS- grid with no additional steps necessary as based clusters, a capability that was in previous portal instances where users developed for this project. The flexibility needed to generate a proxy certificate and the workflow and parallelisation outside of the portal and upload it to a capabilities coming with the user-friendly myproxy server. Akos Balasko [email protected] MTA SZTAKI
96 | EGI User Forum 2011 user environmentS • • • • • • SOMA2 – gateway to grid-enabled molecular IMPACT CONCLUSION modelling workflows in www-browser SOMA2 gateway provides an easy to SOMA2 source code is distributed openly use and intuitive single user interface for all interested parties. At CSC, SOMA2 to scientific applications and it hides all is also available as a service for CSC’s technicalities from end-users. The system academic users providing access to 14 automates repeating tasks and eliminates different molecular modelling applications, redundant work so that end-users can which are seamlessly integrated within the focus on the actual scientific task instead system. The system is fully integrated with of dealing with technical issues requiring the local computing infrastructure. OVERVIEW the data to CML format and preparing it manual work. for the application workflow. With the web Within EGI-InSPIRE WP6-SA3, we plan SOMA2 gateway is a molecular modelling application, the user is able to set up the With the new DCI-support features of to extend the current SOMA2 service to workflow environment developed and workflow, configure chosen applications, SOMA2, we can really say that SOMA2 include DCI-enabled applications. This deployed by CSC – IT Centre for Science submit projects for computation, review the does not only integrate applications but service will be first introduced to EGI. Later Ltd. The SOMA2 environment is used status of the project and access the results. also different computing infrastructures on, we plan to make the service available with WWW-browser and it allows users making complex computing environments for other communities in EGI. to combine scientific applications into In SOMA2, scientific applications and reachable for all users. We think that tech unique application workflows, which are their execution is described in pluggable nical complexity is still a major bottleneck, In addition to common enhancements in automatically executed in the underlying capsules, which make use of interfaces which prevents users from fully benefiting the web application, SOMA2 development computing system. SOMA2 offers a to manage the internal data. A capsule from the distributed resources. SOMA2 plans include investigating other flexible framework for integrating consists of an XML description, used for system can alleviate this problem a lot. middleware to be used in SOMA2 DCI- and executing molecular modelling example, to generate an application web integration and new applications to be applications, and facilitates automated form, and scripts and file templates to The framework for describing scientific integrated in SOMA2. molecular data exchange. SOMA2 enable scientific application execution and applications in SOMA2 facilitates transfer source code is distributed under the processing of the programme output. of technical know-how from experts to www.csc.fi/soma GPL open source licence. service users so that everything remains SOMA2 system includes a workflow in machine readable form. In addition, For end-users, SOMA2 offers a secure, manager programme which is responsible core SOMA2 software is separated from personalised and easy to use environment for internal data transfers and controlling the programme descriptions and basically for utilising computing infrastructure. execution of the capsules. no real programming skills are required In SOMA2, scientific applications are to create a SOMA2 capsule. Flexibility, presented and configured via web forms, Recently we have added DCI-support in application oriented approach for reusable which guide users to correctly configure SOMA2. In SOMA2 web application and workflows and open source licence make a programme by supplying default values, its utilities, this includes handling of users' SOMA2 system very unique in its domain. thresholds, runtime help and validation. X509 certificates, requesting available For experts, SOMA2 offers a framework resources from the grid and generating to make virtually any molecular modelling proxy certificates upon project submission. application accessible to the end-users. We make use of grid middleware to submit the jobs. This work has been conducted as DESCRIPTION part of EGI-InSPIRE project's WP6-SA3. Currently, we have support for Nordugrid SOMA2 enables communication and Arc middleware but other middleware data exchange between applications by could be used as well. employing a common data exchange format, CML (Chemical Markup Language). The common data format is crucial for seamless integration of different applications within the SOMA2 system. The SOMA2 web application handles user's initial molecular data by converting T apani Kinnunen [email protected] CSC
98 | EGI User Forum 2011 user environmentS • • • • • • Data-driven portal for hadron therapy IMPACT CONCLUSION The portal is an easy to use tool for all We demonstrate how we use grid services the actors involved in the hadron therapy to securely integrate medical data from domain: medical doctors can securely various sources. The combination of import, transfer and view patient related enterprise portal Liferay and Vine Toolkit information and researchers can construct provides an easy to use grid-enabled complex queries. Moreover, patients environment where additional services could record their quality of life during could be prototyped and tested. To improve and after the treatment. Specific roles and the portal services, data provenance and OVERVIEW DESCRIPTION privileges are protecting data and actions data mining aspects of our platform we of actors using policies enforced by hope to establish close interactions with PARTNER grid prototype is a data-driven Using the Vine Toolkit [http://fury.man. VOMS credentials. other communities and projects in EGI. platform for recording and sharing medical poznan.pl/vinetoolkit], we connected data within the hadron therapy community. Liferay portal [www.liferay.com] with http://partner.web.cern.ch The idea is to provide secure cross-border VOMS [https://voms.ific.uv.es:8443/voms/ access to heterogeneous repositories while vo.partner.eu] to enable strong grid security leaving data where it was created and not and to use a DPM Storage Element as copying it centrally (data warehousing) due a grid resource to store DICOM images. to data privacy and ownership concerns. For domain specific data recording we Our solution is to describe local data using developed forms using web-services, metadata and use federated queries to Java and Flex portlets. We work on a provide users with a unified view over data virtualisation layer that coupled to a distributed resources as a virtual database. metadata registry [https://cancergrid.org/] enables semantic queries over different data repositories.
F austin Laurentiu Roman [email protected] CERN
100 | EGI User Forum 2011 user environmentS • • • • • • Job Submission Tool, web interface and IMPACT CONCLUSION WebDAV data management Researchers are often not familiar with The JST has shown a good reliability X509 certificates and grid technology: in and efficiency and few improvements are recent years JST has proved the capability needed. The development team is just to solve the problem of submitting huge working on the integration of the interface challenges on the grid with little grid in other portals. Since JST has been competence and small human effort. developed inside the BioinfoGRID and LIBI project, so far all the work planned The first goal achieved by JST was to split has been focused on the bioinformatics OVERVIEW We have tried to go one step behind, trying applications born to be executed on single applications, but the generalisation of the to enable final users to use the grid on their machines, in order to reach the same tool allows us to open our technology to The Job Submission Tool (JST) allows the own. For this purpose we have developed results executing single independent jobs other communities. exploitation of the grid computing power a tool, JST, which uses a web graphic in a parallelised way. to many research communities with highly interface, written using PHP, Javascript and compute-intensive applications. The tool XSLT, where a user can authenticate using The JST team ported over the grid about helps to sub-divide large applications into a username and password. Using the same 20 different bioinformatics applications, single independent tasks and execute graphic interface the user uploads the submitting around half a million jobs them on the grid nodes in an optimised input files. Since it was found not feasible representing about 50 years of CPU. time. Furthermore with the implementation to upload big input files (up to 10GB) of an ad hoc graphic interface, users have using the http protocol, a new mechanism After the release of the graphic interface the opportunity to use the grid without based on the ‘webdav’ protocol has been more than 50 challenges have been knowing all the technological details implemented to upload input files on a submitted to grid through the web. A and without taking care of its complex file server in an efficient way and with an total of 359 days of CPU usage on the authentication methods. So far, several automatic registration of the files on the grid worker nodes has been reached bioinformatics applications have been grid Storage Elements. The user can then for the completion of all the tasks. The successfully executed onto the grid using customise the application (changing the submission with the GUI has produced JST, allowing the users to reach important configuration parameters) and specify 19,656 correctly executed jobs and 583 goals on their research. the name of the output files. After the failed ones. Considering other similar tools submission of the web form, JST executes that allow access to the grid, we have DESCRIPTION all the steps required for the submission tried to completely mask the underlying of the application to the grid (slicing of the technology to provide the end-users the Several bioinformatics scientific questions problem, user authentication by means of a chance to use a simple portal that takes often require months or even years to be robot certificate, submission of the jobs to care of the complex steps they would solved using the computational power the grid, retrieving of the output files). At the face if using the grid directly. Furthermore provided by a single cluster. In such end, JST sends the link of the output files we have overcome the limit that a classic cases it is important to exploit the grid to the user by e-mail. portal could give for input file dimensions, in order to reduce the execution to a few setting up the new webdav server for JST days or hours. The collaboration between that offers a new way to manage very big bioinformatics researchers and grid experts files. We decided to use webdav as it is has been found to be very productive: a standard and well supported protocol, the grid expert takes care of splitting indeed each operative system has its the application into independent tasks, own client (this is true for Linux, MacOS filling the central DB with the independent and Windows). tasks, submitting the tasks to the grid and retrieving the output. In other words, the final user does not have to deal with all the grid technicalities which are taken by the grid expert. Several important challenges have been successfully executed with this approach. Giacinto Donvito [email protected] INFN
102 | EGI User Forum 2011 user environmentS • • • • • • • Baseline technologies. We will As a result gUSE and WS-PGRADE is New developments of gUSE and WS-PGRADE enumerate the additional feature already popular for many communities to support e-science gateways set, which facilitates community level since it helped to lower barriers to entry in collaboration among developers e-science environments for researchers. and end-users, such as support of It also increases the potential for e-Infra various repositories. structure usage by non-specialists, as demonstrated in the FP6 and FP7 projects IMPACT (CancerGrid, SHIWA, HP-SEE, MoSGrid).
As the available grid infrastructure achieves CONCLUSION OVERVIEW of different technologies. It enables the larger community adoption and higher design, integration and submission of performance, there are an increasing The developed new feature set of WS- E-science Gateways are the primary sophisticated (layered and parameter number of user communities with well PGRADE/gUSE can be categorised into solutions dedicated to support end- sweep enabled) workflows spreading described needs for an application-specific scientific gateway services, developer users without grid knowledge to do their parallel among grids, web services, gateway. Recently various e-science and end-user services and baseline research on distributed grid infrastructure. clusters, and eventual local resources communities tried to create user-specific technologies. It provides support to build Due to the wide range of research fields, of gUSE. It provides a collaborative, and user-friendly science gateways in up, customise and integrate easy and generic grid workflow management community-oriented grid application their own way. However, in many cases reliable new gateways and to upgrade systems/portal services should be development environment where grid the attempt failed since the community older existing gateway solutions to recent flexible enough to support the different application developers and end-users can underestimated the effort needed to technologies. requirements both at front-end and back- share workflow graphs, workflow templates create a stable, secure and robust gate end. Previously the importance of science and ready-to-run workflow applications via way to be used simultaneously by many It also further facilitates and speeds up the gateways was overlooked. This required a workflow repository. community member as a reliable 24/7 development of community owned applica a huge (time and skill) overhead in usage service. The scattered activities among tion-specific gateways and supports and development due to the required WS-PGRADE portal is a second generation, communities also led to lot of redundancy application developers, end-users / complete understanding of various generic purpose e-science portal that and duplication of efforts. At the end in e-scientists and e-Infrastructure operators distributed infrastructure technologies introduces many advanced features both many cases they have realised that it is in their work. New user communities are and solutions. As a result adoption of at the workflow and architecture level. In much more cost-effective and faster to supported strongly during their e-science these e-Infrastructures has been below this presentation we are focusing on the build the gateway based on an existing gateway development in a collaborative expectation until today. Recognising following feature set of the WS-PGRADE/ generic-purpose DCI gateway. Even those way. As a result, a wide range of other the problem many attempts have been gUSE solution offered for the various types communities that have already successfully European user communities benefit already made to create user/community-specific of user and NGI user communities: deployed their first gateways have identified from the improved e-science services and user-friendly science gateways for • Scientific gateway services. We will new requirements and would like to use the provided by the WS-PGRADE/gUSE the various e-science communities. In show how the solution supports unified latest technologies to keep their research solutions. In addition, the developed feature this oral presentation we introduce the access to and seamless integration competitive. With our newly developed set also bridges into the commercial cloud newly developed features of gUSE a Grid of the underlying networking, WS-PGRADE/gUSE feature set we have application domain, allowing scientists User Support Environment and the WS- computing and data infrastructures targeted the following emerging user and commercial users to use clouds and PGRADE portal solutions. and services for all the major DCI community requirements: easy and reliable grids together. infrastructures (local and remote way to build new gateways, upgrade DESCRIPTION clusters, supercomputers, local and existing gateways to modern technologies, www.guse.hu volunteer DGs, grids, public and private strong customisation capabilities, and www.wspgrade.hu The gUSE and the WS-PGRADE solutions academic or commercial clouds). seamless infrastructure compatibility. are tailored for supporting grid application • Developer and end-user services. development and building up e-science We will detail how the solution provides gateways. support for the design, development and deployment of user-friendly gUSE is an easily usable, highly flexible interfaces which abstract service and scalable co-operative grid application provision from the underlying infra development infrastructure, connecting structure complexities and specific developers and end-users of grid implementations through the use of applications with computational resources various web technologies. D r. Miklos Kozlovszky [email protected] MTA SZTAKI
104 | EGI User Forum 2011 user environmentS • • • • • • Grid computing for biomechanical markers extraction IMPACT CONCLUSION The approach researched in this article While extracting biomechanical markers influences medical procedures to: by various computational procedures a 1. specify more precisely a set of number of methods have to be applied recognisable objects and calculate/ when analysing radiographs. The most separate biomechanical parameters, easily recognisable objects are femurs. 2. examine the adaptability of the When searching for clearly visible pelvic developed methods for diagnostics of arches a set of additional thresholds for other body areas which is based on the parameters due to peculiarities of the OVERVIEW diagnosis in the pelvic area (such as analysis of bone structures visible in equation describing pelvic arches and the dysplasia, arthrosis, dislocation or x-ray images, radiograph itself are done on the basis Visual evaluation of an object (in images) in fracture) and selection of treatment . 3 examine the adaptability of the of data about detected femoral ridges. biomechanical and orthopaedic processes methods relies on a number of methods developed methods for diagnostics After defining the parameters, pelvic is a usual method of contemporary medical based on calculations of biomechanical of other areas which is based on the bones are successfully detected. Various diagnostics. In some cases visual diagnos parameters of objects under analysis. visual analysis of objects in 3D images. sequences of calculations and parameters tics is relatively simple (for example, in the These parameters can be conveyed in of detected objects involve more case of non-complicated bone fracture) geometrical expressions: points, distances sophisticated diagnostic methods, involve and does not require complex medical among objects, positions of objects drug selection and shorten diagnostic methods and calculations. However, to each other and angle formations. In procedures. This also opens broader such cases are rare and also mere visual general, the diagnostic process involves suggestions for further explorations. These inspection is sometimes not sufficient. The the following steps: computational procedures are obviously diagnosis may require more complicated 1. pain diagnostics, suitable to implement in cloud computing measurements, as well as extraction of 2. anamnesis, or virtualisation environments. biomechanical parameters of an object. 3. patient examination (temperature, visual However, this is a time-consuming process changes, palpation, pace, stand, also prone to mistakes. This article presents movement extent), computer vision algorithms, developed 4. radiology test and extraction of and implemented to improve diagnosis and biomechanical parameters, increase the reliability of medical decisions. 5. additional diagnostic methods Algorithms, set in a grid environment, are (computed tomography, radioisotope able to accelerate and unify the diagnosis, bone scan, ultrasound testing, leaving to a physician mainly a function magnetic resonance imaging, of interpretation (assessment of medical arthroscopy). condition), selection of an appropriate treatment method and evaluation of This article presents a number of its efficiency. computationally-based methods for extracting biomechanical parameters, DESCRIPTION intensively using a computer vision approach. Such parameters are used in Different dysfunctions of an acetabulofemoral primary diagnostics, certain calculations joint result in pain and/or motility restrictions. are made when observing the course Such dysfunction may be congenital, caused of treatment, while other methods are by trauma or joint operation conditions employed only in exceptional cases when (such as long-term joint strain exceeding standard methods fail to provide enough joint stress limits), natural wear on joints information. Grid platform provides ‘on or may be a result of other pathologies. demand’ computational resources, Modern medicine can help all patients suffer- as well as suitable services for such ing from these types of joint dysfunctions; computing procedures. however, treatment resources are not Algimantas Juozapavicius [email protected] VU • Ramunas Markauskas sufficient. Establishment of pathological [email protected] Vilnius University
106 | EGI User Forum 2011 user environmentS • • • • • • IMPACT The improvement and further development Socially impacting grid services from on the distributed platform of EGI of the chemistry Heavy User Community The impact of our work has materialised the molecular level components of this into the offer to the members of our software is, in fact, a clear citizens- community of GEMS, a prototype simulator oriented target that impacts not only the able to start from first principles and end work of chemists, materials technologists, up, in a complete a priori fashion, by engineers, biologists etc. working on the evaluating the signal of crossed beam development of faster and more efficient experiments and the values of reactive modelling of complex systems starting cross sections and rate coefficients. The from the microscopic level but also (and OVERVIEW products already implemented in tools used for that purpose (the KEPLER even more) the everyday activities of EGI. The first of them is targeted to workflow, the data models of Q5cost and students and generic citizens. The work of the community of computa foster the communication between D5cost, the grid framework of GriF) drive tional chemists and molecular scientists the EGI grid and other platforms the user through the massive calculations CONCLUSION is at the core of several innovations and (like that of ENEA). The second of needed by the above mentioned packages technological advances of modern society. them is a workflow used for the to allow a realistic modelling based on a The porting of key ab initio approaches to To this end we have joined our efforts assemblage of the grid empowered complex combination of building blocks molecular simulations onto the grid, using (carried out within the COMPCHEM VO) molecular simulators. The third one is in which ab initio treatments, dynamics, tools developed by other communities and with those of some members of the a framework based on a JAVA Service kinetics, fluid dynamics, statistical sampling developing new tools ad hoc designed for GAUSSIAN (CYFRONET) belonging Oriented Architecture (SOA) designed can be combined together with statistical the chemistry and molecular science, has to the same computational chemistry to facilitate the use of the grid by non- treatments and rendering techniques to shown to be highly effective not only for area to prepare the ground for joining specialists by properly selecting the build versatile user centric instruments. further development of grid technologies other communities like MoSGrid (UCO) user interface, the computing element and related chemical applications but also and ENEA-GRID (ENEA) in building a and the storage element. Examples of the application of this approach for paving the way for developing higher Heavy User Community (HUC). Our • Electronic structure: this is a layer of have been worked out for software education virtual campus technologies, effort has focused on adopting (with application software made of highly devoted to the construction of scenarios governing emergencies, designing appropriate adaptations) some basic successful quantum chemistry packages for the production of secondary pollutants, innovative processes and materials for grid tools, including those designed and (like GAMESS, GAUSSIAN, MOLPRO, to the labelling of chemical products and to the quality of life, energetic problems and implemented by other HUCs to support DALTON, MOLCAS, NWCHEM, the assemblage of distributed repositories environmental issues of high social impact. their users, by assembling suitable TURBOMOLE, etc.), which are at present for learning objects. The development workflows for chemistry and molecular solid foundations of any determination of of grid implemented versions of this science applications, by managing related molecular structures and properties. software that is being taken care of in our repositories and proposing appropriate • Dynamics: this is a layer of molecular laboratory is aimed at making sustainable, mechanisms to foster the formation of dynamics codes like either VENUS, within the chemistry community, the collaborative research within the community. DL_POLY, GRO-MACS, CPMD, development of grid tools and applications CP2K, NAMD, etc. treating molecular and at producing as well high social and DESCRIPTION systems as an ensemble of particles or economic benefits by tackling areas of RWAVEPR, MCTDH, FLUSS, ABC, etc. applications relevant to environmental, The work focused on the design and treating molecular systems as quantum health and educational aspects. development of ab initio grid empowered waves and therefore requiring a larger molecular simulators of complex physical size of node memory. phenomena and technologies to the end • Multiscale and statistical treatments: of assembling study cases for building grid this is a set of services based on grid services of social relevance. simulations of natural structures and processes based on multi-scale and Among the software tools and computer multi-physics approaches starting applications considered for this purpose are: from the nanoscale level and including • Middleware, workflows and frame when appropriate fluid dynamics, works: this is a set of tools like FARO, randomisation, virtual reality, data KEPLER and GriF that are being mining, etc. analysed and compared with other Antonio Lagana [email protected] University of Perugia
108 | EGI User Forum 2011 user environmentS • • • • • • e-Social Science development in Taiwan: scalable IMPACT CONCLUSION social simulation using commodity computing The e-Social Science community in A fruitful international collaboration of Taiwan has been built up within the e-Social Science has been established EUAsiaGrid project. The population-scale between Europe and Asia. This project also social simulation model using commodity shows a promising result for a feasibility computing power can be tractable under study of a scalable social simulation. current simulation frameworks. The agent- based simulation has been encouraged as a complementary social science research OVERVIEW DESCRIPTION method for the social scientists in Taiwan.
This talk describes the overview of the This project aims to build up social process of developing the agent-based simulation models in several application modelling of simulation, the current domains, including models of demo development of model construction, graphics and migration in Taiwan, and how to utilise multiple CPU cores and the electorate voting model for political an investigation of the scalability of the science. In this project, we describe resulting code using grid/commodity experiences made in the development of computing. an example model that utilises multiple CPU cores and investigate the scalability of the resulting code. The feasibility study and stability analysis of the model has been completed. We argue that commodity compute resources and commoditised simulation frameworks can now be used to simulate real-world populations and to use these simulations to investigate social phenomena such as migration.
H sin-Yen Chen [email protected] ASGC • Jing-Ya You [email protected] ASGC
110 | EGI User Forum 2011 user environmentS • • • • • • as a bridge between the project and the CONCLUSION Access earth science data from the EGI infrastructure earth science community in EGI. We will discuss the available OGC services Problems, recent developments and and will show that an OPeNDAP solution different approaches to solution will be is upcoming. discussed in this session.
IMPACT
The access to data centres from the EGI OVERVIEW DESCRIPTION infrastructure supports existing and new Earth Science application and will be used Accessing data stored outside of EGI We will present the work of the earth for new deployments. during the execution of a job has been science activity of EGI-InSPIRE SA3, which in discussion since the grid was first is devoted to earth science data access. used by Earth Science (ES). There is, for The activity is investigating and following example, high interest in the community the development of specialised data in using the OPeNDAP protocol and access and management methods in the corresponding Hyrax Data Server. Due area of earth science. to the many different technologies, data centres, standards and pseudo standards, Accessing data stored outside of EGI however, it seems that no general solution during the execution of a job has been in can be found. Even if Open Geospatial discussion since the grid was first used Consortium (OGC) compliant web services by ES. There is, for example, high interest are adopted, interfaces are required. in the community in using the OPeNDAP protocol and corresponding Hyrax Data Server. Hyrax offers many interesting features that go beyond high performance access to distributed datasets, such as an extensible component-based architecture, multiple data representations, static and dynamic THREDDS catalogues and more. Due to the many different technologies, data centres, standards and pseudo standards, however, it seems that no general solution can be found. Even if OGC compliant web services are adopted, interfaces are required.
The activity can be seen as the follow- up to the work that was carried out in EGEE III based on a memorandum of understanding with the Ground European Network for earth science Interoperations – Digital Repositories (GENESI-DR) project. The work of GENESI-DR is continued in the GENESI Digital Earth Community (GENESI- DEC) project that has started in 2010 and aims to extend and improve both the infrastructure and target audience of the projects network. The activity tries to act H orst Schwichtenberg [email protected] Fraunhofer SCAI
112 | EGI User Forum 2011 user environmentS • • • • • • Grid computing for electrophysiology IMPACT CONCLUSION The grid computing services designed The Lithuanian grid computing and and developed for EEG analysis, are technical resources, together with the implementing the operational process for combination of grid technology and electrophysiology. The segmented EEG HPC procedures, enabled us to use parts, as initial objects of the database minimal amount of efforts and in a short of EEG signals, together with suitable range of time to create and develop the computational procedures, reflect the efficient model of electrophysiological following attributes: processes, as well as to support doctors OVERVIEW neurons and it's widely used in diagnosis 1. correlation of spike number in EEG and in their diagnostics and drug selection of neurological disorders related to spike parameters with type of epilepsy processes to save them time. Such a In electrophysiology many diagnostic epilepsy. The seizure of epilepsy leads to and clinical data, system is highly evaluated by medical processes are based on digital signal transient disturbances of the EEG signal. 2. spikes and segments clustering, researchers and doctors. detection and processing procedures. Thus usually epileptic EEG data contains 3. locating damaged areas of the brain, In epilepsy diagnosis and treatment the transient components and background 4. numerical estimation of amount of electroencephalography (EEG) signals activities. The epileptic transient activity waves of different rhythms (wave types are one of the main tools. However the usually appears as sharp spikes in signal of EEG) and their correlation with the inspection of EEG in a ‘manual way’ is which occur randomly with short duration type of epilepsy, sleepiness, etc. very time consuming. Automatic extraction of 20-70ms. One of the main tasks in Thus grid resources offer a possibility to of important EEG features provides an EEG signal processing is spike detection, manipulate data, to process big amounts entire new level for doctors, for disease as their recognition is significant for of data and to produce various analysis diagnostics and exploration, it also saves clinical diagnosis of epileptic disorders. on it. Grid computing procedures are doctors' time. EEG analysis using data In hospitals a method of visual/manual produced on EEG segments (from the mining methods gives powerful tools for detection of spikes in EEG recordings by database), on entire EEGs, primary and further exploration. In this article, methods an experienced neurologist is widely used. secondary attributes of them (such to extract some of these features – drowsi In the case of long EEG recording, this as KDS, KSS, spike index, slow eye ness score and centre-temporal spikes process is more time consuming. Thus movement recognition, distribution of spike are analysed. For spike detection a new automatic spike detection methods are appearance time, amplitude, duration, method based on morphological filters is needed. Such a detection method, based shape, others). Theoretical models of such used. Also a database design is proposed on mathematical morphology algorithms, data manipulation and computational in order to allow easy EEG analysis and was developed by the authors, EEG modelling are based on attributed graphs provide data accessibility for data mining data being provided by Vilnius University representation, where nodes and edges algorithms to be developed in the future. Children's hospital. Input files are of have many parameters and pre-computed Grid computing platform is an efficient way European Data Format (EDF). The grid attributes. In the case of full EEG data for implementation of algorithms needed. infrastructure is used to create a virtual (for one patient) the computational model repository for such data, making data to be performed requires a lot of CPU DESCRIPTION accessible for authorised manipulations by hours (simultaneously involving hundreds hospitals, doctors, computing processes. of cores). Grid technology allocates such One of the main subjects of research is The functionality of a virtual data repository resources dynamically, actually producing measurement of brain activity recorded includes ‘raw’ data storage (together with all such computing ‘on demand’. Grid is as electroencephalograms (EEGs). A suitable indices), data anonymisation, data also able to allocate enough resources for multichannel EEG signal from one person segmentation into attributed segments, data mining analysis (data clusterisation, is usually a collection of about 12 to 128 data warehousing for complex algorithms, pattern recognition, etc.) of all data signals, which are recorded with 200Hz which are required by doctors for diagnostics, accumulated in the data repository. rate or similar and may last from some selection of drugs, shortening decision time. minutes to some hours. This data requires large amounts of storage, sophisticated data manipulation, complex data transformations. EEG is a summation of Algimantas Juozapavicius [email protected] VU • Gytis Bacevicius electrical activities generated by cortical [email protected] Vilnius University
114 | EGI User Forum 2011 user environmentS • • • • • • Grid-empowered molecular simulators IMPACT CONCLUSION of crossed beam signals The main impact of this work is to be The design and the implementation of recognised in the possibility of simulating the last segment of the GEMS simulator a chemical reaction from first principles has been carried out. This means that the and comparing calculated results directly simulator is now ready to be used as a with those of the related CMB experiment. powerful instrument in quantum reaction This affects both the theoretical and dynamics. We are still in the test phase experimental sides of dynamical studies. In with respect to the observables block fact, thanks to this approach the feasibility of the simulator in order to extend the OVERVIEW the angle and the velocity of the scattered of a CMB experiment could be predicted workflow to four atom reactions. products (expressed in the laboratory a priori by simulating cross sections and In the last years, several theoretical and frame) starting out of the related computed possible product distributions. At the same experimental efforts have been spent in dynamical quantities in the Centre of time, on the theoretical side the validity of the field of quantum reactive scattering Mass (CM) frame (angular distribution and a proposed potential energy surface could and this has fostered the development of translational energy distribution). be characterised to a very fine detail if an molecular simulators. Our group is working access to grid platforms can be obtained. at extending GEMS (Grid Empowered As a study case we have chosen the Moreover, the development of the Molecular Simulator), to simulate Cl + H2 system. To this end, an accurate chemical simulator on a grid environment the experimental signal of Crossed ab initio Potential Energy Surface is also important from a computational Molecular Beams (CMB) experiments. (PES) has been incorporated inside the science point of view because it prompts In this communication we report on the quasi-classical trajectory programme the production of specific innovative implementation of a prototype version of (VENUS96.F). In order to simulate the technological solutions. the GEMS block generating the virtual observables of the experiment, we intensity of the product beam of a CMB have implemented inside the VENUS reactive scattering experiment out programme the calculation of the fixed of the integration of a large number of angle velocity distribution of the products in quasi-classical trajectories for a three the laboratory frame out of the theoretical atom reaction. CM angular and translational energy distributions. DESCRIPTION Production runs of the simulator were GEMS is made of different blocks each performed for the system proposed for of which is devoted to a particular task. the study case. To this end, the modified In particular, the skeleton of the simulator version of VENUS implemented on the grid (the production of ab initio electronic was run for 400,000 trajectories at different energy values, the fit of these values values of the rotational quantum number j. to work out a potential energy surface Finally, the comparison between calculated and the block devoted to dynamics and experimental observables has been calculations) has already been designed carried out. and implemented whereas the block devoted to the assemblage of the virtual experimental observables is still in the test phase. For this reason, as already mentioned our efforts have been spent in implementing the last block of the simulator. As a matter of fact, we have designed and implemented on the grid a Fortran code (called gmtsigma.f) best fitting the measurements of the intensity of the CMB experiments taken as a function of Antonio Lagana [email protected] University of Perugia
116 | EGI User Forum 2011 user environmentS • • • • • • compose environmental reports for CONCLUSION Earth science oriented data processing tools and decision makers and citizens; applications for the Black Sea catchment basin 5. Decision maker and citizen application – The presentation mainly highlights provides the decision makers with the issues and the experiments on the interactive and graphical tools interoperability between geospatial to access the private environmental and grid platforms in order to support reports. The user may visualise data high power computation requirement that make possible statistical analysis applications. The SWAT model covers and predictions. the huge area of the Black Sea catchment region. It is a quite complex model that OVERVIEW The portal consists of a set of Web IMPACT requires the high performance resources applications through which the users of the grid. EnviroGRIDS (Black Sea Catchment access the system resources such The regular users visualise the reports Observation and Assessment System as spatial data, hydrologic models, generated by the specialists as a result The experiments have revealed the supporting Sustainable Development) environmental scenarios, data processing of executing different environmental potential of the grid related technologies. [www.envirogrids.net] is a 4-year FP7 tools, visualisation facilities, environmental scenarios. The input data for the reports New user interaction techniques and Project aiming to address the subjects of reports, and training materials. are built up by the specialists by running platform interoperability are the main ecologically unsustainable development hydrological models of the Black Sea directions of the future research work. and inadequate resource management. The main user application categories catchment area and by processing related The project develops a Spatial Data provided by the portal are: satellite data. All data sets required to enviroGRIDS Project: www.envirogrids.net Infrastructure of the Black Sea catchment 1. data management – provides the build up the hydrological models, environ region. The geospatial technologies user with spatial data management mental scenarios, and spatial models are http://users.utcluj.ro/~gorgan offer specialised functionalities for earth and operations. The user may enter provided and entered into the system by science-oriented applications as well data and metadata, visualise, modify, the data providers. as the grid-oriented technology that is update, and remove spatial data from able to support distributed and parallel the data repositories; EnviroGRIDS functionality gathers services processing. One challenge of the 2. hydrologic model management – provided by various technologies such enviroGRIDS project is the interoperability provides earth science specialists as SWAT related modules, Collaborative between geospatial and grid infrastructures with hydrologic model configuration, Working Environment (CWE), Uniform by providing extended features of both scenario and model development, Resource Management (URM), gProcess technologies. The enviroGRIDS system model calibration and scenario running. and ESIP Platforms, and experience on resources are accessible to the large One of the water quality models other research projects like GiSHEO, community of users through the BSC-OS that will be used is SWAT (Water SEE-grid-SCI, and EGEE. Portal that provides Web applications Assessment Tool) [http://www.brc. for data management, hydrologic model tamus.edu/swat/index.html]. SWAT is EnviroGRID system is developed on the calibration and execution, satellite image a model designed to estimate impacts gLite middleware available on EGI, SEE- processing, report generation and of land management practices on grid and enviroGRIDS VO infrastructures. visualisation, and a virtual training centre. water quantity and quality in complex watersheds. The SWAT model requires DESCRIPTION specific information about weather, soil properties, topography, vegetation, The portal publishes through the Web and land management practices of the applications the geospatial functionality watershed; provided through Web technologies and 3. Satellite data processing – the specialist the high power computation supported may process satellite data and by the grid technologies. There are five images in order to search for relevant categories of users such as data providers, information (e.g. land cover, vegetation, earth science specialists, decision makers, water, soil composition, etc.); citizens, and system administrators. 4. Data visualisation and report – the specialists visualise various spatial data in different formats and views and Dorian Gorgan [email protected] Technical University of Cluj-Napoca
118 | EGI User Forum 2011 user environmentS • • • • • • The grid rocks IMPACT CONCLUSION The primary benefits of these projects are In addition to facilitating use of the that they will allow earth scientists at the database for larger and more complex University of Glasgow to investigate the analysis, this work offers insights into how potential of batch computing for this type existing web based databases could be of analysis. An additional benefit is that transitioned to the grid when they outgrow these project types lend themselves to the their existing infrastructure, and how the wider European Earth Science community user experience may be re-examined through grid deployments over specific as part of such a transition. The overall OVERVIEW DESCRIPTION departmental projects. performance of the database within a grid environment rather being solely web-based The development of web based As PaleoDB has grown, the primary will be discussed and limitations of both technologies has seen a proliferation of methods for accessing the data have systems explored. e-science related projects, globally. One not kept pace with the user demands. such project is the Paleobiology database In collaboration with the Earth Sciences www.scotgrid.ac.uk/Projects.html (http://paleodb.org). This system "has department at the University of Glasgow, been organised and operated by a multi- we have deployed a mirror of the database disciplinary, multi-institutional, international inside our existing grid cluster, and worked group of paleobiological researchers. Its on ways to provide combined compute and purpose is to provide global, collection- data access to this via grid interfaces. based occurrence and taxonomic data for marine and terrestrial animals and plants of An additional Earth Science project is the any geological age, as well as web-based deployment of modelling software within software for statistical analysis of the data." the cluster at Scotgrid, Glasgow to allow (Source http://paleodb.org.) for the detailed analysis of prehistoric movements in tectonic plates including We intend to work closely with the geo-seismic activity. This research will Paleodb.org as described below. be conducted at both the regional and continental scales. A full analysis of the technical implementation and issues surrounding this type of research will be discussed including the benefits and drawbacks of batch computing in a grid framework on this type of research.
M ark Mitchell [email protected] GridPP
120 | EGI User Forum 2011 user environmentS • • • • • • e-science on earthquake hazard mitigation IMPACT CONCLUSION With accurate regional seismic data, domain- This work is a great example of the specific services and the e-Infrastructure, added value of e-Science and regional we are able to greatly enhance earthquake collaboration. Further federation of various hazard alleviation and conduct unique regional resources and customisation is seismology research in Asia. planned, which will be compatible with the European Grid Infrastructure.
OVERVIEW DESCRIPTION
In EUAsiaGrid, ASGC coordinates the Under the EUAsiaGrid framework, we construction of a pilot virtual research have been implementing a series of environment (VRE) on earthquake disaster forward wave propagation simulations and mitigation, by streamlining the primary analysis applications. From the generation research process and components; of accurate earthquake wave propagation, seismic sensor network, seismic wave on-demand simulation of seismograms propagation simulations and earthquake for any place on earth for any event, to the data centres. Based on these, Strain cost reduction of seismic wave propagation Green's Tensor (SGT) methodology was analysis by using the pre-calculated Strain also deployed to support hazard modelling Green's Tensor (SGT) function method. more efficiently in Taiwan. All the VRE and The work is not just porting designated e-Science applications are now supporting computational models to gLite-based Asia-wide collaboration and customisation. Grids, but also establishing research- oriented production services and long- term collaboration mechanisms among the partners. Given the SGT database (a 50-station 20 second case study of Taiwan running for 76,800 CPU hours has been done), hazard mapping and disaster potential analysis could be done much more efficiently without re-calculating the required seismograms. Development of the VRE on seismic hazard mitigation with the aforementioned applications will be described in this presentation.
E ric Yen [email protected] Academia Sinica • Hsin-Yen Chen [email protected] Academia Sinica
122 | EGI User Forum 2011 user environmentS • • • • • • An overview of CMS workload management IMPACT CONCLUSION for data analysis CRAB is currently starting the re-writing The CRAB Workload Management process with the aim to better cope with System is widely used for accessing CMS the foreseen growing amount of data and distributed data by end-users. During the user community. This process is providing first year of the data taking the system an extremely important opportunity for coped well with the physics needs. In developers to share a set of functionality order to reduce the operations load and already implemented by other Workload improve the scalability, CRAB is going to Management Systems (WMS). These be re-written. This process has also the OVERVIEW DESCRIPTION latter are the systems which have been aim to share common functionality with developed by other LHC Experiments to other existing WMS. The system is being The Compact Muon Solenoid (CMS) is one CRAB adopts a client server architecture perform grid submissions, such as Panda, developed in a very close collaboration of the four main experiments (VOs) at the implemented in Python. The client, Dirac, Alien and GANGA. with the developers of other Workload LHC and relies heavily on grid computing. which provides the user with a batch- Management Systems as well as the The CMS computing model defines how like command line application, has the Experiment Dashboard team. the data is to be distributed and accessed responsibility of generating the user’s to enable physicists to efficiently run their proxy, packaging up the pre-compiled analysis over the data. The CMS Remote user code, and submitting the job package Analysis Builder (CRAB) is the specific to the CRAB server. The intermediate Workload Management System, that allows Analysis Server has the responsibility of the end-user to transparently access data submitting the jobs to the chosen grid in a heterogeneous pool of resources middleware, resubmitting jobs as needed, distributed across several continents. and caching the users’ output. CRAB’s design allows a transparent interaction with different grids and batch systems. CRAB has been in production and in routine use by end-users since Spring 2004. It has been extensively used between the initial definition of the CMS Computing Model in 2004 and the start of high energy collisions in 2010, attending numerous scaling tests and service challenges.
CMS is currently in the process of replac ing its workload management system. This next generation tool is named WMCore, a Python based workflow library which is used for so-called task lifecycle manage ment. The workflow library is the result of a convergence of three subprojects that respectively deal with scientific analysis, simulation and real time data aggregation from the experiment. CRAB will be part of this migration.
D aniele Spiga [email protected] CERN
124 | EGI User Forum 2011 user environmentS • • • • • • Examples of viewing mechanisms are Lastly, we think that the insight that our Getting a grip on the grid: a graphical interface to analyse the solution provides in all the data and a knowledge base to trace grid experiments provenance graphs and a query interface resources, will present an important added to find events of interest. value to the operational aspects of running a science gateway. These benefits vary IMPACT from data retention strategies, assistance for workflow debugging, and producing all The described solution delivers a set kinds of usage statistics. of tools that have a potential diverse audience. We foresee three main areas of CONCLUSION OVERVIEW Manual data collection is an error-prone impact: scientific, usability, operational. task and requires enormous manpower, The knowledge base for e-Bioinfra was Information management is challenging due to the amount of logs registered by Because of the wide and distributed scale achieved by implementing OPM and DC in science due to the variety of data the processes. An automated solution is of the resources, the ‘grid’ often becomes data models into a relational database produced by the physical instruments and needed. a black box to users, due to the fact that: management system. The rich and the amount of information generated daily 1. jobs may get scheduled arbitrarily on complex data available within the e-Bioinfra by scientists. In addition to supporting the Our approach to building a knowledge distributed resources, application were challenging enough to experiment execution, it is currently crucial base is a three-folded mechanism. First, 2. log-files will be generated on the remote test and validate our approach. These for the scientific applications to trail their the EbioCrawler is designed to gather nodes selected by the scheduler, data relate to a few thousand scientific mechanisms of performing experiments, so automatically already existing logs that 3. the user may lose access to the remote experiments, which have been performed that it is possible to trace back the resulting contain information generated by different nodes where those files reside, and using Moteur/Diane workflow system. scientific data. application systems (e.g. MOTEUR and 4. the logs may get cleaned without DIANE), including workflow descriptions prior notification. In addition, with the Provenance data for e-Bioinfra, In this work, we describe an approach and execution reports, system outputs, increased volume of data, resources, including workflow descriptions and for building a knowledge base for the communication accounts and status reports. methods and collaborators in the domain log files, is collected automatically from scientific experiments performed using the of e-science, it becomes more and the various e-Bioinfra components and e-Infrastructure for bioscience (e-Bioinfra). Secondly, a provenance repository is built more difficult to perform repeatable, transformed into a knowledge base The e-Bioinfra platform provides grid around the notion of graphs outlined by scalable, and traceable experiments. repository. This knowledge base can be workflow management and monitoring the OPM model. The repository is defined accessed for analysis of the experiment services for biomedical researchers that using a relational database schema that The distributed information is now provenance in various forms (e.g. GUI for use the Dutch grid. Our approach focuses captures the concepts of the OPM model collected in a structured way that can be provenance graphs) on gathering meaningful information from but extending it to support Events. Event- queried for various goals like experiment these services and populating it into the driven systems are common in scientific results, operational statistics, tracing error, Although establishing the knowledge knowledge base, within its proper context. environments, but their provenance is customised reporting, etc. By tracing base is fundamentally for documentation, For this, an agent-based software tool not well captured by the OPM alone. the whole history of the resources up to specific tools need to be implemented is designed and developed to retrieve, The provenance data collected by the the current state, it is possible to confirm in order to better explore the information classify and transform existing data into EbioCrawler is stored into the repository or provide evidence of the scientific in the knowledge base. Each tool could meaningful information. using the Application Programme Interface work done. Analysis and comparative be tailored to the specific needs of the (API) from the Provenance Layer Infra mechanisms, scientific opinions or application domain and the type of users. DESCRIPTION structure for E-Science Resources (PLIER), interpretations, and the results of various allowing developers to build, store and kinds of examinations may provide further www.ebioscience.amc.nl A comprehensive knowledge base gathers share (by XML serialisation) graphs using ways to improve (or debug) the actual relevant information to help scientists the OPM model. scientific experiment (workflow). clarify their research questions and to validate operational tasks. However, The third mechanism enables the analysis building and populating such a knowledge of the provenance data. Using the base, with proper and detailed information homogeneous view of the information in resulting from different sources, is a the repository based on the OPM ontology, challenge in itself. Although the information the provenance can be now transformed, is usually accessible, e.g. in logs, it is or serialised, into specific formats (RDF, Ammar Benabdelkader [email protected] PCC UvA • Mark Santcroos [email protected] not trivial to correlate pieces of data. XML, etc.) or other representations. Academic Medical Center Amsterdam • Victor Guevara Masis [email protected] Nikhef
126 | EGI User Forum 2011 user environmentS • • • • • • Shared services and tools based on the Ganga job IMPACT CONCLUSION definition and management framework The end-users of grid computing resources The Ganga team has deployed new demand that the tools they use are reliable, tools and core-functionality designed to efficient and flexible enough to meet their enhance the experience of end-users. needs. Most users, irrespective of the These developments include the advent research community to which they belong, of an interactive user-interface for task are generally not interested in developing monitoring, and an intuitive mechanism grid-access tools, and nor should they for generating error-reports that can be be. Their role is to exploit the resources accessed by user-support teams. OVERVIEW service, which allows users to track the available as effectively as possible, and status of their tasks submitted from the with minimum knowledge of how the Future developments will see an Ganga is a user-targeted job management Ganga/Diane environments. underlying technologies function. improvement in the Ganga communication tool designed to provide a homogeneous channels, including the launch of a blog to environment for processing data on a The Ganga team has also deployed a A wide range of grid-enabled tools have keep users informed of developments and variety of technology ‘back-ends’. service capable of receiving application been developed which aim to shield the provide usage tips. ‘crash reports’ (comprising a snapshot of user from the complexity of distributed We will present examples that illustrate the configuration, job parameters, input/ infrastructure technology. With this goal Longer term goals include the development how Ganga, originally developed by and for output files and command history), and in mind, Ganga has been engineered to of advanced core-features, enhancement LHC experiments, has since been adopted presenting this information in a form that provide a homogeneous environment of the data-management interface and by other, non-HEP, communities and how can be viewed and downloaded by user- for processing data on a range of packaging Ganga for inclusion into popular it can play a key role within a sustainable support teams. The Ganga client provides technology "back-ends", ranging in scale Linux distributions. support model. a simple method by which users can send from a solitary user's laptop, up to the such reports, but this service can also be integrated resources of the Worldwide LHC http://ganga.web.cern.ch/ganga DESCRIPTION exploited with the Ganga environment. Computing Grid.
Initially developed within the high-energy The error-reporting tool will be described, Ganga is, by design, a tool that can be physics (HEP) domain, Ganga has since with specific reference to how it has been extended for a given user-community to been adopted by a wide variety of other adopted by the CMS VO, a community who add new functionality, whether that be user communities as their default analysis have their own task-management system enhancements for data-analysis routines, and task-management system. The in place of Ganga, yet who were able to or integration of tools aimed at improving modular nature of Ganga means that, if make use of the web-based interfaces and the end-user experience and support desired, disparate communities are able to underlying technology to deploy an error- mechanisms. develop their own suite of tools that remain reporting service for their users. independent of both the core code and those of other communities.
This presentation will use case-studies to illustrate the ease with which non-LHC communities (such as those engaged in medical research, or running Monte Carlo simulations on the Grid), have adopted Ganga as their chief job-submission tool.
In addition to providing a stable platform with which to conduct user analysis, the Ganga development team has deployed U lrik Egede [email protected] Imperial College London • Mike Kenyon [email protected] a range of supporting tools and interfaces. CERN • Jakub Moscicki [email protected] CERN • Ivan Antoniev Dzhunov We will present developments of the job [email protected] University of Sofia • Lukasz Kokoszkiewicz lukasz.kokoszkiewicz@ monitoring interface, a lightweight tool cern.ch CERN • Laura Sargsyan [email protected] CERN/Yerevan Physics Institute • Edward integrated into the Dashboard monitoring Karavakis [email protected] CERN • Julia Andreeva [email protected] CERN
128 | EGI User Forum 2011 user environmentS • • • • • • Cheminformatics platform for drug discovery IMPACT CONCLUSION application on grid We introduce GVSS, a user-friendly graphical Molecular docking simulation is a time- user interface desktop application for using consuming process that searches this grid-enabled virtual screening service. exhaustively all correct conformations Through the GUI, the end-users can of a compound. However, the massive easily take advantage of grid computing in-silico processes benefit from the high- resources for large-scale virtual screening. throughput computing grid technology. Furthermore, they can even upload their Providing intensive computing power own target and ligands, and do the same and effective data management, the OVERVIEW However, massive molecular docking docking process, visualisation and analysis production e-Infrastructure (EUAsia VO) required intensive computing power with this GUI, of course including the enables opportunities for an in-silico drug To extend e-science applications to public and effective data management. A grid advanced refinement docking simulations. discovery platform. We deployed this high- health, especially in Asian countries, the computing framework was established for The end-users can finally have a real grid- throughput in-silico massive molecular EUAsiaGrid in the past two years has AutoDock 3.0.5 and evaluated for its ability enabled desktop utility for their daily research. docking service benefiting from state-of- established an in-silico high-throughput to process large-scale molecular docking. the-art grid technology to the activities drug screening platform, GAP Virtual Grid is an ideal environment, which can To improve the application domain and for designing the cheminformatics Screening Service (GVSS). Providing provide large-scale and on-demand bridge the gap between experimental and platform on EUAsiaGrid infrastructure intensive computing power and effective resources, including computing and in-silico endeavour, firstly, we will focus since 2010. Furthermore, these activities data management, the production storage resources. on common interest and also on the most also facilitate more biomedical e-science e-Infrastructure (such as EGI and important issues of public health. The applications, such as other diseases and EUAsiaGrid) enables opportunities for Inspired by the successful experiences activity, therefore, has two approaches compounds profiling. in-silico drug discovery on neglected and on Avian Flu Data Challenges, ASGC aimed to accelerate and optimise the drug emerging diseases, for instance, Avian developed the GVSS application package discovery process by computing method: Influenza and Dengue Fever. that incorporates the EGEE gLite middle firstly, to increase the collection of the ware DIANE2 and AMGA. Therefore, compound library, so as to extend the To extend the compound 3D structure ASGC coordinated the Dengue Fever present compound structure database; database and compound information Data Challenge via the EUAsiaGrid VO in secondly, to execute in-silico to disease management, we will create 3D structure June 2009. The objective is to use grid target, such as critical proteins in cell wall conformers of known compounds for in- enabled high-throughput screening for formation and viral replication pathway. silico screening and post structure-activity structure-based computational methods Examples are: penicillin-binding proteins relation analysis. to identify small molecule protease (PbP) for antibiotics development and inhibitors. In addition to the ZINC CDI non-structure protein (e.g. NS3 and NS5) in DESCRIPTION 300,000 compounds library we used, we anti-flavivirus drug development. Validating established the new ZINC 800,000 ligands the computer work with experimental Cheminformatics is able to come into its and Chembridge 300,000 ligands, a free assay is also the focus of this activity, which own as a discipline and as an enabling database of available compounds library bridges e-science activity and bioscience. science. The structure based molecular selected for virtual screening. docking simulation is a common method for predicting potential interacting complexes of small molecules in protein binding sites. By promoting the integration of data, applications and workflows, it is providing scientists with broad access to information and methods. As cheminformatics systems become more prevalent, we can expect them to appreciably enhance our use of previous knowledge and our ability to generate new knowledge – significantly improving the drug discovery and development process. H sin-Yen Chen [email protected] ASGC
130 | EGI User Forum 2011 user environmentS • • • • • • IMPACT our portals in dedicated servers exploiting VisIVO, visualisation in different grid environments, existing grid infrastructures. We envisage from gLite to desktop grid The main characteristic of modern that using smart phones via these portals, astronomical datasets is extremely large potential users (researchers, citizen sizes, typically requiring storage in a scientists, science centre visitors) would distributed way. Gaining an insight into be able to exploit VisIVO functionality such datasets requires very powerful on-the-go, accessing archives of large- statistical and data analysis algorithms scale datasets and generating images that can be fully realised, in a cost- or movies by exploiting the power of grid effective manner, only through gateways infrastructures. OVERVIEW services based on the VisIVO Server function- to grid and cloud infrastructures. ality and VisIVOSmartphone – a web Forthcoming astronomical surveys are CONCLUSION The availability of distributed computing application allowing modern smart phones expected to collect petabytes of raw infrastructures has been growing rapidly to exploit the VisIVOServer functionality. data resulting in massively large-scale Our future work will focus on fully porting in the last few years. Many scientific fields distributed databases requiring that VisIVO Server and VisIVO Web to grid can now produce high-resolution numerical We are currently in the process of porting astronomers interact with them using environments, more specifically using simulations with multi-dimensional datasets VisIVO Server on grid environments. VisIVO very high performance data analysis and desktop grids and gLite. The exploitation in the order of several petabytes. An Server consists of three core components: visualisation tools. of grid technologies will enhance essential part of modern scientific research VisIVO Importer, VisIVO Filter and VisIVO performance significantly by supplying the is the necessity to employ computer Viewer respectively. To create customised INAF-OACT is involved in numerous required computational capability. A very graphics and scientific visualisation tools views of 3D renderings from astrophysical grid activities, e.g. the COMETA exciting possibility is the deployment of this for appropriately displaying such datasets, data tables, a two-stage process is consortium that is developing a large grid technology for visual discovery in scientific so as to allow scientists to perform efficient necessary. First, VisIVO Importer is used infrastructure, not only for research but disciplines apart from astrophysics. visual discovery. We present VisIVO, a to convert user datasets into VisIVO also for industrial applications. INAF is also Furthermore, a pilot application built upon powerful environment for exploring highly Binary Tables (VBTs). Then, VisIVO Viewer a full partner of the Italian Grid Initiative VisIVOSmartphone is underway for public complex multi-dimensional astrophysical is invoked for display. VisIVO Filters are (IGI) which regards science gateways as an engagement activities in astrophysics. Our datasets on the grid. collections of data processing modules extremely valuable service to be provided ultimate vision is users (researchers, citizen able to extract interesting features in to the scientific community. IGI has scientists or even science centre visitors) VisIVO provides an integrated suite of tools datasets for performing visual discovery. expressed a strong interest in developing being able to reserve resources for data and services that can be used in many visualisation tools on the grid. IGI is also analysis and visual discovery as required other scientific fields. VisIVO allows users Depending on the structure and size part of the European Grid Infrastructure (possibly involving several production grids) to visualise meaningfully highly-complex, of datasets in consideration, the data together with many other European NGIs. and retrieving results on smart phones on- large-scale datasets and create movies of exploration process could take several the-go irrespective of internet access these visualisations based on distributed hours of CPU for creating customised We are also working with science centres or geographical location. infrastructures. The deployment of VisIVO views, and the production of movies could on a pilot experiment to identify novel ways on the DG and gLite is carried out with the potentially last several days. For this reason for exploiting the functionality of VisIVO http://visivoweb.oact.inaf.it support of EDGI and EGI-InSPIRE projects. an MPI parallelised version of VisIVO can using smart phones for large-scale public play a fundamental role in increasing engagement activities. The use of DG DESCRIPTION performance, e.g. it could be deployed could significantly improve this experiment. automatically on nodes that are MPI aware. Within the activities in the EDGI and EGI- VisIVO supports high-performance, multi- A central concept in our development InSPIRE projects we will port and install dimensional visualisation of large-scale is thus to produce unified code that can astrophysical datasets. Users can rapidly run as necessary either on serial nodes obtain meaningful visualisations while or in parallel by using HPC oriented grid preserving full and intuitive control of the nodes. Another important aspect, to obtain relevant parameters. VisIVO consists of as high performance as possible, is the VisiVO Desktop – a stand-alone application integration of VisIVO processes with grid for interactive visualisation on standard nodes where GPUs are available. We have PCs, VisIVO Server – a platform for high selected CUDA for implementing a range of performance visualisation, VisIVO Web – computationally heavy modules. a custom designed web portal supporting U GO BEcciANI [email protected] INAF
132 | EGI User Forum 2011 user environmentS • • • • • • • A Virtual Research Environment and infrastructures (import/export/ e-Infrastructures integration with gCube (VRE). Through VREs, groups of publishing) and it supports common users have controlled access to file management features (drag and distributed data, services, storage, and drop, contextual menu); computational resources integrated • A time-series framework offering under a personalised environment. tools to manage large datasets by VREs support cooperative activities supporting the complete time-series such as: metadata cleaning, lifecycle (validation, curation, analysis, enrichment, and transformation by and reallocation). It offers tools to exploiting mapping schema, controlled operate on multi-dimensional statistical OVERVIEW DESCRIPTION vocabulary, thesauri, and ontology; data. It supports filtering, grouping, processes refinement and show cases aggregation, union, mining, and plotting; Delivering an e-Infrastructure service to gCube is a large software framework implementation; data assessment; • An ecological niche modelling suite to large organisations is a complex task designed to abstract over several expert users validation of products predict the global distribution of marine that requires the integration of several technologies and offer them through a generated through data elaboration or species, that generates colour-coded technologies. The complexity of this well-formed set of APIs. gCube consists of simulation; sharing of data and process species range maps using half-degree service resides on: several packages offering: with other users. latitude and longitude blocks by 1. very rich applications and data • Access to several storage back- interfacing several scientific species- collections are maintained by a ends tailored for different needs. For IMPACT databases and repository providers. multitude of authoritative providers; example, it offers a storage server: for It allows the extrapolation of known 2. different problems require different multiple-version software packages; for gCube doesn’t hide the infrastructure species occurrences to determine execution paradigms: batch, map- scientific data-sets stored as tables; for middleware. It is not another layer. Rather it environmental envelopes (species’ reduce, synchronous call, message- time series with an OLAP interface; for turns infrastructures into a utility by offering tolerances) and to predict future queue, etc.; structured document objects; for geo- single submission, monitoring, and access distributions by matching species 3. key distributed computation middleware coded datasets compliant with OGC; facilities. It offers a common framework to tolerances against local environmental exist: gLite, Globus, and Unicore for and finally for storing files; programming in the large and in the small. conditions (e.g. climate change and grid-based wide resource sharing; • Management of metadata in any format It allows concurrent exploitation of private sea pollution). Condor for site resource sharing; and schema that can be consumed virtualised resources organised in sites Hadoop and Cassandra for cluster by the same application in the same with resources provided by IaaS and PaaS CONCLUSION resource sharing, etc.; Virtual Organisation; cloud providers. 4. several standards exist in the • A process execution engine to manage gCube is currently deployed in the same domain. the execution of software elements By using its set of facilities, several D4Science production infrastructure and in a distributed infrastructure under scientific applications have been provides to the Fishery and Aquaculture gCube, the D4Science empowering the coordination of a composite plan implemented and delivered to the Fishery Resource Management communities a technology, offers solutions to abstract that defines the data dependencies and Aquaculture Resource Management large number of APIs and highly specialised over differences in location, protocols, among its actors. It supports several communities delivering the following scientific applications running on an and models by scaling no less than the computational middlewares without applications (among others): e-Infrastructure service which hides the interfaced resources, by keeping failures performance compromises. A task • A collaboration-oriented suite providing complexity of the underlying heterogeneous partial and temporary, and by being can be designed as a workflow seamless access and organisation set of middleware systems, standards, autonomic reacting and recovering from of invocation of different code facilities on a rich array of objects data types, metadata schemas, etc. a large number of potential issues. components (services, binary (e.g. information objects, queries, executables, scripts, map-reduce files, templates, time-series). It offers www.gcube-system.org jobs, etc.); mediation capabilities between the • A transformation engine to transform external world objects, systems data among various manifestations. This engine is manifestation and transformation agnostic by offering an object-driven operation workflow. It is extensible through the addition of transformation-programme plugins; P edro Andrade [email protected] CERN • Pasquale Pagano [email protected] CNR • Andrea Manzi [email protected] CERN
134 | EGI User Forum 2011 user environmentS • • • • • • Distributed multi-scale computing: IMPACT CONCLUSION the MAPPER project MAPPER is driven by seven exemplar MAPPER started on 1 October 2010, applications from five user communities and is now producing first results. It is (virtual physiological human, computational expected that first demonstrations of biology, fusion, hydrological engineering, Distributed Multi-scale Computing will be nano material science), and these available during the conference. MAPPER communities are specifically targeted. will quickly move forward, and it expects However, our solutions are generic and will to be able to offer a first Distributed Multi- enable distributed multi-scale computing scale Computing environment for external OVERVIEW DESCRIPTION for any multi-scale model fitting into our projects when entering its second paradigm, and MAPPER therefore opens project year. Today scientists and engineers are Driven by seven challenging applications up to other user communities as well. commonly faced with the challenge of from five representative scientific domains The research leading to these results modelling, predicting and controlling (fusion, clinical decision making, systems MAPPER partners have significant trans- has received funding from the European multi-scale systems which cross scientific biology, nanoscience, engineering), Atlantic grid and HPC experience, and Union Seventh Framework Programme disciplines and where several processes MAPPER deploys a computational science have been involved very actively in TeraGrid under grant agreement n° 261507 (the acting at different scales coexist and environment for distributed multi-scale and with the US Department of Energy MAPPER project). interact. Such multidisciplinary multi- computing on and across European laboratories. We collaborate with the US scale models, when simulated in three e-Infrastructures. By taking advantage of TeraGrid to integrate infrastructures across www.mapper-project.eu dimensions, require large scale or even existing software and services, as delivered the globe. extreme scale computing capabilities. The by EU and national projects, MAPPER MAPPER project develops computational will result in high quality components for strategies, software and services for today’s e-Infrastructures. We develop tools, distributed multi-scale simulations across software and services that permit loosely disciplines, exploiting existing and evolving and tightly coupled multi-scale computing European e-Infrastructure. in a user friendly and transparent way. We integrate our applications into the MAPPER environment, and demonstrate their enhanced capabilities.
MAPPER integrates heterogeneous infrastructures for programming and execution of multi-scale simulations. We reuse as much of the existing infrastructural and software solutions as possible. The MAPPER solution is developed on top of existing e-Infrastructures without the necessity to modify already deployed components. The functionality to be delivered is realised as extensions to existing e-Infrastructures. The integration is done using well defined APIs and standard based interfaces, thus reducing potential impact of changes on middleware level components.
Alfons Hoekstra [email protected] University of Amsterdam
136 | EGI User Forum 2011 user environmentS • • • • • • Theompi: a large MPI cluster on the grid IMPACT CONCLUSION for theoretical physics The INFN theoretical physics community The support of the new granularity consists of 700 FTE researchers distributed attributes in the gLite middleware gives a over 28 sites and involved in 60 research real chance to use grid clusters for both projects. In this community there is a MPI and multithreaded applications. We widespread need for massive parallel decided to start with a single large cluster, applications on HPC systems with next but the flexibility of this mechanism will neighbours high speed inter-connections allow the integration of other (old and new) (involving Lattice Quantum ChromoDynamics, clusters in a near future. OVERVIEW DESCRIPTION Fluid Dynamics, Numerical Relativity) that in the past used small or medium-sized http://wiki.infn.it/cn/csn4/calcolo/ We are reporting about the first experience The cluster has been installed at the dedicated local clusters. The Theompi csn4cluster/home using a large grid-enabled MPI-cluster and INFN-Pisa site and is grid-enabled through project aims to provide the community the problematic/advantage of its use for a CREAM based computing element. with a transparent and flexible mechanism massive parallel applications. The production version of the software to share the HPC resources. has been enhanced and customised In recent years EGEE/EGI has proven to with an experimental patch to better be a robust and scalable infrastructure support the parallel job execution on a for sequential scientific computation, but multicore environment, according to the hardly usable for parallel programming. recommendations given by the EGEE The situation changed following the June MPI WG. This patch has been tested and 2010 proposal of the EGEE MPI WG deployed in collaboration with the gLite (Working Group) for JDL attributes to middleware developers. select the slots needed for the execution and allocation of a parallel application. In a preliminary operational period of This recent evolution has convinced the the cluster a set of theoretical physics INFN theoretical physics community applications has been executed on the to install a first large grid-enabled MPI cluster in order to test the functionality of cluster, named Theompi, taking advantage the new architecture for different kinds of an experimental implementation of of parallel computational models, such the granularity attributes. We describe as pure MPI, pure openMP and hybrid the porting of significant applications in algorithms, taking into account also the theoretical physics, executed in this new CPU and memory affinity for the NUMA environment and taking advantage of the architecture of the Worker Nodes. new grid parallel attributes.
Roberto Alfieri [email protected] INFN Parma – Parma University
138 | EGI User Forum 2011 user environmentS • • • • • • CORAL – a relational abstraction layer IMPACT CONCLUSION for C++ or Python applications CORAL provides generic software The CORAL software is widely used libraries and tools that do not target for accessing from C++ and Python specific data models and could therefore applications the data stored by the LHC be used in any other scientific domain to experiments using a variety of relational access relational databases from C++ database technologies (including Oracle, or Python applications. MySQL and SQLite). It provides generic software libraries and tools that do not specifically target the data models of the OVERVIEW DESCRIPTION LHC experiments and could therefore be used in any other scientific domain. The huge amount of experimental data CORAL is a C++ software package that from the LHC and the large processing supports data persistency for several capacity required for their analysis has relational database back-ends. It is one imposed a new approach involving of three packages (CORAL, POOL and distributed analysis across several COOL) that are jointly developed by institutions. The non-homogeneity of the CERN IT Department and the LHC policies and technologies in use at the experiments within the context of the different sites and during the different LCG Persistency Framework project. The phases of the experiment lifetime has CORAL API consists of a set of abstract created one of the most important C++ interfaces that isolate the user challenges of the LHC Computing Grid code from the database implementation (LCG) project. In this context, a variety of technology. CORAL supports several back- different relational database technologies ends and deployment models, including may need to be accessed by the C++ local access to SQLite files, direct client client applications used by the experiment access to Oracle and MySQL servers, for data processing and analysis. The and read-only access to Oracle through Common Relational Abstraction Layer the Frontier/Squid and CoralServer/ (CORAL) is a software package that was CoralServerProxy intermediate server/ designed to simplify the development of cache layers. Users are not required to such applications, by screening individual possess a detailed knowledge of the SQL users from the database-specific C++ APIs flavour specific to each back-end, as the and SQL flavours. SQL commands are executed by the relevant CORAL implementation libraries (which are loaded at run-time by a special plugin infrastructure, thus avoiding direct link-time dependencies of user applications against the low-level back-end libraries).
R affaello Trentadue [email protected] CERN • Alexander Loth [email protected] CERN
140 | EGI User Forum 2011 user environmentS • • • • • • Challenges in the adoption of the EGI paradigm ATLAS are: CONCLUSION • Support for middleware and tools. for an e-Science / Tier 2 centre (ES-ATLAS-T2) • Infrastructure support according to The main conclusions would be: required levels. 1. LHC started again on November 2009 • Not disturbing the current operations and successfully reached 7 TeV. and end-users. 2. IFIC is part of Spanish Tier 2, and defined its Tier 3 to fulfil ATLAS IMPACT requirements. Computing and Storage resources are in place according to This model and the IFIC prototype was 2010 pledges. OVERVIEW tion, the main purpose of the Tier 1s previously presented in EGEE User Forums, 3. Common middleware services and distributed all over the world is to reprove and shown its validity for managing the operations are now supported by EGI The High Energy Physics (HEP) community and to carry out analysis that needs to ATLAS data, but with the finalisation of IFIC. Users submit its analysis jobs has reached great achievements during the access to huge amounts of data and the EGEE project that was supporting where data is, replicating most used last year, with the run of the Large Hadron Tier 2s mainly take care of providing CPU the main software releases of gLite, new datasets to local storage. Collider (LHC) again and obtaining first and storage resources for the various challenges and issues arise, including the 4. Various tools are used, some of them collisions on November 2009, and with physics groups’ analysis and official Monte risks of timeline releases and the availability supported by EGI, HEP being a Heavy a success run at 7 GeV at the beginning Carlo data production. Tier 3 sites are of supporters. EGI-InSPIRE intends to User Community: Ganga, Panda, of 2010. institution-level non-ATLAS funded that continue supporting the production DDM or Dashboards xrootd for participate most frequently in support of infrastructure, with a broader end-user interactive analysis. In this contribution we present the current the particular interests of local physicists. community, but leaving middleware production ready infrastructure of the developments for outside providers, http://ific.uv.es/grid/e-science IFIC institute member of the Spanish In the case of ATLAS, the raw data coming being the main one the releases from https://twiki.ific.uv.es/twiki/bin/view/Atlas/ ATLAS Tier2 (ES-ATLAS-T2) and report from the detector is reconstructed to EMI (European Middleware Initiative). GridComputing its response at different stages, from data produce Event Summary Data (ESD) and taking to experimental results, and we Analysis Object Data (AOD) files. The ESD It reflects the key tools that are being used describe the facilities for data analysis incorporates all of the information of the like DDM (Distributed Data Management), (Tier 3) set up at IFIC. event reconstruction, and is mainly used and Ganga, and how are they supported for detector performance studies. The through the Services for HEP Heavy User We will also present the processes being AOD contains only part of the information Community in EGI. implemented to adopt the new situation available in the ESD and is meant to be with the EGI transition in its different used for physics analyses. Data distribution The main impact would be that this activity phases, with the key requirement of and storage organisation is based on provides continued support for activities being the least disruptive possible on the space tokens. According to the Computing previously supported by EGEE while production operations and on the day-by- Model, these space tokens are controlled they transition to a sustainable support day end-user work. As a member of the through the Distributed Data Management model within their own community or EGI project it will be also presented the (DDM) system, which is working ‘a la grid’, within the production infrastructure. The contributions to several activities in order and they are associated to a path to a support model for High Energy Physics to provide better and stable software at Storage Element (SE). will be described in the document MS603 the end infrastructure. – Services for High Energy Physics and This model is supported by the LCG made available in the next version of the DESCRIPTION project that sums up the participant present deliverable. In this way, maybe the institutes to provide a coherent production main impact will be the end of the lcg-CE The computing model of the ATLAS infrastructure with three grid flavours to move to Cream. The proposal is that experiment at the LHC is based on a including gLite, OSG, and ARC. This model all sites supporting LHC experiments run tiered hierarchy that ranges from Tier 0 is going to change at the beginning of 2011 CREAM and are no longer required to run (CERN) down to end-user’s own resources and the WLCG will decide which of them LCG-CE for LHC. (Tier 3). Levels 0 to 2 are well defined within has the blessing for adoption. the computing model and have been thoroughly tested in the last years. At The transition from EGEE to EGI is being Tier 0 takes place a first event reconstruc done and the important issues that affect S antiago Gonzalez de la Hoz [email protected] CSIC
142 | EGI User Forum 2011 user environmentS • • • • • • Monitoring of the LHC computing activities IMPACT CONCLUSION during the first year of data taking The Experiment Dashboard system During the first year of the data taking plays an important role in the the system coped well with growing computing operations of the LHC Virtual load both in terms of the scale of the Organisations, in particular those of ATLAS LHC computing activities and in terms and CMS, and is widely used by the LHC of number of users. The Experiment community. For example, the CMS VO's Dashboard system became an essential Dashboard server receives up to 5,000 component for the LHC computing unique visitors per month and serves more operations. The variety of its applications OVERVIEW DESCRIPTION than 100,000 page impressions daily. covers the full range of the LHC computing activities. The system is being developed The Worldwide LHC Computing Grid The Experiment Dashboard system in a very close collaboration with the users. (WLCG) provides the grid infrastructure was developed in order to address the As a result, the Experiment Dashboard used by the experiments of the Large monitoring needs of the LHC experiments. manages to respond well to the needs Hadron Collider at CERN, which started It covers data transfer and job processing of the LHC experiments. taking data this year. The computing and and works transparently across the storage resources made available to the various middleware flavours used by the http://dashboard.cern.ch LHC community are heterogeneous and LHC VOs. This presentation will describe distributed over more than a hundred the experience of using the system research centres. The scale of WLCG during the first year of LHC data-taking, computing is unprecedented; the LHC focusing on the dashboard applications Virtual Organisations (VOs) alone run that monitor VO computing activities. 100,000 concurrent jobs and the ATLAS Those applications that monitor the VO can sustain an integrated data transfer distributed infrastructure are the subject rate of 3GB/s. Reliable monitoring of the of a different presentation, ‘Experiment LHC computing activities and the quality Dashboard providing generic functionality of the distributed infrastructure are a for monitoring of the distributed prerequisite for the success of the LHC infrastructure’. Though primarily the target data processing. user communities of the Experiment Dashboard are the LHC experiments, many of the Experiment Dashboard applications are generic and can be used outside the scope of the LHC. Special attention will be given to generic applications such as job monitoring, and the common mechanism that can be used by VO-specific Workload Management Systems (WMS) for reporting monitoring data.
Julia Andreeva [email protected] CERN • Mattia Cinquilli [email protected] CERN • Marco Devesas Campos [email protected] CERN • Edward Karavakis [email protected] CERN • Elisa Lanciotti [email protected] CERN • Gerhild Maier [email protected] CERN • Sergey Mitsyn [email protected] CERN • Michal Nowotka [email protected] CERN • William Ollivier [email protected] CERN • Ricardo Rocha [email protected] CERN • Timur Sadykov [email protected] JINR • Pablo Saiz [email protected] CERN • Laura Sargsyan [email protected] Yerevan Physics Institute • Irina Sidorova [email protected] JINR • Jacobo Tarragon Cros [email protected] CERN • David Tuckett [email protected] CERN
144 | EGI User Forum 2011 user environmentS • • • • • • Optimisation of the job submission IMPACT CONCLUSION and data access in a LHC Tier 2 Using Torque interactive jobs, the user The number of users has considerably connects to a front-end machine and runs grown during the last year, involving several a command to submit a job to the cluster, experiments, VOs and communities (CMS, just like he does for batch local submission; Alice, Glast / Fermi, Pamela, Theophys, the batch manager chooses one CPU to Magic V, computational chemistry, bio execute the job and returns an interactive medicine, bioinformatics, etc.) actively shell. The user will keep that CPU until using the new infrastructure both locally he releases the interactive jobs through a and through EGI. This has also resulted in OVERVIEW DESCRIPTION logout. Using the screen utility, the user a considerable drop in the maintenance can also interrupt his work, log out, and overhead of the site administrator. Since LHC start-up, HEP users have Our work consists first of all in creating then log in back again preserving the increased their requests on the computing a recipe to provide the interactive facility session he had left. infrastructure in terms of performance using worker nodes; Torque interactive jobs and functionality, often going beyond the have been a starting point to deploy such Lustre has been mounted on all the nodes projected requirements of the experiments’ facility. We have also tested many different of the farm, so that users can access their computing model. storage solutions in order to choose data through grid jobs, local batch jobs the one which best fulfilled all the user and interactive jobs, simplifying users' life. Several users want an interactive facility requirements. These storage tests have to test their code before submitting it been performed on many different storage The new storage configuration has been to the grid. Usually such an interactive solutions, using different storage brands tested using CMS analysis jobs, resulting in cluster is deployed on a small number of and different technologies (hardware RAID, a very high CPU efficiency when compared machines. In our experience this is not software RAID based on the Linux kernel; with other storage solutions. the best solution due to scalability limits RAID5, RAID6; fibre channel, external SAS) and maintenance issues. Users also want as one of the primary goals of the test was As a consequence of the design of the to use the local batch submission to run to assure that a heterogeneous storage overall infrastructure, it's very easy for new their analysis quickly and reliably, using a infrastructure could be successfully users to start with local activities, and for controlled environment, on a small dataset built to achieve high performances. new experiments to add new resources in produced by them. Lustre was chosen as it provides POSIX terms of nodes and storage servers. access, the best performances, and easy The network topology and the storage administration. As a result of those tests, infrastructure also needs to be set up our farm migrated from dCache to Lustre. in order to fulfil the I/O requirements of analysis jobs; this can be difficult especially Finally we have successfully connected within a big multi-VO site, where users can Lustre to the grid using consolidated have very different use cases. solutions like StoRM, XROOTD, GridFTP, verifying the work in terms of performance and compliance with grid requirements.
Giacinto Donvito [email protected] INFN
146 | EGI User Forum 2011 user environmentS • • • • • • infrastructure – containing tens/hundreds CONCLUSION The EDGI infrastructure and its usage of thousands of computers – together with for the European Grid user communities our bridging services into EGI. The EDGI desktop grid and cloud infrastructure contains a large number IMPACT (>100,000) of volunteer and institutional desktop resources. The EDGI infrastructure Service (cluster) grids like EGI cannot contains bridging services which are able always provide the required number to automatically forward jobs from service of resources for many VOs. Therefore grids (currently gLite and ARC) to Desktop extending the capacity of these VOs with Grids (currently BOINC and XtremWeb). OVERVIEW The objective of automatically forwarding volunteer or institutional Desktop Grids Moreover, cloud resources (currently gLite jobs to Desktop Grid resources would significantly increase the number of Eucalyptus/Amazon) can also be integrated The EDGI (European Desktop Grid has been already achieved by the former accessible computing resources that can to expand the currently available reliable Initiative) FP7 project is aimed at deploying EDGeS project that built a production particularly advantageously be exploited in and non-reliable resources based on the Desktop Grid (DG) and Cloud Computing infrastructure enabling the extension case of parameter sweep applications. latest developments. services for the EGI research user of gLite VOs with several volunteer and communities that require large-scale institutional Desktop Grids. However, this EDGI is currently focusing on three EGI user communities can gain significant distributed computing resources for version had some scalability issues related different areas: The first main area is to advantages from this huge pool of resources multi-national projects. In order to achieve to the gLite modified CE and to the fact create technical solutions and develop the or if there is a need they can set up their this goal EDGI develops middleware for that all data have been forwarded through necessary software extensions for ARC, own pool based on the EDGI software extending Service Grids (SG) (e.g. ARC, the bridge components which became UNICORE, Attic, 3GBridge, Application components. The solution brought by EDGI gLite, UNICORE) with Desktop Grids a bottleneck. Repository, Monitoring, BOINC, XtremWeb, also ensures that EGI users do not need (e.g. BOINC, XtremWeb) enhanced by Eucalyptus and OpenNebula. With these to change their well-known environment, Academic Cloud infrastructures based on In EDGI one of the focuses is on eliminating developments we can enable the creation since the infrastructure developers Eucalyptus and OpenNebula. Software scalability issues, thus first we changed of a Desktop Grid in order to support provided the access to the desktop grid components of ARC, gLite, UNICORE, gLite CE to CREAM CE, secondly we service grids with a huge number of and cloud resources through gLite and BOINC, XWHEP, Attic, 3GBridge, added URL pass through mechanism to resources. The second main area is to ARC user interfaces. Within a year the OpenNebula and Eucalyptus will be the 3GBridge software in order to be able continuously operate and maintain the UNICORE interface will also be supported. integrated into a SGgDGgCloud platform to forward the reference to files instead of EDGI infrastructure built by these software for service provision and as a result EDGI the files themselves. Moreover, Attic (a P2P components and provided as a production http://edgi-project.eu will extend ARC, gLite and UNICORE file system) has also been adapted to the service for all EGI User communities. The http://desktopgridfederation.org grids with volunteer and institutional DG EDGI bridging components to increase the third main area is to extend the capacity systems. In this way, the whole European efficiency of data transfer among the server of the infrastructure by institutional and e-science ecosystem will benefit from and clients. volunteer desktop and cloud resources Desktop Grid extensions since parameter to continuously keep up with the sweep applications can be directed from In order to seamlessly integrate cloud requirements of computational capacity. the expensive cluster and supercomputer resources into the EDGI infrastructure, we resources to cheap desktop resources. have extended the bridge with a special, In order to provide sustainability for the automatic Eucalyptus cloud handling project, EDGI together with its partner DESCRIPTION mechanism, including instantiation and job project called DEGISCO have recently execution. Due to the plugin framework established an organisation called The SG to DG bridging service which of the 3GBridge this could easily be International Desktop Grid Federation is being developed by the EDGI project implemented by adding a new plugin (IDGF) which aims to bring together people leans on two key components: modified without modifying the core software. With (technical experts, developers, operators, computing element on the Service Grid this extension jobs arriving to the bridge users or any interested people) from all (gLite, ARC, UNICORE) to forward the job can be executed on cloud resources. areas related to desktop Grid computing to a Desktop Grid site and the 3GBridge and aims to share the common knowledge service on the Desktop Grid site which is Based on the solution above, the first among its members. able to receive and transform the job as a software release of the EDGI project workunit for execution for the Desktop Grid has been finished. Currently, we are (BOINC, XtremWeb) site. working on integrating our Desktop Grid Jozsef Kovacs [email protected] MTA SZTAKI
148 | EGI User Forum 2011 user environmentS • • • • • • IMPACT communities. G-Eclipse is the closest to Cross-grids simulation scenarios with our approach, but as far as we are aware the GridSFEA framework The cross-grid set up employed for this of, it lacks operations such as migration of work included the grids RoGrid-NGI (gLite), jobs, handling of scenarios, or simulation DEISA (GT4, GT5), and a test grid (GT4). preview. GridWay is also capable of The WSRF-based simulation services interacting with different middleware, of GridSFEA were deployed on the test but it is a middleware component and grid, application wrappers and simulation has a completely different goal, namely codes on all sites, the client tools of metascheduling. GridSFEA were installed on an end-user OVERVIEW practices available in the grid community. machine located outside these grids. A CONCLUSION The initial target middleware was the limitation of our set-up is the necessity Interoperability of grids is a major concern Globus Toolkit 4, currently client-side of having at least one deployment of GridSFEA has been successfully employed for the scientific community. Whereas most support for GT5, UNICORE, and gLite GT4 in the cross-grid that hosts our grid to gridify complex simulation scenarios. efforts are focused on finding solutions for being available within GridSFEA. services. We computed fluid-structure With the tools of the framework, the middleware interoperability, we propose an interaction scenarios on the three grids, migration of scenarios on heterogeneous approach closer to applications and end- The framework handles simulation scenarios each job running within one site at the time. grids is made possible. With our approach, users. Our approach is based on extending on the grid, from their formulation as grid Continuation of jobs on other sites was user applications can run on heterogeneous the client capabilities of the grid simulation jobs, submission and monitoring of the handled by GridSFEA entirely. Thus, our grids based on GT4, GT5, gLite, and framework for engineering applications status’ execution, attachment to jobs prototype was capable of operating under Unicore. From the user’s point of view, the (GridSFEA), to support middleware such as of tasks for post-processing simulation the strict requirements of production grids grid interaction is the same regardless of Globus Toolkit (GT), UNICORE, and gLite. results, to the manipulation of simulation the entire framework and simulation tools the type of middleware in place. End-users For demonstrating the application-level checkpoints and scenario data (intra- or present in the user space only (excepting of the grid benefit from high-level operations interoperability we present the cross grids cross-grid transfers, downloads to end- the simulation services). With our work, that hide the complexity of the different computations of scenarios of numerical user machines). Thus, high-level operations users can interact with different grids grids. Jobs are translated on the fly into simulations. The prototype provides such as parameter investigations, manage in the same way, with solely knowing a other representations; data handling is high-level end-user operations such as ment of long running simulations, and subset of JSDL. In this way, the framework done using GridFTP. We considered parameter investigations, handling of large preview of remote results are made is a concrete example of a tool providing for the next step the investigation of the jobs, handling of remote simulation data. available to framework users. Scenarios application-level interoperability of grids. interoperability of GridSFEA with GridWay The tools and the programming interface of are annotated; the respective metadata for providing flexible migration policies and the framework act as a mediator between is collected at runtime by application We evaluated community libraries such for interactions with further middleware. This users and applications on one side, and wrappers, and it is registered with the grid as jGlobus, GAT, SAGA, DESHL, and work was partly funded by the EC project grids on the other side, converting the grid services of the framework. These services CREAM and integrated them to some “Infrastructure for Globus in Europe”, interactions into operations specific to a are WSRF-based (GT4 specific). The client extent in GridSFEA. By using the Eclipse contract nr. 261560/21.06.2010 and the grid middleware. Experiences are gathered tools of GridSFEA are based on the Eclipse plugins as enabling technology, the Romanian National Council of Scientific on RoGrid-NGI and on DEISA. plugin technology. The main challenges reuse of plugins from and by other tools Research, contract nr. RU-RP 10/2009. we faced were: First, the integration of such as g-Eclipse is fostered. The export DESCRIPTION client-side libraries specific to different of parameter studies of GridSFEA as http://acal.utcluj.ro middleware proved to be difficult from both high-level operations in the WS-VLAM The development of grid applications that the software engineering and the technical tool brings benefits to further user keep pace with the rapid evolution of the perspectives (e.g. conflicting third-party grid middleware is still a challenging task. libraries required). Second, the JSDL Moreover, we face today a great effort requests from users/clients are mapped put in the direction of the interoperability to specifics of the middleware, entailing of grids, of the development and considerable work, thus only a partial adoption of common open standards. implementation of the JSDL spec In this context, the main objective of the is currently supported. GridSFEA framework is to gridify simulation scenarios stemming from different fields of computational sciences and engineering, I oan Lucian Muntean [email protected] Technical University of Cluj-Napoca, Department following standards, libraries, and best of Computer Science
150 | EGI User Forum 2011 user environmentS • • • • • • searching for potential new applications are expected to donate resources to Integration, sharing and exploitation of and users that could benefit from scientific purposes. national and international e-Infrastructures the research infrastructure. For the newcomers the International Desktop CONCLUSION Grid Federation (IDGF) can provide efficient help; IDGF has been set up to The presented efforts of the DEGISCO exchange experiences about the usage project show the feasible ways and best of desktop grid technology, to expand practices for exploitation of the nationally scientific infrastructures, and in order to and internationally integrated service/ bring together grid operators, application desktop/volunteer DCI particularly from the OVERVIEW These DCIs can be formed inside institutes developers, and other key players. application developers’ and users’ point of and universities (local desktop grids) or view. In order to broaden the existing user The DEGISCO infrastructure support by citizens that voluntarily donate spare IMPACT communities, which are particularly from project inherited a production-level and computing time to science (volunteer various areas of bioscience and physics, hybrid Distributed Computing Infrastructure desktop grids). Both types of grids The project supports the creation of several steps have been performed; best (DCI) from the successfully completed collect large numbers of underutilised new desktop grids in ICPC (International practices and enhanced support services EDGeS project based on a generic grid-to- resources and can offer them for scientific Cooperation Partner Countries) countries are provided by the project, and the grid bridging technology. applications and users. and the connection of these grids to application developers and users can join European DCIs and existing service grids and benefit from the International Desktop The main aim of DEGISCO is the further To be more useful for researchers, in ICPC countries by employing 3G Bridge Grid Federation. extension and exploitation of the DCI in desktop grids have been integrated into technology. terms of: scientific workflows on a regular basis; In the case of the DEGISCO project the role • involved service, desktop, and the elaborated generic bridge between Moreover, building on the solid expertise of ported applications is extremely crucial volunteer grids, desktop grids and traditional service of the DEGISCO partners, the project and two-folded; new generic applications • ported applications, grids together with the appropriate provides recommendations on best with high social impact would attract both • supported user communities, application development methodology practices and defines joint roadmaps volunteers (more resource providers for the • number of volunteers. and transparent access mechanisms/ for ICPC countries and Europe. Thus, infrastructure) and scientists (more users The work is in progress with several achieve- tools foster the convergence of distributed the presented work reinforces the global with more research results). Therefore, ments in strong collaborations worldwide computing infrastructures. relevance and impact of European the key element of the exploitation plan with several ICPC partners, such as distributed infrastructures. in DEGISCO is to reach and keep these Russian Federation, China, and Brazil. The recently launched DEGISCO project potential communities. transfers the knowledge concerning this The well-established application related The operated DCI consists of more than combined DCI towards new communities activities of DEGISCO help use the already www.degisco.eu 150,000 PCs from more than 15 global by supporting the creation, integration, more than two dozen ported applications http://desktopgridfederation.org volunteer and local desktop grids, which and operation of new desktop grids for on new connected DCI's in ICPC countries have been connected to gLite based e-Science worldwide. and support new applications. European VOs from EGI and SEE-Grid, and recently from national infrastructures, As the result of the current activities, the The dissemination and training activities e.g. Russian Data-Intensive grid and TWGRID. project members provide best practices promote via various channels the More than 25 applications have been and well-organised assistance in applica interoperation between Desktop grid and ported to the combined DCI using the EADM tion porting, as well as training about the Service grid infrastructures on a global application development methodology. grid and its usage. scale, which leads to more awareness in the general public of computational DESCRIPTION Several popular and generic applications science and distributed computing are already available: e.g. Autodock, co-funded by the EC. As a result more Traditional desktop grids consist of Blender, MOPAC, or ISDEP, but several citizens, students, and companies computers and other devices, including new applications have been ported as well, desktop PCs and notebooks that are e.g. for solving optimisation problems. used for general purposes but having unused computational (CPU/GPU) and In order to enhance further the exploitation storage capacities. of the infrastructure, DEGISCO is R obert Lovas [email protected] MTA SZTAKI
152 | EGI User Forum 2011 user environmentS • • • • • • Concerning mid-ocean ridge modelling, Molecular dynamics simulations were Multidisciplinary approach for computing we simulate mantle flow dynamics to study performed using NAMD v 2.7, a parallel in Emilia Romagna (Italy) mantle rock serpentinisation, a process molecular dynamics code, to study central to a variety of chemical exchanges the trans-membrane lipid translocation between solid earth, hydrosphere and processes. These processes are the basis biosphere. Sub-lithospheric mantle flow of important properties and functions and mantle thermal structure are obtained of cell membranes. In this study, the from the solution of Stokes and heat dynamics of a lipid bilayer, under the equations by semi-analytical-pseudo- effect of two electric field intensities is spectral and finite difference techniques, investigated. Up to a ten-fold decrease in OVERVIEW DESCRIPTION respectively. The grid infrastructure allowed total computation time has been achieved submitting in a single instance n x m jobs, by using a system with three GPU instead A multidisciplinary context has been set Experience has been gained with applications achieving n x m solutions at approximately of traditional CPUs . up in order to build a common distributed in the fields of genome annotation, mid- the time of one single run. computing infrastructure and support the ocean ridge processes, ensemble methods CONCLUSION applications of each partner. Comput-ER for ocean forecasting and molecular systems Ensemble method technique has been (Computing in Emilia Romagna) is based computation with clusters of grid-accessible used to quantify the forecast uncertainty This paper describes how the above mainly on commodity farms but new GPU- GPU-enabled systems. These applications in short-term ocean forecasting systems. applications have been modified in order based farms for parallel applications are are very CPU demanding and data-driven In this study, we explore the short-term to be executed in a loose parallel way under testing. Comput-ER resources are thus using sequential computation requires ensemble forecast variance generated and shows the advantages obtained by accessed through the gLite middleware. months of CPU time. The peculiar features by perturbing the initial conditions. Grid using most of the available computing These resources are either based on real offered by a grid environment, such as a allowed us to perform several ensemble resources of a distributed infrastructure hardware systems, or are dynamically large number of storage and computation forecast experiments with 1,000 members: and the benefit of the knowledge- provisioned via virtual machines using resources and advanced high level job sub they are completed within five hours of sharing from a multidisciplinary scientific the Worker Nodes on Demand Service mission services, allow to reduce consistently wall clock time after their submission, computing environment. (WNoDeS) grid/cloud virtualisation system. the amount of computation time spreading and the ensemble variance peaks at the the jobs on different resource sites. meso-scale.
The key solution is based on their execution as loose parallel applications, splitting the input data in several parts and using the parametric job submission P aolo Veronesi [email protected] INFN-CNAF (National Institute of Nuclear Physics – National feature of the gLite WMS to optimise the Computer Science Center), Bologna Italy • Marco Bencivenni [email protected] INFN-CNAF (National Institute of Nuclear Physics – National Computer Science Center), Bologna Italy • Daniele Cesini management of the computational tasks. [email protected] INFN-CNAF (National Institute of Nuclear Physics – National Computer Science Center), Bologna Italy • Davide Salomoni [email protected] INFN-CNAF (National Institute of IMPACT Nuclear Physics – National Computer Science Center), Bologna Italy • Antonia Ghiselli antonia.ghiselli@ cnaf.infn.it INFN-CNAF (National Institute of Nuclear Physics – National Computer Science Center), Bologna In particular in the field of genome Italy • Damiano Piovesan [email protected] Biocomputing Group, University of Bologna • annotation we instrumented an application Pier Luigi Martelli [email protected] Biocomputing Group, University of Bologna • Piero Fariselli for extended and robust protein sequence [email protected] Biocomputing Group, University of Bologna • Ivan Rossi [email protected] annotation over conservative non- Biocomputing Group, University of Bologna • Daniele Guerzoni [email protected] Biocomputing hierarchical clusters based on the Bologna Group, University of Bologna • Rita Casadio [email protected] Biocomputing Group, University Annotation Resource v 3.0. Tests have of Bologna • Giovanni Bortoluzzi [email protected] CNR/ISMAR (National Research been done with 150 genomes splitting the Council-Institute of Marine Sciences), Bologna Italy • Marco Ligi [email protected] CNR/ISMAR data in 8,000 pieces and implementing a (National Research Council-Institute of Marine Sciences), Bologna Italy • Stefano Ottani stefano.ottani@ loose parallel computation over the Grid. isof.cnr.it CNR/ISOF (National Research Council-Institute for the Organic Synthesis and Photoreactivity), Bologna Italy • Alessandro Venturini [email protected] CNR/ISOF (National Research The above computation has been done on Council-Institute for the Organic Synthesis and Photoreactivity), Bologna Italy • Nadia Pinardi n.pinardi@ 200 nodes and took two weeks instead of sincem.unibo.it INGV (National Institute Geophysics and Volcanology),Bologna Italy • Enrico Scocimarro months if running in a single computer. [email protected] INGV (National Institute Geophysics and Volcanology), Bologna Italy • Srdjan Dobricic [email protected] CMCC (Euro-Mediterranean Centre for Climate Change), Bologna Italy • Antonio Navarra [email protected] CMCC (Euro-Mediterranean Centre for Climate Change), Bologna Italy
154 | EGI User Forum 2011 user environmentS • • • • • • InSilicoLab – grid environment for supporting IMPACT CONCLUSION numerical experiment in chemistry Computational Chemistry is one of the The InSilicoLab portal facilitates in-silico three science domains that utilise grid experiments performed with the help of the resources most – along with high energy most popular Computational Chemistry physics and life sciences. While many packages – Gaussian [www.gaussian.com], computing centres offer a variety of GAMESS [www.msg.ameslab.gov/GAMESS] scientific packages for chemists, the use and Turbomole [www.cosmologic.de]. of these software suites is hindered by The tool is available to any user with a valid the lack of intuitive interfaces to them. Grid certificate registered in Gaussian VO OVERVIEW DESCRIPTION InSilicoLab, not only provides the users [http://egee.grid.cyfronet.pl/Applications/ from the Computational Chemistry gaussian-vo] or vo.plgrid.pl (a Virtual Although the capacity and technical InSilicoLab is a web portal for the scientists domain with easy access to Grid by the Organisation maintained by the Polish advancement of computational resources from the Computational Chemistry domain means of job management, but, more NGI – PL-Grid [www.plgrid.pl]). It can be offered by the European computing who want to take advantage of Grid importantly, supports scientific experiment accessed at: centres is constantly improving, the Resources. The portal allows them to planning and evolution. While the first progress in development of high-level create and manage scientific experiments functionality is already offered by grid http://insilicolab.grid.cyfronet.pl tools facilitating access to those resources and their results using terms specific to portals, and some aspects of the latter is still slow. In this paper, we present our their domain of science. The multi-layer are covered by specialised tools invented approach to such a high-level, customised architecture of InSilicoLab supports for Computational Chemistry (like WebMO environment, called InSilicoLab – designed this domain-specific interaction by [www.webmo.net] or ECCE [http://ecce. especially for chemists who want to use introducing a ‘scientific’ layer, responsible pnl.gov]), only expensive commercial User the grid infrastructure. for communication with the users, Interfaces (like Accelrys Material Studio while a separate layer is reserved for [http://accelrys.com/products/materials- communication with computational and studio/index.html]) join these features. storage resources. These two layers are Still, these proprietary tools are mostly interconnected via a third – intermediate – desktop applications – which makes them layer, which processes and translates inaccessible from outside a local computer the objects and methods defined by the and hinders collaboration. user to parallel grid jobs and/or calls to other services and tools required by the user. This separation of the domain- and grid-specific layers and concepts related to them makes the InSilicoLab portal more intuitive to users, enabling them to focus on their work instead of on the technical details of configuring the computation, and, thus, making their work more efficient.
M ariusz Sterzel [email protected] CYFRONET • Joanna Kocot [email protected] CYFRONET • Tomasz Szepieniec [email protected] CYFRONET • Daniel Harężlak [email protected] CYFRONET • Klemens Noga [email protected] CYFRONET
156 | EGI User Forum 2011 user environmentS • • • • • • The GRelC Project: main service, IMPACT CONCLUSION functionalities and relevant use cases A grid-database service is fundamental for The GRelC service is a grid-database distributed data-oriented infrastructures, management service that can be exploited production grids, grid testbeds, since both at VO and site level. This service it enables the management of crucial is currently successfully deployed and information. Some examples concern positively evaluated by end-users in the biological sequences in the bioinformatics Earth Science and Environmental contexts domain, spatial metadata information (e.g. CMCC and Climate-G). Moreover it is in the earth science and environmental also available for tutorial purposes in the OVERVIEW (XML DB engines such as eXist, XIndice domains, patient-related information GILDA t-Infrastructure. and libxml2 based documents). into a Health Information System (HIS), Grids encourage and promote the astrophysics databases, etc. The talk www.grelc.unile.it publication, sharing and integration of The GRelC service provides: presents the GRelC service highlighting scientific data, distributed across Virtual • basic functionalities (query submission, also how it provides cross-VO capabilities. Organisations (VO). The complexity of data grid-db management, user/VO/ACL Due to the service nature, architecture and management within a grid environment management, etc.) to access and functionalities common use cases can be comes from the distribution, heterogeneity manage grid-databases; defined across different disciplines. This and number of data sources. In the last • efficient delivery mechanisms helps to identify common requirements, ten years there was a strong interest in leveraging streaming, chunking, formalise exploitation patterns paving the grid-database access and management prefetching, etc. to retrieve data from way towards sustainability. topics. Moreover tools and services able databases in grid providing high level to access relational databases in grid are of performance (in terms of query also strongly required in the ES, LF, A&A response time, number of concurrent User Communities. Within the proposed accesses, etc.); talk we will present in detail the Grid • additional functionalities such as Relational Catalog (GRelC) Project, an asynchronous queries, integrated environment for grid database • a data Grid Portal (GRelC Portal) to management, highlighting the vision/ ease the access, management and approach, architecture, components, integration of grid-databases, as well services and technological issues. The as user/VO/ACL management, etc. most relevant use cases (in particular in the earth science and environmental domains) The GRelC middleware has been included will be described in detail. into the EGEE RESPECT Programme (Recommended External Software DESCRIPTION Packages for EGEE CommuniTies) since it works well in concert with the EGEE gLite The key topic of this talk is the GRelC Service. software by expanding the functionality of the grid infrastructure (w.r.t. database The GRelC service is a GSI/VOMS management in grid). The GRelC service enabled web service addressing is currently adopted as grid metadata extreme performance, interoperability management service in the Climate-G and security. It efficiently, securely and testbed to enable geographical data transparently manages databases on the sharing, search and discovery activities. grid across VOs, with regard to emerging Moreover it is currently used at the Euro- and consolidated grid standards and Mediterranean Centre for Climate Change specifications as well as production grid to manage climate metadata across the middleware. It provides a uniform access Italian CMCC data grid infrastructure interface, in grid, both to access and through the CMCC Data Distribution integrate relational (Mysql, Postgresql, Centre portal. S andro FIORE [email protected] University of Salento • Giovanni ALOISIO giovanni.aloisio@ SQLite) and non-relational data sources unisalento.it University of Salento
158 | EGI User Forum 2011 user environmentS • • • • • • Data management is an area of constant evolution: both experiment and user activity always increase and oblige Virtual Organisations to ensure the future scalability of their systems by automating manual operations, optimising the usage of available resources and adapting to evolving middleware, more performant infrastructure and new technologies. This session will present work from the Data Manage ment projects of ATLAS and CMS, two of the LHC experiments distributing multi-Petabyte data volumes across affiliated computing sites in the Worldwide LHC Computing Grid and EGI sites. The session will give a general overview of ATLAS’s Distributed Data Management and the architecture and operational strategies that have been decided in order D ata to provide a fault-tolerant, highly intelligent system. The presentation will explain how different independent subcomponents are used to fully profit from the deployed resources and make the best usage of space at the sites in an automated fashion. Management Next, the presenters will describe the effort in both experiments dedicated to improving the data transfers and monitoring the throughput rates between sites. Both collaborations base their file transfers on common underlying gLite middleware and pursue the optimal usage of their network and evolution of the distribution topologies in their respective Computing Models. The tendency is to move to a less hierarchic data transfer model where all sites can communicate with each other. An insight into the instrumented tools and steps followed to ensure the commissioning of the links will be given. Jamie Shiers Head of the Experiment Support Group, IT Department, CERN
• • • • • • An insight into the ATLAS IMPACT CONCLUSION Distributed Data Management Given the scale of the project, ATLAS Data management is an area of constant Distributed Data Management can be expansion: both experiment and user considered an important example of activity keep increasing. The ATLAS distributed data management on the grid. DDM team oblige to ensure the future Although the implementation is experiment scalability of the system by: automating specific, it is based on common manual operations, optimising the usage underlying middleware and is composed of available resources and adapting to of independent subcomponents, auxiliary evolving middleware, higher performant OVERVIEW The system is currently managing 50 PB systems and monitoring solutions that can infrastructure and new technologies. of data corresponding to almost 200 be of inspiration and re-implemented to fit ATLAS, one of the four LHC experiments, million files and is being used to achieve the needs of any grid community. To mention one example, DDM is currently fully relies on the usage of grid computing aggregated throughput rates far beyond replacing its messaging infrastructure, for offline data distribution, processing the initial requirement of 2 GB/s, having Relatively simple subcomponents such as based on open protocols such as and analysis. The ATLAS Distributed reached a throughput peak of over 10 GB/s. the storage space accounting, which was http, by more recent, asynchronous Data Management is the project built To ensure further scalability, the core of initially implemented to account the used, message queuing technologies, which on top of the WLCG middleware and is the system has been designed as a set of free and total storage space of grid storage should enhance the independence of responsible for the replication, access and independent agents which work around a endpoints was extended to allow the subcomponents and therefore the overall bookkeeping of the multi-PB ATLAS data global database, the Central Catalogues. breakdown of the used space by different fault-tolerance. across more than 100 distributed grid sites. This architecture allows distributed, fault- metadata, have been claimed to be needed It enforces data management policies tolerant services that interact with the grid by different communities inside and outside Another area of development, given the decided on by the collaboration and resources and checkpoint centrally the the WLCG community. The communication evolution of the network throughput, is defined in the ATLAS computing model. status of the requests. of the storage space accounting with a the development of the hierarchic data It has been in production since 2004 and centralised site exclusion service or an distribution policies towards a more to date is considered one of the largest Given the volume of files managed, it is automatic site cleaning agent are the dynamic, relaxed Computing Model where open data management environments and critical to reduce the load on operations by next steps, in order to avoid replicating gradually regional boundaries are relieved. an example of a global multi-grid hybrid providing compact monitoring information, data to full sites or to trigger the cleaning system. This contribution will give an automatically curing inconsistencies and of full sites respectively. In the case of http://bourricot.cern.ch/dq2/accounting/ overview of the architecture, operational automate systems as far as possible. ATLAS, these simple tools have helped to global_view/30 and deployment strategies of this highly significantly reduce manual operations and automated system, as well as details The talk will give an overview of the general improve day to day operations. about different subsystems and monitor- architecture and furthermore explain in ing solutions that could be of interest for detail several independent subcomponents other communities. targeted towards resource optimisation, such as storage space accounting, the DESCRIPTION automatic site cleaning agent and the centralised site exclusion service and how ATLAS Distributed Data Management is these can be extended to interact between the system that manages the experiment's each other. The ideas and concepts detector, simulated and user data while presented will provide inspiration for enforcing the policies defined in the any Virtual Organisation that is currently Computing Model. It provides functionality planning to move their data to the grid or for data placement, deletion, bookkeeping working on improvements to their usage of and access on a hierarchic grid model grid, network and storage resources. composed of around 100 sites with heterogeneous storage technologies, services and protocols. F ernando Harald Barreiro Megino [email protected] CERN • Simone Campana [email protected] CERN • Vincent Garonne [email protected] CERN • Andrii Tykhonov [email protected] Jozef Stefan Institut
162 | EGI User Forum 2011 Data management • • • • • • Improving CMS data transfers among IMPACT CONCLUSION its distributed computing facilities This global study is collecting statistics on With the help of such statistics the central each individual transfer for the whole CMS team of distributed data transfers of CMS distributed fabric. This has never been is running a campaign to spot issues and done before. In fact, we do not know of highlight them to the sites. Regular usage any other VO that has conducted a study in operations will, of course, give more like this. The impact is the improvement feedback on which are the most relevant of data transfers for CMS by either solving statistics to gather. Some new statistics FTS channel bottlenecks or identifying are already planned to be included. At the OVERVIEW DESCRIPTION configuration problems at the sites, or even moment FTS Monitor Parser is gathering on individual transfer links. data only for the CMS VO, but in the future CMS computing needs reliable, stable This contribution deals with a complete it can be opened to other VOs. In fact, we and fast connections among multi-tiered revision of all FTS servers used by CMS, As an example, the study has already been would like other VOs to be engaged with computing infrastructures. PhEDEx customising the topologies and improving useful to propose a policy to set up the us in a global WLCG data transfer study, provides a data management layer their set up in order to keep CMS data FTS at PIC Tier 1 for PIC to Tier 2 transfers. and we are collaborating with FTS Monitor composed of a series of collaborating transfers to the desired levels in a reliable The impact of the change was the increase developers, so more information can agents, which manage data replication and robust way. by a factor 2 in the overall PIC to Tier 2 be published. at each distributed site. It uses the File transfers, as well as a clear improvement Transfer Services (FTS), a low level data We use the FTS Monitor, a web-based on the data transfer qualities, by means movement service responsible for moving monitoring system developed at the CC- of implementing dedicated FTS cloud sets of files from one site to another, while IN2P3 Tier 1, which provides a graphical channels for low and high throughput allowing participating sites to control the view of the FTS activity. This service transfer connections, based on results network resource usage. FTS servers retrieves data directly from the FTS back- collected by this tool. are provided by Tier 0 and Tier 1 centres, end database to generate summary and need to be setup according to statistics and to provide detailed reports By the time of the conference, these results the grid site’s policies, including all the about transfer activities. The FTS Monitor’s will be available on a web page containing virtual organisations making use of the web pages display channel configuration, all the relevant information and plots, at the grid resources at the site, and properly statistics about transfers in the last 14 days disposal of the operation teams as well as dimensioned to satisfy all the requirements on each channel, and detailed information sites administrators. for them. Managing the service efficiently on all jobs submitted in the last 24 hours, requires knowledge of CMS’s needs, of including the status and throughput of all the transfer workflows that need to be each individual transfer. handled, along with the potential benefits and conflicts to other virtual organisations Each transfer detail is published in machine- (VO) using the same FTS transfer managers. readable XML format, which is parsed daily. The FTS Monitors each CMS Tier 1 to keep the history of all transfers and important values, such as transfer rates per file and per stream, SRM response times, FTS channel congestions, etc. A wealth of information that is extremely useful to spot issues and debug problems.
Josep Molina [email protected] PIC/CIEMAT
164 | EGI User Forum 2011 Data management • • • • • • Evolution of the ATLAS data placement model IMPACT CONCLUSION New data brokering models such as The collected transfer statistics are known PanDA Dynamic Data Placement have directly by the FTS servers, but are not been recently introduced by ATLAS. The made available through an programmatic idea behind this model is to replicate interface. Having access to them is not datasets from any Tier 1 to any Tier 2 only an ATLAS particular interest, but can be after they have exceeded a popularity of use to any grid-based community. This threshold, thereby eliminating the replica contribution wishes to open the discussion tion of unpopular datasets. This work is a between the different user communities OVERVIEW Network connections between the Tier 0 first attempt in the ATLAS Distributed Data with the ultimate goal of moving in the (CERN) and the Tier 1s are guaranteed by Management project to optimise these future towards a common, central solution. The ATLAS experiment at the LHC is fully the Optical Private Network and inside cross-cloud transfers, improve the network relying on the usage of grid computing a cloud are generally performant, while usage and provide the necessary statistics http://bourricot.cern.ch/dq2/ftsmon for its offline data placement, processing inter-cloud links are less guaranteed. Thus, needed for link commissioning activity with and analysis. For data placement the Tier 1s usually act as the access point to the final goal of reducing cloud boundaries. ATLAS computing model defines a set get data in and out of the cloud and direct of policies, which establish a hierarchical cross-cloud communication between A similar initiative exists in the CMS tier organisation according to the network Tier 2s is generally avoided. However, experiment, another of the LHC topology, which was laid out for data since networking capabilities have experiments. Their approach is somewhat distribution. However, since the original significantly evolved, the computing different, as instead of measuring the creation of the computing model, network model is moving towards more dynamic transfer events, the statistics are collected capabilities have significantly increased data management policies and the cloud by parsing html files retrieved from each and it is convenient to gradually start boundaries are gradually being reduced one of the gLite File Transfer Service relaxing some of the imposed boundaries. in a controlled fashion. servers. Also the statistics are not fed This talk will focus on the work that is being back into the system in order to optimise carried out in the ATLAS Distributed Data Consequently, the data distribution transfers, but are only displayed for Management project in order to evaluate framework in the ATLAS Distributed link commissioning. more dynamic constraints and provide the Data Management project has been necessary framework for commissioning instrumented to measure the durations network links. of gLite File Transfer Service (FTS) trans fers between sites and store them in an DESCRIPTION Oracle database. The transfer statistics will be used as feedback to optimise the The ATLAS Distributed Data Management source selection and choose between is the project built on top of the WLCG multi-hop transfers through the Tier 1s or middleware and is responsible for the direct cross-cloud transfers. The statistics replication, access and bookkeeping of the are also visualised in a dynamic web multi-Petabyte ATLAS data across the grid page in order to monitor the throughput while enforcing the policies defined in the performance of the network links. In ATLAS Computing Model. Following this parallel, an ad-hoc load generator will model ATLAS sites are grouped into ten trigger transfers on the complete mesh clouds by geographical and organisational of ATLAS sites and will provide the reasons. Each cloud is formed by one information for a first attempt of Tier 1 that must provide a high level of link commissioning. service and is responsible for data storage and reprocessing and several Tier 2s and Tier 3s, which are used for analysis and Monte Carlo production and depend directly on the Tier 1. F ernando Harald Barreiro Megino [email protected] CERN • Simone Campana [email protected] CERN • Andrii Tykhonov [email protected] Jozef Stefan Institut
166 | EGI User Forum 2011 Data management • • • • • • Establishing a user’s identity, and reaching a decision about a users’ access to a requested resource is a perpetual challenge to Distributed Computing. Centralising the physical location of a user’s grid certificate (claiming an identity) and corresponding private authentication key (giving evidence of the claimed identity) on one physical device allows the user to regain control over when and to whom they wish to give evidence of their identity at any given point in time. However, for users that are not proficient with or new to current distributed infrastructures, tapping into alternative authentication sources other than T echnologies grid certificates provides an easy means to establish user authenticity and access control decisions without having to go through the certificate enrolment process. By centralising the management process and federating the authorisation decision process of for Distributed access policies, infrastructure-wide access decisions are consistent across different user communities. Another aspect of distributed computing concerns site-wide configuration management and information management. To sustain administration of sites, Computing consistent management of installed software on many resources is achieved through profile-based distributed software installation. Information management provides the feedback mechanism detecting deviations from an installation profile, and allows users and higher level processes to choose the most suitable computing resource for their current needs. Therefore scalable and efficient technologies must be employed to allow near-realtime update and dissemination of information in a distributed infrastructure. Michel Drescher Technical Manager, EGI.eu
• • • • • • A new ‘lightweight’ crypto library for supporting an IMPACT CONCLUSION advanced grid authentication process with smart cards The work carried out and reported in this The Java SE platform provides developers contribution is particularly relevant for with a large set of security APIs, algorithms, several user communities, applications, tools and protocols. Among them, we grid portals and/or Science Gateways can point out the native Sun PKCS#11 developers. The library allows using the cryptographic tokens which have been credentials stored into smart cards for used in this work together with the Bouncy generating VOMS-compliant proxies Castle and the cog-jGlobus Java APIs thus enhancing the Java technology to to implement a new module for the gLite OVERVIEW X.509 version 3 certificate by reading the deal with private and public keys. The grid middleware which allows us to sign credentials available in the smart card, benefits introduced in this work are far- the proxy certificate using the digital Many of the existing grid middleware, and while the support with the cog-jGlobus reaching. The new crypto library can be credentials stored in a smart card. The in particular gLite, rely only on the adoption was introduced to establish a secure used, for instance, to help developers to open source solution described in this of a Public Key Infrastructure (PKI) of digital connection with the VOMS server and add design and develop science gateways work can be used by users, applications, certificates for user authentication, and the Attributes Certificate to the original for several scientific communities and grid portals and/or science gateways these credentials must be present on each grid proxy. In this first implementation, the provide, especially for non-expert developers to generate VOMS proxies User Interface (UI) that is used by users library runs with the Aladdin e-Token PRO users, a transparent and easy access to using Java APIs starting from the creden to access the computational and storage 32K directly plugged into a remote 64-bit e-Infrastructures. Last but not least, the tials stored in e-Token smart cards. The resources. Distributing the certificate’s UI based on Scientific Linux 5 where the introduction of smart cards for storing new library provides an asset in raising grid private key on multiple locations is Aladdin’s e-Token PKI Client software digital credentials can massively improve awareness and encourages broader use by considered a security breach, as the (pkiclient-full-4.55-34) was previously the security of grid infrastructures since a wider number of potential users. certificate may be subjected to possible installed. This software enables e-Token they protect sensitive information from fraudulent use by non-authorised people USB operations and the implementation of malicious applications. As we mentioned logged into the UI. Furthermore, there is e-Token PKI-based solutions. The library before, the crypto library has been lack of support for other authentication has been successfully tested with both successfully tested on Aladdin e-Token mechanisms such as smart cards even personal user’s and robot certificates and smart cards only. No compatibility tests if this hardware is available and can help used to generate grid proxies in the new have been performed on smart cards of in keeping these certificates safe and e-Collaboration environment based on other vendors, but these can be done avoid any fraudulent use. The public part Liferay and GENIUS/EnginFrame tech on request. of an X.509 certificate stored on this nologies. It is of course not restricted to hardware can usually be accessed by any grid portals and/or science gateways. users, applications, portals and/or science Concerning the credentials stored into gateways but the corresponding private the smart card, we used robot certificates key can never be copied off the smart card. issued by the INFN Certification Authority (CA) because they are the type of DESCRIPTION certificates that best match the use case of these credentials in our environment. In this contribution we describe our work Nevertheless, nothing prevents using in the design and implementation of a new personal certificates instead if their planned grid authentication method based on the usage is consistent with the restrictions the use of digital certificates stored on smart CAs put on it. cards. The solution we propose extends the native Sun PKCS#11 cryptographic APIs with the Bouncy Castle and the cog- jGlobus APIs library in order to implement a new ‘lightweight’ crypto utility which can be used by users to access the digital certificates stored on a smart card and generate a VOMS proxy. The Bouncy Castle APIs were used to generate a Giuseppe La Rocca [email protected] INFN Catania
170 | EGI User Forum 2011 T echnologies for Distributed Computing • • • • • • Federated access to grids IMPACT CONCLUSION Current identity federations are limited A federated identity is a very convenient in the number and type of applications means for authentication of users and supported. The most prevalent federa passing additional information about them. tions are closely tied with the web (e.g., Current identity federations are not generally Shibboleth) or network access (Eduroam). enough to fit the grid environment, though. However, none of these federations are The aim of this contribution is to introduce suitable for facilitating access to the grid. the Moonshot project, which fills in the The Moonshot architecture, on the other gap between existing identity federations OVERVIEW provides federation for the web and web hand, is open for any type of services and applications that cannot be integrated services, RADIUS and Diameter provide regardless the service protocols and/or with them due to completely different Grids have established their own PKI-based federation for network access. Moonshot interfaces they use. underlying technologies. Authentication & Authorisation Infrastructure integrates RADIUS federation into most (AAI), which is not linked to any existing application protocols, whose users can Having Moonshot supported by the grid Moonshot can make the access to the identity-management systems. The result then leverage from federated access. At environment would increase the number grid easier for the users, since they could is not always well received by users. Even the same time, SAML is fully supported of potential users and also make their mostly use the credentials they use also for if the situation is changing and grids are and provides rich attributes to describe management easier. In particular, we other services. Also the resource providers aiming at utilising identity federations, there federated subjects. The Moonshot infra- propose an integration of the Moonshot could benefit from this integration and rely still remain crucial aspects that prevent structure builds upon components that and PKI so that users can easily obtain on the users' attributes propagated from the smooth usage of federated identities are well-known and understood, and have credentials needed to join the grid, based their identity providers. in grids. been in use for a long time. Unlike other on their federated identities. Identity federation middlewares (e.g. Shibboleth), federations also maintain sets of attributes While the Moonshot project is quite One basic issue is the way of obtaining a Moonshot is not tied with the web environ assigned to the users, which may also be young and the work is still in progress, credential to routinely access grids, which ment and can be utilised by non-web valuable for the grid, too. For instance, we plan to demonstrate its benefits for should be simple to use, yet secure enough. applications. The Moonshot architecture the resource providers and/or VO the grid community to provide a notion Utilisation of federated identities is highly is based on open standards and the managers could specify access control of its potential. desired by the users and several solutions Moonshot community actively contributes rules based on these attributes (e.g. have been adopted recently, e.g. with the to the ABFAB working group of the IETF. users' affiliation). The big advantage of the www.project-moonshot.org Terena TCS certification authority. Another federation model is that the attributes are useful approach is transparent credential Moonshot is a good fit for the grid because kept current and therefore can be relied conversion, which allows obtaining grid many grid components, particularly grid upon by their consumers. credentials without the users’ explicit middleware components, do not use web intervention. We will demonstrate how technologies. We are therefore seeking the federated Moonshot infrastructure ways to find out how grids can benefit can provide such a conversion, opening from the Moonshot infrastructure. In this grids to a larger user base. contribution, we will demonstrate the design and current achievements focusing DESCRIPTION on users running jobs on the grids using a Moonshot-generated id. As the result, the Moonshot is a set of technologies for users will be able to use their federated providing federated access to applications. identity to launch and control jobs. Finally, At a technical level, federation decouples we will also demonstrate the flexibility management of credentials within an of Moonshot by demonstrating how a organisation from authentication proofs particular grid service can be plugged into between organisations. Many existing the Moonshot infrastructure so that users technologies such as Security Assertion could use the service without having to Markup Language (SAML), RADIUS, and obtain a certificate first. For this purpose, Diameter support federation. However we will show how the gLite Logging & D aniel Kouril [email protected] CESNET • Sam Hartman [email protected] these existing technologies are focused Bookkeeping service can be adapted to Painless Security • Josh Howlett [email protected] JANET • Jens Jensen [email protected] on a single application domain. SAML natively support Moonshot. STFC Rutherford Appleton Laboratory • Michal Prochazka [email protected] CESNET
172 | EGI User Forum 2011 T echnologies for Distributed Computing • • • • • • Argus, the EMI Authorisation Service IMPACT CONCLUSION The Argus Authorisation Service is The Argus Authorisation Service is the based on XACML; using a standard for solution to render and enforce consistent authorisation eases the integration with authorisation decisions across the different other grid components, and provides EMI middleware stacks. Using a standard interoperability across EMI middleware XACML based authorisation service stacks. Interfacing Argus into Compute facilitates integration and interoperability Elements as well as Storage Elements will between the different EMI services and enable maintenance of the authorisation eases the authorisation management at OVERVIEW ensures the integrity and consistency of the policies in one service for an entire site. individual sites. authorisation requests received from the The chaining of different PAPs at the Authorisation across EMI middleware PEP clients. Lightweight PEP client libraries international as well as national level https://twiki.cern.ch/twiki/bin/view/EGEE/ stacks is currently not homogeneous, are also provided to ease the integration will lead to effective global banning of AuthorisationFramework and components often have their own and interoperability with other EMI services malicious users during incident handling. mechanisms of handling it. To address or components. An infrastructure-wide this inconsistency and to unify the user banning mechanism is a major A special focus on the site and system authorisation, the Argus Authorisation security feature currently missing in existing administrator will be given in this Service was chosen as the EMI solution. deployments despite its importance. This presentation that will allow him/her to It renders consistent authorisation feature can be effectively implemented better understand the advantages of decisions for distributed services (e.g., leveraging the modularity of Argus that deploying Argus at his/her site and the user interfaces, portals, computing allows linking together PAP services roadmap of integration into other services elements, storage elements). The service deployed at different levels of the over the lifetime of the EMI project. is based on the XACML standard, and infrastructure. Leveraging the Argus policy Argus service deployment and concrete uses authorisation policies to determine distribution mechanism, banning lists can examples of authorisation policies will if a user is allowed or denied to perform be defined at various levels (European, be discussed. a certain action on a particular service. NGI, institutional etc.) and sites can import The presentation introduces Argus to such lists so that local authorisation site and system administrators. Different mechanisms are aware of malicious users. deployment scenarios are presented, In order to preserve site autonomy, Argus as well as tools that Argus provides allows to override policies imported from to manage authorisation policies. The remote PAPs with local ones, leaving the roadmap for integration of Argus in other control of local resources authorisation in EMI components is presented together the site administrators' hands. Authoring with an update on the status of common XACML policies in XACML itself is not authorisation profiles standardised in EMI. straightforward: XML per se is perceived by many users as difficult to read, and editing DESCRIPTION can be prone to error. Argus’s solution to this issue is the Simplified Policy Language The Argus Authorisation Service is (SPL), which facilitates the authoring of composed of three main components. The policies. Site administrators can write Policy Administration Point (PAP) provides policies in the SPL and import them in the tools to author authorisation policies, the PAP. Policies are then transparently organise them in the local repository and converted in XACML and stored in the local configure policy distribution among remote repository. Command line tools are also PAPs. The Policy Decision Point (PDP) provided to add and manipulate commonly implements the authorisation engine, and used authorisation policies. is responsible for the evaluation of the authorisation requests against the XACML policies retrieved from the PAP. The Policy Valery Tschopp [email protected] SWITCH • Andrea Ceccanti [email protected] Enforcement Point Server (PEP Server) INFN-CNAF
174 | EGI User Forum 2011 T echnologies for Distributed Computing • • • • • • Site integral management with Puppet IMPACT CONCLUSION Puppet allows you to administrate a Having a steered configuration system whole site from a central service, easing like Puppet is a huge advantage from the a lot of potential reconfiguration issues or administration point of view in easing speeding up disaster recovery procedures. the daily work of sysadmins.
Having a centralised management of grid Having the whole services configuration of services profiles results in a very easy a site under Puppet provides an invaluable scaling method for adding resources confidence, ensuring a streamlined way for OVERVIEW DESCRIPTION on the fly: push the install button and let service deployment. Puppet do all the rest of the work until After the first LHC phase, experiments Managing hundreds of nodes is one of the service is ready for production. Unifying the work model is also an important and sites focused efforts to deliver the the challenges of any computing centre, feature; all services are defined using demanding WLGC metrics. Now, sites are and PIC is no exception. There are many Potentially Puppet doesn't have to be a site the same pattern in a common location. facing a new challenge to consolidate and tools able to cope with the problem; specific tool but a master service able to Therefore removing the constraint of service- automate the computing services to reach ranging from the very simple to the very do the same work on distributed resources responsible by having a common language a functioning steady-state. The installation complicated. We did a study of some or sites. This advantage can be considerable of administration transversal to all sysadmins. and post-installation mechanics are of the available tools (quattor, cfengine, in centrally managing several sites that two of the most important points for the puppet). Our requirements include the share a common middleware infrastructure, computing centres, in order to avoid initial ability to do incremental configuration (no which is in line with the current strategy of installation and configuration problems. need to bootstrap the service to make it middleware implementation. At Port d'Informació Científica (PIC), manageable by the tool), simplicity in the the adopted solution is to steer all post- description language for the configurations install and dynamic post-configuration and in the system itself, ease of extension using Puppet. of the properties/capabilities of the system, a rich community for assistance and Puppet uses a master entity to easily development, and open-source software. define profiles that get propagated around the cluster, hence fulfilling the necessities We found in Puppet the balanced trade-off of post-install configurations, after the raw between simplicity and flexibility that was installation, and ensuring the persistence the most fitting for our requirements. The of the profile and the defined services once Puppet approach to system management they have been completely installed. is simplistic, non-intrusive and incremental; Puppet doesn't try to control every aspect of the configuration but only the ones you are interested in. Our sysadmins were able to build complex configurations in a short time due to the easy learning curve.
X avier Espinal [email protected] Port d'Informació Científica (PIC) • Arnau Bria [email protected] Port d'Informació Científica (PIC)
176 | EGI User Forum 2011 T echnologies for Distributed Computing • • • • • • A new scheme for efficient distribution IMPACT CONCLUSION of information in grid infrastructures The new scheme can be deployed In this work we present the architecture in combination with the current BDII of the system and show the quantitative solution. It reduces drastically (by orders results to support its advantages. of magnitude) the amount of network bandwidth being used, which will be Our system design and implementation important to accommodate for further are sufficiently generic, thus allowing increases in the number of sites in the applications in other areas, where the infrastructure and the complexity of distribution of information follows the OVERVIEW Moreover, we compress the result, which information about them. It also decreases same patterns. decreases even further the amount of the latency in the movement of data The gLite grid middleware utilises BDII information that needs to be transferred. between the layers. as an information system in a layered A client, which can be co-collocated approach. In our work we have developed with a top-level BDII, can query our an alternative path for distribution of the service over the http protocol, supplying information, which enables considerable a hash of the information that it has decrease in the amount of data that is at a given point. If this information is exchanged. We exploit the fact that much sufficiently recent, within the last 1 – 2 of the information that is exchanged does minutes, then only the gzipped difference not change over time by introducing a is transferred. The client proceeds to versioning scheme. In this way, most of the decompress the patch and applies it to traffic consists of data in a compressed its copy of the information, obtaining the diff-like format. same result as when all the information is downloaded. We have implemented a DESCRIPTION testbed deployment of the new system, where we mimic the infosystems of the We implemented an alternative path for European Grid Infrastructure sites in real distribution of information that is currently time, which enabled us to perform a available via the gLite BDII service. The series of benchmarks and to analyse the new lightweight middleware components performance of the new scheme. are based on the well-known properties of the information that is currently transferred between the various layers in production grids based on gLite, and follow a simple protocol to ensure consistency.
Since the information that a site-level information system displays changes frequently, the problem is how to distribute in an efficient way the changes between successive ‘versions’ towards the nodes that require them (mostly top-level BDIIs). Since the changes are usually limited to certain attributes and the appearance or disappearance of whole sections of information is related to rare events, like failing or new nodes, we use the routines from the ‘diffutils’ package to compute the difference between successive ‘versions’ of the information that a site displays. E manouil Atanassov [email protected] IICT-BAS
178 | EGI User Forum 2011 T echnologies for Distributed Computing • • • • • • IMPACT used. Therefore, it enables the develop Coarse-grained interoperability of ment of such inter-disciplinary and inter- heterogeneous grid workflow systems Coarse-grained workflow interoperability organisational workflow applications that enables the coordinated execution of were not possible before. workflows created in different workflow systems, from a host workflow. The The results of the project are directly utilised created ‘meta-workflows’ may span by selected user communities from the multiple heterogeneous grid infrastructures. areas of medical imaging, bioinformatics and chemistry. The communities provide Workflow interoperability allows workflow pilot application workflows that serve as OVERVIEW native workflow engines if they have to sharing to support and foster the adoption benchmarks for the developed platform execute embedded workflows. of common research methodologies, and form the basis of a larger user com Several grid workflow management improves efficiency and reliability of munity that will utilise SHIWA services in systems (WfMS) have emerged in the last The current approach facilitates the research by reusing these common the future. decade. These systems were developed embedding of Kepler, Taverna, Triana, methodologies, increases the lifetime of by different scientific communities for GWES and Moteur workflows into workflows, and reduces development time CONCLUSION various purposes, and many applications P-GRADE workflows. GEMLCA is for new workflows. have been ported to the grid using them. capable to either invoke pre-deployed The experience gained so far in this project Reusing workflows in different WfMS workflow engines through a variety of grid The SHIWA project develops a coarse- has indicated that the reusability of work and building new experiments by utilising middleware solutions (including GT2, GT4 grained interoperability solution that enables flows can be improved by following simple existing workflows as building blocks is and g-Lite), or to submit and execute the the publishing, searching and sharing guidelines to reduce dependencies on the currently not supported. WfMS typically engines on resources of infrastructures of workflow engines and workflows, in specific environment adopted by the user have their own user interfaces or APIs, built with these types of middleware. a repository. Using the repository, users community that develops and runs them. description languages, provenance can select already existing workflows or However some dependencies on the strategies, and enactment engines, which The first version of the SHIWA Simulation workflow components and embed them description language, workflow system, are not standard and do not interoperate. Platform (SSP) uses the NGS P-GRADE into their native workflow. The developed credential handling and file access protocols portal and resources of the Westfocus solution supports the above functionalities cannot be controlled at the workflow level, The European SHIWA project aims to VO of the UK NGS to execute embedded without requiring users to understand the so they need to be solved at the SHIWA leverage existing solutions and enable workflows. The platform is currently nature and technical details of the workflow platform level. cross-workflow and inter-workflow being tested and utilised by the SHIWA system that hosts the embedded workflow. exploitation of DCIs by applying user community. The first considered The embedded workflow is considered as The SHIWA project is currently completing coarse- and fine-grained strategies. workflows implement pilot applications a black-box, and is represented by its input the implementation of SSP including a The coarse-grained approach treats from different areas: neuroimaging (Charité and output parameters and the executing coarse-grained interoperability mechanism. workflow engines as distributed black box Berlin, D-Grid, GWES); DNA sequence workflow engine only. Future work consists of enacting the pilot systems, while the fine-grained approach alignment (AMC Amsterdam, DutchGrid, workflows in different workflow systems addresses language interoperability by MOTEUR); simulation of chemical reactions The new services developed and deployed using the SSP and preparing other (more defining an intermediate representation. (Charité Berlin, EGI, MOTEUR) and medical by SHIWA allow different scientific complex) applications for publication on This contribution concentrates on results simulation (CNRS Creatis, EGI, MOTEUR). communities to exchange applications, th e S S P. of the coarse-grained approach. The existing workflows had to be adapted workflows, and data resources regardless to increase their portability respective to of the workflow system in which they are www.shiwa-workflow.eu DESCRIPTION credentials needed to access the work flow description and DCI resources for The coarse-grained interoperability work enactment; flexibility and neutrality of inside the SHIWA project refers to the invocation and management of input data T amas Kiss [email protected] University of Westminster • Tamas Kukla nesting of different workflow systems and output results; data location and [email protected] University of Westminster • Dagmar Krefting [email protected] to achieve interoperability of execution transfer protocols; description language; Charité – Universitätsmedizin Berlin • Vladimir Korkhov [email protected] Academic Medical Center of the University of Amsterdam • Gabor Terstyanszky [email protected] University of frameworks. The solution integrates and implicit functionality of the workflow Westminster • Noam Weingarten [email protected] University of Westminster • Peter different workflow engines to a grid system; etc. These workflows are now Kacsuk [email protected] MTA SZTAKI • Gabor Kecskemeti [email protected] MTA SZTAKI • application repository and submitter available at the SHIWA repository and Johan Montagnat [email protected] CNRS I3S Lab, UMR 6070 • Ketan Maheshwari [email protected] service, called GEMLCA, and allows grid can be used to compose more complex CNRS I3S Lab, UMR 6070 • Silvia D. Olabarriaga [email protected] Academic Medical Center based workflow systems to access non- “meta-workflows”. of the University of Amsterdam • Tristan Glatard [email protected] CREATIS CNRS UMR 5220
180 | EGI User Forum 2011 T echnologies for Distributed Computing • • • • • • EGI.eu Science Park 105 1098 XG Amsterdam The Netherlands www.egi.eu [email protected]
The EGI-InSPIRE (INFSO-RI-261323) and EMI (INFSO-RI-261611) projects are partially funded by the European Commission.
ISBN 978 90 816927 1 7