Innovative services and tools for NorduGrid (NGIn)

(rehearsal of talk at) Nordunet 2008, 11.04.08

Jon Kerr Nilsen e-mail: [email protected] University of , Department of Physics PhD student, NGIn The NorduGrid Collaboration

A research collaboration that:

Develops and supports ARC middleware Provides middleware to research groups and national Grids Coordinates the various inputs to the KnowARC code Nordic Grid Neighbourhood

3/28/2008 www.nordugrid.org 2 ARC middleware deployed on 50+ sites in 15+ countries all around the world

http://www.nordugrid.org/monitor

28/03/2008 www.nordugrid.org 3 Nordic Data Grid Facility

NDGF pilot project started 2003 Goal: Create a Nordic Grid infrastructure Builds on history and competence of Nordic Grid collaboration NorduGrid/ARC middleware chosen as basis Funded (2 M.EUR/year) by National Research Councils of the Coordinates and hosts major Grid projects (e.g. the Nordic LHC Tier-1) Operates Nordic storage facility for major projects NOS-N Develops grid middleware and DK SF N S services Nordic Data Grid Facility

28/03/2008 www.nordugrid.org 4 NDGF Facility - 2007Q3 EU KnowARC project: 10 partners from 7 countries

NorduGrid members and research teams in medicine, bioinformatics, physics, engineering, automotive industry apps, IT

Univ. of Oslo Univ. of Lund Univ. of Copenhagen Univ. of Uppsala Sweden NIIF Hungary Geneva Univ. Hospitals Switzerland Univ. of Kosice Slovakia Univ. of Lübeck Germany science+computing ag Germany SUN Microsystems Hungary

28/03/2008 www.nordugrid.org 6 Goals of KnowARC

The mission of KnowARC is . to create a novel, powerful Next Generation Grid middleware – extend and re-design ARC – ARC in major Linux distributions – New platforms: Windows, Solaris, Mac OS-X . to promote Grid standardization and ARC interoperability

academia – interoperability with other Grid solutions , , (EGEE and OGF compliant grids)

– possibilities to include (or to be included

middleware

Existing Existing

developers middleware

Experienced in) other infrastructures Industry

application developers application . to prove usage in Health care, Industry and Science – Bioinformatics: Autoimmune Diseases – Medical Informatics: Lung Diseases – Automotive industry

28/03/2008 www.nordugrid.org 7 Nordic Grid Neighbourhood DENMARK: RUSSIA: University of Copenhagen (NBI, DIKU) St. Petersburg State University, Dep. of Physics Southern Danish University, Dep. of Computer Science Petrodvorets Telecommunication Center Project:Aalborg University,… , NO Dep.27 of, NaboComputer 05 Science-06, Nordic Grid Dubna?Neighbourhood Project no.: 172724SWEDEN:/V11 ESTONIA: WeLund have University, read Department and approved of Physics your final academicNational and Institute financial of Chemical project Physics and report Biophysics for (NICPB) Linköping University, National Supercomputer Center the funding period 2005-2006. Estonian Educational and Research Network (EENet) Umeå University, Department of Computing Science and HPC2N Tartu University, Institute of Technology WeUppsala are pleased University, Departmentthat most of Radiation of the Sciences planned goals of the network have been reached : through,Royal Institute i.a. ofworkshops, Technology, NADA/PDC, conferences KTH and tutorials. University of Helsinki, Helsinki Institute of Physics WeStockholm are impressed University and by KTH, the Department number of Physics of partners and participants in the various LITHUANIA: NORWAY: activities. Vilnius University, Institute of Materials Science and Applied , Physics institute Research University of Bergen, Department of Physics LATVIA University of Tromsø, Department of Computer Science ICELAND NTNU, Trondheim, Department of Computer and Information Science NTNU, Trondheim, IT division (ITEA) ... 9 countries, 24 institutions NGIn for Research and Education

•Innovative services and tools for NorduGrid. PI: Farid Ould-Saada, University of Oslo. Nordunet3-funded Internet Research Program 4 year grant Oslo, Lund, Uppsala, Copenhagen, Helsinki, Reykjavik Mainly:  Data Management: storage, indexing and movement  Education and expert training in

5 PhD grants starting January 2007 http://www.nordugrid.org/ngin Grid school http://www.nordunet3.org/

call for collaboration with Baltic and Russia … ??

28/03/2008 www.nordugrid.org 9 NGIn management

Project leader Prof. Farid Ould-Saada, Univ. of Oslo, Norway Technical coordinator Prof. Alex Read, Univ. of Oslo, Norway Steering Group Prof. Farid Ould-Saada, Univ. of Oslo, Norway Prof. Paula Eerola, Univ. of Lund, Sweden Prof. Tord Ekelöf, Univ. of Uppsala Dr. Miika Tuisku, Helsinki Institute of Physics, Finland Dr. Anders Wäänänen, Niels Bohr Institute, Denmark Participating institutions Physics Department, Univ. of Oslo, Norway Physics Department, Univ. of Lund, Sweden Physics Department, Univ. of Uppsala, Sweden Department of Radiation Sciences, Univ. of Uppsala, Sweden Niels Bohr Institute, Univ. of Copenhagen and Univ. of Aalborg, Denmark Helsinki Institute of Physics (HIP), Univ. of Helsinki, Finland CSC, Finnish IT Centre for Science, Helsinki, Finland Univ. of Iceland, Reykjavik, Iceland

28/03/2008 www.nordugrid.org 10 NGIn - training new Grid experts

From the original application: ”The goal of this project is twofold: To extend the excisting middleware and to train new Grid experts...” NGIn obtained 30% of funding applied for The primary goal of the NGIn project is now to train new Grid experts Training programme includes: Grid PhD students A Nordic Grid school (first held in Copenhagen in September 2007) The NGIn activities will go in line with the other NorduGrid related projects Budget 2008: PhD students: 160 k€ Researcer training: 15 k€ Other expenses: 3 k€ Total: 178 k€

28/03/2008 www.nordugrid.org 11 NGIn PhD students

Four PhD students hired, fifth student soon to be hired In chronological order: PhD 1, Jon Kerr Nilsen, Univ. of Oslo, Norway PhD 2, Mika Silander, Helsinki Institute of Physics, Finland PhD 3, Salman Toor, Univ. of Uppsala, Sweden PhD 4, Phillippe Gros, Univ. of Lund, Sweden PhD 5, to be announced soon, Univ. of Oslo, Norway The students are funded 50% by Nordunet3, 50% by local sources All the PhD projects involves data management

28/03/2008 www.nordugrid.org 12 Jon Kerr Nilsen, Oslo

Thesis title: Data Management in Grid Middleware Address limitations and implement solutions in data management and data handling in ARC middleware Use the solutions in ATLAS data analysis Work in collaboration with both NGIn and the Scientific Computing Group at Univ. of Oslo Current status Got familiar with ARC and data management (and Grid acronyms) Survey on data management in conjunction with Grid technology presented at NorduGrid2007 Set up a test cluster with dCache and ARC Storage Element (in collaboration with Salman Toor)

28/03/2008 www.nordugrid.org 13 Jon Kerr Nilsen, Oslo

Near future Storage in conjunction with virtualization Virtualization provides possibilities to run Linux applications in Windows Challenge to get data out of virtual machine and store them securely and efficiently Client side of the next generation ARC storage solution Will work with KnowARC in implementing client- side tools for accessing data in the next generation ARC Look at data management of local versus external dCache instances in current ARC Is there any gain in prefering dCache instances in the same network as the computing resource?

28/03/2008 www.nordugrid.org 14 Mika Silander, Helsinki

Thesis title: Fine-grained rights delegation in Grids  Problem: delegation is too coarse-grained in current middleware, e.g.  Users delegate all rights (Impersonation)  Limited proxy certificates (exclude shell access)  Delegating the least set of rights is desirable  Work so far on  Analysing ARC components for types of operations and objects for rights modeling  Analysing SAML for extending it to support fine- grained rights expressions  Evaluation of existing solutions with applicable features, e.g. SecPAL

27/03/08 www.nordugrid.org 15 Mika Silander, Helsinki

Potential solution candidates  SAML + XACML  Pros: other Grid mw committed to these standards  Cons: complexity in policy definition and evaluation, fine-grained rights not modeled  Security Policy Assertion Language (SecPAL)  Pros: broad set of security functionality including rights restrictions, policies expressed with easily understandable syntax  Cons: restrictive license and implementation languages

27/03/08 www.nordugrid.org 16 Near future plans

Near future plans  Explore possibility of combining pros of the two former for ARC:  Easily understandable syntax similar to SecPAL, but,  Implementation relying on C/C++ and adhering to pure SAML & XACML  Ensure this is compatible with the security architecture of ARC v1.x being developed  Evaluate SAML + XACML solution of gLite  Applicable, parts reusable?

27/03/08 www.nordugrid.org 17 Salman Toor, Uppsala

Thesis title: Data indexing and data movement in grid environments Topic: Management of scientific data in Grid systems Making a Grid Storage Resource Manager A layer on top of various Storage Managers (dCache, DPM, Castor, GPFS...) A complete cost based estimation system to find the best SM amongst the SMs registered

28/03/2008 www.nordugrid.org 18 Salman Toor, Uppsala

Progress Deployed dCache test site in Uppsala (with a pool- node in Oslo) Started study of SRM specifications in conjunction with different Storage Managers Deployed the full Globus toolkit and implemented test services and clients

Next step  write a service that gather the information (owner, permissions etc) of data stored in dCache replicate the data in a different Storage Manager

28/03/2008 www.nordugrid.org 19 Philippe Gros, Lund

Thesis title: Analysis and management of high energy physics data with NorduGrid  Physics oriented project  become an expert user of AliEn  more Grid oriented project  work on the AliEn-ARC integration Current status:  AliEn expertise  still a basic user, which is already a first step (submitting jobs on the Grid)  AliEn-ARC integration  project formulated recently  studying the current state, existing simple interface NGIn Grid School

The NGIn training programme includes: Grid PhD students A Nordic Grid school

The NorduGrid 2007 Grid school was held at Niels Bohr Institute in Copenhagen 26-27 September 2007 Organized by CSC, Finland

More than 35 present for the introductory lectures on first day Tutorials and demos had 15 participants Tutorials in topics from High Energy Physics to bio- informatics and medical imaging From beginner to specialized user in two days! Several participants spotted on the Grid later on!

28/03/2008 www.nordugrid.org 21 NGIn School program

 Day 1 (General Grid Introduction)  Intro to Grid  Intro to ARC  First steps with ARC (tutorials)  Day 2 (Specialized tutorials)  HEP distributed analysis  ARC in bioinformatics, ARC in medical imaging  ARC-gLite interoperability  Grid Job Manager  Dynamic Runtime Environments  Day 3 (ARC Development)  Sys-admin and developer training Next Grid School

 Next Grid School planned to be held during NorduGrid2008 in Budapest, Fall 2008  Similar program as in the first school

 See you there! Summary

NorduGrid is a collaboration developing and deploying the ARC middleware NDGF provides infrastructure and deploys and contributes to the current ARC KnowARC develops the next generation ARC NGIn educates and trains new Grid experts, both through PhD projects and Grid schools The PhD projects contributes to both current ARC and the next generation ARC Next Grid school in Budapest, Fall 2008

28/03/2008 www.nordugrid.org 24