Introduction to GRID com p uting a nd ov e rv ie w of th e E urop e a n Da ta Grid P roj e ct

The European DataGrid Project

http: / / w w w . edg . org Overview

What is GRID computing ?

What is a GRID ?

Why GRIDs ?

GRID pr oj e cts w or l d w id e

T he E ur ope an Data Gr id
Overview of EDG goals and organization
Overview of th e EDG m iddleware c om p onents

Introduction to GRID Computing and the EDG 2 The Grid Vision

The GRID: networked data p roc es s i ng c entres and Res earc hers p erf orm thei r ” m i ddl eware” s of tware as the ac ti v i ti es reg ardl es s “ g l u e” of res ou rc es . g eog rap hi c al l oc ati on, i nterac t wi th c ol l eag u es , s hare and ac c es s data

S c i enti f i c i ns tru m ents and ex p eri m ents p rov i de hu g e am ou nt of data Federico.C a rm in a t i@cern .ch

Introduction to GRID Computing and the EDG 3 What is GRID computing :

coordinated resource sharing and problem solving in dy namic, multi-institutional virtual organiz ations. [ I.Foster]
A V O i s a c ol l ec ti on of u sers sh a ri n g si m i l a r n eed s a n d req u i rem en ts i n th ei r a c c ess to p roc essi n g , d a ta a n d d i stri b u ted resou rc es a n d p u rsu i n g si m i l a r g oa l s.

Ke y c o n c e p t :
ability to negotiate resource-sh aring arrangem ents am ong a set of p articip ating p arties ( p rov id ers and consum ers) and th en to use th e resulting resource p ool f or som e p urp ose [I.Foster]

Introduction to GRID Computing and the EDG 4 The GRID distributed computing idea 1/2

Once upon a time……..

mma a iinnf f rra a mmee

MMiiccrrooccoommppuutteerr

MMiinnii CCoommppuutteerr

CClluusstteerr

(by Christophe J a c q u et, a f ter N . O . W . )

Introduction to GRID Computing and the EDG 5 The GRID distributed computing idea 2/2

…and today

(by Christophe Jacquet)

Introduction to GRID Computing and the EDG 6 Differences between grids and distribu ted ap p l icatio ns

Distributed applications already exist, but they tend to be specialised system s intended f or a sing le purpose or user g roup

G rids g o f urther and tak e into account:
Different kinds of resources
Not always the same hardware, data and applications
Different kinds of intera ction
U ser g rou ps or applications want to interact with g rids in dif f erent ways
Dy na m ic na ture
R esou rces and U ser G rou ps added/ remov ed/ chang ed f req u ently

Introduction to GRID Computing and the EDG 7 Main characteristics of a grid architecture

Service providers
Publish the availability of their services via information systems
S uch services may come-a n d -g o or ch a n g e d ynamically
E . g . a testbed site that offers x C PU s and y G B of storag e
Service b rok ers
R eg ister and categ oriz e p ublished services and p rovid e search cap abilities
E . g . 1 ) E D G R esource B rok er selects the best site for a “ j ob” 2) C atalog ues of d ata held at each testbed site
Service req u est ers
S ing le sig n-on: log into the g rid once
U se brok ering services to find a need ed service and emp loy it
E . g . C M S p hysicists submit a simulation j ob that need s 1 2 C PU s for 6 hours and 1 5 G B w hich g ets sched uled , via the R esource B rok er, on the C E R N testbed site

Introduction to GRID Computing and the EDG 8 GRID security

Resource providers are essentially “opening themselves up” to itinerant users

S ecure access to resources is req uired
X.509 Public Key Infrastructure

U ser’ s identity has to b e certif ied b y ( mutually recogniz ed) national C ertif ication A uthorities ( C A s)

Resources ( node machines) have to b e certif ied b y C A s

T emporary delegation f rom users to processes to b e ex ecuted “in user’ s name” ( prox y certif icates )

C ommon agreed policies f or accessing resource and handling user’ s rights across dif f erent domains w ithin V irtual O rganiz ations

Introduction to GRID Computing and the EDG 9 Why G R I D s

Scale of the problems
frontier research in many different fields today requires world-wide collab orations ( i. e. multi-domain access to distrib uted resources)
G R I D s prov i d e a c c ess to lar g e d at a p r o ces s i n g p o w er a n d h u g e d at a s t o r ag e p o s s i b i li t i es
A s the g rid g rows its usefulness increases ( more resources av ailab le)
L a rg e c ommu n i ti es of possi ble G R I D u sers :
H ig h E nerg y P hysics
E nv ironmental studies: E arthquak es forecast, g eolog ic and climate chang es, oz one monitoring
B iolog y, G enetics, E arth O b serv ation
A strop hysics,
N ew comp osite materials research
A stronautics, etc.

Introduction to GRID Computing and the EDG 1 0 High Energy Physics The LHC Detectors CMS A T L A S

~ 6 -8 P eta B y tes / y ea r 8 ~ 1 0 ev en ts/ y ea r 3 ~ 1 0 b a tch a n d i n tera cti v e u sers ¤ 

 ¤ ¢ ¤ ¤

¤   ¦ £ ¡ © ¡   ¡ ¡ £

© £ £ © ¥ ¦¨§ ¡ ¥ ¡ L H Cb

Introduction to GRID Computing and the EDG 1 1 Earth Observation

ESA missions: • a b ou t 1 0 0 G b y t e s of d a t a p e r d a y (ER S 1 / 2 ) • 5 0 0 G b y t e s, f or t h e ne x t EN V I SAT mission (2 0 0 2 ) .

D a t a G r id c ont r ib u t e t o EO : • e nh a nc e t h e a b il it y t o a c c e ss h ig h l e v e l p r od u c t s • a l l ow r e p r oc e ssing of l a r g e h ist or ic a l a r c h iv e s • imp r ov e Ea r t h sc ie nc e c omp l e x a p p l ic a t ions (d a t a f u sion, d a t a mining ,

mod e l l ing … ) S ource: L . F us co, J une 2 0 0 1 Fe d e r i c o .C a r m i n a t i , E U r e v i e w p r e s e n t a t i o n , 1 M a r c h 2 0 0 2

Introduction to GRID Computing and the EDG 1 2 Biology – BioI n f or m a t ic s

Introduction to GRID Computing and the EDG 1 3 GRID projects world wide
EU
EDG ( EU -I S T ) – R & D EU GR I D p r o j e c t [ w w w . e d g . o r g ]
C r o s s GR I D – Q o S – i n t e r a c t i v e a p p s . [ w w w . c r o s s g r i d . o r g ]
Da t a T A G - i n t e r -o p e r a b i l i t y ( EU -U S A ) [ w w w . d a t a t a g . o r g ]
L C G – T h e L H C C o m p u t i n g GR I D – De p l o y m e n t [ c e r n . c h /l c g ]
T h e n e w 1 6 , 2 B Eu r o EU V I F r a m e w o r k P r o g . GEA N T b a s e d GR I D p r o j e c t s
US A
Gr i P h y N i V DGL -V DT v 1 P P DG ( N S F , Do E ) [ w w w . g r i p h y n . o r g ] [ w w w . i d v g l . o r g ] [ w w w . p p d g . o r g ]
A s i a
A p Gr i d P r a g m a ( U S A -A s i a ) [ w w w . a p g r i d . o r g ] A n d m a n y m o r e . . .

Introduction to GRID Computing and the EDG 1 4 The European Data Grid

To build on the emerging Grid technology to develop a s us tainable computing model f or ef f ective s hare of computing res ources and data
S tart : J an 1 , 2 0 0 1 E nd : D ec 3 1 , 2 0 0 3
S pecif ic proj ect obj ectives :
Middleware for fabric & Grid management (mostly funded by the E U )
L arge scale testbed (mostly funded by the p artners)
P roduction q uality demonstrations (p artially funded by the E U )
To collaborate w ith and complement other E uropean and U S proj ects
C ontribute to O pen S tandards and international bodies :
C o-founder of Global GR I D F orum and host of GGF 1 and GGF 3
I ndustry and R esearch F orum for dissemination of p roj ect results

Introduction to GRID Computing and the EDG 1 5 The EDG Main Partners

CERN – I n t e r n a t i o n a l ( S w i t z e r l a n d / F r a n c e )

CNRS - F r a n c e

ES A / ES RI N – I n t e r n a t i o n a l ( I t a l y )

I NF N - I t a l y

NI K H EF – T h e Ne t h e r l a n d s

P P A RC - U K

Introduction to GRID Computing and the EDG 1 6 EDG A s s i s t a n t P a r t n e r s

Industrial P artners •D a t a m a t ( I t a l y ) •I B M -U K ( U K ) •C S -S I ( F r a n c e ) Research and Academic Institutes •C E S N E T ( C z e c h R e p u b l i c ) •C o m m i s s a r i a t à l ' é n e r g i e a t o m i q u e ( C E A ) – F r a n c e •C o m p u t e r a n d A u t o m a t i o n R e s e a r c h I n s t i t u t e , H u n g a r i a n A c a d e m y o f S c i e n c e s ( M T A S Z T A K I ) •C o n s i g l i o N a z i o n a l e d e l l e R i c e r c h e ( I t a l y ) •H e l s i n k i I n s t i t u t e o f P h y s i c s – F i n l a n d •I n s t i t u t d e F i s i c a d ' A l t e s E n e r g i e s ( I F A E ) - S p a i n •I s t i t u t o T r e n t i n o d i C u l t u r a ( I R S T ) – I t a l y •K o n r a d -Z u s e -Z e n t r u m f ü r I n f o r m a t i o n s t e c h n i k B e r l i n - G e r m a n y •R o y a l N e t h e r l a n d s M e t e o r o l o g i c a l I n s t i t u t e ( K N M I ) •R u p r e c h t -K a r l s -U n i v e r s i t ä t H e i d e l b e r g - G e r m a n y •S t i c h t i n g A c a d e m i s c h R e k e n c e n t r u m A m s t e r d a m ( S A R A ) – N e t h e r l a n d s •S w e d i s h R e s e a r c h C o u n c i l - S w e d e n

Introduction to GRID Computing and the EDG 1 7 EDG overview: Middleware release schedule

Release schedule

testbed 1: 2 0 0 1

testbed 2 : 2 0 0 2

testbed 3 : 2 0 0 3
I n cr em en t al r eleases b et w een t hese m aj o r dat es
E ach r elease i n cludes
f eedb ack o n use o f p r ev i o us r elease b y ap p li cat i o n g r o up s
p lan n ed i m p r o v em en t s/ ex t en si o n b y m i ddle-w ar e g r o up s
A p p li cat i o n g r o up s ( H E P , E O , B i o -I n f o ) ar e usi n g ex i st i n g so f t w ar e an d t est b ed t o ex p lo r e ho w t hey can b est ex p lo i t g r i ds

Introduction to GRID Computing and the EDG 1 8 EDG overview : current project status

EDG currently provides a set of middleware services
J ob & Data M anag ement
GR I D & N etwork monitoring
S ecurity, A uth entication & A uth oriz ation tools
F ab ric M anag ement
R uns on L inux R ed H at 6 . 2 + updates platform
S ite install & config tools and set of common services availab le
5 principle EDG 1 . 2 . 0 sites currently b elong ing to th e EDG-T estb ed
CERN ( C H ) , R A L ( U K ) , N I K H EF ( N L ) , C N A F ( I ) , C C -L yon( F ) ,
b eing deployed on oth er EDG testb ed sites ( ~ 1 0 )
I ntense middleware development continuously g oing on, concerning :
N ew features for j ob partitioning and ch eck -pointing , b illing and accounting
N ew tools for Data M anag ement and I nformation S ystems.
I nteg ration of network monitoring information inside th e b rok ering polices

Introduction to GRID Computing and the EDG 1 9 EDG structure : work packages
The EDG collaboration is structured in 12 Work Packages:
WP1: Wo r k L o a d M a n a g e m e n t S y s t e m
WP2 : D a t a M a n a g e m e n t
WP3 : G r i d M o n i t o r i n g / G r i d I n f o r m a t i o n S y s t e m s
WP4 : F a b r i c M a n a g e m e n t
WP5 : S t o r a g e E l e m e n t
WP6 : Testbed and demonstrators
WP7 : N e t w o r k M o n i t o r i n g
WP8 : H i g h E n e r g y Ph y s i c s A p p l i c a t i o n s
WP9 : E a r t h O b s e r v a t i o n AApppplliiccaattiioonnss
WP10 : B i o l o g y

}} WP11: D i s s e m i n a t i o n
WP12 : M a n a g e m e n t

Introduction to GRID Computing and the EDG 2 0 EDG Globus-ba se d m i d d le w a r e a r c h i t e c t ur e

Current EDG architectural functional blocks:

Basic Services (authentication, authorization, Replica Catalog , secure f ile transf er, I nf o P rovid ers) rely on G lob us 2
H igher level E D G m id d lew are (d eveloped w ithin E D G )
A pplications (H E P ,BI O ,E O ) S p e c i f i c a p p l i c a t i o n l a y e r ALICE AT LAS CM S LH Cb O t h e r a p p s VOs c o m m o n LHC O t h e r a p p s a p p l i c a t i o n l a y e r G R I D m i d d l e w a r e H ig h l evel G R I D m id d l ew a re

G LO B U S B a sic Services 2

OS & Net services

Introduction to GRID Computing and the EDG 2 1 EDG middleware GRID architecture APPLICATIONS

Local Computing L L ooc c aal l A A p p p p l liic c aattiioonn L L ooc c aal l D D aattaab b aas s e e

Grid G G rriid d A A p p p p lliicca a ttiioon n L L a a y y eerr

J J oob b D D aattaa M M e e ttaad d aattaa M M aannaag g e e mme e nntt M M aannaag g e e mme e nntt M M aannaag g e e mme e nntt

CCoolllleeccttiivvee SSeerrvviicceess

G G rriid d R R e e p p l liic c aa IInnffoorrmmaattiioonn S S c c h h e e d d u u l le e rr M M aannaag g e e rr & & M M oonniittoorriinng g

U U n n d d eerrlly y iin n g g G G rriid d SSeerrvviicceess

S S Q Q L L C C oommp p u u ttiinng g S S ttoorraag g e e R R e e p p l liic c aa A A u u tth h oorriiz z aattiioonn S S e e rrv v iic c e e D D aattaab b aas s e e E E l le e mme e nntt E E l le e mme e nntt C C aattaal loog g A A u u tth h e e nnttiic c aattiioonn IInnd d e e x x S S e e rrv v iic c e e s s S S e e rrv v iic c e e s s S S e e rrv v iic c e e s s aannd d A A c c c c oou u nnttiinng g M / W Grid

F a b ric services F a b ric F a b ric services G LOB U S R R e e s s oou u rrc c e e C C oonnffiig g u u rraattiioonn M M oonniittoorriinng g N N ood d e e F F aab b rriic c S S ttoorraag g e e M M aannaag g e e mme e nntt M M aannaag g e e mme e nntt aannd d IInns s ttaal ll laattiioonn & & M M aannaag g e e mme e nntt F F aau u l ltt T T ool le e rraannc c e e M M aannaag g e e mme e nntt

Introduction to GRID Computing and the EDG 2 2 EDG interfaces

AA ppll icc aa tion L L oocca al lA A p p p p lilcica atitoion n L L oocca al lD D a atatab b a assee DD ee vv ee ll opee rr ss SS cc iee ntiss tss SS yy ss tee m MM aa naa gee rr ss G G rridid A A p p p p lilcica atitoion n L L a ay yeerr Cee rr tiff icc aa tee AA uthh orr itiee ss J ob D a ta M eta d a ta O b j ect to F ile J ob M a n a D g aetm a en t M a M n a eg taedm a etan t O M b j aepctp tion g F ile M M a an n a ag g eem m een n tt M a n a g em en t M a n a g em en t M a p p in g

FF ill ee SS yy ss tee mss CCoolllelecctitvivee S Seervrviciceess I nI n f of orrm m a atitoion n & & R R eep p lilcica a G G rridid UU ss ee rr AA cc cc ountss M M oon n itiotorirnin g g M M a an n a ag g eerr SScch h eed d u u lelerr

U U n n d d eerrlyly inin g g G G rridid S Seervrviciceess

SSQ Q L L D D a atatab b a assee CCoom m p p u u titnin g g SStotorarag g ee R R eep p lilcica a A A u u thth oorriziza atitoion n SSeervrvicicee I nI n d d eex x SSeerrvviciceess E E lelem m een n t t E E lelem m een n t t CCa atatalolog g A A u u thth een n titcica atitoion n SSeervrviciceess SSeervrviciceess a an n d d A A ccccoou u n n titnin g g

F F a ab b rricic s seerrvviciceess

R R eessoou u rcrcee CCoon n f if gig u u raratitoion n M M oon n itiotorirnin g g N N ood d ee F F a ab b rricic S Stotorarag g ee M M a an n a ag g eem m een n tt M M a an n a ag g eem m een n tt a an n d d I nI n sstatalllalatitoion n & & M M a an n a ag g eem m een n tt F F a au u ltl tT T ooleleraran n ccee M M a an n a ag g eem m een n tt

OO pee rr aa ting BB aa tcc hh SS yy ss tee mss S y s te ms S y s te ms PP BB SS ,, LL SS FF ,, ee tcc .. SS torr aa gee MM aa ss ss SS torr aa gee SS yy ss tee mss EE ll ee mee ntss Computing EE ll ee mee ntss HH PP SS SS ,, Caa ss torr

Introduction to GRID Computing and the EDG 2 3 EDG : reference web sites

EDG web site
http://www.edg.org

S o u r c e f o r a l l r eq u ir ed so f twa r e :
http://da ta gri d.i n 2 p3 .f r

EDG testbed web site
http://m a ri a n n e.i n 2 p3 .f r

EDG u ser s g u id e
http://m a ri a n n e.i n 2 p3 .f r/da ta gri d/doc u m en ta ti on /E D G -U s ers -G u i de.htm l

EDG tu to r ia l s web site
http://c ern .c h/edg-tu tori a l s

EDG p r o d u c tio n testbed c u r r en t r ea l tim e u p d a ted set u p
http://tes tb ed0 0 7 .c ern .c h/tb s ta tu s -b i n /i n f oi n dex c ern .pl

Introduction to GRID Computing and the EDG 2 4