Integrated Service and Desktop Grids for Scientific Computing
Total Page:16
File Type:pdf, Size:1020Kb
Integrated Service and Desktop Grids for Scientific Computing Robert Lovas Computer and Automation Research Institute, Hungarian Academy of Sciences, Budapest, Hungary [email protected] Ad Emmen AlmereGrid, Almere, The Nederlands [email protected] RI-261561 GRID 2010, DUBNA Why Desktop Grids are important? http://knowledgebase.ehttp://knowledgebase.e--irg.euirg.eu RI-261561 GRID 2010, DUBNA Introduction RI-261561 WP4 Author: Robert Lovas, Ad Emmen version: 1.0 Prelude - what do people at home and SME’s think about grid computing Survey of EDGeS project Questionnaires all across Europe Get an idea of the interest in people and SMEs to donate computing time for science to a Grid Get an idea of the interest in running a Grid inside an SME RI-261561 GRID 2010, DUBNA Survey amongst the General Public and SME’s RI-261561 GRID 2010, DUBNA Opinions about Grid computing RI-261561 GRID 2010, DUBNA Survey - Conclusions Overall: there is interest in Desktop Grid computing in Europe. However, that people are willing to change their current practice and say that they want to participate in Grid efforts does not mean that they are actually going to do that. Need to generate trust in the organisation that manages the Grid. People want to donate computing time for scientific applications, especially medical applications. They do not like to donate computing time to commercial or defense applications. People want feedback on the application they are running. No clear technical barriers perceived by the respondents: so this does not need much attention. Overall the respondents were rather positive about donating computing time for a Grid or about running applications on a Grid. RI-261561 GRID 2010, DUBNA A Grid definition Connecting a large number of resources and make them available for a set of applications Resources belong to different owners Heterogeneous system (HW, SW, network) Typically for Desktop Grids: it is the Desktop Grid computer that initiates all communications; there is no central control over the desktop computer Most computers in a Desktop Grid are desktop computers, but they can also be notebooks, or servers, or mobile phones Æ limited capacities and vulnerable software environment Service Grids have untrusted applications but trusted resource providers; Desktop Grids have trusted applications, but untrusted resource providers RI-261561 GRID 2010, DUBNA Different owners of resources Citizens’ and SMEs’ PC’s: Desktop Grid Thousands or even millions of owners Company divisions: Enterprise Grid A dozen or so organisations within one enterprize Universities and research institutes - eScience Grid Up to a few hundred (in the case of EGEE, a pan-European Grid) RI-261561 GRID 2010, DUBNA An example AlmereGrid - World’s first CityGrid Almere is a new town in the Netherlands Good cradle for the world’s first CityGrid Desktop Grid used for scientific applications: sharing CPU power Back-Up Grid experiment to share hard disk space (as part of EU BEinGRID project) Build-on-Grid and Virtual Reality based environment to allow people to build their own house in virtual space with all the right data and checking permits before it is built in reality. (Feasibility study for municipality finished.) RI-261561 GRID 2010, DUBNA For science and beyond Applications on BEinGRID HealthGrid backup/ diagnosis Science Grid Build-on-Grid the Grid restore Own house agent based applications e-Education construction using AlmereGrid Grid Broadband network infrastructure Strengthen knowledge infra- structure of the city RI-261561 GRID 2010, DUBNA Desktop Grids - introduction Basic techniques Open source examples XtremWeb BOINC Commercial examples LSF Desktop Datasynapse Parabon RI-261561 GRID 2010, DUBNA Basic structure Resource computers The resource computers (also called workers, clients,..) execute the work User system or interface Interface for the Grid user to submit his/her application, monitor the progress, and retrieve results Grid server Handles the requests from the resource computers to get new work or submit the results Handles the requests from the users to execute jobs RI-261561 GRID 2010, DUBNA Basic desktop Grid (players) Companies Grid organisation Universities & research institutes Residential areas RI-261561 GRID 2010, DUBNA Basic desktop Grid (components) Grid resources Grid server Grid resources Grid users RI-261561 GRID 2010, DUBNA Volunteer desktop Grids Uses a pull model: it is always the resource computer that asks for work and initiates all communication with the server There is an application repository: users can only run validated and trusted applications The resources, however, are untrusted and volatile: you are not sure they deliver the correct answer RI-261561 GRID 2010, DUBNA Local Deskop Grids Organisations (universities, companies) can also run an internal Desktop Grid Can be composed of trusted resources Use spare capacity Can also include servers and clusters RI-261561 GRID 2010, DUBNA Local DGs in practice – University of Westminster (London) as an example 6 1 2 5 1. New Cavendish Street 576 nodes 4 2. Marylebone Campus559 nodes 3 3. Regent Street 395 nodes 4. Wells Street 31 nodes Lifecycle of a node: 5. Little Tichfield Street 66 nodes 1. PCs basically used by students/staff 6. Harrow Campus 254 nodes 2. If unused, switch to Desktop Grid mode Total: 1881 nodes 3. No more work from DG server -> shutdown (green solution) RI-261561 GRID 2010, DUBNA 17 Desktop Grid applications There is --in general-- no communication between the resources Hence you can only run a subclass of parallel application: bag of tasks, master-worker, parameter sweep or monte-carlo simulation In volunteer computing the network bandwidth and the HDD/CPU capacity can be rather limited But there can be large numbers of resources RI-261561 GRID 2010, DUBNA XtremWeb Developed by IN2P3 and INRIA in France http://xtremweb.net Written in Java Can be used for all type of Grids, from simple local to advanced volunteer Desktop Grids RI-261561 GRID 2010, DUBNA BOINC Middleware designed for volunteer Desktop grid computing http://boinc.berkeley.edu/ Used for the largest Grid in the world: seti@home More than 4 million computers world wide have BOINC installed Many other projects use BOINC RI-261561 GRID 2010, DUBNA Some other Desktop Grids World Community Grid - IBM (http://www.worldcommunitygrid.org/) 300.000 PCs Leiden Classical Grid - Education on Grid (http://boinc.gorlaeus.net/) 17.000 PCs SZTAKI - Hungarian initiative (http://www.desktopgrid.hu/) 70.000 PCs AlmereGrid (http://almeregrid.nl) 3.000 PCs PS3GRID (Based on Playstations) (http://ps3grid.net/) RI-261561 GRID 2010, DUBNA The problem we addressed • High costs, Supercomputer • small number of sites and based SGs large number of CPUs (DEISA, TeraGrid) OGF GIN • MPI/OpenMP appls • Moderately expensive, Cluster based • moderate number of sites service grids (SGs) and CPUs (EGEE, OSG, etc.) EDGeS • several appls • Inexpensive, Desktop grids (DGs) • very large number of (volunteer DGs – home computers, CPUs (~100K – 1M) organizational DGs – institutional desktops) • Bag of task appls RI-261561 GRID 2010, DUBNA EDGeS - Enabling Desktop Grids for e-Science Connect Service Grids (EGEE) with Desktop Grids (SZTAKI Desktop Grid, AlmereGrid, Extramadura Grid…) to provide even more alternative resources to scientists Provide a Bridge between these types of Grids for automatic job sharing Provide an Application Development Methodology to port applications to the Grid; Port a number of (new) applications to the Grid Organise users and industry Two year, EU funded project (started 1-1-2008) RI-261561 GRID 2010, DUBNA EDGeS - Grid infrastructure RI-261561 GRID 2010, DUBNA Generic Grid-Grid (3G) Bridge to integrate SGs and DGs 3G Bridge Job source BOINC BOINC client BOINC Job Database Plugin DC-API Job source EGEE GRAM EGEE EGEE EGEE Plugin Queue Manager Job Handler Interface Job Handler GridHandler Interface gUSE/ WS- Xtrem PGRADE gUSE b Web portal job submitter Plugin XtremWe RI-261561 GRID 2010, DUBNA Scenario 1 – DG to EGEE via bridge EGEE EDGeS VO Desktop Grid DG->EGEE WMS bridge and other DG user EGEE services User entry point is DG – using EGEE is completely transparent from user’s point of view RI-261561 GRID 2010, DUBNA For Desktop Grid users wanting to run jobs on EGEE EDGeS did create a new Virtual Organisation in EGEE (VO) called as EDGeS VO Sites in the EDGeS VO support the applications of the EDGeS connected Desktop Grids that are available in the Application Repository Jobs of the connected Desktop Grids can be seamlessly executed in the EDGeS VO RI-261561 GRID 2010, DUBNA EDGeS VO activity on EGEE accounting portal CPU > 1 Day/Week CPU > 1 Month/Week CPU > 1 Year/Week RI-261561 GRID 2010, DUBNA 28 Scenario 2 – EGEE to DG via bridge Desktop Grid 1 EGEE VO Edges Services WMS and other EGEE DG CE + services EDGeS AR EGEE user (using EGEE UI machine or Desktop Grid n portal) − User entry point is EGEE − using DG is transparent from user’s point of view RI-261561 GRID 2010, DUBNA For EGEE users wanting to run on Desktop Grids Bag-of-tasks or parameter sweep type of EGEE applications are selected and ported to BOINC / XtremWeb Porting is a joint work of EDGeS experts and the application developer No certificate support in BOINC: all applications are validated by the EDGeS experts team Validated applications are placed in the EDGeS Application Repository Desktop Grids connected by EDGeS can register applications from the EDGeS Application Repository EGEE users can take applications from the Application Repository and submit via the EGEE->Desktop Grid bridges RI-261561 GRID 2010, DUBNA EGEE VOÆ DG system DG servers DC-API plugin Info provider Bridge CE 3G BOINC1 Report resources (SZDG) EGEE BDII and performance DC-API plugin Submit job DG-LRMS EGEE Bridge Add job 3G BOINC2 WMS Log events EGEE Watch job (UoW) LB Send output EGEE Submit Watch Check job Get output VOMS EXE XWeb plugin X509 Bridge 3G XWeb proxy EDGeS EGEE UI Application IN2P3 Repository RI-261561 GRID 2010, DUBNA Get EXE EADM - EDGeS Application Development Methodology 1.