Introduction to the Gridway Metascheduler
Total Page:16
File Type:pdf, Size:1020Kb
Introduction to the GridWay Metascheduler HPC SYSADMINCongreso MEETING'12 Barcelona,Cuidad, SpainSpain GridWay OctoberMay 15-16, 15, 2007 2012 Dr Ismael Marín Carrión Distributed Systems Architecture Group Universidad Complutense de Madrid e-mail: [email protected] / [email protected] Initiative for Globus in Europe 1/15 Contents Initiative for Globus in Europe GridWay 1. What is GridWay? 2. A Global Vision 3. Initiative for Globus in Europe 4. The GridWay Project DSA Group 2/15 What is GridWay? Initiative for Globus in Europe GridWay ● The GridWay metascheduler enables large-scale, reliable and efficient sharing of computing resources over different grid middleware, providing a single point of access for them ● GridWay provides a LRM-like CLI for submitting, monitoring, synchronizing and controlling jobs ● GridWay implements the OGF standard DRMAA API, assuring compatibility of applications with LRM systems that implement the standard • GridWay provides BES- and GRAM-compliant interfaces, enabling the submission and monitoring of jobs through a standard interface DSA Group 3/15 Contents Initiative for Globus in Europe GridWay 1. What is GridWay? 2. A Global Vision 3. Initiative for Globus in Europe 4. The GridWay Project DSA Group 4/15 /15 5 InitiativeGlobusfor inEurope Globus services Globus interfaces Standard TCP/IP) (e.g. end-to-end open source open management jobexecution brokering resource Standard API (OGF DRMAA) (OGF API Standard Command Interface Line highly dynamic &heterogeneous dynamic highly fault high rate • • • • • • • • • • Applications Grid Meta- Grid Middleware Scheduler Results Infrastructure CLI SGE $> Globus GridWay PBS DRMAA .C, .java .C,.java Application-Infrastructure decoupling A Global Vision GlobalArchitecture of aComputational Grid GridWay GridWay DSA GroupGroup DSA DSA A Global Vision Initiative for Globus in Europe GridWay LocalLocal InterfacesInterfaces BES GRAM RemoteRemote InterfacesInterfaces DRMAA library CLI JobJob Submission Submission JobJob Monitoring Monitoring GridWay Core Request JobJob Control Control Manager JobJob Migration Migration Job Pool Host Pool Dispatch Scheduler Manager Transfer Execution Information Manager Manager Manager GRAM GridFTP GASS CREAM MDS2 MDS4 2-4-5 JobJob PreparationPreparation JobJob TerminationTermination SCP BES SSH BDII Files ResourceResource DiscoveryDiscovery JobJob MigrationMigration ResourceResource MonitoringMonitoring File Transfer Execution Information Services Services Services DSA Group 6/15 Contents Initiative for Globus in Europe GridWay 1. What is GridWay? 2. A Global Vision 3. Initiative for Globus in Europe 4. The GridWay Project DSA Group 7/15 Initiative for Globus in Europe Initiative for Globus in Europe GridWay • The IGE project is a EU FP7 project to coordinate European Globus activities • Main objectives: – Support the European computing infrastructures and their users – Provide a central point of contact in Europe for Globus – Strengthen the influence of European developers and users on the development of Globus IGE serves as a comprehensive service provider for the European e- infrastructures regarding the development, customization, provisioning, support, and maintenance of components of the Globus Toolkit DSA Group 8/15 Initiative for Globus in Europe Initiative for Globus in Europe GridWay Work Plan DSA Group 9/15 Initiative for Globus in Europe Initiative for Globus in Europe GridWay IGE Testbed ● GridWay instance provided by: – Universidad Complutense de Madrid (UCM), Spain • Globus 5 resources provided by: – Technische Universität Dortmund (TUDO), Germany – Poznan Supercomputing and Networking Center (PSNC), Poland – Leibniz-Rechenzentrum, Bayerische Akademie der Wissenschaften (BADW-LRZ), Germany – Universitatea Technica Cluj-Napoca (UTCN), Romania – Stichting voor Fundamenteel Onderzoek der Materie – Institute for Subatomic Physics (FOM-NIKHEF), Netherlands – University of Edinburgh – Edinburgh Parallel Computing Centre (UEDIN-EPCC), United Kingdom • Other resources: – University of Southampton (SOTON), United Kingdom DSA Group 10/15 Contents Initiative for Globus in Europe GridWay 1. What is GridWay? 2. A Global Vision 3. Initiative for Globus in Europe 4. The GridWay Project DSA Group 11/15 Initiative for Globus in Europe More at: http://www.gridway.org/ (Success Stories) /15 12 The GridWay Project IRISGrid Politecnico di Torino Some Projects. and Infrastructures CABGrid (Astrobiology Center) . C2VO (Universidad de Castilla La Mancha) GridWay . ESAC Grid (European Space Agency) . CRO-GRID (Croatia) . Sun Microsystems Solution Center World Grid . EGEE Infrastructure . BeinGRID Project . GridX1 (Canadian Grid for HEP applications) . Universidade do Porto . Madras Institute of Technology . National Center for High-Performance Computing . Life-Sciences Some ApplicationAerospace Porting Areas • Fusion Physics • Computational Chemistry • • DSA Group The GridWay Project Initiative for Globus in Europe GridWay Development Process • Community – Open Source Project. Globus Development Philosophy • Development Infrastructure: • Mailing Lists (thanks to Globus Project!) • Bugzilla (dev.gridway.org) • Readmine • SVN • You are very welcome to contribute: • Reporting Bugs ([email protected]) • Making feature requests for the next GridWay release ([email protected]) • Contributing your own developments (bug fixes, new features, documentation) • Detailed Roadmap: • http://dev.gridway.org/projects/gridway/roadmap DSA Group 13/15 Useful Links Initiative for Globus in Europe GridWay ● The GridWay Project: http://gridway.org/ ● GridWay Development: http://dev.gridway.org/ ● Documentation: http://gridway.org/doku.php?id=documentation ● Support: http://gridway.org/doku.php?id=support ● User discussion: [email protected] ● Announcements: [email protected] ● The IGE Project: http://www.ige-project.eu/ DSA Group 14/15 Introduction to the GridWay Metascheduler Initiative for Globus in Europe GridWay Thank you for your attention! DSA Group 15/15 Installation and Basic Configuration HPC SYSADMINCongreso MEETING'12 Barcelona,Cuidad, SpainSpain GridWay OctoberMay 15-16, 15, 2007 2012 Dr Ismael Marín Carrión Distributed Systems Architecture Group Universidad Complutense de Madrid e-mail: [email protected] / [email protected] Initiative for Globus in Europe 1/19 Contents Initiative for Globus in Europe GridWay 1. Installation 2. Configuration 3. Logging 4. Scheduling Policies DSA Group 2/19 Installation Initiative for Globus in Europe GridWay Hands on! • Available accounts: ige_user001 – ige_user030 • Log in (SSH) the host: gridway.fdi.ucm.es • This host provides the GW instance of the IGE testbed (multi-user) • You can install and configure your own GridWay instance (single-user installation) • Each account has a certificate for authentication within the IGE testbed • Now you are ready! DSA Group 3/19 Installation Initiative for Globus in Europe GridWay Required Software for GridWay Core ● C Compiler ● JDK for building the DRMAA Java binding ● GNU Make ● Globus GSI Proxy Utility Programs ● Sudo command ● Only required for multiple-user mode ● Berkeley Database Library ● Only required to compile the accounting module DSA Group 4/19 Installation Initiative for Globus in Europe GridWay Installation Procedure ● User or multi-user installation? ● If multi-user installation, then create the gwusers group and the gwadmin user ● Download and uncompress gridway-5.12.0.tar.gz ● Change to the directory containing the GridWay distribution ● Set environment variables $GW_LOCATION and $GLOBUS_LOCATION ● Run ./configure –prefix=$GW_LOCATION ● There are more options -- check them out in the manual ● Run 'make' and 'make install' ● If multi-user installation, then some privileges are required to be set in the /etc/sudoers file ● Build and install the needed drivers DSA Group 5/19 Installation Initiative for Globus in Europe GridWay Required Software for GridWay Drivers ● C, C++ and Java Compilers, and GNU Make ● For Static (IM), GRAM5 (EM) and GridFTP (TM) drivers, install the software: – Building: Globus Common Library Development Files, GRAM Client Library, GRAM Client Library Development Files, Globus Gass Copy, GridFTP Client Library Development Files, Globus Gass Copy Development Files, GPT and Globus Common Library Programs – Installation: GridWay Core, GRAM Client Library, Globus Gass Copy, Globus Gass Copy Programs and Globus GSI Proxy Utility Programs ● For BDII (IM), CREAM (EM) and Dummy (TM) drivers, install the software: – Building: Development Files for the Client of the CREAM service – Installation: GridWay Core, Globus Gass Server_ez Programs, C/C++ Libraries for the Client of the CREAM service and the Plugin for gSoap to use glite-security-gss as the communication layer ● For Static (IM), BES (EM) and Dummy (TM) drivers, install the software: – Building: JDK, AXIS, Xalan, XMLBeans and the GridSAM schema – Installation: GridWay Core, JRE, AXIS, WSDL4J, Log4j, Xalan, XMLBeans and the GridSAM schema DSA Group 6/19 Installation Initiative for Globus in Europe GridWay Installation Procedure ● Change to src/{im_mad,em_mad,tm_mad}/{driver} ● Run 'make' and 'make install' ● Configure GridWay through the file $GW_LOCATION/etc/gwd.conf (see next point) ● Configure the available hosts in case of using the Static Files IM driver (see next point) ● If multi-user installation, then some privileges are required to be set in the /etc/sudoers file ● Launch the GridWay daemon from $GW_LOCATION/bin/gwd. ● Test it