Porting Fusion and Biomed Applications using DRMAA API and Globus GridWay
JOINT EGEE AND SEE-GRID SUMMERC oSnCgHreOOsoL ON GRID APPLICATION SUPPORT BudCaupeidsatd, ,H Sunpgaainry GridWay JMuneay 2195, ,2 200707
José Luis Vázquez-Poletti Distributed Systems Architecture Group Universidad Complutense de Madrid
1/23 Contents GridWay 1. A Global Vision 2. GridWay Out There 3. Interacting With GridWay 3.1. Command Line Interface 3.2. DRMAA API (C Binding) 4. A Short Demo
DSA Group 2/23 1. A Global Vision
GridWay s
DRMAA n
Results o i $> t • standard API (OGF a g CLI c i DRMAA) n l i l p • Command Line Interface p ..CC,, .j.javaa p u A
o Grid Scheduling Steps - c r a e e t l d e
u • open source meta- e d M r
e scheduler u d h i t GridWay c r • job execution c S G u management r e t
r • resource brokering s a a r w f e l n • Standard interfaces I d - Globus d
n • Globus services i o
i • end-to-end (e.g. M t
a d TCP/IP) i c i r l G p • highly dynamic p
A PBS SGE Infrastructure • heterogeneous • high fault rate
DSA Group 3/23 1. A Global Vision
GridWay
Workload Management
• Advanced scheduling functionalities and new scheduling policies support • Failure detection and recovery • Accounting service • Simple jobs, job arrays and DAGs
User Interface
• Complete support for the OGF DRMAA Standard (C & JAVA) • DRM-like command line interface
Integration
• Deployment of new services is not needed • Interface with new grid services • Interoperability with other infrastructures
DSA Group 4/23 1. A Global Vision
GridWay
GridWay complements gLite
FEATURES BENEFITS DRMAA OGF Standard support Application compatibility with DRM systems which implement the (C & JAVA) Standard: SGE, Torque, ... DRM-like Command Line Interface CLI similar to that found in local (allows users to submit, kill, migrate, resource managers query state and synchronize jobs) Lightweight Middleware Higher Efficiency for certain application profiles Site-level Accounting and scheduling Use analysis, profiling utilization and policies user monitoring
Minimum installation requisites Easy deployment and mantainment Interoperability Simultaneous access to different infrastructures
DSA Group 5/23 1. A Global Vision
GridWay
DSA Group 6/23 1. A Global Vision
GridWay
Grid Infrastructure
Users GridWay daemon Execution Transfer Information Scheduling Manager Manager Manager Module
GSI- pre-WS pre-WS MDS2 MDS2 Grid-FTP MDS2 MDS2 Grid-FTP MDS2 Blahp GRAM GRAM
BLAHP Condor-C/BLAHP Storage Element PBS Cluster Storage Element PBS Cluster PBS Cluster EGEE-CE EGEE-CE EGEE-SE EGEE-CE EGEE-SE
Future gLite 3.0.2 LCG 2 gLite (based on pre-WS GT)
DSA Group 7/23 1. A Global Vision
GridWay
GridWay / LCG-2 RB Comparative EGEE 1st User Forum (CERN, 1st-3rd March 2006)
Resubmissions/Reschedules 25 32,5 30 22,5 27,5 20 25 22,5 17,5 20 LCG-2 17,5 GridWay 15 15 12,5 LCG-2 12,5 10 GridWay 7,5 10 5 7,5 2,5 0 5 2 Jobs Affected Total Max Resub/Resch per Job 2,5 1,8 Productivity (Every 15') 1,6 50 0 Avg Productivity 45 1,4 40 1,2 35 LCG-2 1 30 GridWay LCG-2 0,8 25 GridWay 20 0,6 15 0,4 10
0,2 5
0 0 Avg. Overhead/job 0' 15' 30' 45' 1h 1h 1h 1h 2h 2h 2h 2h 3h 3h 3h 15' 30' 45' 15' 30' 45' 15' 30'
DSA Group 8/23 Contents GridWay 1. A Global Vision 2. GridWay Out There 3. Interacting With GridWay 3.1. Command Line Interface 3.2. DRMAA API (C Binding) 4. A Short Demo
DSA Group 9/23 2. GridWay Out There
GridWay
Some Projects and Infrastructures
● IRISGrid ● Politecnico di Torino ● CABGrid (Centro de Astrobiología) h ● C2VO (Universidad de Castilla La Mancha) tt p: M ● Grid en ESAC (Agencia Espacial Europea) // (Su w o ● CRO-GRID (Croacia) ww r cce e ● Sun Microsystems Solution Center World Grid .g ss rid a ● Infraestructura EGEE t Sto wa : ● Proyecto BeinGRID r ies y.or ● GridX1 (Canadian Grid for HEP applications) ) g ● Universidade do Porto / ● Madras Institute of Technology ● National Center for High-Performance Computing
Some Application Porting Areas
● Life-Sciences ● Aerospace ● Fusion Physics ● Computational Chemistry
DSA Group 10/23 2. GridWay Out There
GridWay MAssive RAy TRAcing in Fusion Plasmas
Executable: Truba x N • 1,8 MB – 9' – 50 Executions Input files =~ 70 KB
Output files =~ 549 KB CD-HIT
DSA Group 11/23 Contents GridWay 1. A Global Vision 2. GridWay Out There 3. Interacting With GridWay 3.1. Command Line Interface 3.2. DRMAA API (C Binding) 4. A Short Demo
DSA Group 12/23 3.1 Command Line Interface
GridWay Main Commands
● gwps: Shows job information and state ● gwhistory: Shows execution history ● gwkill: Sends signals to a job (kill, stop, resume, reschedule) ● gwsubmit: Submits a job or array ● gwwait: Waits for job's end (any, all, set) ● gwuser: User Monitoring ● gwhost: Host Monitoring ● gwacct: Accounting
Application Model and Lifecycle
DSA Group 13/23 3.1 Command Line Interface
GridWay Job Template (Ex.) # Execution variables EXECUTABLE = job ARGUMENTS = ${TASK_ID} ${TOTAL_TASKS} 100000 ENVIRONMENT = LD_LIBRARY_PATH=/usr/local/lib
# Resource selection parameters REQUIREMENTS = HOSTNAME= "*.dacya.ucm.es" RANK = CPU_MHZ
# I/O files INPUT_FILES = my_inputfile OUTPUT_FILES = my_outputfile
# Standard streams STDOUT_FILE = stdout_file.${TASK_ID} STDERR_FILE = stderr_file.${TASK_ID} … other variables related to checkpointing, fault tolerance, performance,…
DSA Group 14/23 3.2 DRMAA API (C Binding)
GridWay
DRMAA API
● Distributed Resource Management Application API http://www.drmaa.org/ ● Open Grid Forum Standard ● Homogeneous interface to different Distributed Resource Managers (DRM): ● SGE ● Condor ● PBS/Torque ● GridWay ● C ● JAVA ● Soon! ● Perl & Ruby
DSA Group 15/23 3.2 DRMAA API (C Binding)
GridWay Application Profiles with DRMAA
DSA Group 16/23 3.2 DRMAA API (C Binding)
GridWay
Program Structure and Compilation
Include the DRMAA library
Verify the following env variable (.bashrc)
Include the compiling options for DRMAA
DSA Group 17/23 3.2 DRMAA API (C Binding)
GridWay
DRMAA Sessions
Initialize DRMAA Session
IncludFinalize DReMAA the Sess DRMAAion library
Job Template Creation
Job Template Allocation
Scalar Attribute
Vector Attribute (i.e. string with executable arguments)
DSA Group 18/23 3.2 DRMAA API (C Binding)
GridWay
Job Submission
Submit a Job
Wait for the Job
Get the exit code
Get remote resource usage stats
Remove Job Template
DSA Group 19/23 3.2 DRMAA API (C Binding)
GridWay
Job Status and Control
Get Job status
Wait Job ending
Send control signals to Job
Job Arrays
Submit Job array
Get next Job id
DSA Group 20/23 Contents GridWay 1. A Global Vision 2. GridWay Out There 3. Interacting With GridWay 3.1. Command Line Interface 3.2. DRMAA API (C Binding) 4. A Short Demo
DSA Group 21/23 For More Information
GridWay http://www.GridWay.org/
DSA Group 22/23 GridWay Thank you for your attention!
DSA Group 23/23