Porting Fusion and Biomed Applications using DRMAA API and Globus GridWay

JOINT EGEE AND SEE-GRID SUMMERC oSnCgHreOOsoL ON GRID APPLICATION SUPPORT BudCaupeidsatd, ,H Sunpgaainry GridWay JMuneay 2195, ,2 200707

José Luis Vázquez-Poletti Distributed Systems Architecture Group Universidad Complutense de Madrid

1/23 Contents GridWay 1. A Global Vision 2. GridWay Out There 3. Interacting With GridWay 3.1. Command Line Interface 3.2. DRMAA API ( Binding) 4. A Short Demo

DSA Group 2/23 1. A Global Vision

GridWay s

DRMAA n

Results o i $> t • standard API (OGF a g CLI c i DRMAA) n l i l p • Command Line Interface p ..CC,, .j.javaa p u A

o Grid Scheduling Steps - c r a e e t l d e

u • open source meta- e d M r

e scheduler u d h i t GridWay c r • job execution c S G u management r e t

r • resource brokering s a a r w f e l n • Standard interfaces I d - Globus d

n • Globus services i o

i • end-to-end (e.g. M t

a d TCP/IP) i c i r l G p • highly dynamic p

A PBS SGE Infrastructure • heterogeneous • high fault rate

DSA Group 3/23 1. A Global Vision

GridWay

Workload Management

• Advanced scheduling functionalities and new scheduling policies support • Failure detection and recovery • Accounting service • Simple jobs, job arrays and DAGs

User Interface

• Complete support for the OGF DRMAA Standard (C & JAVA) • DRM-like command line interface

Integration

• Deployment of new services is not needed • Interface with new grid services • Interoperability with other infrastructures

DSA Group 4/23 1. A Global Vision

GridWay

GridWay complements gLite

FEATURES BENEFITS DRMAA OGF Standard support Application compatibility with DRM systems which implement the (C & JAVA) Standard: SGE, Torque, ... DRM-like Command Line Interface CLI similar to that found in local (allows users to submit, kill, migrate, resource managers query state and synchronize jobs) Lightweight Middleware Higher Efficiency for certain application profiles Site-level Accounting and scheduling Use analysis, profiling utilization and policies user monitoring

Minimum installation requisites Easy deployment and mantainment Interoperability Simultaneous access to different infrastructures

DSA Group 5/23 1. A Global Vision

GridWay

DSA Group 6/23 1. A Global Vision

GridWay

Grid Infrastructure

Users GridWay daemon Execution Transfer Information Scheduling Manager Manager Manager Module

GSI- pre-WS pre-WS MDS2 MDS2 Grid-FTP MDS2 MDS2 Grid-FTP MDS2 Blahp GRAM GRAM

BLAHP Condor-C/BLAHP Storage Element PBS Cluster Storage Element PBS Cluster PBS Cluster EGEE-CE EGEE-CE EGEE-SE EGEE-CE EGEE-SE

Future gLite 3.0.2 LCG 2 gLite (based on pre-WS GT)

DSA Group 7/23 1. A Global Vision

GridWay

GridWay / LCG-2 RB Comparative EGEE 1st User Forum (CERN, 1st-3rd March 2006)

Resubmissions/Reschedules 25 32,5 30 22,5 27,5 20 25 22,5 17,5 20 LCG-2 17,5 GridWay 15 15 12,5 LCG-2 12,5 10 GridWay 7,5 10 5 7,5 2,5 0 5 2 Jobs Affected Total Max Resub/Resch per Job 2,5 1,8 Productivity (Every 15') 1,6 50 0 Avg Productivity 45 1,4 40 1,2 35 LCG-2 1 30 GridWay LCG-2 0,8 25 GridWay 20 0,6 15 0,4 10

0,2 5

0 0 Avg. Overhead/job 0' 15' 30' 45' 1h 1h 1h 1h 2h 2h 2h 2h 3h 3h 3h 15' 30' 45' 15' 30' 45' 15' 30'

DSA Group 8/23 Contents GridWay 1. A Global Vision 2. GridWay Out There 3. Interacting With GridWay 3.1. Command Line Interface 3.2. DRMAA API (C Binding) 4. A Short Demo

DSA Group 9/23 2. GridWay Out There

GridWay

Some Projects and Infrastructures

● IRISGrid ● Politecnico di Torino ● CABGrid (Centro de Astrobiología) h ● C2VO (Universidad de Castilla La Mancha) tt p: M ● Grid en ESAC (Agencia Espacial Europea) // (Su w o ● CRO-GRID (Croacia) ww r cce e ● Solution Center World Grid .g ss rid a ● Infraestructura EGEE t Sto wa : ● Proyecto BeinGRID r ies y.or ● GridX1 (Canadian Grid for HEP applications) ) g ● Universidade do Porto / ● Madras Institute of Technology ● National Center for High-Performance Computing

Some Application Porting Areas

● Life-Sciences ● Aerospace ● Fusion Physics ● Computational Chemistry

DSA Group 10/23 2. GridWay Out There

GridWay MAssive RAy TRAcing in Fusion Plasmas

Executable: Truba x N • 1,8 MB – 9' – 50 Executions Input files =~ 70 KB

Output files =~ 549 KB CD-HIT

DSA Group 11/23 Contents GridWay 1. A Global Vision 2. GridWay Out There 3. Interacting With GridWay 3.1. Command Line Interface 3.2. DRMAA API (C Binding) 4. A Short Demo

DSA Group 12/23 3.1 Command Line Interface

GridWay Main Commands

● gwps: Shows job information and state ● gwhistory: Shows execution history ● gwkill: Sends signals to a job (kill, stop, resume, reschedule) ● gwsubmit: Submits a job or array ● gwwait: Waits for job's end (any, all, set) ● gwuser: User Monitoring ● gwhost: Host Monitoring ● gwacct: Accounting

Application Model and Lifecycle

DSA Group 13/23 3.1 Command Line Interface

GridWay Job Template (Ex.) # Execution variables EXECUTABLE = job ARGUMENTS = ${TASK_ID} ${TOTAL_TASKS} 100000 ENVIRONMENT = LD_LIBRARY_PATH=/usr/local/lib

# Resource selection parameters REQUIREMENTS = HOSTNAME= "*.dacya.ucm.es" RANK = CPU_MHZ

# I/O files INPUT_FILES = my_inputfile OUTPUT_FILES = my_outputfile

# Standard streams STDOUT_FILE = stdout_file.${TASK_ID} STDERR_FILE = stderr_file.${TASK_ID} … other variables related to checkpointing, fault tolerance, performance,…

DSA Group 14/23 3.2 DRMAA API (C Binding)

GridWay

DRMAA API

● Distributed Resource Management Application API http://www.drmaa.org/ ● Standard ● Homogeneous interface to different Distributed Resource Managers (DRM): ● SGE ● Condor ● PBS/Torque ● GridWay ● C ● JAVA ● Soon! ● & Ruby

DSA Group 15/23 3.2 DRMAA API (C Binding)

GridWay Application Profiles with DRMAA

DSA Group 16/23 3.2 DRMAA API (C Binding)

GridWay

Program Structure and Compilation

Include the DRMAA library

Verify the following env variable (.bashrc)

Include the compiling options for DRMAA

DSA Group 17/23 3.2 DRMAA API (C Binding)

GridWay

DRMAA Sessions

Initialize DRMAA Session

IncludFinalize DReMAA the Sess DRMAAion library

Job Template Creation

Job Template Allocation

Scalar Attribute

Vector Attribute (i.e. string with executable arguments)

DSA Group 18/23 3.2 DRMAA API (C Binding)

GridWay

Job Submission

Submit a Job

Wait for the Job

Get the exit code

Get remote resource usage stats

Remove Job Template

DSA Group 19/23 3.2 DRMAA API (C Binding)

GridWay

Job Status and Control

Get Job status

Wait Job ending

Send control signals to Job

Job Arrays

Submit Job array

Get next Job id

DSA Group 20/23 Contents GridWay 1. A Global Vision 2. GridWay Out There 3. Interacting With GridWay 3.1. Command Line Interface 3.2. DRMAA API (C Binding) 4. A Short Demo

DSA Group 21/23 For More Information

GridWay http://www.GridWay.org/

DSA Group 22/23 GridWay Thank you for your attention!

DSA Group 23/23