The Workbench an Introduction to Scientific Workflows Robert Bridgart | Software Engineer September 2013

The Workbench An Introduction to Scientific Workflows Robert Bridgart | Software Engineer September 2013

CSIRO LAND & WATER / WATER FOR A HEALTHY COUNTRY FLAGSHIP Workflow

2 | The Workbench: An Introduction to Scientific Workflows | Robert Bridgart Scientific Workflows

What is a scientific workflow?

“A scientific workflow system is a specialized form of a workflow management system designed specifically to compose and execute a series of computational or data manipulation steps, or a workflow, in a scientific application.”

http://en.wikipedia.org/wiki/Scientific_workflow_system

3 | The Workbench: An Introduction to Scientific Workflows | Robert Bridgart Project Trident A Scientific Workflow Workbench

Origin - Microsoft Research

Currently - Open Source project w/Outercurve Foundation

Microsoft .NET based implementation of a scientific workflow system

Uses Microsoft .NET Workflow Foundation

4 | The Workbench: An Introduction to Scientific Workflows | Robert Bridgart Project Trident A Scientific Workflow Workbench

Provides tools and capabilities to

• Create, store and execute workflow steps (aka Activities)

• Manage workflow/activity revisions

• Record execution history of workflows (what, who and when)

• Persist data products (inputs, intermediate results and or outputs)

Competing technologies – Taverna and Kepler (both Java based)

5 | The Workbench: An Introduction to Scientific Workflows | Robert Bridgart The Workbench

• A suite of tools developed to extended and increase the useability of Workflow Foundation based scientific workflows

• Provides a selection of generic Activities and data types • Activities for importing / transforming / reporting datasets • Activities for calibration / optimisation (using HPC) • Definitions for custom data types i.e. Time Series

• Activity Generator – used to facilitate the embedding of existing programming tools and models into Activities • Programming languages – R, Python and .NET • Surface water models – Source, IQQM and Realm • Groundwater models - MODFLOW

• Utilities – database backup/restore, activity/workflow packing and distribution.

6 | The Workbench: An Introduction to Scientific Workflows | Robert Bridgart Surface water modelling example

7 | The Workbench: An Introduction to Scientific Workflows | Robert Bridgart eReefs project High level workflow

Estuary Lagoon Source Visualisation Modelling Modelling (Catchments) Platform (SHOC) (SHOC)

8 | The Workbench: An Introduction to Scientific Workflows | Robert Bridgart Demonstration

13 | The Workbench: An Introduction to Scientific Workflows | Robert Bridgart Why should I adopt the Workbench?

• Provides a framework for capturing and managing workflows in a central location

• Captures the provenance of output data

• Defines common data types (to facilitate activity/model interaction)

• Comes with a suite of predefined activities

• Provides support for a range of existing technologies (i.e. R, Source, etc.)

• Good support for calibration, optimisation and parallelisation using HPC

• Easily create new activities

• Accountability

14 | The Workbench: An Introduction to Scientific Workflows | Robert Bridgart Thank you CSIRO Land & Water / Environmental Information Systems Robert Bridgart Software Engineer t +61 2 6246 5885 e [email protected] w www.csiro.au

CSIRO LAND & WATER / WATER FOR A HEALTHY COUNTRY FLAGSHIP Getting Started with Trident

Prerequisites

• Microsoft Windows Vista SP3, 7 or Server 2008 • Microsoft SQL Server 2008 (Express or Full Edition) • Microsoft .NET 4.0 Framework or later

Optional

• Silverlight 3.0 with IIS 7.0 • Microsoft Office Word w/Trident Document Add-in package

Project Trident: A Scientific Workflow Workbench http://tridentworkflow.codeplex.com/

16 | The Workbench: An Introduction to Scientific Workflows | Robert Bridgart