iPlant Program SSWAP: Simple Semantic Web Architecture And Protocol

PLANT AND ANIMAL GENOME CONFERENCE JANUARY 2012

DAMIAN GESSLER, Ph.D. SEMANTIC WEB ARCHITECT UNIVERSITY OF ARIZONA

dgessler (at) iplantcollaborative (dot) org The Field

Science is not a spectator sport

http://www.agriculture.com/ag/slideshow/slideShow.jhtml?slideid=/templatedata/ag/slideshow/data/1144946148737.&page=2 w w w . i p l a n t c o l l a b o r a t i v e . o r g 2 From Simple Actions Emerge Complex Systems

Sugar cane

Rain Forest

http://img.timeinc.net/time/daily/2008/0803/a_wbiofuels_0407.jpg http://www.agriculture.com/ag/slideshow/slideShow.jhtml?slideid=/templatedata/ag/slideshow/data/1144946148737.xml&page=2 w w w . i p l a n t c o l l a b o r a t i v e . o r g 3 Non‐Linear Dynamical Systems

http://royalsociety.org/page.asp?tip=1&id=4762 http://img.timeinc.net/time/daily/2008/0803/a_wbiofuels_0407.jpg http://www.agriculture.com/ag/slideshow/slideShow.jhtml?slideid=/templatedata/ag/slideshow/data/1144946148737.xml&page=2 w w w . i p l a n t c o l l a b o r a t i v e . o r g 4 Non‐Linear Dynamical Systems

th ng tthhee eeaarrth le iinnhhaabbiittiing illioonn ppeeoopple SSeevveenn bbilli 201111 tionn FFuunndd 20 UN PPooppuullaatio UN http://royalsociety.org/page.asp?tip=1&id=4762 http://img.timeinc.net/time/daily/2008/0803/a_wbiofuels_0407.jpg http://www.agriculture.com/ag/slideshow/slideShow.jhtml?slideid=/templatedata/ag/slideshow/data/1144946148737.xml&page=2 w w w . i p l a n t c o l l a b o r a t i v e . o r g 5 Evidence‐based Decision‐making

http://blog.lib.umn.edu/ellis271/arch1701/bigstockphoto_Global_Warming_217540%203.jpg http://www.smartpower.org/blog/wp‐content/photos/field_turbines.jpg

Decisions have downstream and unintended consequences; analyses and decisions about our Natural world that utilize a scientific approach bias our odds towards viable solutions.

w w w . i p l a n t c o l l a b o r a t i v e . o r g 6 Evidence‐based Decision‐making Requires … Evidence

http://www.kegg.jp/kegg/atlas Maddison WP 1997 Syst. Biol. 46(3): 523‐536 w w w . i p l a n t c o l l a b o r a t i v e . o r g 7 iPlant Integrative, Collaborative, Science

iPToL Ultra High‐Throughput Sequencing iPG2P (200+ Gbp/run)

Association Genetics / QTL Mapping Evolution Technology Advances (cognitive model) (drivers) (scientific and social return) w w w . i p l a n t c o l l a b o r a t i v e . o r g 8 iPlant Integrative, Collaborative, Science

s ific iiddeeaas f scciieennttific valluuee oof s an tthhee ns: xt,, aanndd va astteerrtthhan e reeaassoons: t, coonntteext a rraattee ffas TThhrreee r inemmeennt, c ngeess aatt a ing,, rreeffine ; itt cchhaang e meeaanning iscooveverryy; i ific:: TThhe m wiitthh ddisc ctuurreess s iittiiss Scciieennttific hannggeess w nfraassttrruuct cit.. TThhuus S ceppttss ccha maattiicc iinfr ot eexxpplliicit g iiss andd ccoonnce y iinnffoorrm licciitt,, nnot s; ssccaalliinng an off mmaanny cs iiss iimmppli serrvviiccees; ife ccyyccllee o emaannttiics ta aanndd se llife g aanndd ssem erggee ddaata meeaanniinng t anndd mmer : Muucchh m conntteexxt a chnniiccaall: M exttrraacctt co ndeerr TTeech sivvee ttoo ex lineess,, uund itthh or iinntteennsi ntiiaall.. s ddiisscciipplin lturreess,, wwi llaabbor exppoonneent d aaccrroosss renntt ccuultu iPToL ar,, nnoott ex eneerraatteed in ddiiffffeere iPG2P lliinneear Ultra High‐Throughputs aarree ggen Sequencinguttiioonnss,, in cooveverriiees nt iinnssttiittu anndd ddiissc diiffffeerreent l: VVaalluuee a deellss,(200+, iinn d Gbp/run) SSoocciiaal: dinngg mmood entt ffuunndi ctuurreess ddiiffffeerren rdd ssttrruuct ntt rreewwaar ddiiffffeerreen Association Genetics / QTL Mapping Evolution Technology Advances (cognitive model) (drivers) (scientific and social return) w w w . i p l a n t c o l l a b o r a t i v e . o r g 9 Data Integration is Hard

Three reasons:

Scientific: The meaning, refinement, context, and value of scientific ideas and concepts changes with discovery; it changes at a rate faster than the life cycle of many informatic infrastructures

Technical: Much meaning and semantics is implicit, not explicit. Thus it is labor intensive to extract context and merge data and services; scaling is linear, not exponential.

Social: Value and discoveries are generated across disciplines, under different funding models, in different institutions, in different cultures, with different reward structures

w w w . i p l a n t c o l l a b o r a t i v e . o r g 10 Distributed Nature of Modern Science

For today’s problems …

•Effective responses to infectious diseases (including crops) •Feeding the world •Bio‐fuels research and production •Many more …

The relevant data sets and expertise span disciplines, institutions, and countries. It is, by its nature, distributed, both geographically and cognitively.

w w w . i p l a n t c o l l a b o r a t i v e . o r g 11 The Challenge

Integrate across web sites, across data sets, across scientific models and procedures ... .

This is, of course, hugely challenging and not naively attainable, but significant progress can be made.

Here, I’ll show you our approach, starting with the architecture and proceeding into the platform and developer tools.

Our architecture is open; any one from any web site can be part of it and all our code is free and open source.

w w w . i p l a n t c o l l a b o r a t i v e . o r g 12 The Actors

You

w w w . i p l a n t c o l l a b o r a t i v e . o r g 13 The Actors

You

Community MODS (Model Organism ) and CODS (Clade Oriented Databases) w w w . i p l a n t c o l l a b o r a t i v e . o r g 14 The Actors

You

Community MODS (Model Organism Databases) iPlant Computational and Resources (e.g., TACC) CODS (Clade Oriented Databases) w w w . i p l a n t c o l l a b o r a t i v e . o r g 15 The Actors

The World You Your lab

Community The World MODS (Model Organism Databases) iPlant Computational and Resources (e.g., TACC) CODS (Clade Oriented Databases) w w w . i p l a n t c o l l a b o r a t i v e . o r g 16 The Actors

The World You Your lab

Community The World MODS (Model Organism Databases) iPlant Computational and Resources (e.g., TACC) CODS (Clade Oriented Databases) w w w . i p l a n t c o l l a b o r a t i v e . o r g 17 The Actors

The World You Your lab

Open Semantic Mediation Layer

Community The World MODS (Model Organism Databases) iPlant Computational and Resources (e.g., TACC) CODS (Clade Oriented Databases) w w w . i p l a n t c o l l a b o r a t i v e . o r g 18 Bridging HPC, Enterprise, and Web assets with a single iPlant cyberinfrastructure

Web

Semantic Integration; e.g., iPlant Semantic Pipelining

Enterprise Class; e.g., Discovery Environment, Atmosphere

Foundation Infrastructure; e.g., TACC

w w w . i p l a n t c o l l a b o r a t i v e . o r g 19 The Antibody Analogy as the Mediation Layer

w w w . i p l a n t c o l l a b o r a t i v e . o r g 20 iPlant Semantic Architecture

Semantic documents described in this talk: REPOSITORY KB PDG: Provider Description Graph RDG: Resource Description Graph BROKER Semantic Broker / Discovery Server RIG: Resource Invocation Graph RRG: Resource Response Graph EXPLICIT INTERFACE RDG RQG RRG RQG: Resource Query Graph

Web

EXPLICIT INTERFACE PDG RDG RIG RRG RIG RRG RQG PDG RDG RIG RRG RESTful API RIG RRG RQG INTERPRETER

Web Ontology Protocol Ontology Protocol Client Resource Servers Ontology

Algorithm Data Algorithm OWL OWL RDF/XML RDF/XML DataData and ServiServicece ProProvidersviders Ontologies Clients

w w w . i p l a n t c o l l a b o r a t i v e . o r g 21 The Technology Stack

Domain model

Higher level APIs Programmatic control

OWL formal semantics and logic Modeling abstraction RDFS, XSD, basal semantics SPARQL and data types

Triple stores, RDF basal modeling layer graph DBs, etc.

XML messaging layer parsing

Universal namespace IRIs, URIs, HTTP and web protocols w w w . i p l a n t c o l l a b o r a t i v e . o r g 22 Requirements

Common syntax: W3C RDF/XML

Computable semantic: W3C OWL DL, with graceful fail‐over to OWL Full and RDFS semantics

Shared semantic: Ontologies; yours and those of others

Protocol: SSWAP: Simple Semantic Web Architecture and Protocol

Platform: put it all together; iPlant Semantic Web Program

w w w . i p l a n t c o l l a b o r a t i v e . o r g 23 Architecture

Web service provider (data and/or services)

w w w . i p l a n t c o l l a b o r a t i v e . o r g 24 Architecture: Protocol

http:GET

OWL DL Description

Mapping from local schema to OWL via Non‐semantic our HTTP API at http://sswap.info/api JSON

Web service provider (data and/or services)

w w w . i p l a n t c o l l a b o r a t i v e . o r g 25 Architecture: Protocol

http:GET

OWL DL Ontologies Description

Non‐semantic Mapping from local schema to OWL via JSON our HTTP API at http://sswap.info/api

Winner of 2011 International Semantic Web Conference Best Student Paper Web service provider (data and/or services)

w w w . i p l a n t c o l l a b o r a t i v e . o r g 26 HTTP API: sswap.info/api

w w w . i p l a n t c o l l a b o r a t i v e . o r g 27 W3C OWL

w w w . i p l a n t c o l l a b o r a t i v e . o r g 28 Formal Logic

There are many logic systems. We concentrate on one called First‐order,

First‐order —We make statements and inferences about things, their relations to other things, and sets of things. We do not talk about properties of properties, or classes of classes.

Description logic —We describe things: Individuals: “things” Properties: relations between things Classes: sets of things—all things that share common properties

w w w . i p l a n t c o l l a b o r a t i v e . o r g 29 A Foundational Comfort Zone that Pays Dividends

Formal logics has advantages (and disadvantages): Pros: yWe ground ourselves in a discipline. There is a formal computer science and logics study of the system (e.g., complexity, decidability, etc). Compare this to the arbitrary semantics inferred from the ad hoc syntax of conventional protocols. yWe gain a formal semantics; e.g., rdfs:sub Class Of has an explicit and clear definition; the formal subClass concept is implemented in a technology, rather than a technology creating a semantic that has no formal model. yWe gain important guarantees and checks into validity (only “truths” can be derived), soundness, completeness (all “truths” can be derived), consistency (lack of contradictions), and decidability (determination in finite time with finite resources). Cons: yLogics can be unintuitive; this can require retraining and new expertise yDL’s have a complexity of NExpTime‐Complete O(2p(n))—this is “harder” than NP‐complete

w w w . i p l a n t c o l l a b o r a t i v e . o r g 30 Architecture: Publication

RDG

OWL DL Description

/publish

Web service provider (data and/or services) iPlant Discovery Server ( Engine) RDG: Resource Description Graph w w w . i p l a n t c o l l a b o r a t i v e . o r g 31 Behind the Scenes

•Read the file as valid RDF/XML OWL SSWAP;

•Retrieve relevant terms;

• Disambiguate and validate the graph;

• Determine on OWL species for reasoning guarantees;

•Generate inferred truths (Pellet: http://clarkparsia.com/pellet)

•Evaluate consistency: do any statements imply a logical impossibility?

• Fully classify: make an explicit, complete subsumption hierarchy;

• Fully realize: assign each and every individual to its most direct class

w w w . i p l a n t c o l l a b o r a t i v e . o r g 32 Architecture: Web site annotation

OntologiesOntologies

Web page Javascript snippet

Web site providers (pages with data) w w w . i p l a n t c o l l a b o r a t i v e . o r g 33 Architecture: Discovery from web sites

You

Web page

Web site providers (pages with data) w w w . i p l a n t c o l l a b o r a t i v e . o r g 34 Architecture: Discovery from web sites

You

iPlant Discovery Server (Semantic Search Engine) w w w . i p l a n t c o l l a b o r a t i v e . o r g 35 Architecture: Discovery from web sites

You

iPlant Discovery Server (Semantic Search Engine) w w w . i p l a n t c o l l a b o r a t i v e . o r g 36 On‐demand Transaction‐time Reasoning for Resolving Semantic Querying

Arbitrary Arbitrary Arbitrary Complex Complex Complex Class Class Class

rdf:type mapsTo

Resource Subject Object

Query: RQG (Resource Query Graph)

subclass super class subclass

Data store: RDGs (Resource Description Graphs) w w w . i p l a n t c o l l a b o r a t i v e . o r g 37 iPlant Semantic Pipeline Discovery from any web site

Mock page for proof‐of‐principal only w w w . i p l a n t c o l l a b o r a t i v e . o r g 38 iPlant Semantic Pipeline Automatically presents semantic results

w w w . i p l a n t c o l l a b o r a t i v e . o r g 39 iPlant Semantic Pipeline Generic and automatic input browser

w w w . i p l a n t c o l l a b o r a t i v e . o r g 40 iPlant Semantic Pipeline Drag‐n‐drop (DnD) in the browser pipeline building

w w w . i p l a n t c o l l a b o r a t i v e . o r g 41 iPlant Semantic Pipeline Automatic semantic match‐making on DnD drop

w w w . i p l a n t c o l l a b o r a t i v e . o r g 42 iPlant Semantic Pipeline Real‐time reasoning on pipeline sanity

w w w . i p l a n t c o l l a b o r a t i v e . o r g 43 iPlant Semantic Pipeline DnD positioning for constrained service matching

w w w . i p l a n t c o l l a b o r a t i v e . o r g 44 iPlant Semantic Pipeline DnD pipeline editing with …

w w w . i p l a n t c o l l a b o r a t i v e . o r g 45 iPlant Semantic Pipeline …automatic enforcement of semantic sanity

w w w . i p l a n t c o l l a b o r a t i v e . o r g 46 iPlant Semantic Pipeline Save (and load) data from your iPlant account with …

w w w . i p l a n t c o l l a b o r a t i v e . o r g 47 iPlant Semantic Pipeline … interactive file chooser

w w w . i p l a n t c o l l a b o r a t i v e . o r g 48 iPlant Semantic Pipeline ‘Play’ enabled when data conditions are met

w w w . i p l a n t c o l l a b o r a t i v e . o r g 49 iPlant Semantic Pipeline ‘Pause’ between independent web services

w w w . i p l a n t c o l l a b o r a t i v e . o r g 50 iPlant Semantic Pipeline 100% idempotent and RESTful via JSON API to a separate, underlying pipeline manager

w w w . i p l a n t c o l l a b o r a t i v e . o r g 51 iPlant Semantic Pipeline Copy, share, name, and manipulate pipelines

w w w . i p l a n t c o l l a b o r a t i v e . o r g 52 iPlant Semantic Pipeline Generic data viewer

w w w . i p l a n t c o l l a b o r a t i v e . o r g 53 iPlant Semantic Pipeline Generic, automatic semantic support for required and optional service parameters

w w w . i p l a n t c o l l a b o r a t i v e . o r g 54 iPlant Semantic Pipeline Generic, automatic semantic support for required and optional service parameters

w w w . i p l a n t c o l l a b o r a t i v e . o r g 55 iPlant Semantic Pipeline Automatic data tagging on uploads from the desktop or web

w w w . i p l a n t c o l l a b o r a t i v e . o r g 56 Status

• Everything shown today is public; try it at http://sswap.iplantcollaborative.org • But as of today, only a minimal set of “test” image conversion services • Our 2012 work: add community services • Currently working with the UC Davis Tree Biology group on a suite of services • Parallel work on semantic phylogenetic pipeline services • For your services, contact me at sswap @ iplantcollaborative . org

w w w . i p l a n t c o l l a b o r a t i v e . o r g 57 iPlant Semantic Pipeline Discovery from any web site

Mock page for proof‐of‐principal only w w w . i p l a n t c o l l a b o r a t i v e . o r g 58 HTTP API

• Generate SSWAP RDF/XML OWL by sending JSON to sswap.info/api • No programming necessary

• Self‐documenting API at sswap.info/api • Wiki at sswap.info/wiki

• Web service providers use it to create their RDGs • Example at sswap.info/api/makeRDG • Web site providers use it to test their RRGs • Example at sswap.info/api/makeRRG • Get started at Discovering Semantic Web Services wiki page

RDG: Resource Description Graph RRG: Resource Response Graph w w w . i p l a n t c o l l a b o r a t i v e . o r g 59 HTTP API: /makeRRG

w w w . i p l a n t c o l l a b o r a t i v e . o r g 60 HTTP API: /makeRRG

w w w . i p l a n t c o l l a b o r a t i v e . o r g 61 iPlant Semantic Web Wiki: sswap.info/wiki

w w w . i p l a n t c o l l a b o r a t i v e . o r g 62 Architecture: Protocol

http:GET

OWL DL Ontologies Description

Non‐semantic Mapping from local schema to OWL via JSON our HTTP API at http://sswap.info/api

Winner of 2011 International Semantic Web Conference Best Student Paper Web service provider (data and/or services)

w w w . i p l a n t c o l l a b o r a t i v e . o r g 63 HTTP API: /makeRDG

w w w . i p l a n t c o l l a b o r a t i v e . o r g 64 HTTP API: /makeRDG

w w w . i p l a n t c o l l a b o r a t i v e . o r g 65 Architecture: Publication

RDG

OWL DL Description

/publish

Web service provider (data and/or services) iPlant Discovery Server (Semantic Search Engine) RDG: Resource Description Graph w w w . i p l a n t c o l l a b o r a t i v e . o r g 66 Java API

• Run your own transaction‐time reasoning servlet by using our SDK • SDK (Software Development Kit) at sswap.info/sdk • Javadocs at sswap.info/docs/API/Java‐API • Extend the single class info.sswap.api.servlet.AbstractS S W A P S ervlet by overriding the single method handleRequest()to perform your service specifics • Examples available at …. • Get started at Hosting Semantic Web Services wiki page

RDG: Resource Description Graph RRG: Resource Response Graph w w w . i p l a n t c o l l a b o r a t i v e . o r g 67 info.sswap.api.servlet.AbstractSSW A P S ervlet.handleRequest()

w w w . i p l a n t c o l l a b o r a t i v e . o r g 68 iPlant Semantic Web Wiki: sswap.info/wiki

w w w . i p l a n t c o l l a b o r a t i v e . o r g 69 Semantic Translation

RDG: TaxonomyRecord Medicago truncatula

RIG:MySpeciesRecord TaxonomyRecord RRG:MySpeciesRecord

TaxonomyRecord

rdfs:subClassOf

MySpeciesRecord

Clients Providers (Data and/or Services) OntologiesOntologies

w w w . i p l a n t c o l l a b o r a t i v e . o r g 70 On‐demand Semantic Translation Enables “future‐proof” Extensibility

RDG: accessionID = 5 RIG:myAccessionID = 3 accessionID = 3 RRG:myAccessionID = 3

accessionID

rdfs:subPropertyOf

myAccessionID

Clients Providers (Data and/or Services) OntologiesOntologies

w w w . i p l a n t c o l l a b o r a t i v e . o r g 71 Workshop

Two‐day developer workshop in June 2012 in Santa Fe, NM

Details will be posted at www.iplantcollaborative.org

w w w . i p l a n t c o l l a b o r a t i v e . o r g 72 Semantic Web Program at iPlant

Academics Infrastructure

St. John’s College, Santa Fe, NM Two work‐study students

GraGrandnd ChallengesChallenges

Industry Research

Meetings

Clark and Parsia LLC (Washington D.C.) expert semantic web firm • Plant & Animal Genome 2012 • International Semantic Web Conference • June 2012 Semantic Web Workshop

Attribution: W3C Semantic Web Logo w w w . i p l a n t c o l l a b o r a t i v e . o r g 73 Acknowledgements

Special thanks to:

•iPlant Collaborative

•Blazej Bulka of Clark and Parsia, LLC

•NSF grants #0943879 and #EF‐0735191

w w w . i p l a n t c o l l a b o r a t i v e . o r g 74 iPlant Semantic Pipeline

w w w . i p l a n t c o l l a b o r a t i v e . o r g 75